Upon awaiting a meeting with XO from my last post we uncover an XO issue regarding IP allocation

Intelletrace deployed an internet connection using XO Communications. Before install we request a /30 interface and a /28 Lan block. During the install process we found the /30 IP range was currently in use. On the surface, not a big issue, but annoying. We wondered why they couldn’t keep this straight, but din’nt give it much thought. We deployed the circuit and were running clean. The customer was running clean and configured their firewall.

This circuit was deployed 5 months prior to this issue surfacing. We now find out that XO deploys all LAN blocks associated with an interface block. When XO deployed a new /30 they also allocated a new /28 without our knowledge. Our customer went hard down due to inability to route the IP addresses he was running on for the prior 5 months. He was using a VOIP conference phone and a firewall with secure tunnels to multiple locations.

Intelletrace decided to take the path of least resistance. We spoke to our customer and they stated it would take time but could be accomplished. Intelletrace reconfigured the router and granted an SLA credit due to the issue.

After a follow up meeting we found the cause of the issue. When a new net block is allocated, the engineer makes the request. While viewing the request the block is not locked. A simultanious request for a net block of the same size can pull up the same block being viewed. This effectively causes this issue. XO has stated as of today they now temporarly lock the IP block at the time of request. Then perminantly take it out of the pool at allocation.

XO has taken responsibility and told us the change has been made to resolve this issue. I want to mention that all of us in the technical community experience issues out of what we expect. It is not what caused the issue that is of importance to me. The importance is to focus on how to resolve the issue and prevent it from occuring going forward. In my oppinion XO is making this effort.

I have two items in regards to Verizon Business to report this week.  The first is a simple T1, however how carriers respond to this level of service is just as important as the OC-12 level of service.

First circuit I want to reference is a T1 in Northern California. Verizon Business is the carrier with Verizon Local as the LEC.  Circuit Installed Early  No Issues.  They even turned it up early and turned up BGP on the spot without any complex forms or issues.  Verizon Thank you for keeping it Simple. 

The second circuit is an OC-12 out of Sacramento. The entire circuit was within the Verizon Business network.  It was ordered as concatenated. At the time of turn-up it was found the customer wanted a channelized OC12 but did order a concatenated by mistake.  This meant the circuit needed to be changed.  Verizon resolved this within 24 hours including a dispatch.   Again, great service. 

I continue to be impressed with Verizon Business and their engineers.  A few of the long time engineers I have worked with are now gone, however the new staff seems to keep up the standards I expect from Verizon Business.

Intelletrace ordered a traditional private MPLS backbone.  We didn’t want any Internet, we didn’t want anything special.   All of the sites were a single T-1 with a single site being a 2xT1 at 3Mbps.  All routing was BGP with a unique private ASN at each location.  Sounds fairly simple? It seemed to me as well.

How does XO provision MPLS?
XO likes to deploy each link in one of two ways.  The first is PPP or MPP if you need to bond two links together.  The second is using frame relay protocol. One DLCI is your MPLS the second is an Internet connection.  This is nice option if you want it, but we didn’t.  Each site has a Public interface IP address.  This IP address is not accessible from the outside because it is within a VRF.  XO is willing to accommodate the unique ASN at each site but the default is the same ASN at each site.

How was this network deployed?
Lets see, I can sum it up in one word WRONG.  We found that our Project Coordinator of Service Delivery Account  Manager had placed processed the order incorrectly after it was specified incorrectly by the Sales Engineer.  We asked to see the network deployment document (An excel Spreadsheet) and found many issues incorrect.  We addressed them with her. Obviously she they knew better than we and didn’t address the issues.  (Edited for Accuracy 2-24-10)

1. Each site was configured with Frame-Relay even though there was no Internet connection at ANY site.
2. The 2xT1 we ordered had two /30 networks on it. Can you say WHY?
3. Every site had the same ASN.
4. BGP was not configured (no routing protocol was configured, not even static)

To our dismay the network was in the process of deployment before we confirmed the deployment document.   We tried to address the issues yet again and were told that this is how it was suppose to be.  We immediately requested the install engineer to help clarify.  The same engineer we spoke with before we placed the order and said all of our requests were no problem.  He reviewed the setup and stated that it wasn’t what we had intended from the beginning and he would help submit a change order.  In fact he stated he would do it himself.    He created the change order and the new network deployment document.  Intelletrace reviewed it and approved it.  PPP or MPP on each interface   Single /30 on the 2xT1 and a unique ASN at each site.   WOW that was easy.  Again one word PREMATURE!

Upon deployment we found that XO uses an AUTO provisioning tool. During our deployment of the ORIGINAL configuration it was stopped and didn’t finish.  Our change order required a MANUAL update of each site.  Greg F.  If you read this AWSOME JOB!  You at least restored my faith in the XO Install Engineering Team, even if Service Delivery the Account Rep’s and the Dispute Department are a severe let down.    Every dispute gets an immediate deny.  We believe the first level dispute individual we deal with only has the power to deny and uses it with pride. I can create an auto responder to state “no”. It takes us escalating to VP level on every occurrence to resolve disputes.  Come on XO where is your common sense.  (Edited for Accuracy 2-24-10)

Our Accout Executive Eric W.  Is an individual that helped us navigate the painful process of resolving the dispute process.  Without him coordinating the proper people we would probably still be dealing with the dispute.  (Added 2-24-10 )

The next note of mention is even if you have a dispute in, XO will threaten to shut you off.  I was going to put in a valid dispute, however per our first level dispute person at XO no dispute is valid.  None the less after months of process and many man hours it was resolved with us getting a full credit as we had submitted it.

Summary:
The Technical Expertise at XO is at the top of the list.  I have a second issue we are working on now regarding a Point to Point Ethernet circuit I am waiting to see how it resolves before I write about that one.  So far it has been similar issues.  XO Dispute department, I am trying to find a word or series of words that are not severely degrading.  I can’t find any that express my full opinion that is appropriate for this forum,  so it is just better left at that.

Results of this Post: ( Addded 2-24-10)
Our president was contacted by XO to address this specific blog post.  I think this is great.  XO does seem to care what their customers say.  A meeting was requested by XO.  I will post their comments, if they don’t comment directly on this post, and the results of the the conversation.

Update as of 3-8-10:
Nothing to report.  No Meeting, no talk. No further interest in this from XO.

Update as of 3-23-10
David G. Sales Director of XO Communications came to visit with us to discuss issues Intelletrace has experienced with XO.   The meeting was not only productive, but in my opinion very productive.  We identified issues that Intelletrace could of done better and where XO could of done better.  The end result was due to a breakdown in communications.  Starting with the incorrect form being filled out by Intelletrace.  We were unaware that it was incorrect but it was incorrect regardless.  The project coordinator should be getting more information to properly identify if what the customer is verbally asking for matches which forms should be submitted.

Finally we learned that XO has two types of SLA credits.  The first is if it meets the Written SLA exactly.  Second is, Yes, it deserves an SLA credit but is on the fringe or is interpreted differently. To our understanding if you fall in the second category your initial request must be denied.  This seems frustrating to me.  We have requested to have SLA denials accompanied with a reason for denial.  We haven’t been promised it would be accommodated but we were told they would make every effort.  We commend the effort to acknowledge the issue and hope that our input will help streamline the process within XO.

Intelletrace is a IXC (InterExchange Carrier).  All Carriers at some point purchase capacity where their infrastructure lacks or does not exist.  This purchased capacity then becomes part of the IXC’s network.  Intelletrace discloses the carrier that maintains the actual cable that Intelletrace Purchases. 

The order was placed for a Fast Ethernet connection with a 10Mbps Commit burstable to 100Mbps. RCN deployed this on a Copper 10/100/1000 Copper port.  Due to the distance limitation of Copper, RCN deployed two Media converters.  The Media Converters were of a higher quality allowing the Fiber segment to be 1Gbps.  The Copper segment was 10/100/1000 or Auto.  In effect they were 2 port switches.  I prefer these devices due to their flexibility.  I was happy to find out they were using this type of product.  Most providers will do an actual copper to Ethernet converter at 100Mbps this makes a potential upgrade to 1Gbps harder.

Customers complaint.
The customer initially connected to this circuit and preformed some traditional speed tests.  The customer was amazed that he was getting 144Mbps on his 100Mbps connection.  This was not an issue when I explained how the circuit was laid out.  Obviously the Carriers interface and the media converter on their side synced up at 1Gbps, and the customer was plugged into an auto sensing 10/100/1000 port as well.  This port obviously synced up at 1Gbps.

A few days later the customer turned up the circuit with BGP routing.  A consultant provided by corporate installed the new equipment.  The customer re-tested there speed tests but now were Inbound at 10Mbps and outbound at 1Mbps.  (This is where you start hearing the music for House) 

Intelletrace offered to assist in diagnosing the issue however the customer had no access to the new edge router because it was completely managed by the consultant.  The consultant, due to policy, refused to allow read only SNMP access to the router.  In addition the consultant animatedly stated it was the internet connection.  In addition we found out the reason for BGP was that they were Multihued with another provider. We had nothing to go on but a few assumptions.  The customer was testing from a computer inside the network.

1. The carrier was running a rate limiter.
2. Customer has BGP incorrectly configured causing packets  out one path then in another.
3. Possible Firewall issue
4. Possible Internal network issue.

The only item we could address was the carrier running a rate limiter.  A ticket was opened with RCN’s Provisioning  to confirm network deployment. (Most carriers refer you back to the turn up group until the circuit has been passing data for a month.)  The turn up group stated everything was deployed correctly and reminded us the customer tested the circuit clean and it ran over the 100Mbps speed.  Dilemma time, Customer has a complaint, everyone is stating “ Nothing has changed”.  Every good engineer knows to ask this question at least 3 to 5 times.  We did just that.  To all parties involved. Here is what we Found.

RCN Metro Network
First I want to give some thumps up with a high five to the engineers at RCN.  They went above and beyond to assist with this next segment including sending me copies of their configurations.  Guss M. and Ron W. Keep up the great work.  We found that indeed there was a rate limiter put on the port at 10Mbps.  This was a mistake in reading the order thinking the 10Mb commit was a 10Mbps rate limit.  This was corrected immediately.  The RCN Engineers offered to walk through their network hop to hop. 

It was found that the media converters were set for port tracking.  This means that if the fiber segment or the copper segment went down it would take the link state down on the other segment.  This is good in my opinion.  On their switch they did have a log entry that the port dropped and restored.  The timing correlates to a period just before the customer went live.  The port was not auto negotiating to 10Mbps.  This was a surprise.  A Tech was dispatched to do a coordinated change to the media converter and the RCN switch forcing these ports to 100Mbps Full Duplex.

The Customer re-tested and now they were seeing 40Mbps download with 1Mbps upload. (This is where you start scratching your head.) We went back to RCN and went through the network with them.  They even sent us copies of the configurations and interface statics.  After this exercise I was extremely confident there was not an issue on the RCN network.

Customers Network
We went back to the customer stating that we personally checked every segment of the RCN network and are confident the circuit should run at 100Mbps.  We asked to re-run the tests and have the consultant be involved to check for internal issues.  The consultant was still insistent it was an ISP issue.  Intelletrace offered to look at the router firewall and the switches to help diagnose the issue. ( Note to self.  If a carrier offers to go beyond the circuit being provided this is RARE take them up on it. The normal response is, “The circuit tests clean. Thank you for calling good bye.”

It took two days of the customer being unhappy and playing email tag with the consultant.  Finally we had the customer on the phone the consultant and Intelletrace.  We started to discuss the issue.  It was re-confirmed that the download was at about 40Mbps and the upload was at 1Mbps.  I asked to step through the network and put an SNMP monitor on the router at least 5 second intervals. We started the test.  I asked to have the switch looked at and see if the link to the server had any errors or if the uplink had any errors.  We found errors on the port going to the server.  It was determined the switch port was set to Auto and the server was set to Auto.  The switch Negotiated to 100 Full Duplex and the server Negotiated to 100 Half Duplex.  There was a brief argument between the consultant and Intelletrace.  The consultant stated that can’t happen.  It only happens like that if the switch is forced to Full and the Server is auto.  Our statement was regardless of what should happen it is happening.    The server and port were both forced to 100Mbps and Full Duplex.  Speeds became wire speed for both upload and download.

The takeaway from this is that no matter what we think the issue may be until it is tested it could be anything.  Sometimes the only way to diagnose is to find out what it is not.

I have decided to add an incident by incident carrier review on circuits and carriers we work with.  Over time I think it will be a good realistic review of a carrier. I will categorize them so over time you can look at posts I have on a specific carrier.

Intelletrace is an IXC (Interexchange carrier). Do not expect the same level of inside access we have if you buy direct.  In many cases I deal directly with the carrier groups.  These groups don’t usually talk to end users.

Please look for current posts or jump directly to one of the categories.

RCN Metro
Verizon Business