Intelletrace ordered a traditional private MPLS backbone.  We didn’t want any Internet, we didn’t want anything special.   All of the sites were a single T-1 with a single site being a 2xT1 at 3Mbps.  All routing was BGP with a unique private ASN at each location.  Sounds fairly simple? It seemed to me as well.

How does XO provision MPLS?
XO likes to deploy each link in one of two ways.  The first is PPP or MPP if you need to bond two links together.  The second is using frame relay protocol. One DLCI is your MPLS the second is an Internet connection.  This is nice option if you want it, but we didn’t.  Each site has a Public interface IP address.  This IP address is not accessible from the outside because it is within a VRF.  XO is willing to accommodate the unique ASN at each site but the default is the same ASN at each site.

How was this network deployed?
Lets see, I can sum it up in one word WRONG.  We found that our Project Coordinator of Service Delivery Account  Manager  had placed  processed the order incorrectly after it was specfied incorrectly by the Sales Engineer.  We asked to see the network deployment document (An excel Spreadsheet) and found many issues incorrect.  We addressed them with her. Obviously she they knew better than us and didn’t address the issues.  (Edited for Accuracy 2-24-10)

1. Each site was configured with Frame-Relay even though there was no Internet connection at ANY site. 
2. The 2xT1 we ordered had two /30 networks on it. Can you say WHY? 
3. Every site had the same ASN.
4. BGP was not configured (no routing protocol was configured, not even static)

To our dismay the network was in the process of deployment before we confirmed the deployment document.   We tried to address the issues yet again and were told that this is how it was suppose to be.  We immediately requested the install engineer to help clarify.  The same engineer we spoke with before we placed the order and said all of our requests were no problem.  He reviewed the setup and stated that it wasn’t what we had intended from the beginning and he would help submit a change order.  In fact he stated he would do it himself.    He created the change order and the new network deployment document.  Intelletrace reviewed it and approved it.  PPP or MPP on each interface   Single /30 on the 2xT1 and a unique ASN at each site.   WOW that was easy.  Again one word PREMATURE!

Upon deployment we found that XO uses an AUTO provisioning tool. During our deployment of the ORIGINAL configuration it was stopped and didn’t finish.  Our change order required a MANUAL update of each site.  Greg F.  If you read this AWSOME JOB!  You at least restored my faith in the XO Install Engineering Team, even if Service Delivery the Account Rep’s and the Dispute Department are a severe let down.    Every dispute gets an immediate deny.  We believe the first level dispute individual we deal with only has the power to deny and use it with pride. I can create an auto responder to state “no”. It takes us escalating to VP level on every occurrence to resolve disputes.  Come on XO where is your common sense.  (Edited for Accuracy 2-24-10)

Our Accout Executive Eric W.  Is an individual that helped us navigate the painfull process of resolving the disute process.  Without him cwordinating the proper people we would probably still be dealing with the dispute.  (Added 2-24-10 )

The next note of mention is even if you have a dispute in XO will threaten to shut you off.  I was going to put in valid dispute, however per our first level dispute person at XO no dispute is valid.  None the less after months of process and many man hours it was resolved with us getting a full credit as we had submitted it.   

Summary:
The Technical Expertise at XO is at the top of the list.  I have a second issue we are working on now regarding a Point to Point Ethernet circuit I am waiting to see how it resolves before I write about that one.  So far it has been similar issues.  XO Dispute department, I am trying to find a word or series of words that are not severely degrading.  I can’t find any that express my full opinion that is appropriate for this forum,  so it is just better left at that.

Results of this Post: ( Addded 2-24-10)
Our president was contacted by XO to address this specific blog post.  I think this is great.  XO does seem to care what their customers say.  A meeting was requested by XO.  I will post their comments, if they don’t comment directly on this post, and the results of the the conversation.

Update as of 3-8-10:
Nothing to report.  No Meeting, no talk. No further interest in this from XO.

Update as of 3-23-10
David G. Sales Director of XO Communications came to visit with us to discuss issues Intelletrace has experienced with XO.   The meeting was not only productive, but in my opinion very productive.  We identified issues that Intelletrace could of done better and where XO could of done better.  The end result was due to a breakdown in communications.  Starting with the incorrect form being filled out by Intelletrace.  We were unaware that it was incorrect but it was incorrect regardless.  The project coordinator should be getting more information to properly identify when what the customer is verbally asking for matches what forms are submitted. 

Finally we learned that XO has two types of SLA credits.  The first is if it meets the Written SLA exactly.  Seccond  is Yes it deserves an SLA credit but is on the fringe or is interpreted differently. Toour understanding if you fall in the second bin your initial request must be denied.  This seems frustrating to me.  We have requested to have SLA denials accompanied with a reason for denial.  We haven’t been promised it would be accommodated but we were told they would make every effort.  We commend the effort to acknowledge the issue and hope that our input will help streamline the process within XO.