Intelletrace ordered a traditional private MPLS backbone.  We didn’t want any Internet, we didn’t want anything special.   All of the sites were a single T-1 with a single site being a 2xT1 at 3Mbps.  All routing was BGP with a unique private ASN at each location.  Sounds fairly simple? It seemed to me as well.

How does XO provision MPLS?
XO likes to deploy each link in one of two ways.  The first is PPP or MPP if you need to bond two links together.  The second is using frame relay protocol. One DLCI is your MPLS the second is an Internet connection.  This is nice option if you want it, but we didn’t.  Each site has a Public interface IP address.  This IP address is not accessible from the outside because it is within a VRF.  XO is willing to accommodate the unique ASN at each site but the default is the same ASN at each site.

How was this network deployed?
Lets see, I can sum it up in one word WRONG.  We found that our Project Coordinator of Service Delivery Account  Manager had placed processed the order incorrectly after it was specified incorrectly by the Sales Engineer.  We asked to see the network deployment document (An excel Spreadsheet) and found many issues incorrect.  We addressed them with her. Obviously she they knew better than we and didn’t address the issues.  (Edited for Accuracy 2-24-10)

1. Each site was configured with Frame-Relay even though there was no Internet connection at ANY site.
2. The 2xT1 we ordered had two /30 networks on it. Can you say WHY?
3. Every site had the same ASN.
4. BGP was not configured (no routing protocol was configured, not even static)

To our dismay the network was in the process of deployment before we confirmed the deployment document.   We tried to address the issues yet again and were told that this is how it was suppose to be.  We immediately requested the install engineer to help clarify.  The same engineer we spoke with before we placed the order and said all of our requests were no problem.  He reviewed the setup and stated that it wasn’t what we had intended from the beginning and he would help submit a change order.  In fact he stated he would do it himself.    He created the change order and the new network deployment document.  Intelletrace reviewed it and approved it.  PPP or MPP on each interface   Single /30 on the 2xT1 and a unique ASN at each site.   WOW that was easy.  Again one word PREMATURE!

Upon deployment we found that XO uses an AUTO provisioning tool. During our deployment of the ORIGINAL configuration it was stopped and didn’t finish.  Our change order required a MANUAL update of each site.  Greg F.  If you read this AWSOME JOB!  You at least restored my faith in the XO Install Engineering Team, even if Service Delivery the Account Rep’s and the Dispute Department are a severe let down.    Every dispute gets an immediate deny.  We believe the first level dispute individual we deal with only has the power to deny and uses it with pride. I can create an auto responder to state “no”. It takes us escalating to VP level on every occurrence to resolve disputes.  Come on XO where is your common sense.  (Edited for Accuracy 2-24-10)

Our Accout Executive Eric W.  Is an individual that helped us navigate the painful process of resolving the dispute process.  Without him coordinating the proper people we would probably still be dealing with the dispute.  (Added 2-24-10 )

The next note of mention is even if you have a dispute in, XO will threaten to shut you off.  I was going to put in a valid dispute, however per our first level dispute person at XO no dispute is valid.  None the less after months of process and many man hours it was resolved with us getting a full credit as we had submitted it.

Summary:
The Technical Expertise at XO is at the top of the list.  I have a second issue we are working on now regarding a Point to Point Ethernet circuit I am waiting to see how it resolves before I write about that one.  So far it has been similar issues.  XO Dispute department, I am trying to find a word or series of words that are not severely degrading.  I can’t find any that express my full opinion that is appropriate for this forum,  so it is just better left at that.

Results of this Post: ( Addded 2-24-10)
Our president was contacted by XO to address this specific blog post.  I think this is great.  XO does seem to care what their customers say.  A meeting was requested by XO.  I will post their comments, if they don’t comment directly on this post, and the results of the the conversation.

Update as of 3-8-10:
Nothing to report.  No Meeting, no talk. No further interest in this from XO.

Update as of 3-23-10
David G. Sales Director of XO Communications came to visit with us to discuss issues Intelletrace has experienced with XO.   The meeting was not only productive, but in my opinion very productive.  We identified issues that Intelletrace could of done better and where XO could of done better.  The end result was due to a breakdown in communications.  Starting with the incorrect form being filled out by Intelletrace.  We were unaware that it was incorrect but it was incorrect regardless.  The project coordinator should be getting more information to properly identify if what the customer is verbally asking for matches which forms should be submitted.

Finally we learned that XO has two types of SLA credits.  The first is if it meets the Written SLA exactly.  Second is, Yes, it deserves an SLA credit but is on the fringe or is interpreted differently. To our understanding if you fall in the second category your initial request must be denied.  This seems frustrating to me.  We have requested to have SLA denials accompanied with a reason for denial.  We haven’t been promised it would be accommodated but we were told they would make every effort.  We commend the effort to acknowledge the issue and hope that our input will help streamline the process within XO.

With all the buzz about Twitter as a customer service tool and many well known companies diving in, you might be asking yourself, should your company use Twitter to connect with your customers. There are a number of things to consider as you answer this question but the most important (by far) is “are your customers using Twitter?”

If they are not, that will affect how you use Twitter. You can’t connect to people who are not there.

Mashable estimates about 12% of the US population will be using Twitter by the end of 2009 with 18% by 2010.

Consider where you do business. Some states have many more people using Twitter. For example, here is a ranking of the top 10 states in the USA based on their Twitter users

1. California
2. Texas
3. New York
4. Flordia
5. Illinois

With all of this in mind, what Telecom Carriers are using twitter at all, and who is using it as a traditional one way push marketing tool.

Carrier Support Twitter Accounts
@askintelletrace            Intelletrace Inc.
@verizonsupport           Verizon Business
@sprintcare                    Sprint
@talktoqwest                  Qwest Communications

Carriers with Marketing ONLY or unknown usage.
@lvlt                                Level3 – Unused Many Followers no tweets
@attnews                       AT&T – Marketing
@xolearning                  XO – Marketing
@Globalcrossing          Global Crossing - Marketing
@integra_cares            Integra Telecom – Many Followers no tweets
@grandecom                 Grande Communications – Marketing
@abovenet                    Abovenet Communications – Many Followers no tweets

No Twitter Accouts Found at all
Yipes
Electric Lightwave
Cogent Communications
Time Warner Telecom
360 Networks

Do the carriers with no twitter presence even know they are being talked about?  Are the Carriers with only a marketing presence doing a traditional PUSH method or are they actually using it as a tool? For the few Carriers Intelletrace, Verizon Business, Sprint, and Qwest Communications, my Congrats for being a telecommunications leader in support.

A great example is a post recently sent out here.  Are you following yet?  It describes an outage with Level3 and Verizon Business.

I have focused on terrestrial telecommunication carriers.  If you want your information to be added to this post, I will be happy to do it at any time.  Just Tweet me @dzerbe or Post a comment to this blog.