This is a upcoming problem that we are experiencing in Northern California on new build outs of DS3’s. We have experienced it to date 4 times.   Each time the LEC engineer is surprised.  It is not a problem with AT&T or any other carrier directly, but it does require the LEC to resolve the issue.

This was orginally experienced on a Verizon Business DS3 using AT&T as the LEC.  Verizon Business was very persistent.  They were well aware and acknowledged there was an issue but admittedly did not know what it was.  We worked with 3 engineers at Verizon Business simultaneously.   AT&T was difficult.  The third engineer we dispatched was great and was willing to work through the problem.  Persistence pays off.

When we ran to loops provided by the LEC at the MUX, all was Clean.  When we placed a Hard Loop on the Adtran DS3 NIU, all was clean. When we connected to the router we took errors.  Placing a Hard loop on the Router, all was clean. The router was a Cisco 7206 VXR with a PA-T3.  The distance from the NIU and the router was about 20 Feet.  Replacing the DS3 cables 3x did not change the issue.  Changing all of the settings in the router ( Encapsulation Protocol, Line Build Out, Clocking, Scramble. Etc…) did not resolve the issue.

Resolve:
After multiple dispatches we had an AT&T engineer on site.  He pulled the NIU card and all of the errors went away.  The AT&T engineer replaced the card with a new card and the errors returned.  I had heard of Line Build Out (LBO) being an issue, but ran into it usually if the cable run was too long.  I asked the AT&T Engineer to look at it.  He stated there was no option on the MUX but he could look at the NIU. The NIU was software configurable with a serial connection.  There were two options.  Normal and Short.  Default is set to Normal. We set the NIU to short and all of the errors went away.

If a company needs a bandwidth for the connection of their voice and data applications on networks that have almost always choose a T1 connection or T3. What is a T1 and a DS3 connection? Why is it better than DSL and cable for commercial use?

T1 (1.544 Mbps) and DS3 (45 Mbps), the connections are in business class traditional leased digital telephone lines. The technology is already pre-cable and DSL. The main advantage is the service. We rely on the phone company to resolve these issues in as fast of a manner as possible. Companies rely on these connections for their income. If these lines are down it can cost thousands of dollars per hour. You also have time-tested surveillance and diagnostic capabilities to keep them so they are more reliable.

A T1 is 1.5 Mbps and average of between $ 300 – $ 500.

A 45Mbps DS3 will cost several thousand dollars a month. However, T3 (or DS3) prices have steadily declined in recent years and can be found in rule for between $ 3-4000 in many areas. but that still makes the cost of impractical for small businesses, in many cases.

Often a corporate capacity and the cost is optimized by using a fraction of a DS3 rather commit to the full bandwith. The big advantage is …. cost reduction and scalability for the periods in which the Surge capacity is required, or business growth strategy is designed and built.

There are also Point-to-Point circuits who are not connected to the Internet. They connect from customer to customer. This type of connection will allow you have a line between two private companies without risk to Internet security.

DS3′s and T1′s are symmetrical, unlike the majority of DSL cable, you get the same speed in both directions. Also important for businesses, in particular the exchange of data between departments.

DSL and Cable services are not a direct connection to a Tier One carrier and generaly are shared causing slowness at peak hours.

Not surprisingly, that can be expensive for a small business, DS3 is so absurd. Costs depend mainly on the distance. This also works on it, two simple parts, the local network (the cost of fees for local historians signal the end of headquarters-user market, otherwise known as CO, at present, otherwise known as POPs, the carrier) and the port (the cost of accessing the telephone network or the Internet), through the network of the carrier. Normally, the price brings the speed of access, and the annual commitment level is based, while the cycle is based on geography.The higher the CO and POP, plus the cost of the cycle.

Unless you run a business that can not afford to go for a half day, you are better off with cable or DSL. Otherwise …. the reliability, scalability, performance and capacity that would be better to evaluate the options using a T1 or DS3 bandwidth.

Intelletrace ordered a traditional private MPLS backbone.  We didn’t want any Internet, we didn’t want anything special.   All of the sites were a single T-1 with a single site being a 2xT1 at 3Mbps.  All routing was BGP with a unique private ASN at each location.  Sounds fairly simple? It seemed to me as well.

How does XO provision MPLS?
XO likes to deploy each link in one of two ways.  The first is PPP or MPP if you need to bond two links together.  The second is using frame relay protocol. One DLCI is your MPLS the second is an Internet connection.  This is nice option if you want it, but we didn’t.  Each site has a Public interface IP address.  This IP address is not accessible from the outside because it is within a VRF.  XO is willing to accommodate the unique ASN at each site but the default is the same ASN at each site.

How was this network deployed?
Lets see, I can sum it up in one word WRONG.  We found that our Project Coordinator of Service Delivery Account  Manager had placed processed the order incorrectly after it was specified incorrectly by the Sales Engineer.  We asked to see the network deployment document (An excel Spreadsheet) and found many issues incorrect.  We addressed them with her. Obviously she they knew better than we and didn’t address the issues.  (Edited for Accuracy 2-24-10)

1. Each site was configured with Frame-Relay even though there was no Internet connection at ANY site.
2. The 2xT1 we ordered had two /30 networks on it. Can you say WHY?
3. Every site had the same ASN.
4. BGP was not configured (no routing protocol was configured, not even static)

To our dismay the network was in the process of deployment before we confirmed the deployment document.   We tried to address the issues yet again and were told that this is how it was suppose to be.  We immediately requested the install engineer to help clarify.  The same engineer we spoke with before we placed the order and said all of our requests were no problem.  He reviewed the setup and stated that it wasn’t what we had intended from the beginning and he would help submit a change order.  In fact he stated he would do it himself.    He created the change order and the new network deployment document.  Intelletrace reviewed it and approved it.  PPP or MPP on each interface   Single /30 on the 2xT1 and a unique ASN at each site.   WOW that was easy.  Again one word PREMATURE!

Upon deployment we found that XO uses an AUTO provisioning tool. During our deployment of the ORIGINAL configuration it was stopped and didn’t finish.  Our change order required a MANUAL update of each site.  Greg F.  If you read this AWSOME JOB!  You at least restored my faith in the XO Install Engineering Team, even if Service Delivery the Account Rep’s and the Dispute Department are a severe let down.    Every dispute gets an immediate deny.  We believe the first level dispute individual we deal with only has the power to deny and uses it with pride. I can create an auto responder to state “no”. It takes us escalating to VP level on every occurrence to resolve disputes.  Come on XO where is your common sense.  (Edited for Accuracy 2-24-10)

Our Accout Executive Eric W.  Is an individual that helped us navigate the painful process of resolving the dispute process.  Without him coordinating the proper people we would probably still be dealing with the dispute.  (Added 2-24-10 )

The next note of mention is even if you have a dispute in, XO will threaten to shut you off.  I was going to put in a valid dispute, however per our first level dispute person at XO no dispute is valid.  None the less after months of process and many man hours it was resolved with us getting a full credit as we had submitted it.

Summary:
The Technical Expertise at XO is at the top of the list.  I have a second issue we are working on now regarding a Point to Point Ethernet circuit I am waiting to see how it resolves before I write about that one.  So far it has been similar issues.  XO Dispute department, I am trying to find a word or series of words that are not severely degrading.  I can’t find any that express my full opinion that is appropriate for this forum,  so it is just better left at that.

Results of this Post: ( Addded 2-24-10)
Our president was contacted by XO to address this specific blog post.  I think this is great.  XO does seem to care what their customers say.  A meeting was requested by XO.  I will post their comments, if they don’t comment directly on this post, and the results of the the conversation.

Update as of 3-8-10:
Nothing to report.  No Meeting, no talk. No further interest in this from XO.

Update as of 3-23-10
David G. Sales Director of XO Communications came to visit with us to discuss issues Intelletrace has experienced with XO.   The meeting was not only productive, but in my opinion very productive.  We identified issues that Intelletrace could of done better and where XO could of done better.  The end result was due to a breakdown in communications.  Starting with the incorrect form being filled out by Intelletrace.  We were unaware that it was incorrect but it was incorrect regardless.  The project coordinator should be getting more information to properly identify if what the customer is verbally asking for matches which forms should be submitted.

Finally we learned that XO has two types of SLA credits.  The first is if it meets the Written SLA exactly.  Second is, Yes, it deserves an SLA credit but is on the fringe or is interpreted differently. To our understanding if you fall in the second category your initial request must be denied.  This seems frustrating to me.  We have requested to have SLA denials accompanied with a reason for denial.  We haven’t been promised it would be accommodated but we were told they would make every effort.  We commend the effort to acknowledge the issue and hope that our input will help streamline the process within XO.

Every computer user utilizes energy.  If it is not the laptop or web enabled phone battery, it is in the datacenter that the computers connect to.  This technology being released by bloom energy is another possible technology to lower our cost of computing.   Data centers take an enormous amount of energy.  Normally I would ignore such hype as this however it is reported this is in production with Google and EBay for over a year. 

In the data center I have toyed with the idea of running a turbine off of natural gas taking the heat of the turbine through a converter to create Cold water, and then run that cold water through all of the various CRAC (Computer Room Air Conditioner) units.  This is unique because it does not burn the hydrocarbon but it does use it.  It produces half of the carbon dioxide a normal power plant does when producing a similar amount of energy.  The big barrier is cost.  If the cost can be dropped significantly we have just found that product to help reduce one of the highest costs in the internet economy, Power.

For the home user, I see an additional cost savings.  Fuel cells create heat.  Route your pipes away from the hot water heater and through this.  Could it be the tank less water heater on steroids?

There is a countdown for the news release at Bloomenergy. 

The 60 Minutes interview is here in its entirety including a video’s.

This is an interesting topic and I expect to get some heat from this one.  There are a few things you are almost guaranteed in telecom.  Most are avoidable and all are able to be engineered around as to mitigate impact.  I love the statement “ This is costing me Millions of Dollars”  this may be true, but then why is it not worth avoiding?

Unrealistic Expectations and Resolves.

• If this circuit is not installed by XYZ date “ I am losing millions of dollars”.
  • Valid Argument, however if you are ordering any service expect delays.  Most all services above a T1 enter on Fiber.  Yes even a DS3 because they are delivered at least on an OC3.  If there is no fiber present a build out must occur.  If fiber is present there must be capacity and the equipment carded out properly.  This takes time.  If anyone else is on the current infrastructure it must be scheduled.  You should get continuous updates.  You should know in advance if you are going to miss a target date.  If you need it in by a certain date,  give it twice the expected time. A T1 is 4 to 6 weeks  Give it 8 to 12 Weeks.  It is from the time you place the order.  Don’t sit on it until it is only a week out then say you need it in 5 days. Did I mention a facility check will not happen until an order is placed?
  • T1’s can have an issue as well.  It is more common in rural areas. Lack of copper Pairs.
  • Ethernet is another fun one.  Ethernet can deliver over fiber or copper (EoC) and can take some time to turn-up.
• The circuit is down and it needs to be back up now because I am losing millions of dollars.
  • If you need a circuit at its most critical point it will fail.  Let’s face it Murphy’s law is alive and true.  If this is true it screams redundancy.  The type of redundancy is dependent on your need.  If you don’t know how important it is then I encourage you to go pull the wire on the T1 or remove a cable on the DS3. If your business screeches to a stop you need redundancy.  If your consultant or technician turns red in the face and steam comes out of his ears it is one of two things.
    • (S)HE has not thought of the issue. You need to find a new consultant or engineer.
    • (S)HE has been told no to proposals for whatever reason. And you are about to get hit with something hard so run.  This is your fault, because remember “I am losing millions of dollars”.
• Ethernet for WAN connectivity.  Expect delays in diagnosis.  A tech may need to be physically dispatched to each connection point to test.  This takes time.  Much more time than traditional telecom services.  It is difficult to coordinate multiple people and takes time.  Did I mention this takes time?  Did I mention Redundancy because this takes time?  Did you pull the plug yet?
• The SLA says 99.99999999999999999999999999999999999 Up time.
  • Who in their right mind believes more 9’s = more up time?
    • This is a token.  You are paying $ 14 a day for a T1. Your circuit is down for 1 to 4 hours.   An SLA will not get you up any faster. 
    • NO Carrier can Guarantee you will NEVER GO DOWN.  No Auto Manufacture can guarantee you will never have a Car Problem (Toyota are you listening? Probably not.)
    • SLA = Money back for an outage not to excede the cost of the circuit.

Real current issue.
Spending thousands of dollars for a radio add.  Internet Connection goes down and all of the voice calls were directed to a VOIP phone number.  Bad Choice, but Ok  How could It be made better? 

This is an opinion and there are numerous ways to prevent form loosing these potential inbound calls.  It all comes down to planning.

If you pull the plug and your tech still sits there with his feet on the desk eating his sandwich and drinking his soda.  Sit and wait for 5 to 10 Min.  If you don’t have half the company ringing his phone off the hook or running in.  You have an amazing tech give him a raise and a well deserved hi five.

If the Tech logs into his computer and does something for a minute or two and goes back to his sandwich.  Again High five and give him a slightly bigger budget.  This one is thinking out of the box and doing the best with what you are giving him.

Happy testing.