I have two items in regards to Verizon Business to report this week.  The first is a simple T1, however how carriers respond to this level of service is just as important as the OC-12 level of service.

First circuit I want to reference is a T1 in Northern California. Verizon Business is the carrier with Verizon Local as the LEC.  Circuit Installed Early  No Issues.  They even turned it up early and turned up BGP on the spot without any complex forms or issues.  Verizon Thank you for keeping it Simple. 

The second circuit is an OC-12 out of Sacramento. The entire circuit was within the Verizon Business network.  It was ordered as concatenated. At the time of turn-up it was found the customer wanted a channelized OC12 but did order a concatenated by mistake.  This meant the circuit needed to be changed.  Verizon resolved this within 24 hours including a dispatch.   Again, great service. 

I continue to be impressed with Verizon Business and their engineers.  A few of the long time engineers I have worked with are now gone, however the new staff seems to keep up the standards I expect from Verizon Business.

This is a upcoming problem that we are experiencing in Northern California on new build outs of DS3’s. We have experienced it to date 4 times.   Each time the LEC engineer is surprised.  It is not a problem with AT&T or any other carrier directly, but it does require the LEC to resolve the issue.

This was orginally experienced on a Verizon Business DS3 using AT&T as the LEC.  Verizon Business was very persistent.  They were well aware and acknowledged there was an issue but admittedly did not know what it was.  We worked with 3 engineers at Verizon Business simultaneously.   AT&T was difficult.  The third engineer we dispatched was great and was willing to work through the problem.  Persistence pays off.

When we ran to loops provided by the LEC at the MUX, all was Clean.  When we placed a Hard Loop on the Adtran DS3 NIU, all was clean. When we connected to the router we took errors.  Placing a Hard loop on the Router, all was clean. The router was a Cisco 7206 VXR with a PA-T3.  The distance from the NIU and the router was about 20 Feet.  Replacing the DS3 cables 3x did not change the issue.  Changing all of the settings in the router ( Encapsulation Protocol, Line Build Out, Clocking, Scramble. Etc…) did not resolve the issue.

Resolve:
After multiple dispatches we had an AT&T engineer on site.  He pulled the NIU card and all of the errors went away.  The AT&T engineer replaced the card with a new card and the errors returned.  I had heard of Line Build Out (LBO) being an issue, but ran into it usually if the cable run was too long.  I asked the AT&T Engineer to look at it.  He stated there was no option on the MUX but he could look at the NIU. The NIU was software configurable with a serial connection.  There were two options.  Normal and Short.  Default is set to Normal. We set the NIU to short and all of the errors went away.

Intelletrace ordered a traditional private MPLS backbone.  We didn’t want any Internet, we didn’t want anything special.   All of the sites were a single T-1 with a single site being a 2xT1 at 3Mbps.  All routing was BGP with a unique private ASN at each location.  Sounds fairly simple? It seemed to me as well.

How does XO provision MPLS?
XO likes to deploy each link in one of two ways.  The first is PPP or MPP if you need to bond two links together.  The second is using frame relay protocol. One DLCI is your MPLS the second is an Internet connection.  This is nice option if you want it, but we didn’t.  Each site has a Public interface IP address.  This IP address is not accessible from the outside because it is within a VRF.  XO is willing to accommodate the unique ASN at each site but the default is the same ASN at each site.

How was this network deployed?
Lets see, I can sum it up in one word WRONG.  We found that our Project Coordinator of Service Delivery Account  Manager  had placed  processed the order incorrectly after it was specfied incorrectly by the Sales Engineer.  We asked to see the network deployment document (An excel Spreadsheet) and found many issues incorrect.  We addressed them with her. Obviously she they knew better than us and didn’t address the issues.  (Edited for Accuracy 2-24-10)

1. Each site was configured with Frame-Relay even though there was no Internet connection at ANY site. 
2. The 2xT1 we ordered had two /30 networks on it. Can you say WHY? 
3. Every site had the same ASN.
4. BGP was not configured (no routing protocol was configured, not even static)

To our dismay the network was in the process of deployment before we confirmed the deployment document.   We tried to address the issues yet again and were told that this is how it was suppose to be.  We immediately requested the install engineer to help clarify.  The same engineer we spoke with before we placed the order and said all of our requests were no problem.  He reviewed the setup and stated that it wasn’t what we had intended from the beginning and he would help submit a change order.  In fact he stated he would do it himself.    He created the change order and the new network deployment document.  Intelletrace reviewed it and approved it.  PPP or MPP on each interface   Single /30 on the 2xT1 and a unique ASN at each site.   WOW that was easy.  Again one word PREMATURE!

Upon deployment we found that XO uses an AUTO provisioning tool. During our deployment of the ORIGINAL configuration it was stopped and didn’t finish.  Our change order required a MANUAL update of each site.  Greg F.  If you read this AWSOME JOB!  You at least restored my faith in the XO Install Engineering Team, even if Service Delivery the Account Rep’s and the Dispute Department are a severe let down.    Every dispute gets an immediate deny.  We believe the first level dispute individual we deal with only has the power to deny and use it with pride. I can create an auto responder to state “no”. It takes us escalating to VP level on every occurrence to resolve disputes.  Come on XO where is your common sense.  (Edited for Accuracy 2-24-10)

Our Accout Executive Eric W.  Is an individual that helped us navigate the painfull process of resolving the disute process.  Without him cwordinating the proper people we would probably still be dealing with the dispute.  (Added 2-24-10 )

The next note of mention is even if you have a dispute in XO will threaten to shut you off.  I was going to put in valid dispute, however per our first level dispute person at XO no dispute is valid.  None the less after months of process and many man hours it was resolved with us getting a full credit as we had submitted it.   

Summary:
The Technical Expertise at XO is at the top of the list.  I have a second issue we are working on now regarding a Point to Point Ethernet circuit I am waiting to see how it resolves before I write about that one.  So far it has been similar issues.  XO Dispute department, I am trying to find a word or series of words that are not severely degrading.  I can’t find any that express my full opinion that is appropriate for this forum,  so it is just better left at that.

Results of this Post: ( Addded 2-24-10)
Our president was contacted by XO to address this specific blog post.  I think this is great.  XO does seem to care what their customers say.  A meeting was requested by XO.  I will post their comments, if they don’t comment directly on this post, and the results of the the conversation.

Update as of 3-8-10:
Nothing to report.  No Meeting, no talk. No further intrest in this from XO.

Every computer user utilizes energy.  If it is not the laptop or web enabled phone battery, it is in the datacenter that the computers connect to.  This technology being released by bloom energy is another possible technology to lower our cost of computing.   Data centers take an enormous amount of energy.  Normally I would ignore such hype as this however it is reported this is in production with Google and EBay for over a year. 

In the data center I have toyed with the idea of running a turbine off of natural gas taking the heat of the turbine through a converter to create Cold water, and then run that cold water through all of the various CRAC (Computer Room Air Conditioner) units.  This is unique because it does not burn the hydrocarbon but it does use it.  It produces half of the carbon dioxide a normal power plant does when producing a similar amount of energy.  The big barrier is cost.  If the cost can be dropped significantly we have just found that product to help reduce one of the highest costs in the internet economy, Power.

For the home user, I see an additional cost savings.  Fuel cells create heat.  Route your pipes away from the hot water heater and through this.  Could it be the tank less water heater on steroids?

There is a countdown for the news release at Bloomenergy. 

The 60 Minutes interview is here in its entirety including a video’s.

This is an interesting topic and I expect to get some heat from this one.  There are a few things you are almost guaranteed in telecom.  Most are avoidable and all are able to be engineered around as to mitigate impact.  I love the statement “ This is costing me Millions of Dollars”  this may be true, but then why is it not worth avoiding?

Unrealistic Expectations and Resolves.

• If this circuit is not installed by XYZ date “ I am losing millions of dollars”.
  • Valid Argument, however if you are ordering any service expect delays.  Most all services above a T1 enter on Fiber.  Yes even a DS3 because they are delivered at least on an OC3.  If there is no fiber present a build out must occur.  If fiber is present there must be capacity and the equipment carded out properly.  This takes time.  If anyone else is on the current infrastructure it must be scheduled.  You should get continuous updates.  You should know in advance if you are going to miss a target date.  If you need it in by a certain date,  give it twice the expected time. A T1 is 4 to 6 weeks  Give it 8 to 12 Weeks.  It is from the time you place the order.  Don’t sit on it until it is only a week out then say you need it in 5 days. Did I mention a facility check will not happen until an order is placed?
  • T1’s can have an issue as well.  It is more common in rural areas. Lack of copper Pairs.
  • Ethernet is another fun one.  Ethernet can deliver over fiber or copper (EoC) and can take some time to turn-up.
• The circuit is down and it needs to be back up now because I am losing millions of dollars.
  • If you need a circuit at its most critical point it will fail.  Let’s face it Murphy’s law is alive and true.  If this is true it screams redundancy.  The type of redundancy is dependent on your need.  If you don’t know how important it is then I encourage you to go pull the wire on the T1 or remove a cable on the DS3. If your business screeches to a stop you need redundancy.  If your consultant or technician turns red in the face and steam comes out of his ears it is one of two things.
    • (S)HE has not thought of the issue. You need to find a new consultant or engineer.
    • (S)HE has been told no to proposals for whatever reason. And you are about to get hit with something hard so run.  This is your fault, because remember “I am losing millions of dollars”.
• Ethernet for WAN connectivity.  Expect delays in diagnosis.  A tech may need to be physically dispatched to each connection point to test.  This takes time.  Much more time than traditional telecom services.  It is difficult to coordinate multiple people and takes time.  Did I mention this takes time?  Did I mention Redundancy because this takes time?  Did you pull the plug yet?
• The SLA says 99.99999999999999999999999999999999999 Up time.
  • Who in their right mind believes more 9’s = more up time?
    • This is a token.  You are paying $ 14 a day for a T1. Your circuit is down for 1 to 4 hours.   An SLA will not get you up any faster. 
    • NO Carrier can Guarantee you will NEVER GO DOWN.  No Auto Manufacture can guarantee you will never have a Car Problem (Toyota are you listening? Probably not.)
    • SLA = Money back for an outage not to excede the cost of the circuit.

Real current issue.
Spending thousands of dollars for a radio add.  Internet Connection goes down and all of the voice calls were directed to a VOIP phone number.  Bad Choice, but Ok  How could It be made better? 

This is an opinion and there are numerous ways to prevent form loosing these potential inbound calls.  It all comes down to planning.

If you pull the plug and your tech still sits there with his feet on the desk eating his sandwich and drinking his soda.  Sit and wait for 5 to 10 Min.  If you don’t have half the company ringing his phone off the hook or running in.  You have an amazing tech give him a raise and a well deserved hi five.

If the Tech logs into his computer and does something for a minute or two and goes back to his sandwich.  Again High five and give him a slightly bigger budget.  This one is thinking out of the box and doing the best with what you are giving him.

Happy testing.