Intelletrace ordered a traditional private MPLS backbone.  We didn’t want any Internet, we didn’t want anything special.   All of the sites were a single T-1 with a single site being a 2xT1 at 3Mbps.  All routing was BGP with a unique private ASN at each location.  Sounds fairly simple? It seemed to me as well.

How does XO provision MPLS?
XO likes to deploy each link in one of two ways.  The first is PPP or MPP if you need to bond two links together.  The second is using frame relay protocol. One DLCI is your MPLS the second is an Internet connection.  This is nice option if you want it, but we didn’t.  Each site has a Public interface IP address.  This IP address is not accessible from the outside because it is within a VRF.  XO is willing to accommodate the unique ASN at each site but the default is the same ASN at each site.

How was this network deployed?
Lets see, I can sum it up in one word WRONG.  We found that our Project Coordinator of Service Delivery Account  Manager had placed processed the order incorrectly after it was specified incorrectly by the Sales Engineer.  We asked to see the network deployment document (An excel Spreadsheet) and found many issues incorrect.  We addressed them with her. Obviously she they knew better than we and didn’t address the issues.  (Edited for Accuracy 2-24-10)

1. Each site was configured with Frame-Relay even though there was no Internet connection at ANY site.
2. The 2xT1 we ordered had two /30 networks on it. Can you say WHY?
3. Every site had the same ASN.
4. BGP was not configured (no routing protocol was configured, not even static)

To our dismay the network was in the process of deployment before we confirmed the deployment document.   We tried to address the issues yet again and were told that this is how it was suppose to be.  We immediately requested the install engineer to help clarify.  The same engineer we spoke with before we placed the order and said all of our requests were no problem.  He reviewed the setup and stated that it wasn’t what we had intended from the beginning and he would help submit a change order.  In fact he stated he would do it himself.    He created the change order and the new network deployment document.  Intelletrace reviewed it and approved it.  PPP or MPP on each interface   Single /30 on the 2xT1 and a unique ASN at each site.   WOW that was easy.  Again one word PREMATURE!

Upon deployment we found that XO uses an AUTO provisioning tool. During our deployment of the ORIGINAL configuration it was stopped and didn’t finish.  Our change order required a MANUAL update of each site.  Greg F.  If you read this AWSOME JOB!  You at least restored my faith in the XO Install Engineering Team, even if Service Delivery the Account Rep’s and the Dispute Department are a severe let down.    Every dispute gets an immediate deny.  We believe the first level dispute individual we deal with only has the power to deny and uses it with pride. I can create an auto responder to state “no”. It takes us escalating to VP level on every occurrence to resolve disputes.  Come on XO where is your common sense.  (Edited for Accuracy 2-24-10)

Our Accout Executive Eric W.  Is an individual that helped us navigate the painful process of resolving the dispute process.  Without him coordinating the proper people we would probably still be dealing with the dispute.  (Added 2-24-10 )

The next note of mention is even if you have a dispute in, XO will threaten to shut you off.  I was going to put in a valid dispute, however per our first level dispute person at XO no dispute is valid.  None the less after months of process and many man hours it was resolved with us getting a full credit as we had submitted it.

Summary:
The Technical Expertise at XO is at the top of the list.  I have a second issue we are working on now regarding a Point to Point Ethernet circuit I am waiting to see how it resolves before I write about that one.  So far it has been similar issues.  XO Dispute department, I am trying to find a word or series of words that are not severely degrading.  I can’t find any that express my full opinion that is appropriate for this forum,  so it is just better left at that.

Results of this Post: ( Addded 2-24-10)
Our president was contacted by XO to address this specific blog post.  I think this is great.  XO does seem to care what their customers say.  A meeting was requested by XO.  I will post their comments, if they don’t comment directly on this post, and the results of the the conversation.

Update as of 3-8-10:
Nothing to report.  No Meeting, no talk. No further interest in this from XO.

Update as of 3-23-10
David G. Sales Director of XO Communications came to visit with us to discuss issues Intelletrace has experienced with XO.   The meeting was not only productive, but in my opinion very productive.  We identified issues that Intelletrace could of done better and where XO could of done better.  The end result was due to a breakdown in communications.  Starting with the incorrect form being filled out by Intelletrace.  We were unaware that it was incorrect but it was incorrect regardless.  The project coordinator should be getting more information to properly identify if what the customer is verbally asking for matches which forms should be submitted.

Finally we learned that XO has two types of SLA credits.  The first is if it meets the Written SLA exactly.  Second is, Yes, it deserves an SLA credit but is on the fringe or is interpreted differently. To our understanding if you fall in the second category your initial request must be denied.  This seems frustrating to me.  We have requested to have SLA denials accompanied with a reason for denial.  We haven’t been promised it would be accommodated but we were told they would make every effort.  We commend the effort to acknowledge the issue and hope that our input will help streamline the process within XO.

On Monday, Verizon Wireless announced its entry into the market for femtocells, small home cell sites expressly designed to allow mobile phones to place calls within a 5000 Square foot radius.  This creates a Mini Cell Site.

Verizon’s “Network Extender” device is priced at $249.99, a flat fee that will not be supplemented – or subsidized – by any monthly pricing. The device is manufactured by Samsung.

Sprint also provides a similar device, the Airrave, while T-Mobile’s HotSpot@Home uses Wi-Fi to connect its handsets. To date, AT&T has not announced a femtocell offering, but is currently testing 3G MicroCell.

Verizon uses the home’s broadband connection as a backhaul, essentially translating the cellular call into a VOIP connection. Who needs it? People in rural areas that may live on the outskirts of a cell site. One drawback: EV-DO is not supported, including those services that depend on it.

Using the Network Extender doesn’t incur additional monthly charges, however, users will be charged against their plan, in minutes, for any calls they make.

The service is open to Verizon customers as well as Family Share Plan members, the company said. But Verizon customers can also use a built-in management program on the Wireless Network Extender to prevent neighboring Verizon subscribers from placing unauthorized calls.

The Wireless Network Extender will be available from Verizon Wireless stores and through its Web site, the company said.

I am on the fence on this one.  Being a parent I see the benefit of me subscribing to a service or service provider that filters its content so I don’t need to or know how to.  Products that come to mind are Cyber Sitter, Net Nanny or equivalent..  On the other hand, I believe in education.  I need to teach my children about the internet.  There may be situations I need access to information out there that may be blocked for my children not for me. 

Is it possible to get both? I think yes. I don’t think service providers should block content without letting me know in advance.  This is so I can decide if their service is good for me.  If they are going to offer this service, let people know, or offer two types of services.  Let the free market decide.  I don’t have a problem with them prioritizing traffic, or de-prioritizing traffic as the case might be.  I dont mind them requiring you to relay your outbound mail through their server.  I think those help the net in general and it is the obligation of the provider to protect them self.

Intelletrace Offers Dedicated (unshared no aggregation) direct internet access to the internet.  This is how I believe it should be done, this is what we advertise.  This is what we do. 

Why this blog?  this is a post from the Washington Post regarding the following article.

FCC looks at ways to assert authority over Web access

As a recent court hearing and industry opposition have cast doubt on its power over Web service providers.
The FCC, which regulates public access to telephone and television services, has been working to claim the same role for the Internet. The stakes are high, as the Obama administration pushes an agenda of open broadband access for all and big corporations work to protect their enormous investments in a new and powerful medium.
“This is a pivotal moment,” said Ben Scott, director of policy at the public interest group Free Press. The government wants to treat broadband Internet as a national infrastructure, he said, like phone lines or the broadcast spectrum. But federal regulators are grappling with older policies that do not clearly protect consumers’ access to the Web, their privacy or prices of service.
The issue may have reached a turning point last week when a federal appeals court questioned the limits of the FCC’s authority in a 2008 case involving Comcast. The agency had ordered the Internet and cable giant to stop blocking subscribers’ access to the online file-sharing service BitTorrent. But in an oral hearing last Friday, three judges grilled an FCC lawyer over whether the agency had acted outside the scope of its authority.
The appeals court is still hearing the case, but analysts predict that the FCC will lose and that the ruling could throw all of its efforts to oversee Internet access into question. A loss could undermine the legality of FCC Chairman Julius Genachowski’s push for policies that would prohibit service providers from restricting customers’ access to legal Web content — the concept known as net neutrality — and throw into doubt the agency’s ability to oversee pricing and competition among Internet service providers.

This is a Follow-up blog from a prior post in June of 09

The indirect channel for communications network services has evolved somewhat differently than those for voice and data telephony gear. Those channels have a relatively straightforward structure with the vendor working directly with a dealer/reseller or using a distributor to supply product to its indirect sales partners. In the carrier world, there is a more complex structure (see graphic, Communications Services Supply Chain). Understanding the different players will help an agent understand his role in the supply chain

Carriers

Carriers, also called network service providers, are regulated companies that provide wide area network (WAN) communications services. WAN services are those beyond the customer LAN. (Think telephone and Internet services.) Carriers fall into a few primary categories:

Local Exchange Carriers (LECs) provide local telephone service. The largest of these are the former regional Bell operating companies (or RBOCs), which include Verizon Communications Inc., Qwest Communications International and AT&T Inc. There are other independent operating companies, such as CenturyTel and a number of rural telephone companies.

Interexchange Carriers (IXCs) provide domestic U.S. long-distance services. In the past, this was a company like AT&T, Verizon Business, Sprint or Intelletrace. Today, almost all interexchange carriers also provide local service.

Wireless Carriers provide mobile wireless services, e.g., cellular wireless service. Today, the leading U.S. wireless carriers are Verizon Wireless, Sprint, AT&T and T-Mobile.

Competitive Local Exchange Carriers (CLECs) are LECs that are competitive to the RBOC or incumbent LECs. These companies are facilities-based (e.g., they own switching facilities). They may own some fiber/copper, but often lease ILEC facilities for local termination. Examples include O1 Communications, TelePacific Communications and many of the cable TV providers, such as Comcast.

International Carriers provide connectivity from the United States to other countries. Some of these are based in the United States, e.g., Global Crossing, Sprint, Level 3, AT&T, and others are non-U.S. operators, e.g., China Telecom America, that have set up points of presence in large U.S. hubs like New York City, Miami or Los Angeles.

Wholesaler or wholesale carrier is a term you might run across that refers to a provider that sells services at wholesale to other carriers or resellers. There are some providers that do this exclusively, but most do it to varying degrees alongside their retail business. Usually, agents do not buy directly from wholesalers because that would make them a reseller (see below), but there are exceptions. Intelletrace operates part of its business as a Wholesaler to other Carriers and Resellers.

Service Providers

There are a number of companies that offer voice and data communications services that are only loosely regulated.

Internet Service Providers (ISPs) offer access to the Internet through dial-up or dedicated connections. There are some pure-play ISPs, like America Online and Earthlink, but most telcos and cable TV companies also are ISPs. Internet access services can be wireline (e.g., DSL, cable modem, T1, Ethernet) or wireless.

VoIP Service Providers also called Internet Telephony Service Providers (ITSPs) deliver voice over the Internet (e.g., Skype or Vonage) or over private IP networks. VoIP providers may also deliver PBX-replacement services, or hosted IP telephony. They are subject to CALEA, USF and E911 regulations. Most carriers and cable companies also provide VoIP services as do many resellers, since the cost of entry is relatively low.

Resellers

A word about resellers: In the carrier world, reseller is used differently than in the hardware/software world. Resellers of gear and software buy at a discount and mark up the product and bill the customer. Value-added resellers (VARs) add value by wrapping professional services around the product. In both cases the vendor branding remains intact. Compare that to these reseller models:

Facilities-based Resellers provide the services of a telephone carrier by combining their own switches with lines leased from an incumbent carrier. These services are purchased wholesale and billed as part of a total service. If they provide local, long-distance or international voice services, they must be certified as a regulated telephone company. Early competitive long-distance companies and most CLECs are facilities-based resellers.

Switchless Resellers also provide the services of a telephony carrier but without facilities. They rebrand, market and rebill those services. Again, the end user usually is not aware of the underlying network provider, although in some cases they understand that the service rides one or more of the facilities-based carrier/reseller networks. A primary value proposition for a switchless reseller is being able to provide local services nationwide. Examples of switchless resellers include TMC Communications, TNCI and Intelletrace.

Aggregators are similar to switchless resellers, but tend to refer to companies that aggregate broadband services from multiple providers to provide a national footprint. Examples include Covad, MegaPath and New Edge Networks.

Virtual Network Operators (VNOs) are switchless resellers by another name. This term is used more commonly in the wireless market, e.g., Mobile Virtual Network Operator.

Agents

As with the service providers, there are many types of indirect sales partners or agents.  Intelletrace offers these types of services but is not an agent.  Billing comes directly from Intelletrace.

Independent Agents are individuals or companies that have a direct agreement with a service provider and do not serve as a master agent for other indirect sales partners. In some circles, they are known as Independent Marketing Representatives.

Brokers have contracts with many vendors, carriers and/or master agents and sell through all other levels. Brokers also are middlemen in carrier-to-carrier deals as well as carrier-to-business transactions.

Master Agents are companies that have negotiated agreements with suppliers to sell a large volume of services and to manage a network of subagents beneath them. Master agents are similar to distributors in the hardware/software industry.

Subagents usually are individuals, but may also be companies, who sell telecom services through a master agent. They are supported and paid by the master agent.

Referral Agents, also known as affiliated agents, usually are individuals, but also may be companies, who refer business to a service provider, master agent or agent in exchange for a one-time bounty. Referral agents often are in a related field, e.g., PBX dealers that don’t want to become full agents, or a complementary one, e.g., premises security companies.

Consultants are individuals or companies who recommend telecom services to their clients. In some cases, they are part of an agent program in order to get help with quoting and provisioning even while taking a 0 percent commission.

You must outsource your data telecommunication needs unless you are going to get all of the agreements in place then lay your own fiber. All carriers outsource services, including technical support. Carriers buy capacity from other carriers when they don’t have the resources. Outsourcing no matter what you relabeled it to is outsourcing.

The Internet giants don’t keep all of their support calls within the US. If I call the colossal big boys of the internet I could be in the Philippines, Argentina Australia, Brazil, Chile China or India. If I get India I hang up and try again. I find that outsourced Indian support is one of the least expensive, so not much is spent on training. I have actually called with a DS3 issue and was asked what operating system I was running. They spend more time on process then support.

Image: Danilo Rizzuti / FreeDigitalPhotos.net

In a nut shell vendors must balance user satisfaction with desire to curb costs.

I found a few interesting statements.
“…At companies with revenues of at least $5 billion, as many as one quarter of IT jobs will be moved offshore by 2010.”ComputerWorldMay 2009

“You may not have signed a contract with your third-party IT services provider solely to cut costs. Maybe you were seeking specific skills or farming out non-core work to focus on strategic initiatives, or looking for a back door into an emerging economy.”CIO
June 2009
Read the rest of this entry »