Upon awaiting a meeting with XO from my last post we uncover an XO issue regarding IP allocation

Intelletrace deployed an internet connection using XO Communications. Before install we request a /30 interface and a /28 Lan block. During the install process we found the /30 IP range was currently in use. On the surface, not a big issue, but annoying. We wondered why they couldn’t keep this straight, but din’nt give it much thought. We deployed the circuit and were running clean. The customer was running clean and configured their firewall.

This circuit was deployed 5 months prior to this issue surfacing. We now find out that XO deploys all LAN blocks associated with an interface block. When XO deployed a new /30 they also allocated a new /28 without our knowledge. Our customer went hard down due to inability to route the IP addresses he was running on for the prior 5 months. He was using a VOIP conference phone and a firewall with secure tunnels to multiple locations.

Intelletrace decided to take the path of least resistance. We spoke to our customer and they stated it would take time but could be accomplished. Intelletrace reconfigured the router and granted an SLA credit due to the issue.

After a follow up meeting we found the cause of the issue. When a new net block is allocated, the engineer makes the request. While viewing the request the block is not locked. A simultanious request for a net block of the same size can pull up the same block being viewed. This effectively causes this issue. XO has stated as of today they now temporarly lock the IP block at the time of request. Then perminantly take it out of the pool at allocation.

XO has taken responsibility and told us the change has been made to resolve this issue. I want to mention that all of us in the technical community experience issues out of what we expect. It is not what caused the issue that is of importance to me. The importance is to focus on how to resolve the issue and prevent it from occuring going forward. In my oppinion XO is making this effort.

Re-Posted from 03/10/10 at 1:32pm from Zacks
US telecom behemoth AT&T (T) reported that it has successfully completed a real-time field trial of 100-Gigabit Ethernet technology, a next-generation backbone network standard. The technology is expected to be available commercially over the next few years. 

In the trial, AT&T tested Cisco’s (CSCO) newly unveiled CRS-3 core router in a live network environment. Moreover, the carrier used leading IP network test solution provider Ixia’s (XXIA) “K2″ 100-gigabit Ethernet test solution and Opnext’s optical equipment. The trial demonstrated a single-carrier 100-gigabit data transmission on a 900-kilometer ultra-long-haul transport link between New Orleans, Louisiana and Miami, Florida.

The 100-gigabit standard supports a data rate of 100 gigabits per second (Gbps), representing a significant increase from the current peak speeds of 40 Gbps enabled by the existing industry standard of 40-gigabit Ethernet. The technology has been designed to effectively support the burgeoning volumes of wireless and wireline broadband data traffic.

Leading carriers across the globe are increasingly focused on accelerating Internet network speeds, given the rapidly growing demand for greater bandwidth. Total bandwidth consumption significantly increased in 2009, driven by the growth of video and other high-bandwidth applications.

100 Gbps networks have emerged as the next major Ethernet standard. Demand for 100 Gbps service in enterprise data centers is ten times greater than the existing fastest deployments. At this speed level, users can enjoy super fast data transmission speeds, which can transmit a 2-hour high-definition movie in 9 seconds or fully loaded 500-gigabyte hard drive in 46 seconds.

AT&T’s trial follows a similar live trial by Verizon (VZ), which was recently conducted using Juniper’s (JNPR) core router and NEC America Inc’s network routes in North Dallas, Texas. Verizon is optimistic in deploying the 100 Gb technology in late 2010.

The ongoing initiative to deploy 100 Gbps capability should render greater overall network efficiency while improving cost effectiveness. Additionally, this advancement will enable AT&T to address increasing customer demand for higher network throughput for both wireless and wired Internet applications.

“The telecommunications industry has experienced more change in the last decade than in its entire history,” says IBM Institute for Business.

Consider that, in 1999, only 15 percent of the world’s population had access to a telephone; by 2009, nearly 70 percent had mobile phone subscriptions. If that seems unremarkable, consider that it took 150 years to add the first billion phone users. Then it took a decade to add the second billion users.

So where will the industry be in five years, in 2015? While nothing is certain, forecasters at the IBM Institute for Business Value say they see four possible outcomes, and none of them are rosy for the telecom business.

Keep in mind IBM believes it will take only five years for one of these scenarios to develop.

Scenario 1 – Survivor Consolidation:
Consumer spending for communications drops, leading to industry “stagnation or decline.”
In this rather-bleak scenario, developed market operators have not significantly changed their voice communications and “closed” connectivity service portfolios and also have failed to expand horizontally or into new verticals. That will trigger an Investor loss of confidence in the telecommunications sector, which produces a cash crisis and leads to industry consolidation.

Scenario 2 – Market Shakeout
Carriers are structurally reshaped into separate wholesale and retail businesses, and the market is further fragmented by government, municipality and alternative providers.
 In this scenario private capital is available only to dense urban areas. Telecom provider growth occurs in large part through sales of services to business partners.

Scenario 3 – Clash of Giants�
Carriers consolidate, cooperate and create alliances to compete with “over the top” providers and device manufacturers or even equipment suppliers.

Scenario 4 – Generative Bazaar
Open access infrastructure leads to more competition from “asset light” and over the top competitors.

It is easy to dismiss the level of change the last 10 years has wrought. It might be easy to dismiss the level of change IBM believes can happen in just another five years. As always, the forecast might be too aggressive in terms of its timetable.

The major implication, though, is that the telecom industry might well be a very-different sort of business by 2020, if not by 2015. If you look at revenue sources, it is virtually certain that in developed markets, less revenue–in some cases far less revenue–will be earned from voice and text services.

More revenue will be earned from broadband services, and possibly from business partners rather than end users.

Under such circumstances, ecosystem conflict is all but inevitable.

I have two items in regards to Verizon Business to report this week.  The first is a simple T1, however how carriers respond to this level of service is just as important as the OC-12 level of service.

First circuit I want to reference is a T1 in Northern California. Verizon Business is the carrier with Verizon Local as the LEC.  Circuit Installed Early  No Issues.  They even turned it up early and turned up BGP on the spot without any complex forms or issues.  Verizon Thank you for keeping it Simple. 

The second circuit is an OC-12 out of Sacramento. The entire circuit was within the Verizon Business network.  It was ordered as concatenated. At the time of turn-up it was found the customer wanted a channelized OC12 but did order a concatenated by mistake.  This meant the circuit needed to be changed.  Verizon resolved this within 24 hours including a dispatch.   Again, great service. 

I continue to be impressed with Verizon Business and their engineers.  A few of the long time engineers I have worked with are now gone, however the new staff seems to keep up the standards I expect from Verizon Business.

This is a upcoming problem that we are experiencing in Northern California on new build outs of DS3’s. We have experienced it to date 4 times.   Each time the LEC engineer is surprised.  It is not a problem with AT&T or any other carrier directly, but it does require the LEC to resolve the issue.

This was orginally experienced on a Verizon Business DS3 using AT&T as the LEC.  Verizon Business was very persistent.  They were well aware and acknowledged there was an issue but admittedly did not know what it was.  We worked with 3 engineers at Verizon Business simultaneously.   AT&T was difficult.  The third engineer we dispatched was great and was willing to work through the problem.  Persistence pays off.

When we ran to loops provided by the LEC at the MUX, all was Clean.  When we placed a Hard Loop on the Adtran DS3 NIU, all was clean. When we connected to the router we took errors.  Placing a Hard loop on the Router, all was clean. The router was a Cisco 7206 VXR with a PA-T3.  The distance from the NIU and the router was about 20 Feet.  Replacing the DS3 cables 3x did not change the issue.  Changing all of the settings in the router ( Encapsulation Protocol, Line Build Out, Clocking, Scramble. Etc…) did not resolve the issue.

Resolve:
After multiple dispatches we had an AT&T engineer on site.  He pulled the NIU card and all of the errors went away.  The AT&T engineer replaced the card with a new card and the errors returned.  I had heard of Line Build Out (LBO) being an issue, but ran into it usually if the cable run was too long.  I asked the AT&T Engineer to look at it.  He stated there was no option on the MUX but he could look at the NIU. The NIU was software configurable with a serial connection.  There were two options.  Normal and Short.  Default is set to Normal. We set the NIU to short and all of the errors went away.