Intelletrace is a IXC (InterExchange Carrier).  All Carriers at some point purchase capacity where their infrastructure lacks or does not exist.  This purchased capacity then becomes part of the IXC’s network.  Intelletrace discloses the carrier that maintains the actual cable that Intelletrace Purchases. 

The order was placed for a Fast Ethernet connection with a 10Mbps Commit burstable to 100Mbps. RCN deployed this on a Copper 10/100/1000 Copper port.  Due to the distance limitation of Copper, RCN deployed two Media converters.  The Media Converters were of a higher quality allowing the Fiber segment to be 1Gbps.  The Copper segment was 10/100/1000 or Auto.  In effect they were 2 port switches.  I prefer these devices due to their flexibility.  I was happy to find out they were using this type of product.  Most providers will do an actual copper to Ethernet converter at 100Mbps this makes a potential upgrade to 1Gbps harder.

Customers complaint.
The customer initially connected to this circuit and preformed some traditional speed tests.  The customer was amazed that he was getting 144Mbps on his 100Mbps connection.  This was not an issue when I explained how the circuit was laid out.  Obviously the Carriers interface and the media converter on their side synced up at 1Gbps, and the customer was plugged into an auto sensing 10/100/1000 port as well.  This port obviously synced up at 1Gbps.

A few days later the customer turned up the circuit with BGP routing.  A consultant provided by corporate installed the new equipment.  The customer re-tested there speed tests but now were Inbound at 10Mbps and outbound at 1Mbps.  (This is where you start hearing the music for House) 

Intelletrace offered to assist in diagnosing the issue however the customer had no access to the new edge router because it was completely managed by the consultant.  The consultant, due to policy, refused to allow read only SNMP access to the router.  In addition the consultant animatedly stated it was the internet connection.  In addition we found out the reason for BGP was that they were Multihued with another provider. We had nothing to go on but a few assumptions.  The customer was testing from a computer inside the network.

1. The carrier was running a rate limiter.
2. Customer has BGP incorrectly configured causing packets  out one path then in another.
3. Possible Firewall issue
4. Possible Internal network issue.

The only item we could address was the carrier running a rate limiter.  A ticket was opened with RCN’s Provisioning  to confirm network deployment. (Most carriers refer you back to the turn up group until the circuit has been passing data for a month.)  The turn up group stated everything was deployed correctly and reminded us the customer tested the circuit clean and it ran over the 100Mbps speed.  Dilemma time, Customer has a complaint, everyone is stating “ Nothing has changed”.  Every good engineer knows to ask this question at least 3 to 5 times.  We did just that.  To all parties involved. Here is what we Found.

RCN Metro Network
First I want to give some thumps up with a high five to the engineers at RCN.  They went above and beyond to assist with this next segment including sending me copies of their configurations.  Guss M. and Ron W. Keep up the great work.  We found that indeed there was a rate limiter put on the port at 10Mbps.  This was a mistake in reading the order thinking the 10Mb commit was a 10Mbps rate limit.  This was corrected immediately.  The RCN Engineers offered to walk through their network hop to hop. 

It was found that the media converters were set for port tracking.  This means that if the fiber segment or the copper segment went down it would take the link state down on the other segment.  This is good in my opinion.  On their switch they did have a log entry that the port dropped and restored.  The timing correlates to a period just before the customer went live.  The port was not auto negotiating to 10Mbps.  This was a surprise.  A Tech was dispatched to do a coordinated change to the media converter and the RCN switch forcing these ports to 100Mbps Full Duplex.

The Customer re-tested and now they were seeing 40Mbps download with 1Mbps upload. (This is where you start scratching your head.) We went back to RCN and went through the network with them.  They even sent us copies of the configurations and interface statics.  After this exercise I was extremely confident there was not an issue on the RCN network.

Customers Network
We went back to the customer stating that we personally checked every segment of the RCN network and are confident the circuit should run at 100Mbps.  We asked to re-run the tests and have the consultant be involved to check for internal issues.  The consultant was still insistent it was an ISP issue.  Intelletrace offered to look at the router firewall and the switches to help diagnose the issue. ( Note to self.  If a carrier offers to go beyond the circuit being provided this is RARE take them up on it. The normal response is, “The circuit tests clean. Thank you for calling good bye.”

It took two days of the customer being unhappy and playing email tag with the consultant.  Finally we had the customer on the phone the consultant and Intelletrace.  We started to discuss the issue.  It was re-confirmed that the download was at about 40Mbps and the upload was at 1Mbps.  I asked to step through the network and put an SNMP monitor on the router at least 5 second intervals. We started the test.  I asked to have the switch looked at and see if the link to the server had any errors or if the uplink had any errors.  We found errors on the port going to the server.  It was determined the switch port was set to Auto and the server was set to Auto.  The switch Negotiated to 100 Full Duplex and the server Negotiated to 100 Half Duplex.  There was a brief argument between the consultant and Intelletrace.  The consultant stated that can’t happen.  It only happens like that if the switch is forced to Full and the Server is auto.  Our statement was regardless of what should happen it is happening.    The server and port were both forced to 100Mbps and Full Duplex.  Speeds became wire speed for both upload and download.

The takeaway from this is that no matter what we think the issue may be until it is tested it could be anything.  Sometimes the only way to diagnose is to find out what it is not.