AFN outage was a perfect storm of things going wrong all at once

On Friday, June 15, many of us endured an extended interruption in our Internet service when Ashland Fiber Network staff attempted a system upgrade to improve performance. On behalf of the entire staff of the city of Ashland, I apologize for the inconvenience this caused, whether you were directly or indirectly affected.

I want to assure you this was not a matter of the lack of staff capability as some who have contacted me have suggested. I continue to have full faith in the abilities and professionalism of our staff. This was a perfect storm of things going wrong all at once and, as inconvenient as it was, we learned valuable lessons that we will apply the next time we seek to conduct this system upgrade, which we will do in mid-August.

The goal of the upgrade was (and still is) to increase bandwidth and thereby improve upload and download speeds for all users. The upgrade will, in fact, quadruple bandwidth on the system without increasing cost to you, the end user. In order to do this, the system has to be taken off-line and then re-started.

Unfortunately, there is no perfect time to take the system off-line. The period between midnight and 6 a.m. is the lowest-use time, but there are nonetheless many businesses affected by a service interruption during this period. Similarly, some people have suggested that we should have done this on a weekend morning. But weekends are a critical time for many businesses, especially those in the hospitality industry. Weekends are also important to residential AFN customers who use the Internet for streaming videos, social media and video calling.

Again, while there is no "good" time, it appears that Monday night into Tuesday morning might be better, or at least "less worse," than other times. We also need to do a better job of communicating a planned interruption in advance.

AFN staff did several dry runs before work began on the actual upgrade and we had every reason to believe that the operation was well-planned and would run smoothly. In fact, the initial stages of the upgrade did go smoothly. At midnight, the system was shut down, the physical plant changes were made quickly and it was turned back on at 1 a.m. At that point, each individual modem must reconnect to the system; a process we knew would take several hours.

However, by 3 a.m., it was clear that modems were not reconnecting in the numbers we expected. Our network administrators corrected a bug in the new programming and, thinking the problem had been solved, we continued to re-boot the system.

By 6 a.m., staff realized that modem counts were not coming up as they should have and began working on a hardware-based solution. But the tests we ran on the hardware showed everything was working normally.

The problem is akin to driving down the interstate in a car that's working just fine, stopping for a bite to eat, but then the car won't start when you're done with the meal.

Shortly after 11 a.m., when neither the programming fix nor the hardware fix had worked, a decision was made to abandon the redesign and restore the previous settings. This meant another hour to physically reconfigure the system and four hours to let all of the modems reconnect.

Without getting overly technical, the final problem turned out to be a piece of equipment called a "combiner/splitter" that was actually sending signal in the wrong direction. As a result, a large number of modems did not recognize the system was back on-line.

Just as an auto manufacturer cannot guarantee that their car will start every time you turn the key in the ignition, no telecommunications or broadband provider can guarantee there will never be another network interruption. But we want to assure you that with the lessons learned from June 15, we believe this upgrade can be accomplished with minimal down time. In any event, the upgrade will not be attempted until all known potential issues have a better contingency plan to address problems.

Again, we know this network interruption created significant problems for many in the community. We apologize for that and pledge to use the experience to improve our service as we move forward. Thank you for supporting AFN, Ashland's community broadband service.

Dave Kanner is Ashland's city administrator.

Share This Story