CenturyLink released the following statement on Monday about Washington’s state recent 911 outage:
On April 10, CenturyLink experienced a 911 outage in Washington. The outage was not caused by any failures or malfunctions of CenturyLink’s network and was not related to any OpenSSL issues such as Heartbleed. The outage was due to a technical error in a third-party vendor’s call router, which prevented the system from properly processing calls. CenturyLink and Intrado Communications, its vendor partner, have taken steps to implement an enhanced monitoring process and have addressed the router issue. Our 911 system has been returned to normal operations and we are confident these steps have addressed this issue.
Approximately 770 911 calls were completed and 4,500 failed between 12:36 a.m. PST and 6:26 a.m., when 911 service was fully restored. This occurred across 127 public safety answering points (PSAPs) in Washington; other PSAPs successfully rerouted calls.
“CenturyLink’s top priority is customer safety and reliable communications,” Brian Stading, Northwest Region President said. “We are working closely with our vendor partner to fully to understand this outage. At this time, we are confident that the 911 system is fully operational and stable.”
It’s unclear at this time why CenturyLink’s personnel did not verify whatever Intrado Communications was doing with the problematic router on early Thursday evening but Sigfred Dahl from the state’s E911 Coordination Office is actively looking into it:
“We contract with CenturyLink to serve as our E911 ESI network provider (to ensure delivery of 911 calls to a call center), and under that contract, CenturyLink is responsible for the ESI network running smoothly and efficiently,” Dahl said. “To ensure it is sound, the Utility and Transportation Commission, which has authority over telecommunications within our state, has opened an investigation which will look into the cause of the outage, the company’s emergency preparedness and response, and restoration efforts. And again, EMD will also be conducting a thorough after action review.”
Last December, The FCC adopted new rules to improve 911 reliability after studying the multi-state, days long E911 outages that occurred in the mid-west after the June 2012 “Derecho storm”. The long and fascinating document posted on their website at http://www.fcc.gov/document/fcc-adopts-rules-improve-911-reliability cites an ex parte document submitted to the FCC by CenturyLink on June 20, 2013 which states:
CenturyLink currently provides service to approximately 1,117 PSAPs. CenturyLink is unable to readily determine how many single routes it has to selective routers. A labor intensive, time-consuming manual audit of CenturyLink’s entire 9-1-1 network would be required to obtain this information. CenturyLink has processes in place to prevent adverse impacts to 9-1-1 network design. While CenturyLink does not formally “lock down” 9-1-1 circuits in order to preserve the ability to readily perform maintenance or emergency work on them, CenturyLink personnel verify that any design changes will not adversely affect 9-1-1.
—
Correction to the previous article: Mr. Dahl was named to head of the Washington State E911 coordinator’s office in August of 2013 (not November as previously stated).
Reference: Ex Parte June 20, 2013 from CenturyLink to the FCC http://apps.fcc.gov/ecfs/document/view?id=7520923795
Chip Phillips is an IT consultant based in the Seattle area, and basically rocks.
