Software issue knocked out 911 service for several provinces last month, Bell report says

A 911 outage that lasted nearly an hour and a half in several Canadian provinces last month was caused by a previously undetected bug in Bell’s software, the company said in a report on Friday.

The report said a “latent software issue” triggered the outage across Manitoba, Ontario, Quebec, New Brunswick, Nova Scotia and Prince Edward Island at 3:13 a.m. ET on Sept. 17.

“At the outset, we note that this incident was not a cyber attack and did not result from any change activity, planned or unplanned,” Philippe Gauvin, assistant general counsel for Bell, wrote in the report to the Canadian Radio-television and Telecommunications Commission (CRTC).

The software bug affected components of Bell’s next generation 911 system, known as NG9-1-1, Gauvin said. The service is designed to allow the public to send more detailed information about emergencies to dispatchers to help police, fire and paramedics better understand an emergency before arriving at the scene.

Bell said it identified the root cause and restored service at 4:40 a.m., after 87 minutes. The company implemented “a permanent solution to the problem” later that same morning.

On the day of the outage, Bell confirmed the issue disrupted service across six provinces.

Its report to the CRTC on Friday said the “vast majority” of calls during the disruption were successfully managed by backup systems, but “legacy” voice calls routed through older systems were missed if they were destined for a call centre already using the next generation technology.

The report said none of the missed calls led to a “critical public safety incident.”

In a statement to CBC News, Bell declined to say how many calls were missed.

“Bell takes its responsibility as a 9-1-1 carrier seriously and understands the gravity of any interruption to that service. We have taken steps to ensure the issue experienced on September 17 does not recur,” read an email, which noted the outage as the first of its kind since the system launched in early 2022.

“We are committed to the reliability, resilience, and continuous improvement of emergency calling services so Canadians can call for help when they need it.”

Early alarms didn’t work, report says

Friday’s report also said two alarms at the beginning of the outage, one at 3:13 and one at 3:19 a.m., did not trigger a “war room” crisis response as they should have. Engineers and other experts weren’t notified of the issue until a separate alarm system flagged an unusual drop in the number of incoming calls 24 minutes later.

The response team convened at 3:48 a.m. and fixed the issue at 4:40 a.m., the letter said.

Bell said the company is still investigating why the first two alarms did not work and has since installed “independent monitoring mechanisms as a further failsafe.” The company said it is also reviewing the system “end to end,” debriefing call centre staff and sharing lessons with other providers.

In the report released publicly, Bell redacted specific information around the flaw that caused the outage and other details about its wider system. The company said revealing those details could create the risk for hackers to find and exploit “weaknesses in the 911 infrastructure.”

“Disclosure of such information could result in direct harm to Bell Canada and the 9-1-1 network serving the public and, accordingly, to the public interest,” wrote Gauvin.

Across Canada, 911 service is connected to regional legacy telephone providers.

Bell provides that service in Manitoba, Ontario and Quebec through to the Atlantic provinces. B.C. and Alberta are serviced by Telus, while Saskatchewan has SaskTel.

Holly Barkwell, the Canadian region director for the National Emergency Number Association (NENA), said last month that the nation’s 911 system “across Canada is extremely resilient,” but noted no network can be perfect. 

“There are always issues with technology,” Barkwell told CBC News at the time.

The CRTC has said all legacy 911 systems in Canada need to be decommissioned by March 31, a deadline the agency extended by two years in early 2025.