Software glitch kills Verizon 911 network in South Bay, CA
An equipment disruption also affected regular phone service as well as some Internet services for 11 hours.
By Larry Altman
Daily Breeze

Tens of thousands of South Bay residents lost the ability to call 911 for 11 hours on Tuesday when equipment failed at a Verizon telephone switching station.

Regular phone service and some AT&T and Verizon Internet services also were affected, knocking out the ability to get online. Some emergency radio transmitters that serve police and firefighters in Redondo Beach and other parts of the South Bay stopped working, and several ATM machines and credit card machines were affected.



Police and fire agencies immediately shifted into 1980s mode when the system failed at 2:20 a.m., calling residents through automated computer systems and broadcasting alerts on radio and television stations stating the departments could be reached with old-fashioned seven-digit numbers that existed before the more convenient 911 system evolved.

No major problems were reported. SBC and other phone company customers were not affected.

"Luckily it was a slow day because of the rain and people anticipated the rain," said Redondo Beach police Sgt. Phil Keenan.

Verizon spokesman Jon Davies said corrupted software in routing equipment caused the problem at a Long Beach switching station. The equipment links, routes and processes 911 calls.

"It was a software error on a piece of equipment that connects two parts of our network together," Davies said.

Technicians copied software from a backup, reloaded the software and rebooted the computer to fix the problem. By 1:30 p.m., 911 operations in the South Bay were working.

"When the system came up with the previous version of software it resolved itself," Davies said.

Throughout the early morning hours and much of the day, residents with Verizon service in Manhattan Beach, Redondo Beach, Hermosa Beach, and parts of Torrance and the Palos Verdes Peninsula lost the ability to use their land lines to access the three-digit 911 system.

Instead, they received busy signals or "all circuits are busy" recordings.

After putting out word about regular telephone numbers, police officers and firefighters in the affected cities conducted extra patrols, driving through areas to make themselves visible in case of an emergency. Redondo Beach put all of its officers, including detectives, on the street, Keenan said.

Although much of Torrance's telephone service is handled by SBC, police put notices on the city's cable and radio channels to alert residents, Torrance police Lt. Rod Irvine said.

In Hermosa Beach, police kept watch on normally busy areas of the city.

"We've got the officers out doing extra patrol along the highway where we would have the biggest problems with traffic," Hermosa Beach police Sgt. Paul Wolcott said.

Ralph Mailloux, executive director of the South Bay Regional Public Communications Authority, which handles 911 operations for affected cities Hermosa Beach and Manhattan Beach, said three of its eight broadcast towers went down, but an emergency backup system kicked in to keep the dispatch center in touch with its fire and police officers in the field.

Residents in Manhattan Beach received two phone calls in the middle of the night to alert them about the 911 problem, one from a community alert network and another from a reverse 911 system at the dispatch center, Mailloux said.

Through the morning, the center had few emergency calls, but had a number of people calling to ask about the problem.

The Verizon equipment failure also affected Long Beach, Seal Beach, Los Alamitos, Huntington Beach, Laguna Beach, Downey, Bellflower and communities along the Orange County-Los Angeles County border.

In some communities, residents were able to make local calls, but not long distance.

Davies said Verizon would conduct an investigation into why the failure occurred. They also would look at why backup systems also failed.

"As part of our routine operations we always look at any kind of failure -- minor or major"

---------------------

Ah, a few drops call.....