After being put on notice, IBM proposes solution to Oregon’s troubled phone system

Oregon continues to have problems with the $41 million phone system it bought from IBM a few years ago, and while the state’s chief information officer says things are looking up, it will be awhile until the process is finished.

The state government purchased a voice over IP (VOIP) phone system, which it calls Project MUSIC, in 2015. And it hasn’t been working right since installation began in early 2016. That means everything from service downtime to spotty sound quality and disappearing voicemails. But in a memo sent to state agency directors on Friday, Pettit explained that IBM has submitted a “final root cause analysis” and “mitigation plan,” which the state has accepted. 

It isn’t the first time IBM has proposed a solution, and the state’s connectivity challenges are far from over.

“Several technical changes have been executed or will be addressed to improve the resilience and reliability of the solution,” Pettit wrote in his memo.

One of the things that has already improved, Pettit told StateScoop, is that as of two weeks ago a new connection is in place between the two IBM data centers for the project — one in Hillsboro, Oregon, and another in Las Vegas. The Las Vegas data center was added as a “failover” site that the state could use when the connection to the primary data center wasn’t working. The thing that really upset his office, Pettit said, was when IBM told them that it had tested the failover and that it would work — and then it didn’t.

“When the phones can’t get through to Hillsboro, they reregister their IP address in Las Vegas,” Pettit said. “Well, the firewall in Las Vegas was set so that with 26,000 phones trying to get a new IP address, it thought that it was a DDoS attack and it shut them down.”

Instead of booting the phones right back up, it took hours to correct the problem, Pettit said. The result was 30 hours of downtime in January. According to the state contract, IBM is allowed about five minutes of downtime per month. Pettit says, however, that he is optimistic about the progress IBM is making.

“They’re moving toward that goal,” he said. “We feel confident that they’re moving toward that goal.”

Pettit, who serves in the Department of Administrative Services, has recruited consultancy Gartner “to review documentation, provide recommendations and observations” as the state continues to work with IBM on the phone system. 

New year, new stance

Any confidence comes with an asterisk. This marks at least the second time that IBM has pledged to fix the system. The vendor told a legislative oversight committee in April 2017 that it could troubleshoot problems if allowed to stop work for four months.

Work did stop, with 20,000 of the state’s approximately 30,000 phones converted, but outages  —  including the infamous Las Vegas failure — and other problems persisted.

After that happened, the state put its foot down. It issued a formal notice of default on Jan. 26.

“IBM has failed to provide the state with a stable, reliable managed communications solution,” wrote Lori Nordlien, a procurement officer at the state Department of Administrative Services. “IBM has materially failed to perform.”

The state then notified IBM that it had 30 days to find a solution.

“We apologize for the continued disruptions,” Pettit wrote to agency directors in January, according to records acquired by Willamette Week. “Put plainly, the outages and associated business impacts are unacceptable, and we have lost faith and confidence in the system.”

And while some of the state’s confidence appears to have been regained, Pettit said officials are now pressing IBM on whether it trusts its technical subcontractor, Unify Software and Solutions, a former business division of Siemens based in Munich, Germany.

“Do you stand behind your sub?” is what Pettit said they’re asking IBM, and added that the company hasn’t responded yet.

When StateScoop requested more information, an IBM spokesperson provided this statement: “IBM will continue to provide the services that the State has contracted us for. We are committed to the continued success of this project and we are working with the Department of Administrative Services to resolve the State’s concerns.”

Eaten alive

The state brought in IBM to consolidate several different aging phone systems. Some of the equipment is 30 to 35 years old, Pettit said. Originally, the state targeted a July 1, 2018, deadline to migrate workers off a dying CenturyLink contract. Today, the state looks to migrate an additional 1,571 phones onto the new system by May, but the work won’t be done there, even if IBM is able to stabilize service.

Because Oregon operates its phones over leased circuits, the cost of internet service is $1,039 per megabyte per month, according to internal documents. The state is paying about $11 million a year for connectivity. 

In Oklahoma, where Pettit was previously CIO, the state paid about $1 per megabyte per month, he said.

“If our design objective was to create the slowest network at the highest price, we’ve accomplished that,” Pettit said of Oregon’s legacy infrastructure. “That’s obscene. I’m getting eaten alive.”

But part of the IBM phone system upgrade is to offset the state’s connectivity costs. In his recent memo to agency heads, Pettit shared a map that shows where the state’s fiber lies — red lines connect major cities and cross the Blue Mountains and Cascade Range. Those closest to fiber will be connected first, Pettit said, and as DAS migrates users off leased circuits and onto state-owned fiber, those connectivity costs will begin to evaporate.

“But it’s not going to happen overnight,” Pettit said. “It’s going to take us about three years to do it.”