Virginia overhauls election results website to combat crashes

The drama of an electoral contest for a wide open White House will likely attract millions of visitors to Virginia’s election results website in the coming months, and IT workers with the state said they’re finally ready to handle that kind of traffic after some 2014 missteps.

The Department of Elections’ IT team is now putting the finishing touches on a new cloud platform for its Web offerings, hoping to avert the slowdowns that plagued the state’s site in the last round of general elections when Virginia primary voters go to the polls in March.

“Going into 2016, I’m not worried about my election night reporting system,” Matthew Davis, the department’s chief information officer, told StateScoop. “I have plenty of other things to worry about, but that’s not one of them, which is great because it’s been many years since we’ve not had to worry about that site.”

Several of the site’s problems first emerged during the 2008 election, when the traditionally red state went for President Barack Obama. After Virginia developed a reputation as a battleground state, the site struggled to handle the influx of users who wanted to track returns.

“We’ve had firewall crashes, concurrent connection issues in cloud-hosting environments,” Davis said. “We’ve had databases just give up because of the load, and we’ve had servers fail.”

But the site’s performance during 2014 elections was what truly spurred the department to make big changes, Davis said.

When then-House Majority Leader Eric Cantor was upset in a primary contest in June, Davis noted that the state unexpectedly became national news overnight, putting a huge burden on the site that it often struggled to meet. Similarly, Democratic Sen. Mark Warner’s razor thin re-election in November caused substantial slowdowns of the site and even occasional crashes.

Those issues meant that Davis’ group was given “pretty heavy instructions to get a solution in place immediately” from the governor’s office, with a $30,000 appropriation to make it happen.

“We learned in our failure in November 2014 that we needed to eliminate database connectivity on our election night reporting site,” Davis said. “We needed to get the static content, because the amount of traffic and spike in traffic is wildly unpredictable.”

Initially, Davis said his team considered a move to a “third-party-hosted environment with standard-test servers that would scale up and scale down in the month before an election and the month after an election.” But the more they looked at that option, the more they found it was expensive and labor intensive, Davis said.

Instead, they settled on working with cloud provider Akamai to build a platform that can respond to those unpredictable spikes in traffic more seamlessly.

“We decided to go with a static website that we build on the back end, we have an automated process that builds the static views of every page every 10 minutes, and we simply upload that to the Akamai cloud,” Davis said.

The department paid the company $49,000 (with the help of some federal funds) to stand up the new system, with “no technical lift by my team to make this happen,” according to Davis.

Davis’ group then had the chance to test the new system both during June’s primaries and local elections, and in November when every seat in the state Legislature was up for election, and he believes it passed with flying colors.

“Our statewide reporting site handled over 25 million hits in a four-hour window, and nothing broke,” Davis said. “We didn’t receive a single media call about any issues or problems with our election night reporting system, everything just worked beautifully.”

With bigger tests yet approaching this March and November, Davis hopes to move the rest of the site’s pages to the Akamai cloud instead of just the results portion, as well as bolster the site’s firewall. But without the site’s stability to worry about, Davis said his staff will get the chance to focus on more innovative projects as well.

“We’re able to focus on adding additional information to the site instead of worrying about whether the site will stay up or not,” Davis said. “We’re working to add additional content that normally has not been something we’re able to provide the general public in a daily updated, or even hourly updated, format.”

Most of all, for a department that’s used to living by the ethos that “if it can go wrong, it will on election day,” Davis said his team is looking forward to building off this newfound success in 2016.

“I don’t know if I can say it without creating major problems for myself down the road, but I feel like, for once, we finally avoided Murphy’s law for an election,” Davis said.