Big data storage continues evolution in state, local government

The way state and local governments are thinking about data storage is changing.

Shaun Bierweiler, vice president of public sector for Hortonworks, says in an interview with StateScoop that state and local governments are evolving that storage mentality due to culture change, volume of data and expectations around data.

We like to say that every agency is a data agency, and that stems from the evolution and the significance that data has taken in the lives and in the missions of our customers,” Bierweiler says.

In the past, data in traditional data warehouses was used in a transactional way but now, it’s at the center of every decision, Bierweiler explains.

The structure of big data itself has evolved, he says.

Previously, you knew what was going in and what was coming out,” Bierweiler says in the interview. “Today, now you have data from an infinite number of sources. You have images, you have videos, you have data encrypted within those items. The data itself has become very much more complex in terms of structure.

The volume of data, however, is the biggest change.

Agencies are drowning in data because theres so much of it, Bierweiler says. You have to be able to store it, you have to be able to process it. You have to be able to extrapolate the value from that data. And so thats become much more complicated and complex.

Expectations for the use of that data has changed drastically as well, Bierweiler explains.

Not only do you have more data that has more information that varies much more greatly, but now users expect to do more with it, “Bierweiler says.” And they not only expect to do valuable things with their data, but they expect to extrapolate information and sharing data from other users data. What used to be very traditionally stove-piped and siloed now is a mesh of data thats expected to be shared.

With such an array of data types, sizes and uses, Bierweiler advocates for state and local agencies to embrace enterprise open source platforms to address users many needs.

If you look at a traditional proprietary technology, the lifecycle for them tends to be much longer, and the development cycle even longer, Bierweiler says. When you get a new release of a proprietary solution, its often with very old or antiquated solutions and its solving the problems that existed when the technologies development model started.

An enterprise open source platform like Hortonworks harnesses the development model of community people that aren’t paid by Hortonworks. What you get then is a very open solution that not only solves what people are trying to address today, but problems they foresee for tomorrow, Bierweiler tells StateScoop.

Because there aren’t barriers or proprietary interfaces on open source platforms, it lends itself to a true best-of-breed solution.

Consider everything as possible, he recommends to agencies and offices considering open source. Its often difficult to make that cultural shift from something that you’ve always done and you convince yourself that thats the only way. Technology has come a very long way and there are creative ways to do things better, cheaper, faster, smarter. So oftentimes, the biggest challenge we have is not a technical hurdle its a cultural shift.

Learn more about how Hortonworks’ open source solutions can help you manage your data.

This podcast was produced by StateScoop and underwritten by Hortonworks.