The other day I Googled, “the problem with a modern data architecture.” Of course, at Attivio we’re big evangelists for an MDA, but it’s always interesting to see what the contrarians have to say. There were over three million returns, but none on the first two pages said a word about problems. Lots of articles about how to develop an MDA or how to optimize an MDA or why you had to have an MDA. You get the picture.
Once Chief Data Officers have identified, confronted, and hopefully overcome the challenges at the bottomand in the middleof the Big Data stack, what’s next? As Andrew Brust of Datameer notes, “At the top of the stack, there are seemingly endless choices. Whether Enterprise BI stalwarts, BI 2.0 challengers, or big data analytics players, the number of vendors and their similar positioning makes it really hard for customers.”
In my last post, I wrote about what Chief Data Officers tell us are the most pressing challenges they face at the bottom of the Big Data stack —mainly creating a hybrid data architecture that can accommodate modern and legacy data sources. Once that architecture is in place and we move up the stack, CDOs encounter another set of challenges.
Writing on the O’Reilly.com site back in August, CEO Jessie Anderson of Smoking Hand, a training company for Big Data technologies, commented on the overall complexity of Big Data, NoSQL technologies, and the distributed systems that deploy them.
Chief Data Officers certainly have first-hand knowledge of this complexity and the hurdles it presents to extracting the maximum value out of business data. Complexity takes a variety of forms throughout the Big Data stack. Let’s start at the bottom.
Chief Data Officers have a lot of things on their plates. And one of those things is giving users freer access to the data they need. This is what we call "data democracy." Most CDOs like the idea of data democracy in theory. But in practice, the CDOs we talk to find that efforts to create a data democracy face at least four common barriers:
If you're a CDO, how would you describe your most important role: as gatekeeper or innovator? Or are you walking a tight rope between the two? Those questions figured prominently at the 10th annual MIT Chief Data Officer & Information Quality Symposium held in July.
When the role first emerged, gatekeeper probably occupied most of a CDOs waking hours. Nevertheless, as more organizations became aware of just how much value could be derived from their data, expectations have changed. Yes, CDOs still need to keep enterprise data safe, but they also need to keep the data supply chain running smoothly for data scientists and business analysts.
At the end of May, the city of Boston named Andrew Therriault as its first CDO—chief data officer. Therriault, former Director of Data Science for the Democratic National Committee, comes with an impressive background. He has a B.A., M.A. and PhD. in politics from NYU. Before joining the DNC, he served as senior data scientist with Greenberg Quinlan Rosner Research, whose client list includes global companies, advocacy groups, and political organizations as well as former presidents. So, he's certainly qualified.
The Chief Data Officer has never been a more necessary role in the organization than it is today. Organizations capture and store more data than ever before, and it’s growing exponentially every year.
Not only is business data growing, but we are seeing new types of data continually entering the mix. Data is structured, unstructured and semi-structured. It’s stored in big data lakes, in business applications, in file shares, and other places across the organization. There’s so much data that even the CDO isn’t completely aware of what’s out there.
And then there’s the external data. CDOs are becoming aware of the need to bring in external data sources that provide relevant, and sometimes essential, information to support decision making.
As the Big Data and analytics parade marches on, I often find that the people we're talking to in large enterprises carry the title chief data officer or CDO. Industry analysts back this up. A 2015 report by PwC found there were 100 CDOs in large enterprises in 2013, more than double the number in 2012. Gartner's most recent tally pegs the number at 950. And it predicts that by 2017, 50 percent of all companies in regulated industries will have a CDO. The CDO role is still evolving, but Debra Logan of Gartner describes the CDO as the "glue between data strategy and metrics."
There is no mistaking the ever greater demand for data — good and usable data for a variety of purposes. The demand is growing fast for data to sharpen focus on customers, help make internal processes run leaner, and to lend certainty to strategic decisions, to name a few. Yet too often the right data can’t be harnessed and remains untapped.
