5 Predictions for BI and Big Data

As Nate Silver points out often—and humorously—in his book The Signal and the Noise, the world is full of noisy data. Inside the untapped potential of Big Data and Business Intelligence is the signal. When the power of data is fully harnessed, it enables executives to transform productivity and act with certainty. My predictions for 2016 are about the trends that move businesses through the noise to truly leverage information as a strategic asset.

The Chief Data Officer Will Emerge As the Driver of Organizational Effectiveness and Competitive Success

Driven by the increased need for faster and more accurate decision making, the Chief Data Officer will be responsible for harnessing the full value of all of an organization’s information, driving innovation and cost take-outs, improving competitive advantage, and developing a full plan on how to make this data more widely accessible.

As competitive pressures demand further differentiation, an increased customer focus, and process efficiencies, the need has never been greater to turn data into a strategic advantage. The CDO will drive increased business value and will leverage open source frameworks like Hadoop, while basking in the innovation and opportunity brought by competition in the BI vendor market to meet this demand.

Businesses can no longer afford to wait for insights to bubble up to the top. By investing in the Chief Data Officer today, companies are prioritizing the overall plan for using data, launching analytics initiatives, and measuring their effectiveness. The CDO will introduce new levels of agility across the organization, driving ROI in both the short and long term by enabling the business to gain a comprehensive view of data and make more insight-based decisions, faster.

Data Dexterity Will Be Recognized as the Last Mile in Agile BI

While most will state their desire to compete on analytics, few are expert at it today.  Consider:

• Forrester’s “Global Business Technographics and Analytics Survey, 2015” states that business satisfaction with analytics output fell by 20% between 2014 and 2015.
• Gartner estimates that by 2017, 90% of information assets in a typical enterprise will be inaccessible.

Without greater access to more data by more people across the enterprise – what we call data democratization – decisions increasingly will be made based on a smaller percentage of an organization’s total intelligence.

The primary inhibitor of data democratization is a systematic lack of insight into what information is available and relevant.  In short, there is no easy way today for organizations to profile, identify, unify, and provision the right data for analysis. This group of key capabilities is what we call “Data Dexterity” and it ultimately:

• Allows the right people to get their hands on data quickly to inform decisions,
• Breaks down silos of information, and
• Turns data into a strategic asset.

Organizations Will Realize That Data Democratization Depends on the Middle Layer of the Data Stack

Visualization-based data discovery and analytic applications have been popular for many years now. These powerful apps will continue to proliferate in 2016 and enable line of business users to spot patterns, trends, and outliers more quickly than by relying on spreadsheets. Moving down the stack, Hadoop file systems for Data Lakes will become yet another data source for IT to manage and for BI professionals to mine.

With both the application layer and the data sources seeming reasonably fixed, therefore, the focus of 2016 will increasingly turn to optimizing the middle layer – the connective tissue that streamlines the interaction between data sources that IT governs and the applications that business users rely on. If you’re not agile in the middle, you’re not agile.

Forrester Research, Inc. states that 80% of any BI initiative is spent on data integration. ETL and cleansing are vital parts of that data prep process, but they address only a tiny fraction of the whole. In fact, Forrester also states that 64% of BI project time is spent just on profiling and identifying data sources.

The need to more efficiently catalogue and provision data for analysis will pick up steam in 2016, helping to drive further justification for existing BI investments.

Organizations Will Finally Realize that Self-Service Data Discovery is the Key to Dominating a Market Via Analytics

Data scientists generally have the knowledge and tools to find the data they need without much help. But they’re expensive, in short supply, and probably always will be. Hiring more data scientists is not the solution getting more value out of your data. It’s not practical and it just doesn’t scale.

It’s line-of-business data users that really need help. They’re the ones who are either making strategic and tactical decisions based on data—or creating the reports that executives need to make those decisions. And these users have self-service analytical tools. But what they don’t have is self-service data cataloging tools.

As the CDO gains more influence, companies will look for efficiencies in their data supply chains so they can deliver data-driven insights at the speed of business. Those that provide data self-service for the business will be way ahead of those that don’t.

 The Migration Away from Legacy Infrastructures Will Continue as Hadoop and Its Offspring Become More Mainstream

When a company like Teradata brings its flagship data warehouse to Amazon Web Services, which it will next year, you know that something major is afoot. Or when Informatica goes private in a $5.3B buyout. Mike Tuchen, CEO of Talend, calls it “a once-in-a-generation redefinition of the entire data-management stack.”

CDOs want a more agile infrastructure. And they’re the ones who will have to navigate the transition from proprietary, hardware-based, on-premise data management solutions to cloud-based, commodity hardware solutions that are optimized for Hadoop and the processing frameworks like Cloudera, Spark, and Hive that can handle big data analytics. The transition won’t be a Big Bang; it will take years.

To navigate the transition, companies will need to determine how to optimize for storage and usage. They’ll need to continue to feed the BI and analytics tools. Those that achieve agility in the middle layer of the data stack will be way ahead of the game.