Business Intelligence vs Data Science? Here’s Why BI Needs AI
A Vice President of BI & Analytics told me his budget is at risk. Why? His CFO perceives his group as a cost center within the broader $3 billion company, not a revenue generator.
This VP realizes things have changed in BI, and wants his team to be the first to take advantage of where things are headed in order to prove his CFO wrong. In order to do that, he'll need to uncover some revenue-generating insights that would otherwise be impossible with their current tool.
Tableau Software, Microsoft PowerBI, and Qlik will remain strong data visualization platforms. But ask any data scientist worth their salt, and they'll tell you that running machine learning models over pre-aggregated data just won't work.
ML is strong at detecting patterns in large datasets that a human would otherwise experience difficulty uncovering on their own. But in order for ML to do it's thing, you need to feed it each individual observation in the raw dataset.
In the race to add AI capabilities, the dominant BI players of yesteryear will find themselves hamstrung by their legacy, semantic-layer based architecture. Don't get me wrong, it made total sense when Qlik, Tableau, and PowerBI were built in 1993, 2003, and 2011 respectively.
Here's why BI platforms struggle
They weren't built for machine learning, they were built for data visualization. Which mattered a whole lot less, as most of the Fortune 500 wasn't ready for ML back then anyway.
Running on pre-aggregated data allowed these BI tools to run faster. This no longer makes a difference on speed. As we all know by now, cloud computing offers lightning fast compute speeds, as long as you have a big enough "box" - or server.
But isn’t BI good at "self-service" analytics?
Sort of. Certainly better than expecting business leaders to know how to query their own data via SQL or Python. But let's be real: how many C-suite executives have you ever seen slice and dice their own data in these BI tools? *If you know anyone who does, comment below!
Enter natural language search. When you've got a question in every day life, you google it and get instant answers. It's like magic, and we've been doing it since Google search launched in 1998. If only it was that easy to get answers in your own company's data.
Moving beyond What happened? to answer Why did things change?
Traditional BI is solid at surfacing what happened. But the next question analytics leaders get asked by their management is a very simple question that is oftentimes far too difficult to answer: “Why?”
Before I started working with a VP of analytics at a leading D2C e-commerce brand, he was managing hundreds of KPIs using Looker’s LookML. While he knew every KPI well, he didn’t know exactly why and how metrics like order frequency, new signups, and churn rate affected his most critical KPI, which he half-jokingly refers to as “money in the bank”.
Traditional BI requires human intuition to form a hypothesis behind why metrics are changing. Once the hypothesis is formed, an analyst has to do the slicing and dicing, and usually does bivariate analysis to see a snapshot of what happened as it relates to two variables. Chris Reuter, a former financial analyst at IBM, will tell you more about his experience doing it this way here.
But the next generation BI platforms that offer an underlying AI engine move beyond bivariate analysis to multivariate analysis to rapidly detect patterns in enormous swaths of data and automate root cause analysis. My favorite part of this capability is when my customers are oriented to insights that are totally counterintuitive. One market access analytics leader in the pharmaceutical industry was able to uncover $2MM in rebates that would have otherwise gone completely unnoticed. It’s like finding a needle in the haystack sitting in your data warehouse, but letting the computer find the needle for you!
"Okay, but why does BI need AI?"
Because you'll sound way smarter when you tell people outside our space what you do for work. What bullet below sounds better?
"I build reports and data visualizations"
"I find, visualize and narrate important findings, such as correlations, exceptions, clusters, drivers and predictions in data that are relevant to my executive leadership. We do it ourselves with some pretty powerful software instead of having to hire an army of data scientists."
Visualization tools only highlight visual relationships; they do not identify statistically significant findings. For example, a user may be able to see differences between bars on a bar chart, but unless the size differences are stark, it would require further analysis to determine whether the differences are relevant, statistically significant and actionable.
My two cents
If you simply need a data visualization platform, you won't get fired for buying one of these market leaders. Pretty data visualizations will always play an important role (as long as they continue to offer PPT export lol). Far too often, year long data science efforts fail to generate actual financial impact, because business leaders can't understand what the heck the model means. Visualizations matter.
But if AI is part of your data roadmap, you really ought to future-proof your organization with a platform that is capable of data visualization, and so much more.
Stay Current in Data Science Technology
Subscribe to our free weekly newsletter.
Subscribe to our weekly Data Science & Machine Learning Technology Newsletter
