In praise of ad hoc analysis

Thinking in repeatable, efficient systems is the bedrock of smart business intelligence and data science strategies. Within business intelligence, the very nature of self-service analytics dashboards is to enable business users to see and understand their data and reduce the need (and turnaround time) for ad hoc analysis. With a successful business intelligence strategy in place, businesses are often able to answer 80% of the questions that come up in less than a minute. There’s no question that as you scale your business intelligence strategy, the nature of your organizational decision-making and your culture evolve right along with it.

When it comes to data science work, there is a huge industry focus on using sound engineering principles and productionizing code. It’s incredibly important to ensure your code uses resources efficiently and is compatible with your production ecosystem. As data science has evolved, specializations such as machine learning engineering and MLOps teams have formed to focus on these processes. It is no wonder that data science leaders and new data scientists alike are building their strategies around repeatability and scalability – organizations don’t get the full value out of data science without such a focus!

Understandably, ad hoc analysis has taken a backseat in many analytics strategies. It’s not just in the data world – “one-off” is a dirty word to any CEO. But, if it is used along with robust systems-first strategies, there should always be a place for ad hoc analysis. Even in organizations where it is implemented, though, ad hoc work is often prioritized below other work in an effort to be scalable and efficient.

There is a very strategic place for ad hoc analysis, and if you don’t plan for it, you will either be missing out on a wealth of insight or caught blindsided by an urgent need that risks putting other deliverables at risk of delay.

Here are some great reasons for analysis that (at least initially) is entirely a one-time effort:

• Deeper dive – go beyond aggregated data and known key drivers, and you might find new ways to tell your data story or new features to use in your models.
• Edge case exploration – dig into the anomalies and outliers, and you might find a whole new set of questions to ask about data collection or other patterns.
• Visualize – create a custom visual that succinctly explains the issue, even if this is just a step in your own understanding.
• Challenge conventional wisdom – analysts are not often directly asked to challenge the business assumptions, but it’s always important to confirm, quantify, or even refute anything everybody already “knows” to be a fact.
• Record what isn’t found – in exploratory analysis, a non-finding is often as useful as a finding. Document inconclusive results and what they mean to the business so nobody else has to spend time repeating this exercise again.
• Create new questions – a good analysis often leads to more and better questions. While frustrating to those not used to it, these new questions and hypotheses are often the most valuable outcomes of ad hoc analysis.

Remember, to make ad hoc analysis successful, you should always:

• Define the business question or issue – take your time to get this right, or you won’t know when you’re done.
• Make it tangible – set your anticipated deadline and what you expect your output to be, so that your colleagues know what to expect when. It can range from “I’ll email you by the end of the day with an answer” to “expect a full presentation with detailed recommendations at the end of the quarter.”
• Don’t start from scratch – there is likely already an existing model or dashboard that comes close to solving your issue, or perhaps the one that spun off the issue, so don’t reinvent the wheel.
• Get feedback early – bring in outside perspectives and stakeholders early and often so that time isn’t wasted and something critical isn’t missed in your final analysis.
• Return to systems thinking – at the end of this process, reflect on how the findings should be integrated into the business intelligence and data science ecosystem of processes and assets within your organization.

Ad hoc analysis is an important part of business intelligence and data science strategies. It can provide valuable insights, generate new questions, and play a key role in moving your data and insights capabilities forward.