Fierce Interrogators Ask a Series of Questions:

What is the source of the data?

Data and analyses rarely arrive at your desk at random. There are often intentions (good ones, but also possibly bad ones) behind how and why the data was collected and how and why the analyses are presented to you. Depending on the source and the intention, there could be possible agendas behind the data delivered to you. A good data interrogator asks: Does the data provider have a reason not to show me the entire data? If so, what are they likely to hide? For example, is it the marketing team that provides evidence about the success of the advertising campaign? Understanding the source of data, and the intent or possible agenda, can inform me about possible issues in the data that you want to pay closer attention to.

Are the metrics provided the ones we expected to see?

If not, why not? Are your data providers showing you the right KPIS? Are you being presented with vanity metrics that make the data provider look good?

How were the metrics calculated?

Many metrics have no clear definition. For example, when a company reports having 10 million customers, you want to ask yourself how customers are defined. Are customers everyone who ever visited the company’s website (even if they never bought anything), people who last bought from the company five years ago or only active customers who purchased in the past year? Depending on the agenda behind the data, the data provider may choose different metrics. Make sure that you understand the metrics, particularly those that are critical to your decision-making.

When and where was data collected? Are the time period, location and context relevant to the decision at hand? Should we make decisions about mobile wallet adoption in 2022 in

Hungary based on mobile adoption data in Austria in 2017? We happen to have accurate and reliable data from Austria, but no readily available good data from Hungary. Am I better off with accurate but less relevant and possibly outdated data from Austria or less precise but more current data from Hungary?

Are the comparisons being made to relevant and comparable alternatives?

Almost every company can look good if compared to the right competitor. If comparisons are made, are the metrics comparable across alternatives? Different companies may measure the same KPI (e.g. the number of customers) in different ways.

Are there other data points that may be relevant? Do you have this data over time so you can explore possible trends?

Is the data I am not seeing similar to the data I am seeing?

What data didn’t you capture? Who was left out? Could you have fallen prey to a nonresponse or a survival bias?

Was there any data the provider could not explain (outliers) and therefore did not show? Is there a pattern in the outliers that may prove valuable?

Note that to be able to ask or answer any of these questions, you don’t need to be a math whiz or a data scientist. You simply need curiosity, critical thinking and a good understanding of the context. Good intuition is the key component of great data interrogation.