In an uncertain world, people would cling to anything presented as a graph.
There’s a lot of research about buyer behavior, winning sales motions, and outstanding marketing tactics. While making data driven decisions, we should use a critical approach to these research papers, regardless of the amount of hyperbolic comments on the Linkedin comments to the post promoting them.
Here’s the way that we do it in PMG:
What is the source of the data?
Look for information about where the data comes from. Is it from a reputable source, like a well-known research firm or an academic institution? The credibility of the source can greatly impact the reliability of the data. I love the data that Kerry from 6sense, Udi from Gong, and the team at Ebsta are sharing. It is usually solid. The same goes for the more known analysts such as Gartner and Forrester.
How recent is the data?
In fast-moving industries like technology, data can become outdated quickly. Ensure the data reflects current trends and market conditions. That’s especially true when talking about data that is from the COVID years.
What was the sample size and demographic?
The larger and more diverse the sample, the more reliable the findings. Small or homogenous samples may not represent the broader market accurately. However, a targeted sample could be accurate if you are targeting that specific niche.
What methodology was used to collect and analyze the data?
Understanding the methodology provides insight into how conclusions were drawn. Look for transparency in how data was gathered, analyzed, and interpreted. As I’ve mentioned I really like the data from Gong and Ebsta as it is based on their own data collection, and they have a large enough customer base to gather meaningful and insightful data.
Are there any potential biases?
It’s amazing to see that every vendor has research that shows that the problem they solve is the number one consideration of their buyers, isn’t it? Consider whether the research could have been influenced by biases, either in the data collection process or from the entity conducting the research. For example, a vendor-sponsored study might skew results in favor of their product.
What assumptions were made?
All research makes some assumptions. Identify what these were and consider how they might affect the findings. I usually ask myself – which questions weren’t asked in this research and why?
How are the results presented?
Be wary of selective presentation of data or conclusions drawn without adequate support from the data. Look for comprehensive results that include both supporting and contradictory evidence.
Are the findings consistent with other research?
If the claims significantly deviate from established knowledge or other research findings, dig deeper to understand why. It’s important to consider the broader context and whether there’s a consensus among experts.
What do experts say?
Look for commentary or analysis from independent experts in the field. They can provide valuable insights and context that might not be obvious from the research alone.
How applicable is the data to your specific context?
Finally, consider how the findings relate to your own business or market. Even the most well-conducted research may not be directly applicable to your specific situation.
Data is just another form of storytelling – make sure that you know which book you are reading – and whether it’s a documentary or a fairy tale.
Data Skepticism: How to Spot Flaws in Data-Driven Claims
March 6, 2024
Kfir Pravda
In an uncertain world, people would cling to anything presented as a graph.
There’s a lot of research about buyer behavior, winning sales motions, and outstanding marketing tactics. While making data driven decisions, we should use a critical approach to these research papers, regardless of the amount of hyperbolic comments on the Linkedin comments to the post promoting them.
Here’s the way that we do it in PMG:
What is the source of the data?
Look for information about where the data comes from. Is it from a reputable source, like a well-known research firm or an academic institution? The credibility of the source can greatly impact the reliability of the data. I love the data that Kerry from 6sense, Udi from Gong, and the team at Ebsta are sharing. It is usually solid. The same goes for the more known analysts such as Gartner and Forrester.
How recent is the data?
In fast-moving industries like technology, data can become outdated quickly. Ensure the data reflects current trends and market conditions. That’s especially true when talking about data that is from the COVID years.
What was the sample size and demographic?
The larger and more diverse the sample, the more reliable the findings. Small or homogenous samples may not represent the broader market accurately. However, a targeted sample could be accurate if you are targeting that specific niche.
What methodology was used to collect and analyze the data?
Understanding the methodology provides insight into how conclusions were drawn. Look for transparency in how data was gathered, analyzed, and interpreted. As I’ve mentioned I really like the data from Gong and Ebsta as it is based on their own data collection, and they have a large enough customer base to gather meaningful and insightful data.
Are there any potential biases?
It’s amazing to see that every vendor has research that shows that the problem they solve is the number one consideration of their buyers, isn’t it? Consider whether the research could have been influenced by biases, either in the data collection process or from the entity conducting the research. For example, a vendor-sponsored study might skew results in favor of their product.
What assumptions were made?
All research makes some assumptions. Identify what these were and consider how they might affect the findings. I usually ask myself – which questions weren’t asked in this research and why?
How are the results presented?
Be wary of selective presentation of data or conclusions drawn without adequate support from the data. Look for comprehensive results that include both supporting and contradictory evidence.
Are the findings consistent with other research?
If the claims significantly deviate from established knowledge or other research findings, dig deeper to understand why. It’s important to consider the broader context and whether there’s a consensus among experts.
What do experts say?
Look for commentary or analysis from independent experts in the field. They can provide valuable insights and context that might not be obvious from the research alone.
How applicable is the data to your specific context?
Finally, consider how the findings relate to your own business or market. Even the most well-conducted research may not be directly applicable to your specific situation.
Data is just another form of storytelling – make sure that you know which book you are reading – and whether it’s a documentary or a fairy tale.