Building Better Products: Leveraging the Pyramid of Evidence
A Step-by-Step Guide to Making Data-Driven Decisions at Every Stage of Product Development
What is the Pyramid of Evidence?
The Pyramid of Evidence is a hierarchical model used to gather and prioritize various types of evidence in product management. This structured approach ensures that decisions are based on the most reliable data available, minimizing biases and enhancing the accuracy of product development decisions.
The research methods that provide the most robust evidence are positioned at the top of the pyramid, whereas those yielding the least dependable evidence are found at the bottom.
The customer research methods at the top of the pyramid are supported by rigorous research design, quantitative analysis, and a systematic application of the scientific method. These methods encompass randomized controlled experiments and meta-analyses of such experiments.
The higher you move up the pyramid, the stronger the quality of the evidence and the lower the risk of bias becomes, however the difficulty also increases as you move higher in the pyramid in terms of technology investment, time, and human effort.
Background
Allot of the models used in tech product management are methodologies borrowed from and used by medical science, A/B tests or randomised controled experiments is how medicine if often tested and validated to be of benefit, the pyramid of evidence is also from the the medical model of clinical evidence.
Importance of Different Types of Evidence
As a product manager, being aware of the different types of evidence is crucial because it allows you to:
Validate assumptions with robust data.
Prioritize user needs and pain points effectively.
Make informed decisions that reduce the risk of product failure.
Systematically address issues with a structured approach, ensuring comprehensive analysis and informed strategic planning.
It's typically advisable to begin at the base of the pyramid, progressively collecting evidence and developing understanding over time and keep in mind to use a combination of customer research methods rather than relying on a single method to cover everything.
Business constraints, such as time and budget, can often limit your options. Nevertheless, consider how you can navigate these constraints to progress from weaker evidence to stronger evidence and assess these risks vs. the size of effort/investment you and your team are going to be put into building the product or feature.
Detailing Each Level and When Should You Use it in your Product Life Cycle
1. Expert Opinion & Best Practices
This level includes advice from industry experts, thought leaders, and established best practices.
While expert opinions and best practices can provide valuable guidance, they are often based on general industry trends rather than specific data about your product or users. Use this level to gain initial insights and formulate hypotheses, but be aware that these insights need further validation.
When to Use: Early in the product development cycle, when seeking direction or inspiration, and when you need to quickly get up to speed on industry standards. Beware of how biased this data is!
2. Case Research
Involves studying specific instances or examples of similar products or features, often documented in case studies or reports.
Case research can provide practical insights and real-world examples of what has worked (or not worked) for other products. This can help in understanding potential challenges and opportunities for your product.
When to Use: When exploring potential solutions or strategies and looking for proven examples from other companies or products. This can also be heavily biased data so be aware.
3. Cohort Analysis
A quantitative analysis method where a group of users is tracked over time to observe how their behavior changes.
Cohort analysis helps in identifying patterns and trends in user behavior, such as retention rates or feature adoption, over time. This level provides more reliable data than case research but still requires careful interpretation.
Keep in mind that in cohort analysis in randomisation is done and this is often used to identify correlation, not causation.
When to Use: During the growth and optimization phases, to understand user behavior trends and improve user engagement and retention.
4. Qualitative Research
In-depth research methods like user interviews, focus groups, and usability testing that provide insights into user motivations, behaviors, and pain points.
Qualitative research offers deep insights into the “why” behind user actions. It helps in uncovering user needs, preferences, and frustrations that are not easily captured through quantitative data.
When to Use: During the discovery and validation phases, to gather detailed user feedback and refine product concepts.
5. Quantitative Research
Data collected through surveys, analytics, and usage data that provide numerical insights into user behavior and preferences.
Quantitative research helps validate insights from qualitative research with statistical data. It allows for measuring user behavior, preferences, and satisfaction at scale, providing a solid foundation for decision-making.
When to Use: Throughout the product development cycle, especially during validation and optimization phases, to measure and validate hypotheses with robust data.
6. A/B Tests
Controlled randomised experiments where two or more variants of a feature are tested simultaneously to determine which performs better.
A/B testing provides empirical evidence of how changes to a product impact user behavior. It helps in making data-driven decisions by comparing the performance of different variations. This is one of the most often used method of validating hypothesis in the larger and more successful tech companies.
This is is the method that is also used to identify causation and the validation/invalidartion of a hypothesis.
When to Use: During the optimization phase, when you need to test specific changes or features and measure their impact on key metrics. This is the phase that a product typically has more traffic and users which facilities running experiments in shorter durations.
7. Meta-Analysis
Comprehensive analysis that combines results from multiple studies or experiments to draw more generalized conclusions.
Meta-analysis provides the highest level of evidence by synthesizing data from various sources. It helps in identifying overall trends and drawing more reliable conclusions.
When to Use: When you have conducted multiple studies or experiments and need to aggregate the findings to inform broader strategic decisions. This is often done by extremely large tech companies that have run tens of thousands of A/B tests and even at companies like Booking.com where at the time I was there and we had ran tens of thousands of experiments, meta analysis wasnt used (at least from my personal experience)
Applying the Pyramid of Evidence as a Product Manager
"Insight tells you what's happening. Context tells you why it's happening. Insight come from customers and competitors, from markets and industries, but most important, from our data.
Context comes from knowledge and experience. It applies perspective to the data.
Combining insights and context gives you the full-screen picture of what is happening, why it's happening and what needs to be done."
From the book Transformed by Marty Cagan
Here is my advise on how to start applying the pyramid:
Start with Anecdotal Evidence:
Gather informal feedback from users, team members, and market observations.
Use this to identify potential issues and areas for further investigation.
Conduct Qualitative User Research:
Organize user interviews and usability tests to delve deeper into user experiences.
Understand the motivations, behaviors, and pain points of your users.
Validate with Quantitative Data:
Implement surveys and analyze usage data to gather statistically significant evidence.
Use A/B testing to measure the impact of changes and validate insights from qualitative research.
Confirm with Experiments and Meta-Analyses:
Run controlled experiments to test specific hypotheses and determine causality.
Conduct meta-analyses to aggregate data from multiple studies for comprehensive insights.
I hope you’ve found this article useful, please subscribe if you haven’t and leave a comment with your thoughts or experience using (or not using) the pyramid.