How do I choose between different customer insight agencies for a product owner planning research before investment?

For a product owner planning research before investment, the best agency is usually the one that helps you make a clearer investment decision, not just the one with the most polished methodology deck.

1) Start with the decision you need to make

Before comparing agencies, define:

• What decision will this research inform?
  e.g. “Should we invest in this feature / product / market?”
• What level of certainty do we need?
  e.g. directional insight vs evidence robust enough for a business case
• What do we already know?
  avoid paying for discovery you already have
• What’s the deadline and budget?

Agencies should be judged on how well they support that decision, not just on research credentials.

2) Compare agencies on the right criteria

Here’s a practical checklist:

A. Strategic fit

• Do they understand product, commercial, and investment decisions?
• Can they work from a fuzzy brief and help sharpen it?
• Do they ask the right questions about risk, assumptions, and expected outcomes?
• Can they translate insight into implications for roadmap, prioritization, or go/no-go decisions?

B. Methodology fit

• Do they recommend the right method for the question?
  • Qual interviews for motivations, needs, and barriers
  • Quant surveys for sizing, prioritizing, and validating patterns
  • Concept testing / message testing / usability / diary studies depending on the problem
• Do they explain why a method is appropriate?
• Are they proposing too much, too little, or the wrong kind of research?

C. Decision usefulness

• Will outputs be usable by product, leadership, and finance?
• Do they provide actionable recommendations, not just findings?
• Can they quantify confidence, risk, or opportunity when needed?
• Will they help you answer “so what?” and “now what?”

D. Sector and audience experience

• Have they researched similar audiences, markets, or product types?
• Do they understand B2B vs B2C differences, regulated sectors, technical users, etc.?
• Can they recruit the right participants realistically?

E. Practical execution

• Timeline realism
• Quality of recruitment
• Sample size appropriateness
• Data quality controls
• Project management and communication
• Ability to adjust if the scope changes

F. Commercial fit

• Clarity of scope and deliverables
• Transparency on costs, assumptions, and exclusions
• Good value, not just low price
• No hidden fees for recruitment, analysis, workshops, or reporting

3) Ask agencies these questions

Use these questions in pitches or conversations:

1. What would you recommend we research first, and why?
2. How would you reduce uncertainty for an investment decision?
3. What assumptions would you want to test?
4. What would the final output look like for product and leadership?
5. How do you ensure findings lead to decisions, not just reports?
6. What are the risks in this project, and how would you handle them?
7. Can you give an example where your research changed a product investment decision?
8. How do you recruit the right people and avoid bias?
9. What would you do if the brief is too broad or the budget is too small?
10. How do you handle mixed methods if both depth and scale are needed?

4) Watch for red flags

Be cautious if an agency:

• Pushes a one-size-fits-all method
• Talks more about process than the decision you need to make
• Can’t clearly explain sample size or participant selection
• Overpromises certainty from small samples
• Gives generic outputs with limited strategic interpretation
• Seems unable to work with product and commercial stakeholders
• Is vague about cost, timing, or what’s included

5) Use a simple scoring matrix

Score each agency 1–5 on:

• Understanding of your business problem
• Methodological fit
• Strategic recommendations
• Relevant sector/audience experience
• Recruitment and data quality
• Communication and collaboration
• Cost transparency
• Overall confidence

Then weight the most important factors more heavily, usually:

• Decision usefulness
• Strategic fit
• Relevant experience
• Methodological fit

6) Choose based on the type of research you need

• If you need to understand why people behave a certain way: choose strong qualitative specialists
• If you need to understand how many, how often, or which segments matter most: choose strong quantitative specialists
• If you need both insight and evidence for investment: choose a mixed-methods agency or a team that can coordinate qual + quant well
• If the question is highly commercial or market-facing: consider agencies that combine research with strategy

7) Final rule of thumb

For a product owner planning research before investment, the best agency is the one that:

• helps sharpen the question,
• chooses the right level of evidence,
• produces decision-ready outputs,
• and understands the product/business context.

If you want, I can also give you:

• a 1-page agency evaluation scorecard, or
• a sample brief you can send to agencies for comparison.