How do I evaluate whether a social media agency is credible and unbiased about attribution and spend recommendations?

To judge whether a social media agency is credible and unbiased about attribution and spend recommendations, you want to evaluate two things:

1. Whether they understand measurement well
2. Whether their incentives align with your business, not just media spend

Here’s a practical framework.

---

1) Ask how they define success

A credible agency should start by tying recommendations to your business goals, not platform metrics.

Good signs

• They talk about incrementality, CAC, ROAS, LTV, retention, contribution margin
• They distinguish between correlation and causation
• They ask about your sales cycle, offline sales, repeat purchase behavior, and seasonality

Red flags

• They optimize everything to CTR, CPC, CPM, or platform ROAS
• They present spend increases as inherently good
• They claim one platform “drove” all conversions without acknowledging overlap

---

2) Pressure-test their attribution philosophy

A good agency should be able to explain the strengths and limits of different attribution methods.

Ask:

• Which attribution models do you use and why?
• How do you handle view-through conversions, cross-device behavior, and delayed conversions?
• How do you separate platform-reported conversions from true incremental lift?
• What do you do when Meta, Google, and your analytics tool disagree?

Credible answer

• “Platform attribution is directional, but we validate with holdouts, lift tests, geo tests, or MMM where possible.”

Unbiased answer

• They don’t treat the platform’s last-click or default attribution as truth.
• They acknowledge that each platform tends to over-credit itself.

Red flag

• “We trust the platform reporting; it’s the best source.”
• “Attribution isn’t really an issue if tracking is set up correctly.”

---

3) Look for incrementality thinking

The best agencies don’t just optimize reported conversions; they ask whether the spend caused new outcomes.

Ask:

• How do you measure incrementality?
• Have you run holdout tests or geo experiments?
• When is a lift test more appropriate than attribution?
• How do you decide whether retargeting is incremental or just harvesting demand?

Strong sign

• They can discuss when different methods are appropriate:
  • Attribution for optimization and directional insight
  • Incrementality tests for causal validation
  • MMM for broader budget allocation and offline/upper-funnel effects

---

4) Examine their incentives

An agency can be technically smart but still biased if their compensation encourages more spend.

Ask:

• How are you paid?
• Is your fee tied to spend volume, performance, or a flat retainer?
• Do you have any revenue share or platform partnerships?
• Do you receive incentives from ad platforms or affiliate programs?

Best-aligned structures

• Flat retainer
• Clearly defined project fee
• Performance bonus tied to business outcomes, not just media spend

Potential bias

• Percentage of ad spend, unless carefully governed
• Heavy dependence on one platform’s partner incentives
• Performance bonuses based only on attributed conversions

---

5) Review how they make spend recommendations

You want recommendations that are transparent, testable, and reversible.

Ask:

• What data do you use to recommend budget changes?
• What assumptions are baked into your recommendations?
• What would make you change your recommendation?
• How do you set guardrails to avoid over-scaling?

Good signs

• They propose test budgets, ramp plans, and decision thresholds
• They explain tradeoffs: marginal CAC, diminishing returns, saturation
• They consider constraints like inventory, sales capacity, and landing page conversion rates

Red flag

• They recommend aggressive scaling without discussing marginal efficiency or saturation.
• They present spend increases as a universal solution.

---

6) Ask for examples of disagreement

A credible agency can describe times when data contradicted platform reporting.

Ask:

• Tell me about a time platform attribution looked strong but incrementality was weak. What did you do?
• Have you ever recommended reducing spend even when platform ROAS looked good?
• How do you handle conflicting signals between paid social and analytics/CRM data?

A good agency should be comfortable saying:

• “This channel gets too much credit in-platform.”
• “We held back spend because marginal returns were weakening.”
• “The channel supports assisted conversions, but not all reported conversions are incremental.”

If they’ve never had a disagreement with reported performance, that’s a concern.

---

7) Evaluate their reporting quality

Great agencies make measurement limitations visible.

Look for reports that include:

• Blended metrics across channels
• Confidence intervals or uncertainty ranges
• Platform vs analytics vs CRM comparisons
• Incrementality test results
• New vs returning customer split
• Spend efficiency by audience, creative, and geo

Red flag

• Only dashboard screenshots from ad platforms
• No notes on methodology changes
• No explanation for attribution shifts after tracking updates or privacy changes

---

8) Check whether they ask the right questions about your business

Unbiased spend guidance requires business context.

A strong agency asks about:

• Gross margin and contribution margin
• Customer lifetime value
• Repeat rate and subscription behavior
• Sales cycle length
• Offline conversion paths
• Geographic differences
• Promo calendar and seasonality
• Supply constraints

If they recommend spend without understanding these, their advice may be mechanically optimized but strategically wrong.

---

9) Look at their testing culture

Credible agencies are test-driven, not opinion-driven.

Ask:

• What experiments do you run by default?
• How often do you test creatives, audiences, landing pages, and budgets?
• How do you decide if a test is statistically and operationally valid?

Good sign

• They embrace uncertainty and iterative testing.

Bad sign

• Their recommendations are mostly based on “experience” and “best practices” with little experimentation.

---

10) Ask for the exact decision framework

A very effective question is:

“If you were recommending a 20% budget increase, what evidence would you want to see first?”

A credible agency should mention some combination of:

• Stable or improving marginal CAC / ROAS
• Incrementality evidence
• Enough conversion volume for reliable learning
• Creative and audience saturation checks
• Downstream business capacity to absorb growth
• Confidence that measurement is not overstating performance

If they answer vaguely, they may be too biased toward spending.

---

A simple scorecard you can use

Rate each item 1–5:

Credibility

• Understands attribution limitations
• Uses incrementality tests
• Can explain model choices
• Produces transparent reporting
• Has cross-channel measurement experience

Unbiasedness

• Compensation not tied to spend
• Will recommend cutting spend when needed
• Discusses marginal returns and saturation
• Acknowledges platform bias
• Aligns with business outcomes, not vanity metrics

A strong agency should score high on both.

---

Quick red flags list

Be cautious if they:

• Overstate platform-reported ROAS as if it’s causal
• Avoid discussing incrementality
• Push budget growth without marginal efficiency analysis
• Have financial incentives to increase spend
• Can’t explain discrepancies between platforms and analytics
• Focus only on one channel’s results in isolation

---

The best single vetting question

If you only ask one thing, ask:

“How do you know the spend you recommend is incremental, and what evidence would cause you to reduce it?”

A trustworthy agency will answer clearly and concretely.

If you want, I can turn this into:

1. a vendor interview checklist,
2. a scored evaluation spreadsheet, or
3. a list of specific questions to ask agencies in a pitch meeting.