How do I evaluate whether a media buying agency is credible and unbiased about spend recommendations and reporting?

To judge whether a media buying agency is credible and unbiased about spend recommendations and reporting, look for evidence in three areas:

1. Incentives and business model
2. Measurement and reporting practices
3. Decision quality and transparency

Here’s a practical framework.

1) Check whether their incentives align with yours

A credible agency should be able to explain how they make money and whether that could influence recommendations.

Red flags

• They get paid a percentage of spend only
• They earn rebates, volume discounts, or platform incentives and don’t fully disclose them
• They push channels where they have preferred partnerships without clear rationale
• They are vague about whether media, labor, tech, and markups are separated

What you want to hear

• “Here’s our fee structure, and here’s how we avoid conflicts.”
• “We disclose all rebates, incentives, and any downstream compensation.”
• “Our recommendations are based on your business goals, not just spend growth.”

Questions to ask

• How are you compensated?
• Do you receive rebates, kickbacks, inventory incentives, or platform credits?
• Are those passed through, shared, or retained?
• Do you have preferred partner arrangements?
• Do you get paid differently depending on where you spend?

If they hesitate, that’s a warning sign.

2) Evaluate whether their recommendations are evidence-based

A strong agency recommendation should be tied to business outcomes, not just platform metrics.

Good signs

• They connect spend to incremental revenue, profit, CAC, ROAS, LTV, conversion lift, or brand outcomes
• They explain why a channel is being increased or decreased
• They distinguish between correlation and incrementality
• They show sensitivity to diminishing returns and saturation

Red flags

• “This channel is performing well” based only on platform-reported ROAS
• Recommendations that simply favor channels with better-looking attribution
• Overconfidence without uncertainty ranges or assumptions
• No mention of holdouts, experiments, or incrementality tests

Ask

• What metric drives your budget recommendations?
• How do you separate attributed conversions from incremental conversions?
• How do you account for diminishing returns?
• What would make you recommend spending less, not more?

A credible partner should be comfortable recommending cuts when a channel is inefficient.

3) Inspect the reporting for completeness and objectivity

Reporting should be designed to help you make decisions, not just to make performance look good.

Strong reporting includes

• Clear definitions of all metrics
• Raw delivery data and spend, not just summarized KPIs
• Attribution model used and its limitations
• Channel-by-channel performance plus total business impact
• Time lag consideration
• Notes on tests, changes, anomalies, and seasonality
• Both successes and underperformance

Red flags

• Only one attribution model, especially platform-native
• No transparency on methodology
• Reporting that cherry-picks favorable date ranges
• “Executive summaries” with no underlying data access
• Metrics that change definitions month to month

Ask

• Can we see the raw data behind the dashboard?
• What attribution model is used, and what are its limitations?
• How do you handle view-through conversions, cross-device behavior, and offline impact?
• Can we compare platform reporting to independent measurement?
• Do you report on tests that failed, not just those that succeeded?

4) Test for independence in recommendation-making

A credible agency should be able to challenge its own assumptions.

Signs of independence

• They propose experiments to validate claims
• They use third-party or client-owned measurement where possible
• They distinguish “what we think is happening” from “what we know”
• They acknowledge uncertainty

Red flags

• Overreliance on platform claims
• No outside validation
• Defensive reactions to skepticism
• They discourage experiments because they’re “too disruptive”

Best practice

Ask them to make at least one recommendation they would not make if they were optimizing only for higher spend. That reveals whether they’re truly acting in your interest.

5) Look at forecast and budget logic

Budget recommendations should follow a rational process.

Good process

• Start with business goals
• Estimate marginal returns by channel
• Consider constraints such as inventory, creative fatigue, seasonality, and target CAC
• Allocate budget to the next best dollar, not just the best historical channel
• Revisit regularly with updated data

Bad process

• Last year’s budget plus 20%
• Spend increases because “there’s room to scale”
• Channel allocations based on anecdotes or preference

Ask

• How do you estimate marginal return at the next dollar of spend?
• How often do you reforecast?
• What data would change your budget recommendation?
• How do you handle channels with uncertain attribution?

6) Ask for proof of past decision quality, not just good results

A smart agency can explain how it made decisions under uncertainty.

Ask for case examples

• Where they reduced spend in a “high-performing” channel because incrementality was weak
• Where reporting contradicted platform data
• Where an experiment caused them to change their strategy
• Where they missed and corrected course

You’re looking for judgment, not just polished wins.

7) Check operational transparency

They should be clear about how campaigns are built and managed.

Good signs

• You have admin access to ad accounts or at least visibility
• Naming conventions and change logs are clean
• Creative, audience, and bidding decisions are documented
• They share tests, learnings, and hypotheses

Red flags

• You don’t control the accounts
• They are opaque about who made changes
• There is no version history or decision log
• They won’t share campaign-level detail

8) Use an outside benchmark

If possible, compare the agency’s recommendations to:

• A second agency’s audit
• A consultant who doesn’t buy media
• Internal finance or analytics review
• Incrementality or geo/holdout testing
• Historical performance under similar conditions

If all parties independently converge on similar recommendations, that increases confidence.

9) Practical scorecard

You can score an agency from 1–5 on each item:

• Compensation transparency
• Disclosure of incentives/rebates
• Measurement rigor
• Use of incrementality testing
• Transparency of raw data
• Clarity of assumptions
• Willingness to recommend spend cuts
• Evidence of past course corrections
• Access to accounts and logs
• Consistency between reporting and business outcomes

If they score poorly on incentives and measurement rigor, be cautious even if the dashboard looks impressive.

10) A simple due diligence script

You can ask:

1. “How do you get paid, and what incentives might affect recommendations?”
2. “What independent evidence do you use to determine incrementality?”
3. “How do you handle platforms that overstate performance?”
4. “Can we review raw data and your assumptions?”
5. “When have you recommended reducing spend in a channel that looked strong in-platform?”
6. “What would cause you to change your recommendation?”
7. “Can we see failed tests and past mistakes?”

A credible agency will answer directly and specifically.

Bottom line

A media buying agency is more likely to be credible and unbiased if it:

• Has transparent compensation
• Discloses all incentives
• Uses incrementality-based measurement
• Provides raw, consistent, auditable reporting
• Makes recommendations based on marginal business value, not just spend growth
• Is willing to recommend lower spend when warranted

If you want, I can turn this into a one-page agency evaluation checklist or a list of interview questions for vendor selection.