How do I evaluate whether a retail media publisher is credible and unbiased?

To judge whether a retail media publisher is credible and unbiased, look at both the business model and the evidence behind their claims. A publisher can be useful and still have incentives that tilt results, so you want to separate marketing spin from verifiable performance.

1) Check who they are financially incentivized to favor

Ask:

• Do they primarily sell their own retail media inventory?
• Do they take fees from brands, agencies, or tech partners?
• Are they also a marketplace, first-party data provider, measurement vendor, or ad network?

Why it matters:

• If their revenue depends on you buying ads through them, they may emphasize metrics that make their platform look better.
• If they also control the marketplace and the measurement, there’s a risk of “self-graded homework.”

2) Look for transparent methodology

A credible publisher should clearly explain:

• How impressions, clicks, conversions, ROAS, and incrementality are defined
• Attribution window and model used
• Whether results are deduplicated across channels
• How they handle view-through vs click-through conversions
• What data sources feed their reporting

Red flags:

• Vague claims like “industry-leading performance” with no methodology
• No explanation of how conversions are attributed
• Benchmarks without sample sizes or date ranges

3) Ask whether results are independently verifiable

Good signs:

• Third-party measurement partners
• Audits or external validation
• Ability to export raw or near-raw data
• Compatibility with independent analytics platforms

Questions to ask:

• Can a third party validate the reported lift?
• Can we compare platform-reported results against our own analytics?
• Are the reports reproducible by another analyst using the same inputs?

4) Evaluate benchmark quality

A publisher may cite benchmarks, but those can be misleading if they’re:

• Based only on their best-performing advertisers
• From outdated periods
• From a narrow category or geography
• Not normalized for budget, seasonality, or campaign objective

Ask for:

• Sample size
• Time period
• Category breakdown
• Median as well as average
• Whether outliers were removed

5) Watch for cherry-picked case studies

Case studies are useful only if they include:

• Starting baseline
• Campaign objective
• Spend level
• Controls or comparison group
• Actual lift, not just attributed sales

Be cautious if:

• Every example is a success story
• No failed campaigns are discussed
• Outcomes are framed without context

6) Assess governance and disclosure

Credible publishers usually disclose:

• Sponsored content or paid partnerships
• Conflicts of interest
• Editorial standards
• Separation between sales and measurement/reporting teams

For retail media specifically, ask:

• Is the measurement team independent from the sales org?
• Do they disclose when results are self-reported?
• Are rankings or “top performer” lists influenced by advertisers’ spend?

7) Compare claims against your own data

The best credibility test is whether their numbers align reasonably with:

• Your ecommerce backend
• Your CRM / sales data
• Your media mix model
• Independent attribution tools

Discrepancies aren’t always a problem, but they should be explainable.

8) Use a simple credibility scorecard

You can rate a publisher on:

• Transparency of methodology
• Third-party validation
• Conflict-of-interest disclosure
• Data access/exportability
• Benchmark rigor
• Consistency with your own results
• Responsiveness to tough questions

If they score poorly on transparency and external validation, treat their claims as directional at best.

9) Ask these direct questions before trusting them

1. How do you define success metrics?
2. What attribution model do you use?
3. What portion of your reporting is self-reported vs independently measured?
4. Do you have third-party audits or partners?
5. Can you provide benchmark methodology and sample sizes?
6. Are sponsored placements clearly labeled?
7. What conflicts of interest should we know about?
8. Can we reconcile your reporting with our own data?

Bottom line

A credible, unbiased retail media publisher is one that is:

• Transparent about methodology and incentives
• Open to third-party validation
• Specific with data and benchmarks
• Consistent with independent evidence
• Clear about conflicts of interest

If you want, I can turn this into a one-page vendor evaluation checklist or a set of RFP questions you can send to retail media publishers.