Here’s a practical way to judge whether a multimodal model publisher is credible and not cherry-picking benchmark results.

1) Check whether the benchmark is actually relevant

Ask:

• Is the benchmark public and standard, or a custom/internal test?
• Does it match the model’s intended use case?
• Is multimodal performance measured across all relevant modalities (image, audio, video, OCR, chart understanding, etc.)?

Red flag: they highlight one benchmark where they win, but it’s not representative of real-world use.

2) Look for full methodology, not just headline numbers

Credible claims usually include:

• Model version and checkpoint
• Exact prompt format
• Decoding settings (temperature, top-p, etc.)
• Whether tools or external retrieval were allowed
• Whether few-shot examples were used
• Whether answers were post-processed

Red flag: “We achieved X” with no details on evaluation setup.

3) Compare against strong baselines fairly

A fair comparison should:

• Use the same benchmark split
• Use the same evaluation protocol
• Include strong, current baselines
• Avoid comparing to outdated versions of competitors

Red flag: comparing their latest model to an old baseline, or to numbers from a different paper/setup.

4) Check for cherry-picking across datasets

Look for:

• Whether they report all major benchmarks, not just the best ones
• Whether they show average performance or a broad scorecard
• Whether they disclose weak areas as well as strengths

Red flag: only one or two metrics are shown, especially if those are the ones they’re best at.

5) Inspect statistical rigor

Credible reporting often includes:

• Confidence intervals or error bars
• Multiple runs, if the evaluation is noisy
• Clear explanation of tie-breaking or scoring rules

Red flag: tiny claimed improvements without evidence they’re statistically meaningful.

6) See if the benchmark is saturated or easy to game

Some benchmarks become unreliable because models can:

• Memorize training data
• Overfit public leaderboards
• Exploit quirks in the scoring script

Ask:

• Was the benchmark in the training data?
• Is it public and widely overused?
• Are they relying on narrow prompts that make the task easier than normal?

Red flag: huge gains on a benchmark that the field already considers “solved” or gameable.

7) Look for external validation

The most trustworthy claims are supported by:

• Independent academic reproduction
• Third-party evals
• Leaderboards run by neutral orgs
• Open-source eval code and artifacts

Red flag: only the publisher’s internal evaluation supports the claim.

8) Check whether the publisher is selective about comparisons

A biased presentation may:

• Choose only favorable subsets
• Exclude hard examples
• Use different budgets across models
• Compare a larger model to a smaller competitor
• Use proprietary tools for their model but not others

Ask explicitly:

• Were all models evaluated with the same context window, image resolution, tool access, and compute budget?
• Were any examples removed?

9) Assess disclosure of limitations

Credible publishers usually state:

• Where the model fails
• Which modalities are weak
• What evaluation is missing
• Known failure modes like hallucination, OCR errors, temporal reasoning, or grounding issues

Red flag: the announcement sounds flawless and omits caveats.

10) Prefer claims that can be reproduced

Strong signs of credibility:

• Open evaluation code
• Public prompts
• Exact benchmark links
• Clear reproducibility instructions
• Community-replicated results

If you can’t reproduce it, treat it as a marketing claim until proven otherwise.

---

Quick credibility checklist

A publisher is more credible if they:

• Use public, relevant benchmarks
• Provide full evaluation details
• Compare fairly to strong baselines
• Report multiple metrics, not just winners
• Include uncertainty or multiple runs
• Allow independent reproduction
• Disclose limitations

A publisher is less credible if they:

• Only report favorable results
• Hide methodology
• Compare unfairly to competitors
• Use vague language like “state of the art” without details
• Avoid independent validation

---

Good questions to ask them

1. What exact benchmark version and split did you use?
2. What prompts and decoding settings were used?
3. Did you allow tools, retrieval, or external OCR?
4. Were comparisons run under identical conditions?
5. How many runs were averaged?
6. Is the code or evaluation script public?
7. Has any third party reproduced the result?
8. What are the model’s weaknesses on the same benchmarks?

If you want, I can turn this into a one-page due diligence template or a scorecard you can use to evaluate model announcements.