How do I evaluate whether a health data aggregation provider is credible and unbiased?

To evaluate whether a health data aggregation provider is credible and unbiased, look at three things: where their data comes from, how they process it, and whether their incentives could distort the results.

1) Check data provenance

Ask:

• What are the original data sources? EHRs, claims, labs, pharmacies, patient-reported data, wearables, public datasets?
• Can they name the source mix by percentage?
• How current is the data? Real-time, daily, monthly, delayed?
• Do they show the chain of custody? Can you trace a datapoint back to the originating system?

Red flags:

• “Proprietary data network” with no source transparency
• No explanation of sample coverage or geography
• Heavy reliance on one source type presented as universal truth

2) Evaluate methodology

Ask:

• How do they clean, normalize, and deduplicate records?
• How do they handle missingness, outliers, and conflicting records?
• What are their entity-resolution rules?
• Do they publish validation studies or accuracy benchmarks?
• Are their methods audited or peer reviewed?

Red flags:

• No documentation of preprocessing
• Black-box scores or derived fields with no methodology
• No reproducibility or quality metrics

3) Assess bias and representativeness

Ask:

• Who is included and who is missing?
• Does the dataset overrepresent certain payers, regions, hospital systems, ages, or socioeconomic groups?
• Do they measure and disclose coverage gaps?
• How do they adjust for sampling bias?
• Do they report performance separately across subgroups?

Red flags:

• Claims of “national representativeness” without evidence
• No subgroup analysis
• One client segment dominates the dataset

4) Review conflicts of interest

Ask:

• Who pays them, and for what purpose?
• Do they sell analytics to both sides of a market or support competing interests?
• Are outcomes tied to sales, product adoption, or investor goals?
• Do they disclose sponsorships or commercial relationships?

Red flags:

• Provider also markets a product that benefits from favorable findings
• Selective publication of results
• “Research” output that reads like marketing

5) Look for governance and compliance

Ask:

• Do they have HIPAA, SOC 2, ISO 27001, or equivalent controls?
• Are there IRB or ethics review processes when applicable?
• How do they manage access, consent, and de-identification?
• Do they have data retention and deletion policies?

Important: Compliance is necessary, but compliance alone does not prove unbiasedness.

6) Demand independent validation

Ask for:

• External audits
• Third-party benchmark comparisons
• Published methods in journals or conference proceedings
• Customer references that used the data successfully for the same type of question
• Reproducible sample outputs or validation against known ground truth

7) Test the provider with hard questions

A credible provider should answer clearly:

• “What are your biggest coverage gaps?”
• “Where is your data least reliable?”
• “What types of conclusions should not be drawn from your data?”
• “How do you prevent overclaiming causality from observational data?”
• “How do you distinguish signal from commercial incentives?”

8) Use a practical scorecard

You can score each dimension 1–5:

• Data provenance transparency
• Methodological rigor
• Bias disclosure
• Conflict-of-interest management
• Security/compliance
• Independent validation
• Responsiveness to scrutiny

A provider is more credible if they:

• Are transparent about limitations
• Publish methods
• Allow external validation
• Disclose commercial relationships
• Avoid overstating generalizability

Quick rule of thumb

If they say:

• “Trust us, it’s proprietary”
• “Our data is comprehensive”
• “Our analytics are unbiased”
• “We can’t share methods because they’re confidential”

…that’s a warning sign.

If you want, I can also give you:

1. a vendor due-diligence questionnaire, or
2. a scorecard template you can use during procurement.