How do I evaluate whether a genomics service provider is credible and unbiased about technical capabilities and assay performance?

To evaluate whether a genomics service provider is credible and unbiased about their technical capabilities and assay performance, use a mix of evidence-based checks, technical due diligence, and bias screening. The key is to look for verifiable data, not marketing claims.

1) Check the provider’s evidence, not just their claims

Ask for:

• Validation reports for each assay
• Limit of detection, precision, accuracy, sensitivity, specificity
• Reproducibility / repeatability data
• Inter-lab or external comparison results
• Failure rates, sample rejection criteria, and re-run rates
• Performance by sample type and input quality
• Benchmarking against known standards or reference methods

A credible provider should be able to show:

• What was validated
• How it was validated
• On what sample set
• Against what comparator
• Under what acceptance criteria

2) Look for independent quality and regulatory signals

Strong indicators include:

• CAP/CLIA accreditation or equivalent, where relevant
• ISO 15189 / ISO 17025 certification
• Participation in proficiency testing
• External quality assessment programs
• Publication of methods in peer-reviewed literature
• Use of validated SOPs and documented change control

These don’t guarantee excellence, but they reduce the chance of unsupported claims.

3) Scrutinize assay performance in context

Assay metrics can be misleading if presented without context. Ask:

• What were the sample types and cohorts used?
• Were the samples clinical, contrived, cell line, FFPE, fresh frozen, etc.?
• Were results stratified by:
  • input amount
  • quality metrics
  • tumor purity / contamination
  • GC content / coverage bias
  • variant allele fraction
  • read depth
• Was the assay validated for the intended use case or just a broader one?

A provider may have excellent performance in an easy sample set but poor performance in real-world specimens.

4) Watch for bias in how they present results

Potential bias red flags:

• Cherry-picked “best-case” examples only
• No mention of limitations or failure modes
• Performance shown only on successful samples
• Comparisons against weaker competitors or outdated methods
• Vague language like “industry-leading” without data
• Overstated clinical claims without regulatory support
• Conflicts of interest not disclosed in white papers or studies

A trustworthy provider will openly discuss:

• where the assay works well
• where it struggles
• what inputs are unacceptable
• what can cause false positives/false negatives

5) Ask about the full workflow, not just the sequencing chemistry

Genomics performance depends on more than the assay:

• sample collection and preservation
• extraction method
• library prep
• sequencing platform
• bioinformatics pipeline
• variant calling thresholds
• annotation and reporting
• manual review criteria

A provider should clearly distinguish between:

• wet-lab performance
• instrument performance
• computational pipeline performance
• interpretation/reporting performance

6) Evaluate transparency and reproducibility

Good signs:

• They provide methods details sufficient for reproduction
• They can explain parameter choices
• They can describe QC thresholds and how they were set
• They have version-controlled pipelines
• They can show how changes are handled over time
• They can reproduce prior results on request

Ask whether their pipeline is:

• locked or periodically updated
• revalidated after changes
• auditable
• standardized across clients

7) Ask for references and real-world use cases

Request:

• customer references in your application area
• case studies with concrete metrics
• sample performance summaries from similar projects
• evidence from independent collaborators

Be cautious if all references are:

• internally generated
• anonymous
• unverifiable
• from a different application domain

8) Assess scientific and commercial alignment

Bias can arise if the provider benefits from overselling capabilities. Ask:

• Do they sell the assay they are evaluating, or a competing product?
• Are they transparent about sponsorships?
• Are their publications sponsored or ghostwritten?
• Do they have a history of overstating performance?

Look for disclosure of:

• funding sources
• author affiliations
• vendor involvement in analysis and manuscript preparation

9) Conduct a pilot or blinded test

The best practical check is to run a small blinded pilot:

• Use known positive and negative samples
• Include edge cases
• Include replicates
• Compare against a trusted reference method
• Predefine acceptance criteria
• Review concordance and discordance carefully

This often reveals more than slides or brochures.

10) Use a structured vendor scorecard

Score each provider on:

• Technical validation quality
• Transparency of methods
• Independent accreditation
• Performance on relevant sample types
• Reproducibility
• Bioinformatics robustness
• Data/report clarity
• Responsiveness to technical questions
• Disclosure of limitations
• Evidence of conflicts of interest

Questions you can ask directly

Here are useful questions:

1. What is the assay validated for, exactly?
2. Can you share validation data and protocols?
3. What are the main failure modes?
4. How do you handle low-quality or borderline samples?
5. What external proficiency testing have you passed?
6. How often do you change the bioinformatics pipeline?
7. What gets revalidated after a change?
8. Can you provide blinded pilot results on our sample type?
9. What are your false positive and false negative rates?
10. What limitations should we know before starting?

Bottom line

A credible and unbiased genomics provider is one that:

• supports claims with data
• shows limitations openly
• uses independent standards
• is transparent about methods and conflicts
• can reproduce performance in your relevant sample type

If you want, I can also give you a vendor evaluation checklist, a red-flag list, or a scorecard template you can use to compare providers side by side.