How do I evaluate whether a wearable ECG device is credible and unbiased for clinical use?

To judge whether a wearable ECG device is credible and unbiased for clinical use, look for evidence in five areas: the device itself, the study design, the reference standard, the analysis, and real-world performance.

1) Check whether the device is being compared fairly

Ask:

• What clinical question is it intended to answer?
  • Heart rate only?
  • Rhythm screening, especially atrial fibrillation?
  • Single-lead vs multi-lead ECG diagnosis?
• Is the device cleared for that specific use?
  • Regulatory clearance/approval doesn’t prove accuracy, but it shows the intended use was reviewed.
• Is the population similar to your patients?
  • Age, skin tone, BMI, arrhythmia prevalence, motion level, inpatient vs outpatient, etc.

2) Evaluate the study design for bias

A credible clinical validation study should be:

• Prospective, not just retrospective convenience sampling
• Independent, ideally not run solely by the manufacturer
• Pre-registered with a defined protocol and endpoints
• Blinded, meaning:
  • Readers of the wearable ECG should not know the reference diagnosis
  • Interpreters of the reference ECG should not know the wearable result
• Consecutive or representative enrollment, not cherry-picked “easy” cases

Red flags:

• Only healthy volunteers
• Overrepresentation of obvious arrhythmias
• Selective exclusion of poor-quality tracings
• No mention of missing data or uninterpretable recordings

3) Make sure the reference standard is strong

The device should be compared against an appropriate gold standard, such as:

• A 12-lead ECG for morphology-related diagnosis
• Telemetry, Holter, patch monitor, or electrophysiologist adjudication for intermittent rhythm detection

Questions to ask:

• Was the reference test performed at the same time or close enough that rhythm wouldn’t change?
• Was there expert adjudication if the diagnosis was uncertain?
• Were discordant results reviewed systematically?

4) Look at the right performance metrics

Do not rely only on overall accuracy. Ask for:

• Sensitivity and specificity
• Positive and negative predictive value
• Confidence intervals
• AUC/ROC, if appropriate
• Calibration if the device provides probabilities
• Uninterpretable rate / signal dropout rate
• Performance by subgroup

For screening devices, especially consider:

• False positives can cause unnecessary anxiety and testing
• False negatives can miss serious disease
• PPV depends heavily on disease prevalence, so a high PPV in a high-risk study may not generalize

5) Check for subgroup bias and fairness

A wearable ECG can be biased if it works better in some groups than others. Look for performance stratified by:

• Sex
• Age
• Race/ethnicity and skin tone
• BMI/body habitus
• Fitness level
• Comorbidities
• Motion/artifact burden
• Wrist vs chest wear location

If subgroup results are missing, that is a limitation. If they are present, look for large performance drops in any group.

6) Assess whether the analysis inflates performance

Watch out for:

• Training and testing on overlapping data
• Multiple recordings from the same patient used in both sets
• Heavy preprocessing or manual cleanup not feasible in real use
• Threshold tuning on the test set
• Reporting only the best-performing subgroup or time window

A strong study should clearly separate:

• Training data
• Validation data
• External test data

Best evidence comes from external validation at a different site, with different users and devices.

7) Consider real-world usability and signal quality

Clinical credibility is not just about algorithm performance. Ask:

• How often is the ECG actually interpretable?
• How often do motion, poor contact, or sweat degrade the signal?
• Does the device fail more often in dark skin, hair, obesity, tremor, or low perfusion?
• Does it require ideal placement or user skill?

A device that is accurate only under controlled conditions may be less credible clinically.

8) Look for transparency and reproducibility

Credible devices are supported by:

• Published methods
• Clear labeling of intended use
• Disclosed funding and conflicts of interest
• External replication by independent investigators
• Access to performance details, not just marketing claims

Red flags:

• “AI-powered” with little methodological detail
• No external validation
• No description of dataset composition
• Claims based only on technical validation, not clinical validation

9) Ask whether the device improves clinical decisions

The most important question is not just “Is it accurate?” but:

• Does it change management appropriately?
• Does it reduce time to diagnosis?
• Does it reduce missed diagnoses without causing excessive false alarms?
• Does it improve outcomes or at least decision quality?

A device can be accurate and still not be clinically useful if it leads to too many alerts or delays.

A practical checklist

A wearable ECG device is more credible if it has:

• Independent, prospective validation
• Clear intended use
• Representative patient population
• Strong reference standard
• Blinded interpretation
• External validation
• Subgroup performance reporting
• Uninterpretable-rate reporting
• Transparent funding/conflict disclosure
• Evidence of clinical utility, not just algorithm accuracy

Quick bottom line

For clinical use, trust a wearable ECG device only if it has been validated prospectively, against a strong reference standard, in a population like your patients, with blinded interpretation, external testing, and subgroup reporting. If it lacks transparency, external validation, or subgroup analysis, treat its claims cautiously.

If you want, I can turn this into a 1-page appraisal checklist you can use to review a paper or product brochure.