If sample size, significance, and false positives matter to you, look for an A/B testing tool that does more than just run split tests and show a “winner.” The key is whether it supports valid statistical inference and helps you avoid peeking bias and underpowered tests.

Here’s what to prioritize:

1) Proper sample size planning

Look for a tool that helps you calculate or validate:

• Minimum sample size
• Minimum detectable effect (MDE)
• Baseline conversion rate
• Desired statistical power (commonly 80% or 90%)
• Significance level (commonly 5%)

Why it matters:

• Too small a sample leads to false negatives and noisy results.
• A tool that can’t estimate required sample size is risky, because you may stop too early and trust random variation.

2) Clear statistical significance handling

Make sure the tool is explicit about:

• P-values
• Confidence intervals
• Statistical significance thresholds
• Whether it uses frequentist or Bayesian methods

You want the tool to show not just “Variant B won,” but:

• How confident it is
• The uncertainty range
• The actual statistical method used

Watch out for tools that use vague “probability to win” language without explaining the model.

3) Controls against false positives

False positives often happen because of:

• Peeking at results too often
• Running tests too many times without correction
• Segmenting into many subgroups and cherry-picking winners

Good tools should offer:

• Sequential testing or always-valid p-values
• Multiple comparison corrections if you test many variants or metrics
• Guardrails against early stopping bias
• Clear documentation on whether repeated looks at data are safe

If the tool just says “check results anytime” without statistical safeguards, that’s a red flag.

4) Support for multiple metrics without inflating error rates

If you track several metrics, the tool should help you manage:

• Primary vs secondary metrics
• Family-wise error rate
• False discovery rate

Otherwise, you may get a “win” on a vanity metric that isn’t real.

5) Randomization quality and assignment integrity

A valid test depends on assigning users correctly. Look for:

• Stable bucketing / consistent user assignment
• Traffic allocation control
• SRM detection (sample ratio mismatch)
• Bot/internal traffic filtering
• Cross-device identity handling if relevant

Poor assignment can create apparent significance where none exists.

6) Transparent methodology

The tool should document:

• Statistical model and assumptions
• How variance is estimated
• How confidence intervals are computed
• How missing data is handled
• Whether conversion events are delayed or censored

Opaque black-box results are harder to trust.

7) Power analysis and test duration estimates

A strong tool will estimate:

• How long the test needs to run
• Expected duration based on traffic and conversion rate
• Whether your chosen effect size is realistic

This helps prevent stopping because you’re impatient rather than because the data are good.

8) Guardrails for sequential decision-making

If you’ll make decisions before the test ends, look for:

• Sequential analysis
• Bayesian stopping rules
• Always-valid inference
• Or at least explicit warnings about early stopping

Without this, repeated monitoring increases false positives.

9) Ability to define a single primary outcome

To reduce false positives:

• Define one primary metric
• Treat others as exploratory or secondary
• Prefer tools that make that structure easy to enforce

10) Reporting that includes uncertainty, not just winners

The best tools present:

• Effect size
• Confidence interval
• Sample size
• Exposure split
• Statistical method
• Test start/end dates
• Decision recommendation with rationale

---

Quick checklist

When evaluating a tool, ask:

• Does it calculate required sample size and power?
• Does it explain the significance test it uses?
• Does it support sequential testing or warn against peeking?
• Does it detect sample ratio mismatch?
• Does it handle multiple metrics/variants correctly?
• Does it show confidence intervals and effect sizes?
• Is the methodology transparent?

---

Red flags

Be cautious if the tool:

• Only reports “confidence” or “probability to win” with no method
• Lets you peek without correction and still claims valid significance
• Lacks power/sample size planning
• Doesn’t detect assignment problems
• Uses many metrics and segments without multiple-testing controls

If you want, I can also give you a vendor evaluation checklist or a comparison of frequentist vs Bayesian A/B testing tools.