To test a new pricing page with an experimentation platform, you’re basically running a controlled experiment where some users see the current page and others see the new one, then comparing behavior.

1) Set up the experiment

Start with a clear hypothesis.
Example: “A simpler pricing page will increase free-trial signups by reducing confusion.”

Choose one primary metric.
Common pricing-page metrics:

• Trial starts
• Checkout starts
• Purchases / conversion rate
• Revenue per visitor
• Plan mix / upgrade rate

Also define guardrail metrics so you don’t “win” on one metric while hurting others:

• Bounce rate
• Page load time
• Refunds
• Support contacts
• Downstream retention / churn

Randomly split traffic between:

• Control: current pricing page
• Treatment: new pricing page

Usually 50/50 is fine for a first test.

2) Make sure randomization is clean

Use the platform to assign users consistently:

• The same user should keep seeing the same version
• Prefer assigning at the user/account level, not just the session level, if purchase decisions span multiple visits
• If logged-out users are involved, be careful about cookies/device switching

3) Run the test long enough

Don’t stop just because one variant looks better early on.

A valid test needs enough:

• Users
• Conversions
• Time coverage
  Include full weekly cycles if behavior varies by day of week.

4) Monitor results carefully

Track:

• Overall conversion lift
• Statistical significance / confidence intervals
• Segment behavior
• Guardrails

If the platform supports it, use sequential testing or always-valid methods; otherwise avoid peeking too often and making decisions too early.

---

Sample size: what to watch out for

1) Base it on the effect size you care about

Sample size depends mainly on:

• Baseline conversion rate
• Minimum detectable effect (MDE)
• Desired confidence / power

Example:

• Current conversion = 5%
• You want to detect a +10% relative lift (to 5.5%)
• That requires much more traffic than detecting a large lift like +30%

If your traffic is low, you may need:

• A longer test window
• A larger effect threshold
• Or fewer variants

2) Don’t underpower the test

If sample size is too small:

• You may miss a real improvement
• Results will be noisy
• Segment-level conclusions become unreliable

A common mistake is checking results after a few hundred visitors when the site normally needs tens of thousands.

3) Watch out for novelty effects

A pricing page redesign may get an artificial bump or dip early on. If possible, run long enough to reduce first-time novelty bias.

4) Avoid too many variants

More variants mean:

• Traffic gets split thinner
• Longer runtime
• Harder interpretation

For pricing page tests, start with one strong alternative unless you have very high traffic.

---

Segmentation: what to watch out for

1) Segmenting too early can mislead you

A segment may look “winning” by chance if:

• It has small sample size
• You’ve looked at many segments
• You’re searching for patterns after the fact

This creates false positives.

2) Predefine important segments

Decide before the test starts which segments matter, such as:

• New vs returning users
• Mobile vs desktop
• Geography / language
• Traffic source
• SMB vs enterprise
• Logged-in vs anonymous

Treat post-hoc segments as exploratory, not proof.

3) Beware of Simpson’s paradox

A variant can win overall but lose in a key segment, or vice versa. For example:

• The new pricing page improves conversion on desktop
• But hurts mobile users enough to offset the gain

Always check whether the treatment effect is consistent across major user groups.

4) Keep segmentation coarse

Over-segmenting leads to tiny cells and unreliable results.
Better to analyze a few meaningful segments than dozens of micro-segments.

5) Consider interaction effects

A pricing page may perform differently for:

• High-intent paid traffic vs organic traffic
• Existing customers vs prospects
• Different price sensitivities by country

If one segment is strategically important, you may want a targeted experiment or a stratified rollout.

---

Practical decision rules

Use the experiment to answer:

1. Did the new page improve the primary metric?
2. Are guardrails acceptable?
3. Is the effect robust across the main segments?
4. Is the lift big enough to justify rollout?

Don’t ship based on:

• A small early lead
• A single segment win
• Multiple uncorrected comparisons
• A result that ignores revenue or retention

---

A simple workflow

1. Define hypothesis and primary metric
2. Estimate baseline conversion and MDE
3. Calculate sample size
4. Launch 50/50 randomized test
5. Monitor overall metrics and guardrails
6. Review preplanned segments
7. Decide based on total evidence, not just p-values

If you want, I can also help you:

• estimate sample size for your pricing page,
• design the experiment setup,
• or build a decision checklist for rollout.