A/B Test Calculator

How to Use the A/B Test Calculator – Step-by-Step

1. Choose your tool: run the Significance Test for completed experiments, or the Sample Size Planner before launching.

2. Enter counts accurately: Visitors represent total exposures, and Conversions represent goal completions.

3. Set α and power thoughtfully: standard defaults are α = 0.05 (95% confidence) and power = 0.80.

4. Click Calculate to get lift, z-score, p-value, confidence, and winner, or the required n per variant.

5. Validation: Visitors and Conversions must be integers ≥ 0; Conversions ≤ Visitors. α in (0, 0.5]; power in [0.5, 0.99]; MDE > 0.

Understanding Your A/B Test Results

Once you click Calculate, you’ll see whether the observed lift is statistically significant at your chosen α.

Interpreting outputs

• p-value & confidence: low p-value (e.g., < 0.05) → high confidence in a real difference.

• Lift (%): relative improvement of B over A; confirm it’s practically meaningful, not just statistically.

• Winner: shown only if the difference is significant; otherwise, report No considerable difference.

When to Use It

• Validate completed tests (post-hoc significance).

• Plan future tests with a realistic MDE and sample size.

• Compare channels, pages, or offers using a consistent method.

Note: For multi-armed or sequential designs, guard against peeking and use appropriate corrections—this calculator assumes a fixed-horizon A/B test.

Industry A/B Testing Benchmarks

Note: Choose MDE based on business impact; smaller MDEs require much larger samples.

Practical Examples

Example 1 — Clear Winner (Significance Test)

• Visitors A: 4,000; Conversions A: 200 → CRA = 0.0500

• Visitors B: 4,000; Conversions B: 260 → CRB = 0.0650

• p̂ = 460 / 8,000 = 0.0575; SE ≈ 0.005205

• z ≈ 2.882; p-value ≈ 0.004 → Confidence ≈ 99.6%

• Lift = +30% → Winner: B (α = 0.05)

Example 2 — Not Significant (Small Lift)

• Visitors A: 10,000; Conversions A: 500 → CRA = 0.0500

• Visitors B: 10,000; Conversions B: 530 → CRB = 0.0530

• Observed lift: +6% but p-value > 0.05 (insufficient evidence)

• Interpretation: Increase sample size or accept equivalence; consider whether +0.3 pp is meaningful for your economics.

Growth Tips & Business Impact

• Define MDE by revenue, not vanity. Pick the smallest lift that materially moves profit; design power around that. (Impacts: CRO, PPC efficiency.)

• Balance traffic & duration. Run across complete cycles to capture weekday/weekend effects and avoid seasonal bias. (Impacts: Data quality.)

• Prevent peeking & bias. Pre-register stop rules; avoid mid-test changes to targeting, pricing, or creatives. (Impacts: Validity.)

• Segment after you win. Claim the global win first; then explore segments to guide personalization. (Impacts: Scaling impact.)

• Harden implementation. QA tags, events, and split logic before launch to prevent leakage. (Impacts: Accuracy, Speed to insight.)

• Pipeline prioritization. Use the Sample Size Planner to stack-rank tests by feasibility and expected value. (Impacts: Roadmap ROI.)