What this calculator tells you
An A/B test shows two groups of people two different versions of something: a subject line, a landing page, a button. Each group converts at some rate. The rates will differ a little even if the two versions are identical, because people are random.
This calculator tells you how likely a gap that large would be if nothing real were going on. That likelihood is the p-value. A small p-value means the gap is unlikely to be luck. The calculator turns it into a plain verdict so you do not have to remember where the lines are.
How to use the calculator
Fill in the four boxes, then pick the kind of test and the confidence level. The page recalculates as you type, and it opens with a worked example so you can see an answer straight away.
- Visitors and conversions for variant A, the version you already had.
- Visitors and conversions for variant B, the new version.
- Two-sided asks whether B is different from A. One-sided asks only whether B is better.
- Confidence of 95% is the usual choice. Use 99% when a wrong call would be costly.
How to read the results
The p-value is the chance of seeing a gap this big by luck alone. Below 0.05 is significant at 95% confidence. The confidence interval is a range for the true difference between the two rates. If it sits entirely above zero, B is very likely better. If it crosses zero, you cannot yet tell.
Power is the other side of the coin. It is the chance your test would have noticed a real difference of this size. Low power means a quiet result is weak evidence, because the test may simply be too small to see the effect. Collect more visitors before you call it a tie.
Common mistakes in A/B testing
Do not stop the test the moment the p-value dips under 0.05. Checking again and again until you win inflates your false alarm rate. Decide how many visitors you need first, then look once. Do not run many variants and pick the best one without adjusting for it either. Each extra comparison gives luck another chance to look like a win.
Significance is not importance. With enough traffic, a tiny lift becomes significant. Look at the size of the lift and the interval as well as the verdict, then ask whether the change is worth shipping.
Frequently asked questions
What does statistical significance mean in an A/B test?
It means the gap between your two variants is bigger than random chance would usually produce. At 95% confidence, a gap this large would show up by luck less than 5 times in 100 if the two variants were really the same.
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the smallest lift you care about. Small lifts on low conversion rates need tens of thousands of visitors per variant. The power figure on this page shows whether your current sample was big enough to notice the difference you saw.
Should I use a one-sided or a two-sided test?
Use two-sided unless you decided before the test that you only care whether B is better. Two-sided also catches the case where B is worse, which is usually the result you most want to know about.
Is my data sent anywhere?
No. The calculation runs in your browser. The numbers you type are never uploaded, stored or logged.