Did my change really work?
Two versions, two counts. Is the gap real or luck, how big could it be, and how many people would settle it.
Try it with your own numbers
+25.3% for B, and luck is an unlikely reason: B wins.
The old sign-up button (A) against one with clearer words (B). Visitors were split at random for two weeks. Made-up counts. Type your own.
The test only means something if people were split between A and B at random.
- A: signed up
- 4.00%
- 192 of 4,800
- B: signed up
- 5.01%
- 238 of 4,750
- Lift
- +25.3%
- B − A = +1.01 pts
- p-value
- 1.7%
- z = 2.38 · under 5%: statistically significant
- 95% range for B − A
- +0.2 pts to +1.8 pts
- where the real difference probably is
- Needed per version
- 6,613
- to catch a lift this size 8 times in 10
B wins. Luck alone would make a gap this big only 1.7% of the time. The real difference is probably +0.18 pts to +1.84 pts.
How this works, step by step
Split people at random.
Show the old version to some people and the new one to others, decided by a coin flip for each person. Random splitting is what makes the two groups alike in everything except the change. If A ran on Monday and B on Saturday, you would be testing the day, not the button.
Compare the rates.
Each version’s rate is how many acted out of how many saw it. The lift is how much bigger B’s rate is, as a share of A’s. Going from 4% to 5% is a 1 point gap, and a 25% lift.
rate = conversions ÷ visitors · lift = (B − A) ÷ A
Ask: could luck alone do this?
Even two identical versions never get identical rates: people are random. The z test measures the gap in units of the wobble you would expect by chance. A z near 0 is ordinary wobble. Past about 2, it is rare for luck alone.
pooled rate p = (conversions A + B) ÷ (visitors A + B) · z = (B − A) ÷ √(p(1 − p)(1/visitors A + 1/visitors B))
Read the p-value.
The p-value answers one question: if the change did nothing, how often would a gap at least this big show up? Under 5% is called statistically significant. It is not the chance that B is better, and it says nothing about whether the lift is big enough to matter.
p = 2 × Φ(−|z|), where Φ is the bell curve’s area to the left · z = 2.38 gives p = 1.7%
Give a range, not one number.
The gap you saw is one draw. The 95% range is where the real gap probably sits. If the range crosses zero, B could be no better, or even worse.
B − A ± 1.96 × √(A(1 − A) ÷ visitors A + B(1 − B) ÷ visitors B)
Count how many you need.
Small lifts need many people. To catch a given lift 8 times in 10 (80% power) at the 5% level, use the count below. Going from 10% to 12% takes about 3,841 people per version. Half the lift takes about four times as many.
n per version = (1.96 × √(2p̄(1 − p̄)) + 0.84 × √(A(1 − A) + B(1 − B)))² ÷ (B − A)², with p̄ = (A + B) ÷ 2
Don’t peek and stop
Decide how many people you need before you start, and stop there. Checking every day and stopping the moment B looks ahead turns luck into a “win” far more often than 5% of the time.
And a real lift can still be too small to be worth the cost of the change. The range tells you how big it could be; whether that is worth it is your call.
For learning, not financial, medical or betting advice.