Hypothesis tests
- Read it, not yetReach the end of the page
- Practice it, not yetAnswer a few questions
- Master it, not yet80% on a drill of 8 or more
A hypothesis test asks: would this data be surprising if nothing were going on? Assume nothing is (H₀). A small p-value means the data would be rare, so reject H₀.
Why it matters
A test asks: if nothing were going on, how weird would this data be?
It guards against seeing patterns in noise, the most expensive mistake in data work.
- A/B testsDid the new checkout really lift sales, or was it a lucky week?
- OperationsIs the machine off target, or is that its normal wobble?
- MedicineDrug trials rest on tests like this one.
Everyday example
A coffee machine should pour 250 ml. 36 cups average 247.5 ml, σ = 6. z = (247.5 − 250) / (6 / √36) = −2.5. The two-sided p-value is 0.012, under 0.05: reject H₀. It pours short.
Drag the sample mean: when does it get surprising?
H₀ says a machine pours μ = 250 ml, with σ = 6. Move the sample average and the sample size, and watch the p-value.
z = (248.5 − 250) / (6 / √36) = -1.5. p-value = 0.134. Fail to reject H₀ at α = 0.05: samples like this are common when the machine is fine. Red is the rejection region, beyond z = ±1.96.
Check yourself
H₀ and H₁ for “the machine pours 250 ml on average”?
H₀: μ = 250, H₁: μ ≠ 250H₀ holds the equals sign. The claim you want evidence for goes in H₁.Test statistic for a mean, σ known?
z = ( − μ₀) / (σ / √n)How many standard errors the sample landed from H₀.z = −2.5, two-sided. p-value?
0.01242 × P(Z < −2.5) = 2 × 0.0062.
The trap
The mistake people make most on this one.
A p-value is not the chance H₀ is true. It is the chance of data at least this extreme under H₀. For a proportion, the SE uses p₀, not p hat.
Formulas
7 formulas and a way to remember them.
- Five steps
- State H₀ and H₁, pick α, compute the test statistic, find the p-value, then decide and say it in words
- Mean, σ known
- z = ( − μ₀) / (σ / √n)
- Mean, σ unknown
- t = ( − μ₀) / (s / √n), with n − 1 df
- Proportion
- z = (p hat − p₀) / √[p₀(1 − p₀) / n]
- Decision
- Reject H₀ when p-value < α, or when the statistic is past the critical value
- Errors
- Type I: reject a true H₀ (chance α). Type II: keep a false H₀ (chance β). Power = 1 − β
- Excel
- Two-sided p for z: 2*(1 − NORM.S.DIST(ABS(z), TRUE)). For t: T.DIST.2T(ABS(t), df)
- Remember it
- Innocent until proven guilty. H₀ is the defendant; the p-value is how unlikely the evidence would be if innocent.
Watch and go deeper
3 free videos and pages.
- jbstatistics: An Introduction to Hypothesis TestingA first look at the logic of a hypothesis test, built around one worked example. About 10 minutes.
- StatQuest: p-values, What they are and how to interpret themWhat a p-value is, how to read one, and what it does not tell you. About 11 minutes.
- jbstatistics: Type I Errors, Type II Errors, and the Power of the TestThe two ways a test can be wrong, how α sets one of them, and what power means. About 8 minutes.