Skip to content
AP Statistics · Inference for Quantitative Data: Means

Statistical significance

Statistical significance appears in 11 reviewed AP Statistics questions and sits in Inference for Quantitative Data: Means. Below are worked examples showing exactly how the exam tests it, with the answer marked and every distractor explained.

How Statistical significance is tested

Example 1Medium
A study finds that a new teaching method produces a statistically significant improvement in test scores (p = 0.02) but the average improvement was only 0.5 points on a 100-point scale. A school board member says this is a major breakthrough. What concept is most relevant here?
  1. AType I error; the result may be a false positive.
  2. BType II error; the true effect may be much larger.
  3. CUndercoverage; the sample may not include all types of students.
  4. DStatistical significance versus practical significance; the improvement is real but too small to be educationally meaningful.Correct

Why D is right

Statistical significance (p < 0.05) indicates that the result is unlikely to occur by chance if there is no effect. However, a statistically significant result may have very small practical importance. An improvement of 0.5 points on a 100-point scale, while statistically detectable (especially in large samples), is likely too small to have educational meaning. This distinction between statistical and practical significance is crucial.
Why the other 3 choices miss
A
A Type I error is falsely rejecting a true null hypothesis. While possible, the most relevant concept here is not whether the result is a false positive but rather that a true small effect is not practically meaningful.
B
A Type II error is failing to detect a real effect (false negative). The study DID detect a significant effect, so Type II error is not the primary concern.
C
Undercoverage is a sampling issue. The question focuses on interpretation of a statistically significant but practically negligible result.
Example 2Medium
A medical study on a new blood pressure drug reports p = 0.001, meaning the result is highly statistically significant. However, the study had n = 50,000 participants. A critic argues that with such a large sample, even trivial effects become significant. This critique relates to:
  1. AStatistical versus practical significance; large samples can detect tiny effects that are not clinically meaningful.Correct
  2. BType II error, because with so many participants the study is likely rejecting a null hypothesis that is actually true.
  3. CSelection bias, because a sample of 50,000 participants is too large to have been drawn truly at random from the population.
  4. DThe placebo effect, because too many participants were given placebos.

Why A is right

With very large sample sizes, hypothesis tests become extremely sensitive and can detect effects so small they have no practical importance. A p = 0.001 with n = 50,000 might reflect a real but clinically negligible difference in blood pressure reduction (e.g., 0.1 mm Hg). Statistical significance does not imply practical or clinical significance.
Why the other 3 choices miss
B
Type II error is failing to detect a true effect (false negative), not rejecting a true null hypothesis (that would be a Type I error). The concern here is about the practical importance of a detected effect.
C
Sample size does not introduce selection bias by itself. A sample of 50,000 can be truly random and representative.
D
The placebo effect relates to participants improving because they believe they are receiving treatment. The critique here is about interpretational issues with large sample sizes.
Example 3Medium
A medication trial finds a statistically significant reduction in LDL cholesterol (bad cholesterol) with p = 0.03. The mean reduction was 2 mg/dL. A cardiologist argues this is not clinically meaningful because typical LDL levels are 100-200 mg/dL and a 2 mg/dL reduction offers negligible benefit. Which concept is the cardiologist applying?
  1. AConfounding: other variables explain the LDL reduction.
  2. BType I error: the result is probably a false positive.
  3. CPractical significance: the effect, while statistically real, is too small to matter in clinical practice.Correct
  4. DPower analysis: the study should have been larger.

Why C is right

The cardiologist is distinguishing between statistical significance (the result is unlikely by chance, p = 0.03) and practical significance (the effect size is too small to be clinically meaningful). A 2 mg/dL reduction in LDL, while real, is far below the 10-30 mg/dL reductions typically considered clinically significant. This is the statistical vs. practical significance distinction.
Why the other 3 choices miss
A
Confounding involves other variables distorting the apparent effect of the treatment. The cardiologist accepts the LDL reduction is real but argues it is too small to matter.
B
With p = 0.03 and a true null hypothesis being unlikely to produce this result, calling it a Type I error is possible but speculative. The cardiologist's argument is about the SIZE of the effect, not whether it's a false positive.
D
Power analysis determines the required sample size to detect an effect of a certain size. The cardiologist is questioning whether the detected effect is meaningful, not whether the study needed more power.

Studied alongside

Other concepts from Inference for Quantitative Data: Means.