150401P - STATISTICAL SIGNIFICANCE vs CLINICAL SIGNIFICANCE

Presentation at the Medicine Grand Round on April 1, 2015 by Professor Omar Hasan Kasule Sr. MB ChB (MUK), MPH (Harvard), DrPH (Harvard)

STATISTICAL VS SUBSTANTIVE

• An investigator starts with a substantive question.
• The substantive question is formulated as a statistical question.
• Data is then collected and is analyzed to provide a statistical answer to the statistical question. The answer to the statistical question is the statistical conclusion.
• The investigator uses the statistical conclusion and other knowledge available to him to reach a substantive conclusion that answers the substantive question.

HYPOTHESES AND THE SCIENTIFIC METHOD

• The scientific method consists of hypothesis formulation, experimentation to test the hypothesis, and drawing conclusions.
• Hypotheses are statements of prior belief. They are modified by results of experiments to give rise to new hypotheses.
• The new hypotheses then in turn become the basis for new experiments.

NULL HYPOTHESIS (H₀) & ALTERNATIVE HYPOTHESIS (H_A):

• The null or research hypothesis, H₀, states that there is no difference between two comparison groups and that the apparent difference seen is due to sampling error.
• The alternative hypothesis, H_A, disagrees with the null hypothesis.
• H₀ and H_A are complimentary and exhaustive. They both cover all the possibilities.
• A hypothesis can be rejected but cannot be proved.
• A hypothesis cannot be proved in a conclusive way but an objective measure of the probability of its truth can be given in the form of a p-value.

DIFFERENT DEFINITIONS OF THE CONCEPT OF P-VALUE

• The probability of wrongfully rejecting H₀ 5% of the time, a ratio of 1:20.
• The probability of rejecting a true hypothesis by mistake.
• P values for large samples that are normally distributed is derived from 4 test statistics that are computed from the data: t, F, c, and β.
• P values for small samples that are not normally distributed are computed directly from the data using exact methods based on the binomial distribution.
• The decision rules are: If the p < 0.05 H₀ is rejected (test statistically significant). If the p>0.05 H₀ is not rejected (test not statistically significant).

INTERPRETATION OF A SIGNIFICANT P-VALUE (p<0.05)

• H₀ is false
• H₀ is rejected
• Observations are not compatible with H₀
• Observations are not due to sampling variation
• Observations are real/true biological phenomena.

INTERPRETATION OF A NON-SIGNIFICANT P-VALUE (p<0.05)

• H₀ is not false (we do not say true)
• H₀ is not rejected
• Observations are compatible with H₀
• Observations are due to sampling variation or random errors of measurement
• Observations are artificial, apparent and not real biological phenomena.

STATISTICALLY SIGNIFICANT BUT NOT CLINICALLY SIGNIFICANT

• Statistical significance may have no clinical/practical significance/importance due to (a) confounding / effect modifying factors that are involved but were not studied (b) Invalid measurements
• Clinically important differences may not reach statistical significance due to (a) small sample size (b) short follow up (c) measurement that are not discriminating enough.