Effect Sizes as Evidence – The single most important pitfall to avoid
Despite widespread use of effect sizes across industries as a standardized measure of impact, effect size calculations remain one of the most incorrectly applied and misinterpreted statistics. An effect size is nothing more than a standardized comparison, or “effect” , that captures the difference between an average value and a meaningful comparison in the metric of standard deviation.
While many statistical formulations and variations of effect sizes exist, meaningful effect size calculations should formulate the comparison group as a control group or an expected result.
The comparison group must represent a control group or an expected result
An effect size is a meaningful statistic only if it represents the effect from a meaningful control, benchmark, or expectation of what is likely to be observed. Too often, especially in the field of education, effect sizes are formulated as a pretest to post-test change score. This before and after formulation of an effect size provides little to no useful information as the expected change is not represented in the calculation.
For example, a particular school district may wish to evaluate if a reading intervention program is effective. The district calculates an effect size by comparing reading scores before the intervention to reading scores after the intervention. This type of an effect size is impossible to interpret without additional information. Without knowing the expected change from the pretest to the posttest absent the intervention (the control) one has no basis for how to interpret the magnitude of the effect.
As a general rule, formulate effect sizes as the difference between a treatment and a control group, or as the difference between an observed value and an expected value. In the reading intervention example, the control should represent one of the following:
- A true control group that captures the typical result without the intervention, but with all other confounding variables being equal
- A projection-model control group that uses additional student data as a means of establishing an expected value while holding other confounding variables constant
If an effect size is not formulated accordingly, be skeptical and very cautious when interpreting the magnitude of an effect.
