Calculating and Reporting Effect Sizes

While a p-value can tell you if an effect exists, an effect size tells you how large that effect is. Moving beyond statistical significance to quantify the magnitude and practical importance of a finding is fundamental to rigorous research, meta-analysis, and meaningful scientific communication. For graduate students, mastering effect sizes is not optional—it's a core competency for interpreting your own results and critically evaluating the literature.

The Core Purpose of Effect Size

An effect size is a quantitative measure of the magnitude of a phenomenon or the strength of the relationship between variables. Unlike p-values, which are conflated with sample size, effect sizes are largely independent of N. This makes them indispensable for two primary reasons. First, they allow you to assess the practical or substantive significance of a result; a statistically significant finding with a trivial effect size may be meaningless in the real world. Second, they enable the comparison and synthesis of results across different studies with different sample sizes and measurement scales, which is the foundation of meta-analysis.

Consider a study finding a significant difference in test scores between two teaching methods (p < .01). Without an effect size, you cannot tell if this difference is educationally meaningful or negligibly small. Reporting the effect size, such as Cohen's d = 0.8, immediately signals a substantial difference that an educator might care about. Your goal is always to report both the statistical significance and the practical magnitude.

Effect Size for Mean Comparisons: Cohen's d

Cohen's d is the standard effect size measure used to accompany independent or paired samples t-tests. It standardizes the difference between two group means by dividing by a pooled standard deviation, expressing the difference in standard deviation units. The formula for an independent samples t-test is:

$$d=\frac{M_1-M_2}{S{D}_{pooled}}$$

Where $S{D}_{pooled}$ is calculated as:

$$S{D}_{pooled}=\sqrt{\frac{(n_1-1)S{D}_1^2+(n_2-1)S{D}_2^2}{n_1+n_2-2}}$$

Often, you may need to compute d from a reported t-statistic and group sizes, using an approximation like: $d\approx t\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}$. Interpretation relies on established benchmarks, where d = 0.2 is considered a small effect, d = 0.5 a medium effect, and d = 0.8 a large effect. Crucially, these are only heuristic starting points. A d of 0.5 might be huge in a clinical drug trial but small in a social psychology context. You must interpret the magnitude within your specific field of study.

Effect Size for Analysis of Variance: Eta-Squared and Partial Eta-Squared

When your analysis involves one-way or factorial ANOVA, the appropriate effect size is a measure of variance explained. Eta-squared (${\eta }^2$) represents the proportion of total variance in the dependent variable that is attributable to a given factor. It is calculated as:

$${\eta }^2=\frac{S{S}_{effect}}{S{S}_{total}}$$

Where $S{S}_{effect}$ is the sum of squares for the factor you're examining, and $S{S}_{total}$ is the total sum of squares. In factorial designs with multiple independent variables, partial eta-squared (${\eta }_p^2$) is often preferred because it isolates the variance explained by one factor while controlling for others. It is calculated as:

$${\eta }_p^2=\frac{S{S}_{effect}}{S{S}_{effect}+S{S}_{error}}$$

You can frequently compute these from standard ANOVA table output. For interpretation, common guidelines suggest ${\eta }^2$ = 0.01 is a small effect, 0.06 is medium, and 0.14 is large. Remember, ${\eta }^2$ is an overestimating descriptive statistic for your sample, while ${\eta }_p^2$ can sum to over 100% in designs with multiple factors—understanding this distinction is key to accurate reporting.

Effect Size for Association and Prediction: Odds Ratios

For analyses dealing with binary outcomes, such as logistic regression, the go-to effect size is the odds ratio (OR). The odds ratio compares the odds of an event occurring in one group to the odds of it occurring in another. If you have a 2x2 contingency table, the odds ratio is $(a/c)/(b/d)$, where a, b, c, and d are the cell frequencies.

In logistic regression, the exponentiated coefficient for a predictor is its adjusted odds ratio. For a continuous predictor, the formula is $OR=e^b$, where b is the regression coefficient. An OR = 1 means no association; OR > 1 indicates increased odds of the outcome; OR < 1 indicates decreased odds. For example, an OR of 2.5 for a treatment group means the odds of the positive outcome are 2.5 times higher in the treatment group than the control group.

Interpretation requires caution: an OR is not a risk ratio, and the difference becomes substantial when the outcome is common. Benchmarks are less standardized but an OR of 1.5 might be considered a small effect, 2.5 medium, and 4.3 large in many fields. Always report the confidence interval with the OR to convey its precision.

Common Pitfalls

1. Over-relying on Generic Benchmarks: Treating Cohen's "small, medium, large" labels as universal truth is a major error. A d of 0.4 might be groundbreaking in some disciplines. Always contextualize your effect size by comparing it to prior literature in your specific field to determine what constitutes a meaningful magnitude.
2. Confusing Odds Ratios and Risk Ratios: Reporting an odds ratio as if it were a risk ratio (or relative risk) overstates the effect when the outcome is common (>10%). Clearly label your measure and, if the outcome is common, consider calculating and reporting the risk ratio separately for clarity.
3. Reporting Without Confidence Intervals: An effect size point estimate (e.g., d = 0.6) gives limited information. Always report the 95% confidence interval around it (e.g., d = 0.6 [0.2, 1.0]). This interval shows the precision of your estimate and allows readers to see if the effect could plausibly be trivial or very large.
4. Selecting the Wrong Measure for the Design: Using Cohen's d for a one-way ANOVA with three groups is inappropriate, as it only compares two means. Similarly, using ${\eta }^2$ for a t-test is unnecessary. Match the effect size statistic to the hypothesis test and research design you actually used.

Summary

• Effect sizes are mandatory complements to significance tests, providing essential information about the magnitude and practical importance of a finding that p-values cannot.
• Match the effect size to your statistical test: Use Cohen's d for t-tests (mean differences), eta-squared or partial eta-squared for ANOVA (variance explained), and odds ratios for logistic regression and tests of association with binary outcomes.
• Calculation often requires only basic output (means/SDs, sums of squares, t-statistics, or regression coefficients), and you must know how to derive the correct effect size from the statistics available to you.
• Interpretation involves both established benchmarks and contextual understanding. While heuristic guidelines (small/medium/large) are useful starting points, the true meaning of an effect size is determined by comparison to prior research and the practical consequences in your field.
• Always report effect sizes with confidence intervals to communicate the precision of your estimate and enable proper synthesis of evidence across studies.