Variance Explained in Models

When you build a statistical model, from a simple linear regression to a complex machine learning algorithm, a fundamental question arises: how good is it? The concept of variance explained provides the most direct answer, quantifying the proportion of your outcome's variability that your predictors account for. For graduate researchers, correctly calculating, selecting, and interpreting these measures is not just a technical step; it's central to evaluating your model's substantive utility and communicating its effectiveness within your academic field.

The Foundation: Understanding R-Squared

At its core, explaining variance means reducing uncertainty. The total variance in your outcome variable, often denoted as $SST$ (Total Sum of Squares), is the baseline amount of "noise" or variability you start with. Your model's job is to account for as much of that as possible. The most common measure for this is $R^2$ (R-squared), also called the coefficient of determination.

Mathematically, R-squared is defined as: $$R^2=1-\frac{SSE}{SST}$$ where $SSE$ is the Sum of Squared Errors (the variance not explained by your model). Conceptually, an $R^2$ of 0 means your model explains none of the variance, while an $R^2$ of 1 means it explains all of it. In a simple linear regression with one predictor, it is literally the square of the correlation coefficient $r$. For example, if you model household energy consumption based on square footage and get an $R^2=0.65$, you would report that 65% of the variance in energy consumption is explained by the home's size.

However, a critical and common misinterpretation is to equate a "high" R-squared with a "correct" or "causal" model. R-squared only measures the strength of a quantitative relationship within your sample. A model can have a high R-squared but be spurious (due to coincidental patterns) or misspecified (by omitting key variables). Its value must always be interpreted alongside diagnostic plots, residual analysis, and theoretical plausibility.

Accounting for Complexity: Adjusted R-Squared

A major pitfall in model building is the allure of simply adding more predictors. Each new variable added to an ordinary least squares (OLS) regression will never decrease the R-squared value; it will either increase it or stay the same. This can lead to overfitting, where your model describes the random noise in your specific sample rather than the true underlying relationship, harming its ability to generalize.

This is where Adjusted R-squared becomes essential. It modifies the R-squared formula to penalize the addition of irrelevant predictors. The formula is: $$R_{adj}^2=1-\left[\frac{(1-R^2)(n-1)}{n-k-1}\right]$$ where $n$ is the sample size and $k$ is the number of predictors. Unlike standard R-squared, the adjusted version can decrease when a predictor adds less explanatory power than would be expected by chance. When comparing models with different numbers of predictors, especially in exploratory research, the adjusted R-squared is the appropriate metric. It provides a more honest assessment of a model's explanatory power in the population, not just the sample.

Beyond Linearity: Pseudo R-Squared Measures

Linear regression and the classic R-squared formula rely on the assumptions of OLS. But what about logistic regression for binary outcomes, or other generalized linear models (GLMs) like Poisson regression? The outcome variance in these models isn't the same as in OLS, so the standard R-squared formula does not apply.

For these non-linear models, statisticians have developed several pseudo R-squared measures. They are "pseudo" because they approximate the idea of explained variance but do not have the exact same interpretation or range as OLS R-squared. Common examples include:

• McFadden's R-squared: Compares the log-likelihood of your model to that of a null model (with only an intercept). Values between 0.2 and 0.4 are typically considered indicative of an excellent fit.
• Cox & Snell R-squared: Attempts to mimic the properties of R-squared but often has a maximum value less than 1.
• Nagelkerke R-squared: Adjusts the Cox & Snell measure to span the full 0 to 1 range, making it more intuitively comparable to the traditional R-squared, though the comparison should still be made cautiously.

The key is that no single pseudo R-squared is universally best. Researchers must choose one appropriate for their model type and, crucially, not report its value alongside OLS R-squared values as if they were directly comparable. They are tools for comparing competing models of the same type (e.g., two different logistic regression models on the same data).

Context is King: Field-Specific Interpretation

Perhaps the most important lesson for a graduate researcher is that there is no universal benchmark for a "good" R-squared value. A model in particle physics explaining 99% of variance might be expected, while a model in psychology explaining 10% of the variance in a complex human behavior could be a landmark finding.

Meaningful effect sizes are domain-specific. In fields studying controlled physical processes (e.g., engineering, chemistry), high R-squared values are the norm. In fields studying human behavior (e.g., sociology, economics, epidemiology), low R-squared values are common due to the inherent complexity and unmeasurable influences on outcomes. An $R^2$ of 0.10 in an educational intervention study might represent a practically significant effect that could improve student outcomes on a large scale. Therefore, you must interpret your variance-explained statistics within the context of your literature. A value should be judged by whether it is meaningful for advancing theory or practice in your specific field, not against an arbitrary threshold.

Common Pitfalls

1. Chasing a High R-squared: Adding irrelevant variables to inflate R-squared leads to overfitted, non-generalizable models. Always prefer adjusted R-squared for model comparison and prioritize theoretical justification for each predictor.
2. Equating High Explanation with Causality: A high R-squared indicates association, not causation. A model linking ice cream sales to drowning rates has a high R-squared but is confounded by temperature (a lurking variable). Causality is established through research design, not statistics alone.
3. Misinterpreting Pseudo R-squared: Reporting a pseudo R-squared value of 0.07 and calling it "low" without understanding that in your field's logistic regression models, values of 0.04 to 0.14 may represent strong, important effects.
4. Ignoring Model Assumptions: An R-squared value from a model that violates core assumptions (like linearity, homoscedasticity, or independence of errors) is misleading. Always validate assumptions before celebrating a high explained variance.

Summary

• $R^2$ (R-squared) quantifies the proportion of variance in the outcome variable explained by the predictor variables in a linear model, but it inherently increases with added predictors.
• Adjusted R-squared corrects for this by penalizing model complexity, providing a better metric for comparing models with different numbers of predictors and guarding against overfitting.
• Pseudo R-squared measures (e.g., McFadden's) are used for non-linear models like logistic regression; they approximate explained variance but are not directly comparable to the OLS R-squared.
• The practical importance of any variance-explained statistic is field-dependent; researchers must interpret values within the context of their discipline's norms and the inherent variability of their subject matter.
• A high value does not validate a model; it must be supported by sound theory, correct specification, and adherence to statistical assumptions.