Logistic Regression and Odds Ratios

Logistic regression is the cornerstone of classification for binary outcomes, allowing you to predict probabilities and understand the influence of predictor variables. Unlike linear regression, it models outcomes that are categorical—like yes/no, success/failure, or 0/1—making it indispensable in fields from medicine to marketing. Mastering its interpretation, especially through odds ratios, is what transforms raw statistical output into actionable insight for decision-making.

From Probability to Log-Odds: The Logistic Function

At its heart, logistic regression predicts the probability that an observation belongs to a particular category. Let's denote the probability of the event (e.g., a patient having a disease) as $p$. The immediate challenge is that probabilities are bounded between 0 and 1, while a linear combination of predictors, ${\beta }_0+{\beta }_1{x}_1+...+{\beta }_k{x}_k$, can range from $-\infty$ to $+\infty$.

The solution is the logit link function, which transforms the bounded probability into an unbounded "log-odds" scale. The odds of an event are defined as $p/(1-p)$. The logit is simply the natural logarithm of these odds:

$$\text{logit}(p)=\ln \left(\frac{p}{1-p}\right)$$

Logistic regression then models this transformed value as a linear function of the predictors:

$$\ln \left(\frac{p}{1-p}\right)={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+...+{\beta }_k{x}_k$$

To get back to a probability, you apply the inverse of the logit function, which is the logistic function (or sigmoid function):

$$p=\frac{1}{1+e^{-({\beta }_0+{\beta }_1{x}_1+...+{\beta }_k{x}_k)}}$$

This elegant S-shaped curve ensures all predicted probabilities naturally fall between 0 and 1.

Interpreting Coefficients: From Log-Odds to Odds Ratios

The coefficients (${\beta }_1,{\beta }_2,...$) in a logistic model are interpreted as changes in the log-odds of the outcome. This isn't intuitively meaningful for most people. This is where odds ratios become powerful.

An odds ratio (OR) quantifies how the odds of the outcome change with a one-unit increase in a predictor. You obtain it by exponentiating the coefficient: $OR=e^{\beta }$. For example, if ${\beta }_{\text{smoking}}=0.8$, then the odds ratio is $e^{0.8}\approx 2.23$.

Interpretation: Holding other variables constant, smokers have 2.23 times the odds of the outcome (e.g., heart disease) compared to non-smokers.

• An OR = 1 means the predictor has no effect.
• An OR > 1 indicates increased odds of the outcome.
• An OR < 1 indicates decreased odds of the outcome.

For a continuous predictor, like age with an OR of 1.05 per year, the interpretation is: For each one-year increase in age, the odds of the outcome multiply by 1.05 (a 5% increase).

Fitting the Model: Maximum Likelihood Estimation

Logistic regression parameters are not estimated using ordinary least squares. Instead, they are found via maximum likelihood estimation (MLE). The goal of MLE is to find the set of coefficients that make the observed data "most likely."

Think of it this way: given our observed outcomes (a series of 0s and 1s), what values for the $\beta$ coefficients would result in predicted probabilities that are closest to these actual outcomes? MLE searches for these values by maximizing the likelihood function, which is the joint probability of observing all the data points under the model.

The search is an iterative, computational process. Software begins with an initial guess for the coefficients and then uses algorithms (like Newton-Raphson) to adjust them step-by-step, increasing the likelihood with each step until no further improvement can be made. The output is the set of coefficients that maximize this function, accompanied by standard errors used for hypothesis testing and confidence intervals.

Assessing Model Fit: Deviance and the Hosmer-Lemeshow Test

Once fitted, you must evaluate how well your model explains the data. Two key metrics are deviance and the Hosmer-Lemeshow test.

Deviance (often labeled -2LL or -2 Log-Likelihood) measures the model's "badness of fit." It is calculated as $-2\times \ln (\text{Likelihood of the model})$. You typically examine two types:

1. Null Deviance: The deviance of a model with only the intercept.
2. Residual Deviance: The deviance of your fitted model with all predictors.

A significant drop from the null to the residual deviance suggests your predictors improve the model. While useful for comparing nested models, deviance alone doesn't tell you if predictions are well-calibrated—that is, if predicted probabilities match observed event rates.

For calibration, the Hosmer-Lemeshow test is a common goodness-of-fit test. It works by:

1. Sorting observations by their predicted probability.
2. Dividing them into 10 groups (deciles).
3. Comparing the number of observed vs. expected events in each group using a chi-squared test.

A non-significant p-value (e.g., > 0.05) indicates good calibration—the model's predictions align well with reality. A significant p-value suggests a poor fit, meaning the model systematically over- or under-predicts in certain probability ranges.

Beyond Binary: Multinomial Logistic Regression

Not all outcomes are yes/no. When your target variable has three or more unordered categories (e.g., travel mode: Bus, Car, Train), you use multinomial logistic regression. It is a natural extension of binary logistic regression.

The model works by designating one category as the reference or baseline. It then constructs a series of binary logistic models that compare each of the other categories to this baseline. For a 3-category outcome (A, B, C) with A as the baseline, the model estimates two equations:

• $\ln \left(\frac{P(Y=B)}{P(Y=A)}\right)={\beta }_{0,B}+{\beta }_{1,B}{X}_1+...$
• $\ln \left(\frac{P(Y=C)}{P(Y=A)}\right)={\beta }_{0,C}+{\beta }_{1,C}{X}_1+...$

Interpretation follows the same logic as odds ratios, but always relative to the baseline category. For instance, an odds ratio for variable $X_1$ in the B-vs-A equation tells you how a one-unit change in $X_1$ affects the odds of being in Category B versus the baseline Category A.

Common Pitfalls

1. Interpreting Odds Ratios as Risk Ratios: This is the most critical error. An odds ratio is not the same as a risk ratio (relative risk). If the baseline probability of an event is high (e.g., 0.8), an OR of 2.0 does not mean the probability doubles to 1.6 (impossible). Always remember you are interpreting multiplicative changes in odds, not direct changes in probability.
2. Ignoring Non-Linearity of the Logit: The effect of a predictor on the probability is not constant; it depends on the values of all other variables. The coefficient describes a constant linear effect on the log-odds, but this translates to a varying effect on the probability itself, which is greatest when the predicted probability is near 0.5.
3. Overlooking Model Diagnostics and Fit: Relying solely on coefficient p-values is insufficient. Failing to check for goodness-of-fit (e.g., Hosmer-Lemeshow) or assess discrimination (with metrics like the Area Under the ROC Curve) can leave you with a poorly calibrated model that makes unreliable predictions.
4. Insufficient Sample Size or Rare Events: Logistic regression requires a substantial number of observations, especially for the less frequent outcome category. With very rare events, maximum likelihood estimates can be biased, and odds ratios can be extreme. Techniques like Firth's penalized likelihood may be necessary.

Summary

• Logistic regression models binary outcomes by linking probability to predictors via the logit link function, which works with unbounded log-odds.
• Coefficients are interpreted by exponentiating them to produce odds ratios, which describe multiplicative changes in the odds of the outcome for a one-unit change in a predictor.
• Models are fit using maximum likelihood estimation, an iterative process that finds the coefficients that make the observed data most probable.
• Model assessment requires checking both deviance for comparing models and goodness-of-fit tests like Hosmer-Lemeshow to evaluate prediction calibration.
• For outcomes with more than two categories, multinomial logistic regression extends the binary framework by comparing each category to a chosen baseline category.