Further Statistics: Continuous Distributions and Estimation

Moving beyond counts and discrete events, continuous random variables allow us to model phenomena like time, weight, and measurement—quantities that can take any value within an interval. Mastering this area is crucial for advanced statistical inference, forming the bedrock for estimating population parameters from sample data, a fundamental skill in data science, economics, and scientific research.

The Foundation: Probability Density Functions and Cumulative Distribution Functions

For a continuous random variable $X$, the probability of it taking any single, exact value is zero. Instead, we describe its behavior using a probability density function (PDF), denoted $f(x)$. The key principle is that probability is represented by area under the curve of the PDF. For any interval $[a,b]$, the probability that $X$ lies in that interval is given by the integral: $$P(a\le X\le b)={\int }_a^{b}f(x)dx.$$ A valid PDF must satisfy two conditions: it is never negative ($f(x)\ge 0$ for all $x$), and the total area under its curve is 1 (${\int }_{-\infty }^{\infty }f(x)dx=1$).

Closely related is the cumulative distribution function (CDF), denoted $F(x)$. This function gives the probability that $X$ is less than or equal to a specific value $x$: $$F(x)=P(X\le x)={\int }_{-\infty }^{x}f(t)dt.$$ The CDF is a non-decreasing function that ranges from 0 to 1. You can move between the PDF and CDF using calculus: the PDF is the derivative of the CDF ($f(x)=F^{\prime }(x)$), and the CDF is the integral of the PDF.

Expectation, Variance, and Percentiles for Continuous Variables

The concepts of average and spread have direct analogs in the continuous world. The expectation (or mean) of a continuous random variable $X$ with PDF $f(x)$ is defined as: $$E[X]=\mu ={\int }_{-\infty }^{\infty }xf(x)dx.$$ Think of this as a continuous weighted average, where each value $x$ is weighted by its density $f(x)$. The variance measures the average squared deviation from the mean and is calculated as: $$Var(X)={\sigma }^2=E[(X-\mu {)}^2]={\int }_{-\infty }^{\infty }(x-\mu {)}^{2}f(x)dx.$$ A frequently easier computational formula is $Var(X)=E[X^2]-(E[X]{)}^2$, where $E[X^2]={\int }_{-\infty }^{\infty }{x}^{2}f(x)dx$.

Percentiles are another vital tool. The $p$th percentile (or the $q$th quantile, where $q=p/100$) is the value $x_p$ such that $P(X\le x_p)=p/100$. You find it by solving the equation $F(x_p)=p/100$ for $x_p$. The 50th percentile is the median, a robust measure of center.

The Theory of Estimation: What Makes a Good Estimator?

We rarely know population parameters; we estimate them using sample statistics. A statistic used to estimate a parameter is called an estimator. Not all estimators are created equal, and we judge them by three key properties:

1. Unbiasedness: An estimator $\hat{\theta }$ is unbiased for a parameter $\theta$ if its expected value equals the parameter: $E[\hat{\theta }]=\theta$. The sample mean $\bar{X}$ is an unbiased estimator for the population mean $\mu$. Bias is the difference $E[\hat{\theta }]-\theta$.
2. Consistency: An estimator is consistent if it converges to the true parameter value as the sample size increases. Formally, as $n\to \infty$, $P(|\hat{\theta }-\theta |<ϵ)\to 1$ for any small $ϵ>0$. A biased estimator can be consistent, but unbiasedness alone does not guarantee consistency.
3. Efficiency: Among unbiased estimators, the one with the smallest variance is called the efficient (or minimum variance unbiased) estimator. Efficiency means the estimator's values are more tightly clustered around the true parameter, yielding more reliable estimates.

Finding Estimates: The Method of Maximum Likelihood

Maximum likelihood estimation (MLE) is a powerful and general method for finding parameter estimates from data. The core idea is simple: choose the parameter values that make the observed sample data most probable.

The procedure involves these steps:

1. Write the Likelihood Function: For a continuous distribution with PDF $f(x;\theta )$, and an independent sample $x_1,x_2,...,x_n$, the likelihood function is $L(\theta )={\prod }_{i=1}^{n}f(x_i;\theta )$.
2. Take the Log to Form the Log-Likelihood: Products are awkward, so we use the natural logarithm: $\ell (\theta )=\ln L(\theta )={\sum }_{i=1}^n\ln f(x_i;\theta )$.
3. Differentiate and Solve: Differentiate $\ell (\theta )$ with respect to $\theta$, set the derivative equal to zero, and solve for $\theta$ to find the maximum likelihood estimate (MLE), denoted $\hat{\theta }$.

Example: For data modeled as $X\sim \text{Exp}(\lambda )$ (exponential distribution), the PDF is $f(x;\lambda )=\lambda e^{-\lambda x}$ for $x>0$. The log-likelihood is $\ell (\lambda )=n\ln \lambda -\lambda \sum x_i$. Differentiating: $\frac{d\ell }{d\lambda }=\frac{n}{\lambda }-\sum x_i$. Setting to zero gives the MLE: $\hat{\lambda }=\frac{n}{\sum x_i}=\frac{1}{\bar{x}}$. This is an intuitive result—the rate parameter is estimated by the reciprocal of the sample mean.

Common Pitfalls

1. Treating the PDF as a Probability: A common error is to interpret $f(a)$ as $P(X=a)$. For continuous variables, $P(X=a)=0$. The PDF's value is a density; only an area under it represents a probability. Always think in terms of integration.

1. Confusing Unbiasedness with Consistency: Students often assume an unbiased estimator is automatically good. An estimator can be unbiased but have a huge variance that doesn't decrease with sample size (e.g., using only the first data point to estimate the mean). Consistency is often a more critical long-term property, ensuring improvement with more data.

1. Incorrectly Applying the Expectation Formula: When calculating $E[g(X)]$, the formula is $\int g(x)f(x)dx$, not $g(E[X])$. For example, $E[X^2]\ne (E[X]{)}^2$. Failing to use the correct integral definition is a frequent mistake in variance calculations.

1. Algebraic Errors in MLE: The log-likelihood step is crucial. Forgetting to take the log, making errors in differentiating sums of logs, or failing to verify that the critical point is a maximum (e.g., by checking the second derivative) can lead to an incorrect estimate.

Summary

• Continuous random variables are modeled using a probability density function (PDF), where probabilities are areas under the curve, and a cumulative distribution function (CDF), which gives $P(X\le x)$.
• The expectation and variance extend to continuous variables via integration: $E[X]=\int xf(x)dx$ and $Var(X)=E[X^2]-(E[X]{)}^2$.
• A good estimator should be unbiased (accurate on average), consistent (improves with more data), and efficient (has minimal variance among unbiased estimators).
• The maximum likelihood estimation (MLE) method finds parameter values that maximize the likelihood (or log-likelihood) of the observed sample, providing a powerful and general framework for estimation.
• Always remember that for a continuous variable, $P(X=a)=0$; probability is only meaningful over an interval.