Point Estimation and Maximum Likelihood

How do you take a collection of data and draw a meaningful conclusion about the larger, unseen world it came from? In data science, statistics, and machine learning, we constantly face this challenge. Point estimation provides the foundational answer, giving us a single "best guess"—an estimator—for an unknown population parameter based on sample data. Among all methods for constructing these estimators, Maximum Likelihood Estimation (MLE) stands out for its power, versatility, and theoretical elegance, forming the backbone of many modern algorithms from logistic regression to advanced neural network training.

What is a Point Estimator?

A point estimator is a statistic—a function of your sample data—used to infer the value of an unknown parameter in a population. It produces a single numerical value as the estimate. Formally, if you have a sample of $n$ independent observations, $X_1,X_2,...,X_n$, a point estimator for a parameter $\theta$ is a function $\hat{\theta }=g(X_1,X_2,...,X_n)$. Crucially, $\hat{\theta }$ is a random variable because it depends on the random sample; the specific numerical value you get from a particular dataset is called the point estimate.

For example, the sample mean $\bar{X}=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ is a point estimator for the population mean $\mu$. Before you collect your data, $\bar{X}$ is a rule (average the values). After you collect data, say {2, 5, 11}, the point estimate is $\bar{x}=(2+5+11)/3=6$. The goal is to find estimators with good statistical properties.

Desirable Properties of Estimators

Not all estimators are created equal. We evaluate them based on their long-run behavior across many hypothetical samples from the same population. Three key properties are unbiasedness, consistency, and efficiency.

An estimator is unbiased if its expected value equals the true parameter value: $E[\hat{\theta }]=\theta$. On average, across infinite repeated samples, an unbiased estimator gets it right. The sample mean is unbiased for the population mean. However, unbiasedness alone is insufficient; an estimator could be unbiased but have enormous variance, making any single estimate unreliable.

Consistency is a more fundamental large-sample property. A consistent estimator converges in probability to the true parameter as the sample size grows: $\hat{\theta }\stackrel{P}{\to }\theta$ as $n\to \infty$. This means that with enough data, your estimate will be arbitrarily close to the truth. Consistency is often a minimal requirement for a good estimator.

Efficiency compares the precision of unbiased estimators. If ${\hat{\theta }}_1$ and ${\hat{\theta }}_2$ are both unbiased for $\theta$, ${\hat{\theta }}_1$ is more efficient if it has a smaller variance: $Var({\hat{\theta }}_1)<Var({\hat{\theta }}_2)$. The most efficient unbiased estimator is often given by the Cramér-Rao lower bound, which defines the theoretical minimum possible variance an unbiased estimator can achieve.

The Method of Moments

The method of moments is an intuitive, older technique for constructing estimators. The core idea is simple: equate sample moments (like the sample mean, variance) with their corresponding theoretical population moments (which are functions of the unknown parameters), then solve the resulting system of equations.

The procedure is straightforward:

1. Compute the first $k$ sample moments. For example, the first sample moment is the mean $m_1^{\prime }=\frac{1}{n}\sum X_i$, the second is $m_2^{\prime }=\frac{1}{n}\sum X_i^2$, and so on.
2. Express the first $k$ population moments as functions of the $k$ unknown parameters. For a normal distribution $N(\mu ,{\sigma }^2)$, the first population moment is $E[X]=\mu$, and the second is $E[X^2]={\mu }^2+{\sigma }^2$.
3. Set the population moments equal to the sample moments and solve for the parameters.

$$m_1^{\prime }=\mu$$ $$m_2^{\prime }={\mu }^2+{\sigma }^2$$ Solving gives the method of moments estimators: ${\hat{\mu }}_{MM}=\bar{X}$ and ${\hat{\sigma }}_{MM}^2=\frac{1}{n}\sum X_i^2-{\bar{X}}^2=\frac{1}{n}\sum (X_i-\bar{X}{)}^2$. Notice this variance estimator uses $n$ in the denominator, not $n-1$, making it biased but consistent.

While simple, method of moments estimators are not always the most efficient. They serve as excellent starting points and are particularly useful for initializing more complex procedures like MLE.

Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation (MLE) is a predominant method for parameter estimation due to its optimal asymptotic properties. The principle is beautifully intuitive: choose the parameter values that make the observed data most probable.

The process begins by defining the likelihood function. For a sample of independent and identically distributed (i.i.d.) observations, this is the joint probability density (or mass) function, viewed as a function of the parameter $\theta$: $$L(\theta ;x_1,...,x_n)=\prod _{i=1}^{n}f(x_i|\theta ).$$

Because products are difficult to differentiate, we almost always work with the log-likelihood function, $\ell (\theta )=\log L(\theta )={\sum }_{i=1}^n\log f(x_i|\theta )$. The logarithm is a monotonic transformation, so the value of $\theta$ that maximizes $L(\theta )$ also maximizes $\ell (\theta )$.

To find the Maximum Likelihood Estimator (MLE), ${\hat{\theta }}_{MLE}$, we take the derivative of the log-likelihood with respect to $\theta$, set it equal to zero, and solve: $$\frac{\partial \ell (\theta )}{\partial \theta }=0.$$ This is known as the likelihood equation. You must also verify you've found a maximum, typically by checking the second derivative is negative.

Deriving MLEs for Common Distributions

Example: MLE for a Normal Distribution Mean (${\sigma }^2$ known) Assume $X_i\sim N(\mu ,{\sigma }^2)$ with ${\sigma }^2$ known. The PDF is $f(x|\mu )=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$. The log-likelihood is: $$\ell (\mu )=\sum _{i=1}^n\left[-\frac{1}{2}\log (2\pi {\sigma }^2)-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right].$$ Dropping constants, we have $\ell (\mu )\propto -\sum (x_i-\mu {)}^2$. Differentiating and setting to zero: $$\frac{d\ell }{d\mu }=\frac{1}{{\sigma }^2}\sum (x_i-\mu )=0⟹\sum x_i-n\mu =0.$$ Solving gives ${\hat{\mu }}_{MLE}=\frac{1}{n}\sum x_i=\bar{x}$. This aligns with our intuitive estimator.

Example: MLE for a Bernoulli Probability Let $X_i\sim \text{Bernoulli}(p)$, where $P(X=1)=p$. The likelihood is $L(p)=\prod p^{x_i}(1-p{)}^{1-x_i}=p^{\sum x_i}(1-p{)}^{n-\sum x_i}$. The log-likelihood is: $$\ell (p)=(\sum x_i)\log p+(n-\sum x_i)\log (1-p).$$ Differentiating: $$\frac{d\ell }{dp}=\frac{\sum x_i}{p}-\frac{n-\sum x_i}{1-p}=0.$$ Solving yields ${\hat{p}}_{MLE}=\frac{1}{n}\sum x_i=\bar{x}$, the sample proportion of successes.

MLEs possess compelling properties: they are consistent, asymptotically normal (converging to a normal distribution as $n$ grows), and asymptotically efficient (achieving the Cramér-Rao lower bound for large samples). They are also invariant: if $\hat{\theta }$ is the MLE of $\theta$, then for any function $g$, the MLE of $g(\theta )$ is $g(\hat{\theta })$.

Fisher Information and The Variance of MLEs

The Fisher information, denoted $I(\theta )$, quantifies the amount of information a random sample carries about an unknown parameter $\theta$. A higher Fisher information implies the parameter can be estimated with greater precision (lower variance). For a single observation from a distribution with PDF/PMF $f(x|\theta )$, the Fisher information is defined as: $$I(\theta )=E\left[{\left(\frac{\partial }{\partial \theta }\log f(X|\theta )\right)}^2\right].$$ Under regularity conditions, an equivalent form is $I(\theta )=-E\left[\frac{{\partial }^2}{\partial {\theta }^2}\log f(X|\theta )\right]$, which is the negative expected value of the second derivative (curvature) of the log-likelihood. Steeper curvature means the log-likelihood is sharply peaked, indicating the data provides strong information about $\theta$.

For an i.i.d. sample of size $n$, the Fisher information is $I_n(\theta )=nI(\theta )$. A key asymptotic result states that the variance of the MLE, ${\hat{\theta }}_{MLE}$, achieves the Cramér-Rao lower bound: $$Var({\hat{\theta }}_{MLE})\approx \frac{1}{I_n(\theta )}=\frac{1}{nI(\theta )}.$$ This provides a practical way to approximate the standard error of your MLE, which is essential for constructing confidence intervals. For example, a 95% asymptotic confidence interval is ${\hat{\theta }}_{MLE}\pm 1.96\times \sqrt{1/I_n({\hat{\theta }}_{MLE})}$.

Common Pitfalls

1. Confusing the Likelihood Function with the Probability Density Function (PDF): A PDF, $f(x|\theta )$, is a function of the data $x$ given a fixed parameter $\theta$. The likelihood function, $L(\theta |x)$, is the same expression but viewed as a function of the parameter $\theta$ given fixed, observed data $x$. They are mathematically identical but have fundamentally different interpretations.
2. Ignoring the Support of the Distribution: The MLE must lie within the parameter space. For instance, estimating a variance ${\sigma }^2$ or a probability $p$, the MLE must be non-negative and between 0 and 1, respectively. Sometimes the calculus solution (like a negative variance) is invalid, and the MLE is found at the boundary of the parameter space (e.g., ${\hat{\sigma }}^2=0$).
3. Not Checking the Second Derivative: Setting the first derivative to zero finds a critical point, which could be a minimum, maximum, or saddle point. Always verify that the second derivative is negative at the candidate $\hat{\theta }$ to confirm it's a maximum of the likelihood function.
4. Applying MLE Blindly to Non-I.I.D. Data: The standard likelihood formulation $L(\theta )=\prod f(x_i|\theta )$ assumes independence. For correlated data (e.g., time series, spatial data), this formula is incorrect and will lead to biased estimates. You must model the joint dependence structure correctly in the likelihood.

Summary

• Point estimation provides a single best guess for an unknown population parameter using a function of sample data called an estimator. Key properties to seek are unbiasedness, consistency, and efficiency.
• The method of moments constructs estimators by equating sample moments to population moments. It is intuitive and provides consistent estimators, though they may not be efficient.
• Maximum Likelihood Estimation (MLE) is a powerful, general method that chooses parameter values to maximize the likelihood function, which represents the probability of observing the collected data.
• In practice, we maximize the log-likelihood to simplify calculations. MLEs are derived by solving the likelihood equation and possess excellent asymptotic properties: consistency, normality, and efficiency.
• The Fisher information measures the expected curvature of the log-likelihood and inversely relates to the minimum achievable variance of an unbiased estimator. The asymptotic variance of the MLE is the inverse of the Fisher information, enabling the construction of confidence intervals.