Exact and Approximate Bayesian Computation

Modern Bayesian data science offers a powerful framework for updating beliefs with data, but its practical success hinges on a critical choice: how to compute the posterior distribution. This distribution, representing our updated uncertainty about model parameters after seeing data, is the cornerstone of Bayesian inference. You face a spectrum of methods, from exact analytical solutions to sophisticated approximations and simulations, each with distinct trade-offs in accuracy, scalability, and implementation effort. Choosing the right tool requires understanding the computational landscape—knowing when to leverage elegant mathematical shortcuts and when to deploy more general, computationally intensive machinery.

The Foundation: Exact Computation with Conjugate Priors

The ideal scenario in Bayesian analysis is exact posterior computation, where the posterior distribution belongs to the same probability family as the prior. This occurs when you use a conjugate prior. For example, using a Beta prior for a binomial likelihood yields a Beta posterior. The update is beautifully simple: you just add the observed "successes" and "failures" to the prior's parameters.

The primary advantage is computational efficiency. Calculating the posterior is instantaneous, as it involves only basic arithmetic. This makes conjugate priors excellent for teaching foundational concepts, for building simple components within larger models, and for situations where real-time updating is required. However, the severe limitation is flexibility. You are constrained to a narrow set of likelihood-prior pairs (e.g., Normal-Normal, Poisson-Gamma). In practice, your prior beliefs or the complexity of your model will rarely conform perfectly to these convenient mathematical forms. Over-reliance on conjugacy can force you to distort your model to fit the computational method, which is the antithesis of principled Bayesian modeling.

Analytical Approximation: The Laplace Method

When models step beyond conjugacy, one of the simplest approximation strategies is the Laplace approximation (or Normal approximation). This method approximates the true, often complex, posterior distribution with a single multivariate Gaussian distribution. The procedure is: first, find the posterior mode (the parameter values that maximize the posterior density). Then, construct a Normal distribution centered at this mode, with a covariance matrix equal to the inverse of the negative Hessian (second derivative matrix) evaluated at the mode.

In essence, you are performing a Bayesian version of maximum likelihood estimation and then using the curvature at the optimum to define uncertainty. The computational cost scales moderately with the number of parameters $p$, typically around $O(p^3)$ for the matrix inversion. The accuracy is good for unimodal, roughly symmetric posteriors, especially as sample size increases (due to Bayesian central limit theory). It becomes poor for skewed posteriors, multimodal distributions, or in low-data regimes. It's a useful, fast method for getting a rough sense of the posterior landscape and is often used to initialize more advanced algorithms.

Optimization-Based Approximation: Variational Inference

Variational Inference (VI) re-frames the problem of posterior computation as an optimization problem. The goal is to find a simpler, tractable distribution $q(\theta )$ (from a chosen family, like a factorized Gaussian) that is close to the true posterior $p(\theta |data)$. Closeness is measured by the Kullback-Leibler (KL) divergence. VI minimizes this divergence, which is equivalent to maximizing a quantity called the Evidence Lower Bound (ELBO).

This turns an integration problem into an iterative optimization one, often using gradient-based methods like stochastic gradient descent. The computational scaling is generally favorable—often linear in the number of parameters and data points—which makes VI exceptionally fast and scalable for very large datasets and complex models, like Bayesian neural networks. The trade-off is two-fold. First, you must choose the approximating family $q$; a poor choice (e.g., a unimodal distribution to approximate a multimodal posterior) leads to a bad approximation. Second, because it optimizes a lower bound, VI tends to produce approximations that are over-confident (have artificially small variance). It excels in production settings where speed and scalability are paramount and some bias in the uncertainty quantification is acceptable.

Simulation-Based Gold Standard: Markov Chain Monte Carlo

Markov Chain Monte Carlo (MCMC) sampling is the most general and widely trusted workhorse for Bayesian computation. Instead of finding a distribution, MCMC generates a correlated sequence of random samples from the posterior. Algorithms like Metropolis-Hastings, Gibbs sampling, and Hamiltonian Monte Carlo (HMC) construct a Markov chain that, after a burn-in period, has the posterior as its stationary distribution.

The power of MCMC is its asymptotic exactness. Given infinite time and a properly tuned algorithm, the samples will eventually represent the true posterior arbitrarily well, capturing all its nuances—skewness, multimodality, and complex correlations. The cost is high computational expense. Running the chain to convergence can take thousands or millions of iterations, and each iteration may involve evaluating the model on the entire dataset. Scaling to ultra-high dimensions or massive data can be challenging, though modern tools like the No-U-Turn Sampler (NUTS) for HMC have made remarkable strides. MCMC is the preferred method for final, high-stakes inference, model checking, and when accurate uncertainty quantification is critical, provided you have the computational budget.

Common Pitfalls

1. Misapplying Conjugate Priors: The pitfall is choosing a prior solely for computational convenience, which misrepresents genuine prior knowledge. Correction: Use conjugate priors only when they genuinely encode reasonable beliefs, or as components in hierarchical models where hyperpriors can add flexibility.

1. Trusting a Laplace or VI Approximation Blindly: These methods can fail silently. A Laplace approximation will not indicate multimodality, and VI will converge confidently to a wrong answer. Correction: Always perform sensitivity analysis. Run a full MCMC on a subset of data or a simplified model to validate that the approximation's shape and summary statistics are credible.

1. Using Raw MCMC Output Without Diagnostics: Assuming a chain has converged because it ran for many iterations is a grave error. Correction: Always use diagnostic tools. Run multiple chains from dispersed starting points and compute the $\hat{R}$ statistic (Gelman-Rubin diagnostic) to assess convergence. Examine trace plots and effective sample size to ensure the samples are reliable for inference.

1. Selecting a Method Based on Habit, Not Problem Constraints: Automatically reaching for MCMC for every problem, or insisting on VI for speed, ignores the problem's specific needs. Correction: Let your model complexity and computational budget guide you. Prototype with fast approximations (Laplace/VI) to explore, then use MCMC to refine and validate for your final, most important models.

Summary

• Exact computation via conjugate priors offers zero-cost, exact inference but is only possible for a limited set of simple models, restricting its general applicability.
• The Laplace approximation provides a fast, analytic Gaussian approximation centered at the posterior mode; it works well for regular, high-data problems but fails for complex posterior geometries.
• Variational Inference transforms inference into optimization, achieving remarkable speed and scalability for large models by approximating the posterior with a tractable distribution, though often at the cost of underestimating uncertainty.
• MCMC sampling is the asymptotically exact, general-purpose gold standard for producing true posterior samples, capable of handling highly complex models but demanding significant computational time and careful convergence diagnostics.
• Your selection strategy should be pragmatic: use fast approximations (Laplace/VI) for model development, exploration, and very large-scale applications, and reserve robust, exact sampling methods (MCMC) for final inference, model checking, and cases where precise uncertainty quantification is non-negotiable.