Signals: Quantization and Signal Digitization

Capturing the continuous, analog world in the discrete, digital domain is a foundational act of modern engineering. While sampling defines when we measure a signal, quantization determines the precision of each measurement. This process of mapping infinite continuous amplitude values to a finite set of discrete levels is what ultimately enables digital storage, processing, and transmission, but it introduces an irreversible error that every signal processing engineer must master. Understanding quantization error, its mathematical modeling, and the strategies to mitigate it is essential for designing everything from high-fidelity audio systems to precise scientific instruments.

The Quantization Process and Quantization Error

After a continuous-time signal is sampled at discrete intervals, each sample still possesses a continuous amplitude value. Quantization is the process of rounding or mapping each of these continuous amplitude values to the nearest level from a predefined finite set. The difference between the original sample's true amplitude and its quantized value is called the quantization error or quantization noise.

Imagine a ruler marked only in whole centimeters. Measuring an object 3.4 cm long forces you to round to 3 cm, creating a 0.4 cm error. In signal processing, the "ruler" is defined by the quantizer. The spacing between adjacent quantization levels is called the step size, denoted by $\Delta$. For a uniform quantizer, $\Delta$ is constant across the entire amplitude range. The number of discrete levels is determined by the bit depth, $b$. A system with $b$ bits can represent $2^b$ distinct levels. For example, a 3-bit system uses $2^3=8$ levels. The full-scale amplitude range $V_{pp}$ that the quantizer can handle is related to the step size by $V_{pp}\approx \Delta \times 2^b$.

Modeling Quantization Noise and Signal-to-Quantization-Noise Ratio

To analyze systems, quantization error is often modeled as additive white noise. This powerful simplification assumes the error is statistically independent of the signal, uniformly distributed, and has a white (flat) power spectral density. For a uniform quantizer with step size $\Delta$, the quantization error, $e$, is typically bounded between $-\Delta /2$ and $+\Delta /2$. If the signal is complex enough to "dither" across many levels, the error can be approximated as having a uniform probability distribution over this interval.

The power (variance) of this error is a key metric. For a uniformly distributed error over $[-\Delta /2,+\Delta /2]$, the noise power, ${\sigma }_e^2$, is calculated as: $${\sigma }_e^2=\frac{1}{\Delta }{\int }_{-\Delta /2}^{\Delta /2}{e}^{2}de=\frac{{\Delta }^2}{12}$$ This result, ${\Delta }^2/12$, is fundamental. The performance of a quantizing system is measured by the Signal-to-Quantization-Noise Ratio (SQNR), expressed in decibels (dB). It compares the power of the input signal, $P_{signal}$, to the power of the quantization noise: $$SQNR=10{\log }_{10}\left(\frac{P_{signal}}{{\sigma }_e^2}\right)=10{\log }_{10}\left(\frac{P_{signal}}{{\Delta }^2/12}\right)$$ For a common scenario—a full-scale sinusoidal input—the signal power is $(V_{pp}/2\sqrt{2}{)}^2$. Substituting and relating $\Delta$ to $V_{pp}$ and bit depth $b$ leads to a crucial rule of thumb: $$SQN{R}_{dB}\approx 6.02b+1.76\text{ dB}$$ This equation reveals that each additional bit improves the SQNR by approximately 6 dB. A 16-bit audio system has a theoretical maximum SQNR of about 98 dB, defining its potential dynamic range.

Uniform vs. Non-Uniform Quantization

The simplest quantizer is the uniform quantizer, where the step size $\Delta$ is constant. It is optimal when the input signal has a uniform probability density function (PDF) across its amplitude range. However, many real-world signals, like speech, have a non-uniform PDF, with lower amplitudes being far more probable than large peaks. A uniform quantizer is inefficient here; it allocates the same precision (step size) to amplitude regions that are rarely used, wasting bits.

Non-uniform quantization addresses this by using a variable step size. Dense quantization levels are used for small, common signal amplitudes (providing high fidelity for quiet sounds), and coarser levels are used for large, less probable amplitudes. This can be implemented in two primary ways: 1) Directly designing a quantizer with a non-linear input-output staircase characteristic, or 2) Companding (COMPressing before expansion). In companding, the signal is first passed through a non-linear compressor (which amplifies low amplitudes more than high ones), then uniformly quantized, and finally expanded by the inverse characteristic. The $\mu$-law and A-law standards used in telephony are classic examples of companding, dramatically improving the perceived quality for voice signals at a given bit rate.

Bit Depth, Dynamic Range, and Applications

Bit depth is the primary control knob for the quality-cost trade-off in digitization. It directly dictates two critical system properties: the resolution (the smallest discernible amplitude change, roughly $\Delta$) and the dynamic range (the ratio between the largest representable signal and the system's noise floor, closely related to SQNR).

In digital audio, a higher bit depth means a lower noise floor and a greater ability to capture subtle nuances in quiet passages. The "16-bit vs. 24-bit" debate centers on this: 16-bit provides ~98 dB of dynamic range, which exceeds the threshold of hearing in a quiet room, while 24-bit provides ~144 dB, useful primarily in professional recording and processing to prevent noise accumulation during mixing.

In measurement systems, such as analog-to-digital converters (ADCs) in scientific instruments, bit depth determines the measurement precision. An 8-bit ADC for a 5V sensor has a resolution of $5V/2^8\approx 19.5mV$. Any signal variation smaller than this is lost in the quantization step. Engineers select the ADC bit depth based on the required measurement accuracy and the inherent noise of the sensor itself; using a 24-bit ADC with a very noisy sensor is an unnecessary expense, as the sensor's own noise will dominate the quantization noise.

Common Pitfalls

1. Assuming SQNR is a fixed system number: The $6.02b+1.76$ dB formula applies specifically to a full-scale sinusoidal input. For a quieter signal, the SQNR is lower because $P_{signal}$ in the ratio is smaller, while ${\sigma }_e^2$ remains constant (if using the additive noise model). Saying "this is a 16-bit system, so the SQNR is 98 dB" is only true when the signal uses the full range.
2. Applying uniform quantization to non-uniform signals: Using a uniform quantizer for a signal like speech without companding results in poor subjective quality. The quantizer will perform well on loud peaks but poorly on the much more frequent low-amplitude sounds, making quiet speech sound grainy or distorted.
3. Overlooking the need for dithering: In very low-amplitude or high-precision scenarios, quantization error is not random and independent; it becomes correlated with the signal, manifesting as harmonic distortion. Dithering—adding a small amount of random noise before quantization—is a critical technique to break this correlation, making the error truly random and noise-like, at the cost of a slight, uniform increase in the noise floor. Forgetting to dither can introduce audible distortion in digital audio or patterned errors in measurement systems.
4. Confusing precision with accuracy: A high-bit-depth (high-precision) quantizer can still be inaccurate if it is not properly calibrated. Offset errors (a shifted zero point) and gain errors (an incorrect step size $\Delta$) affect accuracy, meaning the digital numbers are consistently wrong, even though they are reported with very fine resolution.

Summary

• Quantization is the rounding of a sampled signal's continuous amplitude to one of a finite set of discrete levels, fundamentally necessary for digital representation.
• The resulting quantization error is often modeled as additive white noise with power ${\sigma }_e^2={\Delta }^2/12$, leading to a Signal-to-Quantization-Noise Ratio (SQNR) that improves by approximately 6 dB for each additional bit in bit depth.
• Uniform quantization uses a constant step size and is optimal for signals with a uniform amplitude distribution, while non-uniform quantization (e.g., via companding) allocates levels more efficiently for signals where small amplitudes are more probable, such as human speech.
• Bit depth ($b$) directly determines a system's resolution (smallest detectable change) and its usable dynamic range, which is the span between the noise floor and the maximum signal level.
• Effective system design requires matching the quantization strategy (bit depth, uniform/non-uniform) to the statistical characteristics of the target signal and understanding practical techniques like dithering to manage error behavior.