Maximum likelihood sequence estimation

Maximum likelihood sequence estimation (MLSE) is a mathematical algorithm that extracts useful data from a noisy data stream.

Theory
For an optimized detector for digital signals the priority is not to reconstruct the transmitter signal, but it should do a best estimation of the transmitted data with the least possible number of errors. The receiver emulates the distorted channel. All possible transmitted data streams are fed into this distorted channel model. The receiver compares the time response with the actual received signal and determines the most likely signal. In cases that are most computationally straightforward, root mean square deviation can be used as the decision criterion for the lowest error probability.

Background
Suppose that there is an underlying signal {x(t)}, of which an observed signal {r(t)} is available. The observed signal r is related to x via a transformation that may be nonlinear and may involve attenuation, and would usually involve the incorporation of random noise. The statistical parameters of this transformation are assumed to be known. The problem to be solved is to use the observations {r(t)} to create a good estimate of {x(t)}.

Maximum likelihood sequence estimation is formally the application of maximum likelihood to this problem. That is, the estimate of {x(t)} is defined to be a sequence of values which maximize the functional
 * $$L(x)=p(r\mid x),$$

where p(r | x) denotes the conditional joint probability density function of the observed series {r(t)} given that the underlying series has the values {x(t)}.

In contrast, the related method of maximum a posteriori estimation is formally the application of the maximum a posteriori (MAP) estimation approach. This is more complex than maximum likelihood sequence estimation and requires a known distribution (in Bayesian terms, a prior distribution) for the underlying signal. In this case the estimate of {x(t)} is defined to be a sequence of values which maximize the functional
 * $$P(x)=p(x\mid r),$$

where p(x | r) denotes the conditional joint probability density function of the underlying series {x(t)} given that the observed series has taken the values {r(t)}. Bayes' theorem implies that
 * $$P(x)=p(x\mid r)=\frac{p(r\mid x)p(x)}{p(r)}.$$

In cases where the contribution of random noise is additive and has a multivariate normal distribution, the problem of maximum likelihood sequence estimation can be reduced to that of a least squares minimization.