Minimum Error Entropy
Ideally in adaptive systems under probabilistic setting, the error pdf would be as concentrated at zero as possible. If there is no modeling error nor noise, then at the optimal solution the error would be a Dirac delta function, i.e., always zero. Hence any dispersion measure for the deviation from the optimal solution can be used as a cost function for an adaptive system. The most commonly used dispersion measure is the MSE (Mean square error) criterion, where it measures the variance of the error random variable. Unlike MSE, the minimum error entropy directly measures the dispersion with Information theory. Higher entropy implies more spread error pdf, and lower entropy, a more peaky one. Intuitively speaking, the optimal solution gives the sharpest the error pdf.