Information Theoretic Learning

From ITL Wiki

Jump to: navigation, search

Welcome to Information Theoretic Learning

ITL Brief Introduction

Information Theoretic Learning (ITL) was initiated in the late 90’s at CNEL and has been a center piece of the research effort. ITL uses descriptors from information theory (entropy and divergences) estimated directly from the data to substitute the conventional statistical descriptors of variance and covariance. ITL can be used in the adaptation of linear or nonlinear filters and also in unsupervised and supervised machine learning applications. See the ITL Resource Center for tutorials, examples and Matlab code.


Current Projects

  1. Correntropy Dependence Measure
  2. Nonlinearity tests based on Correntropy
  3. Pitch Detection Based on Correntropy
  4. Nonlinear Granger Causality based on correntropy
  5. Compressive sampling based on correntropy
  6. ITL Feature Extraction for mine recognition
  7. The Principle of Relevant Entropy
  1. On-line KLMS is intrinsically regularized
  2. Nonlinear adaptive filters in RKHS
  3. Active Learning Strategies



Related Publications

  • Books
    • Principe J., Erdogmus D., Information Theoretic Learning, under contract with John Wiley, expected publication...
  • Journal and Conference Papers
    • Erdogmus D. and Principe J., “Information Theoretic Learning”, Encyclopedia of Artificial Intelligence, edited by Dopico, Dorado, and Pazos, IGI Global, 2008.
    • Principe, J., Xu D., Fisher J., Information Theoretic Learning, in Unsupervised Adaptive Filtering, Simon Haykin Editor, 265-319, Wiley, 2000.
    • Erdogmus D., Principe J. “From Linear Adaptive Filtering to Nonlinear Signal Processing” IEEE SP Magazine, vol 23, pp 14-33.
    • Principe J., Xu D., Zhao Q., Fisher J. “Learning from examples with information theoretic criteria”, VLSI Signal Processing Systems, 26, 61-77.

    • Jeong K-H, Liu W., Principe J., “The correntropy MACE filter”, accepted in Pattern Recognition.
    • Pokharel P., Liu W., Principe J., “Kernel Least Mean Sqaure Algorithm with Constrained Growth”, accepted in Signal Processing.
    • Xu J., Paiva A., Park I., Principe J., “A Reproducing Kernel Hilbert space framework for information-theoretic learning", accepted in IEEE Trans. Signal Processing.
    • Gunduz A., Principe J., “Correntropy as a Novel Measure for Nonlinearity Tests”, accepted in Signal Processing
    • Xu J., Principe J., “A Pitch Detector Based on a Generalized Correlation Function”, accepted in IEEE Trans. on Audio, Speech and Language Processing.
    • Liu W., Principe J., “Kernel Affine Projection Algorithms”, accepted in European J. of Signal Processing, Special Issue on Machine Learning for Signal Processing.
    • Xu J., Bakardjian H., Cichocki A., and Principe J., “A New Nonlinear Similarity Measure for Multichannel Signals”, Neural Networks (invited paper), Volume 21, Issues 2-3, March-April 2008, Pages 222-231.
    • Jeong K-W and Principe J., “Enhancing the Correntropy MACE filter with Random Projections” accepted in Neurocomputing.
    • Liu W., Pokarel P., Principe J., “The Kernel LMS Algorithm”, IEEE Trans. Signal Processing, Volume 56, Issue 2, Feb. 2008 Page(s):543 - 554 pdf.
    • Han S., Rao S., Erdogmus D., Principe J., “A minimum error entropy algorithm with self adjusting stepsize (MEE-SAS)”, Signal Processing 87, 2733-2745, Jan 2008 pdf.
    • Li R., Liu W., Principe J., “A unifying criterion for blind source separation based on correntropy”, Signal Processing, Special Issue on ICA, vol 87 , #8, 1872-1881 .
    • R. Jenssen, J. C. Principe, D. Erdogmus and T. Eltoft,"The Laplacian Classifier," IEEE Transactions on Signal Processing, vol 55, #7, 3262-3271, Dec 2006 pdf.
    • Liu W., Pokharel P., Principe J., “Correntropy: Properties and Applications in Non Gaussian Signal Processing”, accepted in IEEE Trans. Sig. Proc., vol 55; # 11, pages 5286-5298, Nov 2007 pdf.
    • Kenneth E. Hild II, Deniz Erdogmus, Jose C. Principe, "An Analysis of Entropy Estimators for Blind Source Separation," Signal Processing, Vol. 86, No. 1, pp. 182-194, Jan 2006 pdf.
    • K.E. Hild II, D. Erdogmus, K. Torkkola, J.C. Principe, "Sequential Feature Extraction Using Information Theoretic Learning," IEEE Trans on Pattern Analysis and Machine Intelligence, vol 28,#9, 1385-1393, Sep 2006 pdf.
    • Jenssen R., Erdogmus D., Hild II K., Principe J., and Eltoft T., "Information Cut for Clustering using a Gradient Descent Approach," Pattern Recognition , 40, pp 796-806, Jan 2008 pdf.
    • Jenssen R., Principe J., Erdogmus D. and Eltoft T., "The Cauchy-Schwartz Divergence and Parzen Windowing: Connections to Graph Theory and Mercer Kernels," Journal of the Franklin Institute , 343, pp 614-629 (Invited paper as ICASSP 2005 Student Paper Award Winner), Sep 2006 pdf.
    • Jenssen R., Erdogmus D., Principe J., Eltoft T., “Some Equivalence between Kernel and Information Theoretic Methods”, in J. VLSI Signal Processing, 45, pp 49-65, Nov 2006 pdf.
    • Jeong K-H, Xu J-W, Erdogmus D., Principe J., “A New Classifier based on Information Theoretic Learning with Unlabeled Data”, Special Issue Neural Networks, vol 18, # 5-6, pp 719-726, Jul 2005 pdf.
    • Santamaria I., Pokharel P., Principe J., “Generalized Correlation Function: Definition, Properties and Application to Blind Equalization”, IEEE Trans. Signal Proc. vol 54, no 6, pp 2187- 2186, Jun 2006 pdf.
    • Kenneth E. Hild II, David Pinto, Deniz Erdogmus, and Jose C. Principe, "Convolutive blind source separation by minimizing mutual information between segments of signals," IEEE Trans. Circuits and Systems--I, Vol. 52, No. 10, pp. 2188-2196, Oct 2005 pdf.
    • Erdogmus D., Agrawal R., Principe J., “A mutual information extension to the matched filter”, Signal Processing, vol 85, #5, 927-936.
    • Lehn-Schioler T., Hegde A., Erdogmus D., Principe J. C., "Information Theoretic Vector-Quantization," Natural Computation, vol 4 #1, 39-51, Jan 2005 pdf.
    • M. Lazaro, I. Santamaria, D. Erdogmus, K.E. Hild II, C. Pantaleon, J.C. Principe, "Stochastic Blind Equalization Methods Based on PDF Fitting Using Parzen Estimator," IEEE Transactions on Signal Processing, vol 53, #2, 696-704, Feb 2005 pdf.
    • Erdogmus D., K.E. Hild II, M. Lazaro, I. Santamaria, J.C. Principe, "Adaptive Blind Deconvolution of Linear Channels Using Renyi's Entropy with Parzen Estimation," in IEEE Trans. on Signal Processing, vol. 52, no. 6, pp. 1489-1498, Jun 2004 pdf.
    • Morejon R., Principe J., “Advanced Parameter Search Algorithms for Information-Theoretic Learning”, IEEE Transactions on Neu¬ral Networks Special Issue on Information Theoretic Learning, vol 15, #4, 874- 884, Jul 2004 pdf.
    • Erdogmus D., Principe J., "Convergence Properties and Data Efficiency of the Minimum Error Entropy Criterion in ADALINE Training," in IEEE Transactions on Signal Processing, vol. 51, no. 7, pp. 1966-1978, Jul 2003 pdf.
    • Erdogmus D., Principe J., Hild II K., "On-Line Entropy Manipulation: Stochastic Information Gradient," in IEEE Signal Processing Letters, vol. 10, no. 8, pp. 242-245, Aug 2003 pdf.
    • Erdogmus D., J. Principe, “Generalized Information Potential for Adaptive Systems Training,” IEEE Trans. Neural Networks, vol 13, #5, 1035-1044, Sep 2002 pdf.
    • D. Erdogmus, J.C. Principe, "Insights on the Relationship Between Probability of Misclassification and Information Transfer Through Classifiers," International Journal of Computers, Systems and Signals, vol. 3, no. 1, pp 40-54, Aug. 2002.
    • Erdogmus D., Principe J.C., “An Error-Entropy Minimization Algorithm for Supervised Training of Nonlinear Adaptive Systems,” IEEE Transactions on Signal Processing, vol. 50, no. 7, pp. 1780-1786, Jul 2002. Best young investigator paper award in IEEE Trans. Signal Processing, 2003 pdf.
    • Erdogmus D., Hild K., Principe J., “Beyond second order statistics for learning: a pairwise interaction model for entropy estimation”, Journal of Natural Computing, vol. 1, no. 1, pp 85-108, May 2002 pdf.
    • Santamaria I., Erdogmus D., Principe J.C., “Entropy Minimization for Supervised Communication Channel Equalization,” IEEE Transactions on Signal Processing, vol. 50, no. 5, pp. 1184-1192, May 2002 pdf.
    • Erdogmus D., Principe J., “Information theoretical lower and upper bounds for error probability of classifiers,” J. of VLSI Signal Proc. Systems, vol. 37, no. 2/3, pp. 305-317, Jun 2004 pdf.
    • Erdogmus D., Hild K.E. II., Principe J.C., “Blind Source Separation Using Renyi’s -Marginal Entropies,” in Neurocomputing Special Issue on Blind Source Separation and Independent Component Analysis, vol 49, #1, 25-38, Dec 2002 pdf.
    • D. Erdogmus, J.C. Principe, K.E. Hild II, "On-Line Entropy Manipulation: Stochastic Information Gradient," IEEE Signal Processing Letters, vol. 10, no. 8, pp. 242-245, Aug 2003 pdf.
    • Gokcay E., Principe J., “Information theoretic clustering”, accepted in IEEE Trans. Pattern Analysis and Machine. Intelligence, vol 24, #2, 158-171, Feb 2003 pdf.
    • Hild K., Erdogmus D., Principe J., “Blind source separation using Renyi’s mutual information”, IEEE Signal Proc. Letters, vol 8, #6, 174-176, Aug 2003 pdf.










Retrieved from "http://www.cnel.ufl.edu/itl/index.php/Information_Theoretic_Learning"
Personal tools
EEL 6935