Readings – Statistical Physics and Information Theory

Readings – Statistical Physics and Information Theory

Statistical Physics, Entropy, and Information Theory

Statistical Physics, Entropy, and Information Theory – Essential Books

More details on statistical physics and information theory books, including abstracts, descriptions, key articles by the authors presenting their main points, etc.

Information Theory – Seminal Papers

Information Theory – Review Papers

  • Burnham, K.P., and Anderson, D.R., Kullback-Leibler information as a basis for strong inference in ecological studies, Wildlife Research (2001), 28, 111-119. Review Paper on Information Theory (reviews concepts and methods – including K-L – in the context of practical applications to experimental data, rather than a deeply mathematical review – good for understanding basics)

Cluster Variation Method – Essential Papers

  • Kikuchi, R. (1951). A theory of cooperative phenomena. Phys. Rev. 81, 988-1003.
  • Kikuchi, R., & Brush, S.G. (1967), “Improvement of the Cluster‐Variation Method,” J. Chem. Phys. 47, 195; online as: http://jcp.aip.org/jcpsa6/v47/i1/p195_s1?isAuthorized=no
  • Pelizzola, A. (2005), “Cluster variation method in statistical physics and probabilistic graphical models,” J. Physics A: Mathematical & General, 38 (33) R308; online as: Pellizola CVM PDF, see also Pellizola-abstract in Cluster Variation Methods page.
  • Yedidia, J.S.; Freeman, W.T.; Weiss, Y., “Understanding Belief Propagation and Its Generalizations”, Exploring Artificial Intelligence in the New Millennium, ISBN 1558608117, Chap. 8, pp. 239-236, January 2003 (Science & Technology Books). Also available online as Mitsubishi Electronic Research Laboratories Technical Report MERL TR-2001-22, online as: http://www.merl.com/reports/docs/TR2001-22.pdf

Additional Cluster Variation Method Readings

Statistical Mechanics – Essential Papers

  • To be filled in.

General Interesting Work on Entropy

Nonlinear Forecasting Methods

Dissipative Systems

Must-Read List – Not Prioritized

Informal Resources – Tutorials, PPT Presentations, Historical

  • Yu, B. (2008). Tutorial: Information Theory and Statistics. ICMLA 2008, San Diego. Nice.pdf
  • Spin-glass theory, Tommaso Castellani1 and Andrea Cavagna, Spin-glass theory for pedestrians. J. Stat. Mech. (2005) P05012. (doi:10.1088/1742-5468/2005/05/P05012)
    A very comprehensive intro (graduate-level) tutorial on spin glasses, with three distinct approaches covered. Well-worth the read.
  • Shannon’s Information Theory, by Lê Nguyên Hoang , a lovely intro-level and intuitive (with nice graphics) tutorial on Shannon’s Information Theory with relationship to the Boltzmann concept of entropy. Very pleasant and well-worth the read.
  • The Essential Message: Claude Shannon and the Making of Information Theory, by Erico Marui Guizzo, study of Shannon and his thoughts leading to information theory.
  • Schreiber, Thomas. Nonlinear Prediction (Web-based tutorial on predictive methods).
  • Markov Chain Applications, Philipp von Hilgers and Amy N. Langville, The Five Greatest Applications of Markov Chains.
  • Letter, digram and trigram frequencies have been tabulated by cryptologists and can be found for example in Secret and Urgent by Fletcher Pratt, Blue Ribbon Books, 1939.
  • I.M. Yaglom, A.M. Yaglom, Probability and Information. Moscow, 1973. Pub. by Hindustan Pub. Co. Has ref to Pratt. (Google Book Result search)

Dynamic Systems

  • Liberzon, D. (2000). Nonlinear feedback systems perturbed by noise: steady-state probability distributions and optimal control. IEEE Trans. Automatic Control (2005) 45 (6), 1116-1130. pdf
    Get back to this one

Information Maximization

  • Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution, Neural Computation 7 1129-1159. pdf
    Classic paper
  • Linsker, R. (1989). An application of the principle of maximum information
    preservation to linear systems. In Advances in Neural Information Processing
    Systems 1
    Ed. D. S. Touretzky (Morgan Kaufmann, San Mateo, CA.) pdf Classic paper