Browsed by
Category: Statistical Mechanics

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Making Machine Learning As Simple As Possible Albert Einstein is credited with saying, Everything should be made as simple as possible, but not simpler. Machine learning is not simple. In fact, once you get beyond the simple “building blocks” approach of stacking things higher and deeper (sometimes made all too easy with advanced deep learning packages), you are in the midst of some complex stuff. However, it does not need to be more complex than it has to be.  …

Read More Read More

The Statistical Mechanics Underpinnings of Machine Learning

The Statistical Mechanics Underpinnings of Machine Learning

Machine Learning Is Different Now:   Actually, machine learning is a continuation of what it always has been, which is deeply rooted in statistical physics (statistical mechanics). It’s just that there’s a culmination of insights that are now a very substantive body of work, with more theoretical rigor behind them than most of us know.     A Lesson from Mom: It takes a lot of time to learn a new discipline. This is something that I learned from my…

Read More Read More

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Essential Statistical Mechanics for Deep Learning   If you’re self-studying machine learning, and feel that statistical mechanics is suddenly showing up more than it used to, you’re not alone. Within the past couple of years, statistical mechanics (statistical thermodynamics) has become a more integral topic, along with the Kullback-Leibler divergence measure and several inference methods for machine learning, including the expectation maximization (EM) algorithm along with variational Bayes.     Statistical mechanics has always played a strong role in machine…

Read More Read More

How to Read Karl Friston (in the Original Greek)

How to Read Karl Friston (in the Original Greek)

Karl Friston, whom we all admire, has written some lovely papers that are both enticing and obscure. Cutting to the chase, what we really want to understand is this equation: In a Research Digest article, Peter Freed writes: … And today, Karl Friston is not explaining [the free energy principle] in a way that makes it usable to your average psychiatrist/psychotherapist on the street – which is frustrating. I am not alone in my confusion, and if you read the…

Read More Read More

Approximate Bayesian Inference

Approximate Bayesian Inference

Variational Free Energy I spent some time trying to figure out the derivation for the variational free energy, as expressed in some of Friston’s papers (see citations below). While I made an intuitive justification, I just found this derivation (Kokkinos; see the reference and link below): Other discussions about variational free energy: Whereas maximum a posteriori methods optimize a point estimate of the parameters, in ensemble learning an ensemble is optimized, so that it approximates the entire posterior probability distribution…

Read More Read More

Brain Networks and the Cluster Variation Method: Testing a Scale-Free Model

Brain Networks and the Cluster Variation Method: Testing a Scale-Free Model

Surprising Result Modeling a Simple Scale-Free Brain Network Using the Cluster Variation Method One of the primary research thrusts that I suggested in my recent paper, The Cluster Variation Method: A Primer for Neuroscientists, was that we could use the 2-D Cluster Variation Method (CVM) to model distribution of configuration variables in different brain network topologies. Specifically, I was expecting that the h-value (which measures the interaction enthalpy strength between nodes in a 2-D CVM grid) would change in a…

Read More Read More

The Cluster Variation Method: A Primer for Neuroscientists

The Cluster Variation Method: A Primer for Neuroscientists

Single-Parameter Analytic Solution for Modeling Local Pattern Distributions The cluster variation method (CVM) offers a means for the characterization of both 1-D and 2-D local pattern distributions. The paper referenced at the end of this post provides neuroscientists and BCI researchers with a CVM tutorial that will help them to understand how the CVM statistical thermodynamics formulation can model 1-D and 2-D pattern distributions expressing structural and functional dynamics in the brain. The equilibrium distribution of local patterns, or configuration…

Read More Read More

The Single Most Important Equation for Brain-Computer Information Interfaces

The Single Most Important Equation for Brain-Computer Information Interfaces

The Kullback-Leibler Divergence Equation for Brain-Computer Information Interfaces The Kullback-Leibler equation is arguably the best place for starting our thoughts about information theory as applied to Brain-Computer Interfaces (BCIs), or Brain-Computer Information Interfaces (BCIIs). The Kullback-Leibler equation is given as: We seek to express how well our model of reality matches the real system. Or, just as usefully, we seek to express the information-difference when we have two different models for the same underlying real phenomena or data. The K-L…

Read More Read More

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method – Application to Text Mining and Data Mining There are three particularly good reasons for us to look at the Cluster Variation Method (CVM) as an alternative means of understanding the information in a system: The CVM captures local pattern distributions (for an equilibrium state), When the system is made up of equal numbers of units in each of two states, and the enthalpy for each state is the same (the simple unit activation energy…

Read More Read More

Statistical Mechanics – Neural Ensembles

Statistical Mechanics – Neural Ensembles

Statistical Mechanics and Equilibrium Properties – Small Neural Ensembles Statistical Mechanics of Small Neural Ensembles – Commentary on Tkačik et al. In a series of related articles, Gašper Tkačik et al. (see references below) investigated small (10-120) groups of neurons in the salamander retina, with the purpose of estimating entropy and other statistical mechanics properties. They provide the following interesting results: Simple scheme for entropy estimation in undersampled region (1), given that only a small fraction of possible states can…

Read More Read More