Browsed by
Category: Free Energy

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Making Machine Learning As Simple As Possible Albert Einstein is credited with saying, Everything should be made as simple as possible, but not simpler. Machine learning is not simple. In fact, once you get beyond the simple “building blocks” approach of stacking things higher and deeper (sometimes made all too easy with advanced deep learning packages), you are in the midst of some complex stuff. However, it does not need to be more complex than it has to be.  …

Read More Read More

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Essential Statistical Mechanics for Deep Learning   If you’re self-studying machine learning, and feel that statistical mechanics is suddenly showing up more than it used to, you’re not alone. Within the past couple of years, statistical mechanics (statistical thermodynamics) has become a more integral topic, along with the Kullback-Leibler divergence measure and several inference methods for machine learning, including the expectation maximization (EM) algorithm along with variational Bayes.     Statistical mechanics has always played a strong role in machine…

Read More Read More

How to Read Karl Friston (in the Original Greek)

How to Read Karl Friston (in the Original Greek)

Karl Friston, whom we all admire, has written some lovely papers that are both enticing and obscure. Cutting to the chase, what we really want to understand is this equation: In a Research Digest article, Peter Freed writes: … And today, Karl Friston is not explaining [the free energy principle] in a way that makes it usable to your average psychiatrist/psychotherapist on the street – which is frustrating. I am not alone in my confusion, and if you read the…

Read More Read More

Approximate Bayesian Inference

Approximate Bayesian Inference

Variational Free Energy I spent some time trying to figure out the derivation for the variational free energy, as expressed in some of Friston’s papers (see citations below). While I made an intuitive justification, I just found this derivation (Kokkinos; see the reference and link below): Other discussions about variational free energy: Whereas maximum a posteriori methods optimize a point estimate of the parameters, in ensemble learning an ensemble is optimized, so that it approximates the entire posterior probability distribution…

Read More Read More

The Cluster Variation Method: A Primer for Neuroscientists

The Cluster Variation Method: A Primer for Neuroscientists

Single-Parameter Analytic Solution for Modeling Local Pattern Distributions The cluster variation method (CVM) offers a means for the characterization of both 1-D and 2-D local pattern distributions. The paper referenced at the end of this post provides neuroscientists and BCI researchers with a CVM tutorial that will help them to understand how the CVM statistical thermodynamics formulation can model 1-D and 2-D pattern distributions expressing structural and functional dynamics in the brain. The equilibrium distribution of local patterns, or configuration…

Read More Read More

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method – Application to Text Mining and Data Mining There are three particularly good reasons for us to look at the Cluster Variation Method (CVM) as an alternative means of understanding the information in a system: The CVM captures local pattern distributions (for an equilibrium state), When the system is made up of equal numbers of units in each of two states, and the enthalpy for each state is the same (the simple unit activation energy…

Read More Read More

Statistical Mechanics – Neural Ensembles

Statistical Mechanics – Neural Ensembles

Statistical Mechanics and Equilibrium Properties – Small Neural Ensembles Statistical Mechanics of Small Neural Ensembles – Commentary on Tkačik et al. In a series of related articles, Gašper Tkačik et al. (see references below) investigated small (10-120) groups of neurons in the salamander retina, with the purpose of estimating entropy and other statistical mechanics properties. They provide the following interesting results: Simple scheme for entropy estimation in undersampled region (1), given that only a small fraction of possible states can…

Read More Read More

Statistical Mechanics, Neural Domains, and Big Data

Statistical Mechanics, Neural Domains, and Big Data

How Neural Domain Activation and Statistical Mechanics Model Interactions in Large Data Corpora (Big Data) I was enthralled. I could read for only a few pages at a time, I was so overwhelmed with the insights that this book provided. And I was about twenty-five years old at the time. I had just discovered this book while browsing the stacks as a graduate student at Arizona State (ASU). The book was The Mindful Brain: Cortical Organization and the Group-Selective Theory…

Read More Read More

Visualizing Variables with the 2-D Cluster Variation Method

Visualizing Variables with the 2-D Cluster Variation Method

Cluster Variation Method – 2-D Case – Configuration Variables, Entropy and Free Energy Following the previous blog on the 1-D Cluster Variation Method, I illustrate here a micro-ensemble for the 2-D Cluster Variation Method, consisting of the original single zigzag chain of only ten units (see previous post), with three additional layers added, as shown in the following Figure 1. In Figure 1, we again have an equilibrium distribution of fraction variables z(i). Note that, as with the 1-D case,…

Read More Read More

Visualizing Configuration Variables with the 1-D Cluster Variation Method

Visualizing Configuration Variables with the 1-D Cluster Variation Method

Cluster Variation Method – 1-D Case – Configuration Variables, Entropy and Free Energy We construct a micro-system consisting of a single zigzag chain of only eight units, as shown in the following Figure 1. (Note that the additional textured units, with a dashed border, to the right illustrate a wrap-around effect, giving full horizontal nearest-neighbor connectivity.) In Figure 1, we have the equilibrium distribution of fraction variables z(i). Note that the weighting coefficients for z(2) = z(5) = 2, whereas…

Read More Read More