Browsed by
Category: Equilibrium

A “First Principles” Approach to General AI

A “First Principles” Approach to General AI

What We Need to Take the Next Tiny, Incremental Little Step: The “next big thing” is likely to be the next small thing – a tiny step, an incremental shift in perspective. However, a perspective shift is all that we need in order to make some real advances towards general artificial intelligence (GAI). In the second chapter of the ongoing book , I share the following figure (and sorry, the chapter itself is not released yet): Now, we’ve actually been…

Read More Read More

How Getting to a Free Energy Bottom Helps Us Get to the Top

How Getting to a Free Energy Bottom Helps Us Get to the Top

Free Energy Minimization Gives an AI Engine Something Useful to Do:   Cutting to the chase: we need free energy minimization in a computational engine, or AI system, because it gives the system something to do besides being a sausage-making machine, as I described in yesterday’s blog on What’s Next for AI. Right now, deep learning systems are constrained to be simple input/output devices. We force-feed them with stimulus at one end, and they poop out (excuse me, “pop out”)…

Read More Read More

Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!

Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!

The Surprising (Hidden) “Gotcha” in This Energy Equation: A couple of days ago, I was doing one of my regular weekly online “Synch” sessions with my Deep Learning students. In a sort of “Beware, here there be dragons!” moment, I showed them this energy equation from the Hinton et al. (2012) Nature review paper on acoustic speech modeling: One of my students pointed out, “That equation looks kind of simple.” Well, he’s right. And I kind of bungled the answer,…

Read More Read More

A Tale of Two Probabilities

A Tale of Two Probabilities

Probabilities: Statistical Mechanics and Bayesian:   Machine learning fuses several different lines of thought, including statistical mechanics, Bayesian probability theory, and neural networks. There are two different ways of thinking about probability in machine learning; one comes from statistical mechanics, and the other from Bayesian logic. Both are important. They are also very different. While these two different ways of thinking about probability are usually very separate, they come together in some of the more advanced machine learning topics, such…

Read More Read More

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method – Application to Text Mining and Data Mining There are three particularly good reasons for us to look at the Cluster Variation Method (CVM) as an alternative means of understanding the information in a system: The CVM captures local pattern distributions (for an equilibrium state), When the system is made up of equal numbers of units in each of two states, and the enthalpy for each state is the same (the simple unit activation energy…

Read More Read More

Statistical Mechanics – Neural Ensembles

Statistical Mechanics – Neural Ensembles

Statistical Mechanics and Equilibrium Properties – Small Neural Ensembles Statistical Mechanics of Small Neural Ensembles – Commentary on Tkačik et al. In a series of related articles, Gašper Tkačik et al. (see references below) investigated small (10-120) groups of neurons in the salamander retina, with the purpose of estimating entropy and other statistical mechanics properties. They provide the following interesting results: Simple scheme for entropy estimation in undersampled region (1), given that only a small fraction of possible states can…

Read More Read More

Statistical Mechanics, Neural Domains, and Big Data

Statistical Mechanics, Neural Domains, and Big Data

How Neural Domain Activation and Statistical Mechanics Model Interactions in Large Data Corpora (Big Data) I was enthralled. I could read for only a few pages at a time, I was so overwhelmed with the insights that this book provided. And I was about twenty-five years old at the time. I had just discovered this book while browsing the stacks as a graduate student at Arizona State (ASU). The book was The Mindful Brain: Cortical Organization and the Group-Selective Theory…

Read More Read More

Visualizing Variables with the 2-D Cluster Variation Method

Visualizing Variables with the 2-D Cluster Variation Method

Cluster Variation Method – 2-D Case – Configuration Variables, Entropy and Free Energy Following the previous blog on the 1-D Cluster Variation Method, I illustrate here a micro-ensemble for the 2-D Cluster Variation Method, consisting of the original single zigzag chain of only ten units (see previous post), with three additional layers added, as shown in the following Figure 1. In Figure 1, we again have an equilibrium distribution of fraction variables z(i). Note that, as with the 1-D case,…

Read More Read More

Visualizing Configuration Variables with the 1-D Cluster Variation Method

Visualizing Configuration Variables with the 1-D Cluster Variation Method

Cluster Variation Method – 1-D Case – Configuration Variables, Entropy and Free Energy We construct a micro-system consisting of a single zigzag chain of only eight units, as shown in the following Figure 1. (Note that the additional textured units, with a dashed border, to the right illustrate a wrap-around effect, giving full horizontal nearest-neighbor connectivity.) In Figure 1, we have the equilibrium distribution of fraction variables z(i). Note that the weighting coefficients for z(2) = z(5) = 2, whereas…

Read More Read More

Analytic Single-Point Solution for Cluster Variation Method Variables (at x1=x2=0.5)

Analytic Single-Point Solution for Cluster Variation Method Variables (at x1=x2=0.5)

Single-Point Analytic Cluster Variation Method Solution: Solving Set of Three Nonlinear, Coupled Equations The Cluster Variation Method, first introduced by Kikuchi in 1951 (“A theory of cooperative phenomena,” Phys. Rev. 81 (6), 988-1003), provides a means for computing the free energy of a system where the entropy term takes into account distributions of particles into local configurations as well as the distribution into “on/off” binary states. As the equations are more complex, numerical solutions for the cluster variation variables are…

Read More Read More