Browsed by
Category: Entropy

Machine Learning: Multistage Boost Process

Machine Learning: Multistage Boost Process

Three Stages to Orbital Altitude in Machine Learning Several years ago, Regina Dugan (then Director of DARPA) gave a talk in which she showed a clip of epic NASA launch fails. Not just one, but many fails. The theme was that we had to risk failure in order to succeed with innovation. This YouTube vid of rocket launch failures isn’t the exact clip that she showed (the “action” doesn’t kick in for about a minute), but it’s pretty close. For…

Read More Read More

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Making Machine Learning As Simple As Possible Albert Einstein is credited with saying, Everything should be made as simple as possible, but not simpler. Machine learning is not simple. In fact, once you get beyond the simple “building blocks” approach of stacking things higher and deeper (sometimes made all too easy with advanced deep learning packages), you are in the midst of some complex stuff. However, it does not need to be more complex than it has to be.  …

Read More Read More

A Tale of Two Probabilities

A Tale of Two Probabilities

Probabilities: Statistical Mechanics and Bayesian:   Machine learning fuses several different lines of thought, including statistical mechanics, Bayesian probability theory, and neural networks. There are two different ways of thinking about probability in machine learning; one comes from statistical mechanics, and the other from Bayesian logic. Both are important. They are also very different. While these two different ways of thinking about probability are usually very separate, they come together in some of the more advanced machine learning topics, such…

Read More Read More

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Essential Statistical Mechanics for Deep Learning   If you’re self-studying machine learning, and feel that statistical mechanics is suddenly showing up more than it used to, you’re not alone. Within the past couple of years, statistical mechanics (statistical thermodynamics) has become a more integral topic, along with the Kullback-Leibler divergence measure and several inference methods for machine learning, including the expectation maximization (EM) algorithm along with variational Bayes.     Statistical mechanics has always played a strong role in machine…

Read More Read More

How to Read Karl Friston (in the Original Greek)

How to Read Karl Friston (in the Original Greek)

Karl Friston, whom we all admire, has written some lovely papers that are both enticing and obscure. Cutting to the chase, what we really want to understand is this equation: In a Research Digest article, Peter Freed writes: … And today, Karl Friston is not explaining [the free energy principle] in a way that makes it usable to your average psychiatrist/psychotherapist on the street – which is frustrating. I am not alone in my confusion, and if you read the…

Read More Read More

Approximate Bayesian Inference

Approximate Bayesian Inference

Variational Free Energy I spent some time trying to figure out the derivation for the variational free energy, as expressed in some of Friston’s papers (see citations below). While I made an intuitive justification, I just found this derivation (Kokkinos; see the reference and link below): Other discussions about variational free energy: Whereas maximum a posteriori methods optimize a point estimate of the parameters, in ensemble learning an ensemble is optimized, so that it approximates the entire posterior probability distribution…

Read More Read More

Brain Networks and the Cluster Variation Method: Testing a Scale-Free Model

Brain Networks and the Cluster Variation Method: Testing a Scale-Free Model

Surprising Result Modeling a Simple Scale-Free Brain Network Using the Cluster Variation Method One of the primary research thrusts that I suggested in my recent paper, The Cluster Variation Method: A Primer for Neuroscientists, was that we could use the 2-D Cluster Variation Method (CVM) to model distribution of configuration variables in different brain network topologies. Specifically, I was expecting that the h-value (which measures the interaction enthalpy strength between nodes in a 2-D CVM grid) would change in a…

Read More Read More

The Cluster Variation Method: A Primer for Neuroscientists

The Cluster Variation Method: A Primer for Neuroscientists

Single-Parameter Analytic Solution for Modeling Local Pattern Distributions The cluster variation method (CVM) offers a means for the characterization of both 1-D and 2-D local pattern distributions. The paper referenced at the end of this post provides neuroscientists and BCI researchers with a CVM tutorial that will help them to understand how the CVM statistical thermodynamics formulation can model 1-D and 2-D pattern distributions expressing structural and functional dynamics in the brain. The equilibrium distribution of local patterns, or configuration…

Read More Read More

The Single Most Important Equation for Brain-Computer Information Interfaces

The Single Most Important Equation for Brain-Computer Information Interfaces

The Kullback-Leibler Divergence Equation for Brain-Computer Information Interfaces The Kullback-Leibler equation is arguably the best place for starting our thoughts about information theory as applied to Brain-Computer Interfaces (BCIs), or Brain-Computer Information Interfaces (BCIIs). The Kullback-Leibler equation is given as: We seek to express how well our model of reality matches the real system. Or, just as usefully, we seek to express the information-difference when we have two different models for the same underlying real phenomena or data. The K-L…

Read More Read More

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method – Application to Text Mining and Data Mining There are three particularly good reasons for us to look at the Cluster Variation Method (CVM) as an alternative means of understanding the information in a system: The CVM captures local pattern distributions (for an equilibrium state), When the system is made up of equal numbers of units in each of two states, and the enthalpy for each state is the same (the simple unit activation energy…

Read More Read More