Browsed by
Category: Information Theory

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Making Machine Learning As Simple As Possible Albert Einstein is credited with saying, Everything should be made as simple as possible, but not simpler. Machine learning is not simple. In fact, once you get beyond the simple “building blocks” approach of stacking things higher and deeper (sometimes made all too easy with advanced deep learning packages), you are in the midst of some complex stuff. However, it does not need to be more complex than it has to be.  …

Read More Read More

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Essential Statistical Mechanics for Deep Learning   If you’re self-studying machine learning, and feel that statistical mechanics is suddenly showing up more than it used to, you’re not alone. Within the past couple of years, statistical mechanics (statistical thermodynamics) has become a more integral topic, along with the Kullback-Leibler divergence measure and several inference methods for machine learning, including the expectation maximization (EM) algorithm along with variational Bayes.     Statistical mechanics has always played a strong role in machine…

Read More Read More

The Single Most Important Equation for Brain-Computer Information Interfaces

The Single Most Important Equation for Brain-Computer Information Interfaces

The Kullback-Leibler Divergence Equation for Brain-Computer Information Interfaces The Kullback-Leibler equation is arguably the best place for starting our thoughts about information theory as applied to Brain-Computer Interfaces (BCIs), or Brain-Computer Information Interfaces (BCIIs). The Kullback-Leibler equation is given as: We seek to express how well our model of reality matches the real system. Or, just as usefully, we seek to express the information-difference when we have two different models for the same underlying real phenomena or data. The K-L…

Read More Read More

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method (CVM) – Simple Application

The 1-D Cluster Variation Method – Application to Text Mining and Data Mining There are three particularly good reasons for us to look at the Cluster Variation Method (CVM) as an alternative means of understanding the information in a system: The CVM captures local pattern distributions (for an equilibrium state), When the system is made up of equal numbers of units in each of two states, and the enthalpy for each state is the same (the simple unit activation energy…

Read More Read More

Why Nonadditive Entropy Is Important for Big Data Corpora Combinations

Why Nonadditive Entropy Is Important for Big Data Corpora Combinations

Non-Additive Entropy – A Crucial Predictive Analysis Measure for Data Mining in Multiple Large Data Corpora Statistical mechanics has an important role to play in big data analytics. Up until now, there has been almost no understanding of how statistical mechanics provides both practical value and a theoretic framework for data analysis and even predictive intelligence (sometimes called predictive analysis). This blogpost focuses on a related – and crucially important – issue: How can we determine the value of combining…

Read More Read More

Nonextensive Statistical Mechanics – Good Read on Advanced Entropy Formulation

Nonextensive Statistical Mechanics – Good Read on Advanced Entropy Formulation

Advances in Thinking about Entropy Starting through Tsallis’s book on entropy; Introduction to Nonextensive Statistical Mechanics. This is a fascinating discussion – really, it’s the roots of philosophy; the real “what-is-so” about the world. Which minimally requires a good solid year or two of graduate-level statistical thermodynamics to even start the read. But worth it. There’s some potential applications of this approach to areas in which I’ve worked before; need to mull this over and jig some ideas about to…

Read More Read More

Non-Equilibrium Information Theory (DARPA group)

Non-Equilibrium Information Theory (DARPA group)

Of possible interest — DARPA group attempting to use non-equilibrium information theory to study mobile ad hoc wireless networks (MANETs). Lots of information theory pubs, not too sure yet they’re really on to what constitutes “non-equilibrium,” worth investigating.