Browsed by
Category: Neural Networks

Neural networks, including network architectures, training methods, deep learning, and more.

New! The YouTube Vid Series: Backpropagation and More

New! The YouTube Vid Series: Backpropagation and More

Comments 0 Comment

If you are branding yourself as an AI/neural networks/deep learning person, how well do you really know the backpropagation derivation? That is, could you work through that derivation, on your own, without having to find a tutorial on it? If not – you’re in good company. MOST people haven’t worked through that derivation – and for good reason. MOST people don’t remember their chain rule methods from first semester calculus. (It probably doesn’t help if I say that the backprop…

Read More Read More

Directed vs. Undirected Graphs in NNs: The (Surprising!) Implications

Directed vs. Undirected Graphs in NNs: The (Surprising!) Implications

Comments 0 Comment

Most of us don’t always use graph language to describe neural networks, but if we dig into the implications of graph theory language, we get some surprising (and very useful) insights! We probably all know that a typical feedforward neural network can be described as a “directed graph.” Many of us also know that a restricted Boltzmann machine (RBM) is an “undirected graph.” In this little difference of terms, there is a wealth of meaning. Salakhutdinov, Mnih, and Hinton (2007;…

Read More Read More

Book Chapter: Draft Chapter 7 – The Boltzmann Machine

Book Chapter: Draft Chapter 7 – The Boltzmann Machine

Comments 3 comments

Chapter 7: Energy-Based Neural Networks This is the full chapter draft from the book-in-progress, Statistical Mechanics, Neural Networks, and Artificial Intelligence. This chapter draft covers not only the Hopfield neural network (released as an excerpt last week), but also the Boltzmann machine, in both general and restricted forms. It deals with that form-equals-function connection, based on the energy equation. (However, we postpone the full-fledged learning method to a later chapter.) Get the pdf using the pdf link in the citation…

Read More Read More

Start Here: Statistical Mechanics for Neural Networks and AI

Start Here: Statistical Mechanics for Neural Networks and AI

Comments 0 Comment

Your Pathway through the Blog-Maze: What to read, and what order to read things in, if you’re trying to teach yourself the rudiments of statistical mechanics – just enough to get a sense of what’s going on in the REAL deep learning papers. As we all know, there’s two basic realms of deep learning neural networks. There’s the kind that only requires (some, limited) knowledge of backpropagation. That’s first semester undergraduate calculus, and almost everyone coming into this field can…

Read More Read More

Generative vs. Discriminative – Where It All Began

Generative vs. Discriminative – Where It All Began

Comments 0 Comment

Working Through Salakhutdinov and Hinton’s “An Efficient Learning Procedure for Deep Boltzmann Machines”   We can accomplish a lot, using multiple layers trained with backpropagation. However (as we all know), there are limits to how many layers that we can train at once, if we’re relying strictly on backpropagation (or any other gradient-descent learning rule). This is what stalled out the neural networks community, from the mid-1990’s to the mid-2000’s. The breakthrough came from Hinton and his group, with a…

Read More Read More

Wrapping Our Heads Around Entropy

Wrapping Our Heads Around Entropy

Comments 5 comments

Entropy – the Most Powerful Force in the ‘Verse:   Actually, that’s not quite true. The most powerful force in the ‘verse is free energy minimization. However, entropy is half of the free energy equation, and it’s usually the more complex half. So, if we understand entropy, then we can understand free energy minimization. If we understand free energy minimization, then we understand all the energy-based machine learning models, including the (restricted) Boltzmann machine and one of its most commonly-used…

Read More Read More

Artificial General Intelligence: Getting There from Here

Artificial General Intelligence: Getting There from Here

Comments 3 comments

What We Need to Create Artificial General Intelligence (AGI):   A brief recap: We know that we want to have neural networks (including deep learning) do something besides being sausage factories. We’ve know that the key missing step – a first principles step – to making this happen is to give the network something to do when it is not responding to inputs. Also, we’ve introduced something that the neural network CAN do; it can do free energy minimization with…

Read More Read More

Figuring Out the Puzzle (in a 2-D CVM Grid)

Figuring Out the Puzzle (in a 2-D CVM Grid)

Comments 7 comments

The Conundrum – and How to Solve It: We left off last week with a bit of a cliff-hanger; a puzzle with the 2-D CVM. (CVM stands for Cluster Variation Method; it’s a more complex form of a free energy equation that I discussed two weeks ago in this blogpost on The Big, Bad, Scary Free Energy Equation (and New Experimental Results); while not entirely unknown, it’s still not very common yet.) We asked ourselves: which of the two grids…

Read More Read More

The Big, Bad, Scary Free Energy Equation (and New Experimental Results)

The Big, Bad, Scary Free Energy Equation (and New Experimental Results)

Comments 8 comments

The 2-D Cluster Variation Method Free Energy Equation – in All Its Scary Glory:   You know, my dear, that we’ve been leading up to this moment for a while now. I’ve hinted. I’ve teased and been coy. But now, it’s time to be full frontal. We’re going to look at a new form of a free energy equation; a cluster variation method (CVM) equation. It deals not only with how many units are in state A or state B,…

Read More Read More

A “First Principles” Approach to Artificial General Intelligence

A “First Principles” Approach to Artificial General Intelligence

Comments 10 comments

What We Need to Take the Next Tiny, Incremental Little Step: The “next big thing” is likely to be the next small thing – a tiny step, an incremental shift in perspective. However, a perspective shift is all that we need in order to make some real advances towards general artificial intelligence (GAI). In the second chapter of the ongoing book , I share the following figure (and sorry, the chapter itself is not released yet): Now, we’ve actually been…

Read More Read More