Browsed by
Category: Deep Learning

Start Here: Statistical Mechanics for Neural Networks and AI

Start Here: Statistical Mechanics for Neural Networks and AI

Your Pathway through the Blog-Maze: What to read, and what order to read things in, if you’re trying to teach yourself the rudiments of statistical mechanics – just enough to get a sense of what’s going on in the REAL deep learning papers. As we all know, there’s two basic realms of deep learning neural networks. There’s the kind that only requires (some, limited) knowledge of backpropagation. That’s first semester undergraduate calculus, and almost everyone coming into this field can…

Read More Read More

Generative vs. Discriminative – Where It All Began

Generative vs. Discriminative – Where It All Began

Working Through Salakhutdinov and Hinton’s “An Efficient Learning Procedure for Deep Boltzmann Machines”   We can accomplish a lot, using multiple layers trained with backpropagation. However (as we all know), there are limits to how many layers that we can train at once, if we’re relying strictly on backpropagation (or any other gradient-descent learning rule). This is what stalled out the neural networks community, from the mid-1990’s to the mid-2000’s. The breakthrough came from Hinton and his group, with a…

Read More Read More

Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!

Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!

The Surprising (Hidden) “Gotcha” in This Energy Equation: A couple of days ago, I was doing one of my regular weekly online “Synch” sessions with my Deep Learning students. In a sort of “Beware, here there be dragons!” moment, I showed them this energy equation from the Hinton et al. (2012) Nature review paper on acoustic speech modeling: One of my students pointed out, “That equation looks kind of simple.” Well, he’s right. And I kind of bungled the answer,…

Read More Read More

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)

Making Machine Learning As Simple As Possible Albert Einstein is credited with saying, Everything should be made as simple as possible, but not simpler. Machine learning is not simple. In fact, once you get beyond the simple “building blocks” approach of stacking things higher and deeper (sometimes made all too easy with advanced deep learning packages), you are in the midst of some complex stuff. However, it does not need to be more complex than it has to be.  …

Read More Read More

A Tale of Two Probabilities

A Tale of Two Probabilities

Probabilities: Statistical Mechanics and Bayesian:   Machine learning fuses several different lines of thought, including statistical mechanics, Bayesian probability theory, and neural networks. There are two different ways of thinking about probability in machine learning; one comes from statistical mechanics, and the other from Bayesian logic. Both are important. They are also very different. While these two different ways of thinking about probability are usually very separate, they come together in some of the more advanced machine learning topics, such…

Read More Read More

Labor Day Reading and Academic Year Kick-Off

Labor Day Reading and Academic Year Kick-Off

Deep Learning / Machine Learning Reading and Study Guide:   Several of you have been asking for guided reading lists. This makes sense.   Your Starting Point for Neural Networks, Deep Learning, and Machine Learning   Your study program (reading and code) depends on where you are. Starting out (High-grass country; St. Louis to Alcove Springs): Basic neural networks and deep learning; architecture for common networks, such as CNNs (convolutional neural networks); learning rules and architecture design. Well on the…

Read More Read More

The Statistical Mechanics Underpinnings of Machine Learning

The Statistical Mechanics Underpinnings of Machine Learning

Machine Learning Is Different Now:   Actually, machine learning is a continuation of what it always has been, which is deeply rooted in statistical physics (statistical mechanics). It’s just that there’s a culmination of insights that are now a very substantive body of work, with more theoretical rigor behind them than most of us know.     A Lesson from Mom: It takes a lot of time to learn a new discipline. This is something that I learned from my…

Read More Read More

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Seven Statistical Mechanics / Bayesian Equations That You Need to Know

Essential Statistical Mechanics for Deep Learning   If you’re self-studying machine learning, and feel that statistical mechanics is suddenly showing up more than it used to, you’re not alone. Within the past couple of years, statistical mechanics (statistical thermodynamics) has become a more integral topic, along with the Kullback-Leibler divergence measure and several inference methods for machine learning, including the expectation maximization (EM) algorithm along with variational Bayes.     Statistical mechanics has always played a strong role in machine…

Read More Read More

Deep Learning: The Fast Evolution of Artificial Intelligence

Deep Learning: The Fast Evolution of Artificial Intelligence

Just one of the slides from a presentation that I’m working up for an upcoming online presentation at Northwestern University, but it tells the story. Just one more thought: here’s the rapid pace of evolution within just the image analysis realm of AI, largely due to multiple layers (sometimes, many, many, MANY multiple layers) of networks, a good fraction of which are Convolutional Neural Networks, or CNNs. Error rates have dropped from over 15% to about 3% within just four…

Read More Read More

Deep Learning: The First Layer

Deep Learning: The First Layer

It’s been something of a challenging week. We’ve kicked off our new PREDICT 490 Special Topics Course in Deep Learning at Northwestern University. I’ve got a full house; there’s been a waiting list since Thanksgiving, and everyone is acutely aware of the business climate surrounding deep learning. However (and I’m not terribly surprised here), most people who want to learn Deep Learning (DL) really don’t have a solid foundation in neural networks just yet. Thus, what we’re really doing is…

Read More Read More