The Uber-Important Chapter 7 – Introduction to Energy-based Neural Networks:

 

I tell students that it’s like being on the Oregon Trail. All of the stochastic gradient-descent networks (up to and including Convolutional Neural Networks, or ConvNets, and Long Short-Term Memory networks, or LSTM networks) can be understood using backpropagation. This requires only that first-semester calculus background. Sure, grunting through the chain rule (many, many times) gets tedious. But it’s doable.

In contrast, the energy-based networks are the heart and soul of deep learning. However, learning about the energy-based neural networks (the Boltzmann machine network, in both general and restricted forms, and the predecessor Hopfield neural network) is like crossing the Rocky Mountains. We need an entirely different skill set.

Pragmatically, very few of us entering the AI and deep learning field have the necessary background. (That is, solid graduate-level coursework in theoretical physics.)

So, I’m writing the book. The Book.

Part I (Chapters 1 – 6) are all on the early part of the Oregon Trail; neural network structures, and training with stochastic gradient descent (e.g., using backpropagation). This has been done before, many times and many ways, by other authors – but it lays the groundwork and gets a consistent framework going.

Part II introduces not only energy-based networks, but also covers the requisite statistical mechanics. Sufficient so that after reading Part II, someone could go to any of Hinton and colleagues’ papers, and actually understand them.

Chapter 7, the kick-off chapter in Part II, starts by connecting the energy equations for both the Hopfield and Hinton (Boltzmann machine) neural networks to their structure. It’s a form follows function kind of chapter.

My gut sense is that this is going to be the best and most readable introduction to energy-based systems available.

Once done, that is.

 

 

Potholes

 

Except that 2018 was just a very tough year – course development for TWO AI courses at Northwestern. And I was a bit behind on life after those were done, so 2019 was off to a slow start.

Basically, I got very little book-writing done in 2018, and not much during the 2018-2019 winter. Everything stalled out.

 

 

James Clear’s Atomic Habits

 

I did get started on the book again, but what really helped me get consistent was James Clear’s book, Atomic Habits. More specifically, one of his blogposts where he wrote about his paperclip strategy.

His post made a lot of sense, so I ordered some strong refrigerator magnetic hooks, and a couple of sets of paperclips. (Multiple colors. I’m girly. I like color. And I like color-coding.)

 

 

Back on Track

 

I’ve started off again – and now am giving myself one paperclip for each day that I work on the book. I tell myself that it doesn’t matter how much I write. Just opening up the file and writing one sentence – just one sentence – is sufficient for the day. (And usually, of course, once I get started, I write a lot more.)

So … I got started a few weeks ago, and then (as soon as the paperclips and magnetic hooks arrived, thanks to Amazon), I began building up the paperclip chain.

And then I was in a slightly off mood one morning, and a girlfriend asked me to meet her for coffee, and without even thinking about it, I said yes. Then I came home, and got caught up in another author’s cry for help. And by the time that I had finished with that, the day was rolling on and there were things that I had to do.

Since I was “playing by the rules,” I had to deconstruct that paperclip chain. (The rules of the game, at least the way that I’m playing it, are that I allow myself certain sanctioned “days off.” They have special colors. But just saying, “Oh, the heck with it!” – that means dis-assembling the paperclip chain and starting all over. Sucks when I have to do that, as I found out.)

So, the next day, I was back on track. New paperclip chain.

I’ve been at it for a week now, as evidenced by the paperclip chain that you can see on my fridge, in the figure below.

 

 

Chapter 7: How Soon You Can Get Your Hands on It

 

I’ve decided to publish certain key chapters as I go along. In part, because a lot of you folks need this. Having Part II will be like having elevators to scale the ice wall in the Game of Thrones.

In part, also, because I need it. I need your feedback and review. (Not to mention a group-effort on nit-finding and error-checking. And my big thanks to those of you who’ve found nits in my previous works, made available to those who’ve Opted-In to The Book..

I’ll let you know as soon as its available.

And, just to keep myself honest, I’ll publish pictures every so often of the paperclip chain. So you’ll know how things are coming along.

 

 

Your Thoughts?

 

What are you looking for in a book? What topics puzzle you the most? What have you tried to learn from other sources? (And then found those sources to be – someohow – not exactly what you needed.)

Right now is the stage when I can most easily incorporate your early feedback, and make sure that this book is something that will be most useful to you. Thank you! – AJM

 

 

Live free or die, my friend –

AJ Maren

Live free or die: Death is not the worst of evils.
Attr. to Gen. John Stark, American Revolutionary War

 

 

Resources

Link to The Book Page

• Book: Statistical Mechanics, Neural Networks, and Artificial Intelligence

Book(s) (Other than Mine)

• Clear, J. (2018). Atomic Habits, New York: Avery (Penguin Random House). Chapter excerpt and ordering online.