Browsed by
Category: Neural Networks

Neural networks, including network architectures, training methods, deep learning, and more.

Third Stage Boost: Statistical Mechanics and Neuromorphic Computing – Part 1

Third Stage Boost: Statistical Mechanics and Neuromorphic Computing – Part 1

Next-Generation Neural Network Architectures: More Brain-Like   Three generations of artificial intelligence.. The third generation is emerging … right about … now. That’s what is shown in this figure, presented in log-time scale. Brief history of AI in log-time scale The first generation of AI, symbolic AI, began conceptually around 1954, and lasted until 1986; 32 years. On the log-time scale shown in the figure above, this entire era takes place under the first curve; the black bell-shaped curve on…

Read More Read More

Neural Network Architectures: Determining the Number of Hidden Nodes

Neural Network Architectures: Determining the Number of Hidden Nodes

Figuring Out the Number of Hidden Nodes: Then and Now   One of the most demanding questions in developing neural networks (of any size or complexity) is determining the architecture: number of layers, nodes-per-layer, and other factors. This was an important question in the late 1980’s and early 1990’s, when neural networks first emerged. Deciding on the network architecture details is even more challenging today. In this post, we’re going to look at some strategies for deciding on the number…

Read More Read More

Selecting a Neural Network Transfer Function: Classic vs. Current

Selecting a Neural Network Transfer Function: Classic vs. Current

Neural Network Transfer Functions: Sigmoid, Tanh, and ReLU   Making it or breaking it with neural networks: how to make smart choices.     Why We Weren’t Getting Convergence   This last week, in working with a very simple and straightforward XOR neural network, a lot of my students were having convergence problems. The most likely reason? Very likely, it’s been my choice for the transfer function. I had given them a very simple network. (Lots of them are still…

Read More Read More

Backpropagation: Not Dead, Not Yet

Backpropagation: Not Dead, Not Yet

Backpropagation: Why It Still Matters   Thirty years ago, at the dawn of the neural networks era, backpropagation was all the rage. In the minds of most people, it was infinitely preferable to the simulated annealing algorithm that Hinton et al. had proposed for their Boltzmann machine. Now, it seems as though the see-saw of algorithm popularity has shifted; we’re focused on energy-based methods. We might be asking: is backpropagation old hat? Good question! Even more than that, someone coming…

Read More Read More

A Tale of Two Probabilities

A Tale of Two Probabilities

Probabilities: Statistical Mechanics and Bayesian:   Machine learning fuses several different lines of thought, including statistical mechanics, Bayesian probability theory, and neural networks. There are two different ways of thinking about probability in machine learning; one comes from statistical mechanics, and the other from Bayesian logic. Both are important. They are also very different. While these two different ways of thinking about probability are usually very separate, they come together in some of the more advanced machine learning topics, such…

Read More Read More

Labor Day Reading and Academic Year Kick-Off

Labor Day Reading and Academic Year Kick-Off

Deep Learning / Machine Learning Reading and Study Guide:   Several of you have been asking for guided reading lists. This makes sense.   Your Starting Point for Neural Networks, Deep Learning, and Machine Learning   Your study program (reading and code) depends on where you are. Starting out (High-grass country; St. Louis to Alcove Springs): Basic neural networks and deep learning; architecture for common networks, such as CNNs (convolutional neural networks); learning rules and architecture design. Well on the…

Read More Read More

The Statistical Mechanics Underpinnings of Machine Learning

The Statistical Mechanics Underpinnings of Machine Learning

Machine Learning Is Different Now:   Actually, machine learning is a continuation of what it always has been, which is deeply rooted in statistical physics (statistical mechanics). It’s just that there’s a culmination of insights that are now a very substantive body of work, with more theoretical rigor behind them than most of us know.     A Lesson from Mom: It takes a lot of time to learn a new discipline. This is something that I learned from my…

Read More Read More

Deep Learning: The Fast Evolution of Artificial Intelligence

Deep Learning: The Fast Evolution of Artificial Intelligence

Just one of the slides from a presentation that I’m working up for an upcoming online presentation at Northwestern University, but it tells the story. Just one more thought: here’s the rapid pace of evolution within just the image analysis realm of AI, largely due to multiple layers (sometimes, many, many, MANY multiple layers) of networks, a good fraction of which are Convolutional Neural Networks, or CNNs. Error rates have dropped from over 15% to about 3% within just four…

Read More Read More

Neural Networks and Python Code: Be Careful with the Array Indices!

Neural Networks and Python Code: Be Careful with the Array Indices!

Our Special Topics class on Deep Learning (Northwestern University, Master of Science in Predictive Analytics program, Winter, 2017) starts off with very basic neural networks: the backpropagation learning method applied to the classic X-OR problem. I’m writing Python code to go with this class, and the result by the end of the quarter should be five-to-six solid pieces of code, involving either the backpropagation or Boltzmann machine learning algorithm, with various network configurations. The following figure shows the dependence of…

Read More Read More

Deep Learning: The First Layer

Deep Learning: The First Layer

It’s been something of a challenging week. We’ve kicked off our new PREDICT 490 Special Topics Course in Deep Learning at Northwestern University. I’ve got a full house; there’s been a waiting list since Thanksgiving, and everyone is acutely aware of the business climate surrounding deep learning. However (and I’m not terribly surprised here), most people who want to learn Deep Learning (DL) really don’t have a solid foundation in neural networks just yet. Thus, what we’re really doing is…

Read More Read More