Browsed by
Category: Neural Network Architectures

Directed vs. Undirected Graphs in NNs: The (Surprising!) Implications

Directed vs. Undirected Graphs in NNs: The (Surprising!) Implications

Comments 0 Comment

Most of us don’t always use graph language to describe neural networks, but if we dig into the implications of graph theory language, we get some surprising (and very useful) insights! We probably all know that a typical feedforward neural network can be described as a “directed graph.” Many of us also know that a restricted Boltzmann machine (RBM) is an “undirected graph.” In this little difference of terms, there is a wealth of meaning. Salakhutdinov, Mnih, and Hinton (2007;…

Read More Read More

Book Chapter: Draft Chapter 7 – The Boltzmann Machine

Book Chapter: Draft Chapter 7 – The Boltzmann Machine

Comments 3 comments

Chapter 7: Energy-Based Neural Networks This is the full chapter draft from the book-in-progress, Statistical Mechanics, Neural Networks, and Artificial Intelligence. This chapter draft covers not only the Hopfield neural network (released as an excerpt last week), but also the Boltzmann machine, in both general and restricted forms. It deals with that form-equals-function connection, based on the energy equation. (However, we postpone the full-fledged learning method to a later chapter.) Get the pdf using the pdf link in the citation…

Read More Read More

Neural Network Architectures: Determining the Number of Hidden Nodes

Neural Network Architectures: Determining the Number of Hidden Nodes

Comments 0 Comment

Figuring Out the Number of Hidden Nodes: Then and Now   One of the most demanding questions in developing neural networks (of any size or complexity) is determining the architecture: number of layers, nodes-per-layer, and other factors. This was an important question in the late 1980’s and early 1990’s, when neural networks first emerged. Deciding on the network architecture details is even more challenging today. In this post, we’re going to look at some strategies for deciding on the number…

Read More Read More

Selecting a Neural Network Transfer Function: Classic vs. Current

Selecting a Neural Network Transfer Function: Classic vs. Current

Comments 2 comments

Neural Network Transfer Functions: Sigmoid, Tanh, and ReLU   Making it or breaking it with neural networks: how to make smart choices.     Why We Weren’t Getting Convergence   This last week, in working with a very simple and straightforward XOR neural network, a lot of my students were having convergence problems. The most likely reason? Very likely, it’s been my choice for the transfer function. I had given them a very simple network. (Lots of them are still…

Read More Read More