Browsed by
Category: A Resource – Article

Third Stage Boost – Part 2: Implications of Neuromorphic Computing

Third Stage Boost – Part 2: Implications of Neuromorphic Computing

Neuromorphic Computing: Statistical Mechanics & Criticality   Last week, I suggested that we were on the verge of something new, and referenced an article by von Bubnoff: A brain built from atomic switches [that] can learn, together with the follow-on article Brain Built on Switches. The key innovation described in this article was a silver mesh, as shown in the following figure. This mesh is a “network of microscopically thin intersecting silver wires,” grown via a combination of electrochemical and…

Read More Read More

Third Stage Boost: Statistical Mechanics and Neuromorphic Computing – Part 1

Third Stage Boost: Statistical Mechanics and Neuromorphic Computing – Part 1

Next-Generation Neural Network Architectures: More Brain-Like   Three generations of artificial intelligence.. The third generation is emerging … right about … now. That’s what is shown in this figure, presented in log-time scale. Brief history of AI in log-time scale The first generation of AI, symbolic AI, began conceptually around 1954, and lasted until 1986; 32 years. On the log-time scale shown in the figure above, this entire era takes place under the first curve; the black bell-shaped curve on…

Read More Read More

Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!

Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!

The Surprising (Hidden) “Gotcha” in This Energy Equation: A couple of days ago, I was doing one of my regular weekly online “Synch” sessions with my Deep Learning students. In a sort of “Beware, here there be dragons!” moment, I showed them this energy equation from the Hinton et al. (2012) Nature review paper on acoustic speech modeling: One of my students pointed out, “That equation looks kind of simple.” Well, he’s right. And I kind of bungled the answer,…

Read More Read More

Neural Network Architectures: Determining the Number of Hidden Nodes

Neural Network Architectures: Determining the Number of Hidden Nodes

Figuring Out the Number of Hidden Nodes: Then and Now   One of the most demanding questions in developing neural networks (of any size or complexity) is determining the architecture: number of layers, nodes-per-layer, and other factors. This was an important question in the late 1980’s and early 1990’s, when neural networks first emerged. Deciding on the network architecture details is even more challenging today. In this post, we’re going to look at some strategies for deciding on the number…

Read More Read More

Selecting a Neural Network Transfer Function: Classic vs. Current

Selecting a Neural Network Transfer Function: Classic vs. Current

Neural Network Transfer Functions: Sigmoid, Tanh, and ReLU   Making it or breaking it with neural networks: how to make smart choices.     Why We Weren’t Getting Convergence   This last week, in working with a very simple and straightforward XOR neural network, a lot of my students were having convergence problems. The most likely reason? Very likely, it’s been my choice for the transfer function. I had given them a very simple network. (Lots of them are still…

Read More Read More

Notational Frenzy

Notational Frenzy

When the Subtle Art of Mathematical Notation Defeats You (and How to Fight Back)   A couple of years ago, I was teaching Time Series and Forecasting for the first time. I didn’t know the subject – at all – but that didn’t bother me. Hey, it was mathematics, right? Anything that’s mathematical, I can eat for lunch, and then want some dessert-equations afterwards. First week, introducing the subject. That went fine. Second week, Simple Exponential Smoothing (SES). That’s simple….

Read More Read More

Labor Day Reading and Academic Year Kick-Off

Labor Day Reading and Academic Year Kick-Off

Deep Learning / Machine Learning Reading and Study Guide:   Several of you have been asking for guided reading lists. This makes sense.   Your Starting Point for Neural Networks, Deep Learning, and Machine Learning   Your study program (reading and code) depends on where you are. Starting out (High-grass country; St. Louis to Alcove Springs): Basic neural networks and deep learning; architecture for common networks, such as CNNs (convolutional neural networks); learning rules and architecture design. Well on the…

Read More Read More

The Statistical Mechanics Underpinnings of Machine Learning

The Statistical Mechanics Underpinnings of Machine Learning

Machine Learning Is Different Now:   Actually, machine learning is a continuation of what it always has been, which is deeply rooted in statistical physics (statistical mechanics). It’s just that there’s a culmination of insights that are now a very substantive body of work, with more theoretical rigor behind them than most of us know.     A Lesson from Mom: It takes a lot of time to learn a new discipline. This is something that I learned from my…

Read More Read More

2025 and Beyond

2025 and Beyond

Artificial Intelligence and Jobs by the Year 2025: One of my biggest take-aways from the recent (May, 2017) NVIDIA GTC (GPU Technology Conference) was less about the technology, and more about the near-term jobs impact of artificial intelligence (AI) and robotics. Making smart education and career decisions is crucial, as the emerging combination of AI and robotics will have a huge impact on jobs. Those of you studying artificial intelligence, deep learning, and neural networks will have a stronger career…

Read More Read More

How to Read Karl Friston (in the Original Greek)

How to Read Karl Friston (in the Original Greek)

Karl Friston, whom we all admire, has written some lovely papers that are both enticing and obscure. Cutting to the chase, what we really want to understand is this equation: In a Research Digest article, Peter Freed writes: … And today, Karl Friston is not explaining [the free energy principle] in a way that makes it usable to your average psychiatrist/psychotherapist on the street – which is frustrating. I am not alone in my confusion, and if you read the…

Read More Read More