Browsed by
Category: Neural Networks

Neural networks, including network architectures, training methods, deep learning, and more.

Neural Networks and Python Code: Be Careful with the Array Indices!

Neural Networks and Python Code: Be Careful with the Array Indices!

Comments 5 comments

Our Special Topics class on Deep Learning (Northwestern University, Master of Science in Predictive Analytics program, Winter, 2017) starts off with very basic neural networks: the backpropagation learning method applied to the classic X-OR problem. I’m writing Python code to go with this class, and the result by the end of the quarter should be five-to-six solid pieces of code, involving either the backpropagation or Boltzmann machine learning algorithm, with various network configurations. The following figure shows the dependence of…

Read More Read More

Deep Learning: The First Layer

Deep Learning: The First Layer

Comments 5 comments

It’s been something of a challenging week. We’ve kicked off our new PREDICT 490 Special Topics Course in Deep Learning at Northwestern University. I’ve got a full house; there’s been a waiting list since Thanksgiving, and everyone is acutely aware of the business climate surrounding deep learning. However (and I’m not terribly surprised here), most people who want to learn Deep Learning (DL) really don’t have a solid foundation in neural networks just yet. Thus, what we’re really doing is…

Read More Read More

Getting Started in Deep Learning

Getting Started in Deep Learning

Comments 5 comments

It’s been a lovely Christmas day. More social than any that I can remember, for a very long time. (Wonderful morning visit with my next-door neighbor. Great mid-day party. A sweet restorative nap afterwards.) And now, the thoughts that have been buzzing through and around my head for the past 48 hours — how to get started with deep learning. Of course there are all sorts of entry points. Historical, functional, mathematical… But what came to me, over these past…

Read More Read More

Approximate Bayesian Inference

Approximate Bayesian Inference

Comments 7 comments

Variational Free Energy I spent some time trying to figure out the derivation for the variational free energy, as expressed in some of Friston’s papers (see citations below). While I made an intuitive justification, I just found this derivation (Kokkinos; see the reference and link below): Other discussions about variational free energy: Whereas maximum a posteriori methods optimize a point estimate of the parameters, in ensemble learning an ensemble is optimized, so that it approximates the entire posterior probability distribution…

Read More Read More

Brain-Based Computing: Foundation for Deep Learning

Brain-Based Computing: Foundation for Deep Learning

Comments 3 comments

Three Key Brain Strategies Used in Deep Learning for Artificial Intelligence   References for Brain-Based Computing (Methodologies for Deep Learning and Artificial Intelligence) Maren, A.J. (2015) How the Brain Solves Tough Problems. In Making Sense: Extracting Meaning from Text by Matching Entities and Terms to Ontologies and Concepts, Chapter 2 Draft. (Dec. 31, 2015). pdf Maren, A.J. (2015). Brain-Based Computing. (PPT Slidedeck) PPT  

Novelty Detection in Text Corpora

Novelty Detection in Text Corpora

Comments 0 Comment

Detecting Novelty Using Text Analytics Detecting novel events – new words, meaning new events – is one of the most important text analytics tasks, and is an important step towards predictive analytics using text mining. On July 24, 2015, The New York Times (and many other news sources) published an article identifying potential inclusion of classified information in the emails which Hillary Clinton had sent via private email and stored on her private email server. How would we use text…

Read More Read More

"Nonadditive Entropy" – An Excellent Review Article

"Nonadditive Entropy" – An Excellent Review Article

Comments 0 Comment

New Advances in Entropy Formulation – “Nonadditive Entropy” Well, chalk it up to being newly returned to the fold – after years of work in knowledge discovery, predictive analysis, neural networks, and sensor fusion, I’m finally returning to my roots and re-invigorating some previous work that involves the Cluster Variation Method. In the course of this, I’ve just learned (as a Janie-come-lately) about the major evolution in thinking about entropy, largely led by Constantino Tsallis. He has an excellent review…

Read More Read More

Equilibrium and Utility: Two Different Realms

Equilibrium and Utility: Two Different Realms

Comments 0 Comment

Continuing with Beinhocker’s Origin of Wealth, it is important to distinguish carefully between some of the ideas that Beinhocker is expounding. While overall, he is doing a good job of bringing in many related thoughts and ideas, there is a slight tendency towards “mushing.” In that note, I’d like to suggest that we discern carefully between ideas involving utility (Origins, hardcover; pp. 34 & 37), and equilibrium. On pg. 34, Beinhocker begins a discussion of how utility is an underlying…

Read More Read More