Neural Networks and Deep Learning

As a teen in the 1950s, Geoffrey Hinton wanted to be a professor and to study AI. He came from a family of noted academics (his great-great-grandfather was George Boole). His mom would often say, “Be an academic or be a failure.”¹⁰

Even during the first AI winter, Hinton was passionate about AI and was convinced that Rosenblatt’s neural network approach was the right path. So in 1972, he received his PhD on the topic from the University of Edinburgh.

But during this period, many people thought that Hinton was wasting his time and talents. AI was essentially considered a fringe area. It wasn’t even thought of as a science.

But this only encouraged Hinton more. He relished his position as an outsider and knew that his ideas would win out in the end.

Hinton realized that the biggest hindrance to AI was computer power. But he also saw that time was on his side. Moore’s Law predicted that the number of components on a chip would double about every 18 months.

In the meantime, Hinton worked tirelessly on developing the core theories of neural networks—something that eventually became known as deep learning. In 1986, he wrote—along with David Rumelhart and Ronald J. Williams—a pathbreaking paper, called “Learning Representations by Back-propagating Errors.” It set forth key processes for using backpropagation in neural networks. The result was that there would be significant improvement in accuracy, such as with predictions and visual recognition.

Of course, this did not happen in isolation. Hinton’s pioneering work was based on the achievements of other researchers who also were believers in neural networks. And his own research spurred a flurry of other major achievements:

• 1980: Kunihiko Fukushima created Neocognitron, which was a system to recognize patterns that became the basis of convolutional neural networks. These were based on the visual cortex of animals.
• 1982: John Hopfield developed “Hopfield Networks.” This was essentially a recurrent neural network.
• 1989: Yann LeCun merged convolutional networks with backpropagation. This approach would find use cases with analyzing handwritten checks.
• 1989: Christopher Watkins’ PhD thesis, “Learning from Delayed Rewards,” described Q-Learning. This was a major advance in helping with reinforcement learning.
• 1998: Yann LeCun published “Gradient-Based Learning Applied to Document Recognition,” which used descent algorithms to improve neural networks.