Annotated History of Modern AI and Deep Learning

This is a wonderful resource on the History of Modern AI and Deep Learning by Jürgen Schmidhuber.

It opens statement with Machine learning (ML) is the science of credit assignment. It takes the technical term of "Credit Assignment" and overlays it with the human approach of assigning credits to ideas shared by predecessors.

It reminds us that we are all standing on the shoulders of the giants.

I enjoyed reading (listening to Audiobook) Genius Makers by Cade Metz. It was information dense that I wanted to keep a note of all the players in the story to form the idea. Fortunately, the book provided the timeline and the people. Here it is, annotated with links against the people to follow their work. It is easier now to form an idea about the timeline with the story and players in action.

Plus, the paper referenced above is far more approachable with this take.

Timeline and People from the Book Genius Makers by Cade Metz

1960s — The Perceptron Era

  • 1960 — Frank Rosenblatt builds the Mark I Perceptron at Cornell Aeronautical Laboratory, the first hardware neural network.
  • 1969 — Marvin Minsky and Seymour Papert publish Perceptrons, demonstrating limits of single-layer networks (the XOR problem). The book contributes to the collapse of neural network research.

1970s — First AI Winter

  • 1971 — Geoff Hinton begins his PhD at the University of Edinburgh.
  • 1973 — The Lighthill Report triggers the first AI winter; funding for neural network research dries up.
  • 1978 — Hinton takes a postdoc at UC San Diego, working with David Rumelhart's group.

1980s — Backpropagation and Revival

  • 1982 — Carnegie Mellon University hires Hinton.
  • 1984 — Hinton meets Yann LeCun at a workshop in France.
  • 1986 — Rumelhart, Hinton, and Williams publish the backpropagation paper in Nature, reviving neural networks. LeCun joins Bell Labs and develops LeNet for handwritten digit recognition.
  • 1987 — Hinton moves to the University of Toronto.
  • 1989 — Dean Pomerleau builds ALVINN at CMU, an autonomous vehicle steered by a neural network.

1990s — Quiet Years

  • 1992 — Bengio and LeCun meet at Bell Labs.
  • 1993 — University of Montreal hires Yoshua Bengio.
  • 1998 — Hinton founds the Gatsby Computational Neuroscience Unit at University College London.
  • 1990s–2000s — Second AI winter; neural networks are out of fashion.

2000s — Rebranding and Resurgence

  • 2000 — Hinton returns to the University of Toronto.
  • 2003 — LeCun joins NYU.
  • 2004 — CIFAR begins funding the Neural Computation and Adaptive Perception (NCAP) workshops, organized by Hinton.
  • 2007 — Hinton coins the term "deep learning".
  • 2008 — Hinton meets Li Deng at the NIPS Whistler workshop.
  • 2009 — Hinton visits Microsoft Research; speech recognition collaboration begins.

2010s — The Boom

  • 2010 — Mohamed and Dahl visit Microsoft to apply deep learning to speech. DeepMind is founded by Hassabis, Suleyman, and Legg in London. Andrew Ng pitches "Project Marvin" to Larry Page at Google.
  • 2011 — Navdeep Jaitly interns at Google. Google Brain is founded. Google deploys deep-learning speech recognition on Android.
  • 2012 — The Cat Paper (Le, Ng, Dean) shows unsupervised learning at scale. Ng leaves Google. Hinton interns at Google Brain. AlexNet (Krizhevsky, Sutskever, Hinton) wins ImageNet by a huge margin. DNNresearch is auctioned to Google.
  • 2013 — Hinton, Sutskever, and Krizhevsky join Google. FAIR (Facebook AI Research) is founded under LeCun.
  • 2014 — Google acquires DeepMind. Goodfellow invents the GAN. Sutskever, Vinyals, and Le publish Seq2Seq.
  • 2015 — Hinton spends summer at DeepMind. AlphaGo beats Fan Hui. OpenAI is founded by Musk, Altman, Brockman, Sutskever, and others.
  • 2016 — DeepMind Health launches. AlphaGo beats Lee Sedol in Seoul. Qi Lu leaves Microsoft. Google Translate switches to deep learning. Trump elected.
  • 2017 — Qi Lu joins Baidu. AlphaGo beats Ke Jie. China announces national AI initiative. Hinton publishes capsule networks. Progressive GANs and deepfakes emerge.
  • 2018 — Musk leaves OpenAI's board. Google employees protest Project Maven. BERT released by Google.
  • 2019 — Activists protest Amazon's facial recognition. Hinton, LeCun, and Bengio receive the Turing Award. Microsoft invests $1 billion in OpenAI.

2020 — Beyond

  • 2020 — Covariant (Pieter Abbeel's startup) deploys a picking robot in a Berlin warehouse, signaling deep learning's arrival in industrial robotics.

The Players

At Google

At DeepMind

At Facebook (Meta)

At Microsoft

At OpenAI

At Baidu

  • Robin Li — founder, CEO
  • Lu Qi — former COO
  • Kai Yu — former head of Institute of Deep Learning, founder of Horizon Robotics

At Nvidia

At Clarifai

In Academia

At the Singularity Summit

In the Past (Founding Era)