TTIC 31230: Fundamentals of Deep Learning

David McAllester

Winter 2020

TA: Adam Dziedzic

Grader: Gray Mackall

Announcement: The office hours have been moved to Monday, Friday 2:00-4:00 in the library rm 435.

In 2020, as in 2019, there will be no machine problems or class projects. The course will again be treated as an "algorithms class" rather than a "programming class".

The grade will be based entirely on four in-class quizes and a final exam with the final worth 30% and each quiz worth 17.5%. The quizes from laset year, with solutions, are given at the bottom of this page. Also the course material includes additional paractice problems with solutions. The quizes and the final will include problems from the practice set --- you just need to memorize the solution --- as well as original problems.

Lectures Slides and Course Material:

  1. Information Theory: The Fundamental Equations of Deep Learning
  2. Back-Propagation and Frameworks
  3. Convolutional Neural Networks (CNNs)
  4. Language Modeling, Recurrent Neural Networks (RNNs), Machine Translation and Attention
  5. Variants of Stochastic Gradient Descent (SGD)
  6. Generalization and Regularization
  7. Connectionist Temporal Classification (CTC)
  8. Deep Graphical Models
  9. More Information Theory: Avoiding Differential Entropy
  10. Rate-Distortion Autoencoders (RDAs)
  11. Expectation Maximization (EM), The Evidence Lower Bound (The ELBO) and Variational Autoencoders (VAEs)
  12. Generative Adversarial Networks (GANs)
  13. Pretraining
  14. Reinforcement Learning (RL)
  15. AlphaZero
  16. Gradients as Dual Vectors, Hessian-Vector Products, and Information Geometry
  17. The Black Box Problem
  18. The Quest for Artificial General Intelligence (AGI)

2019 Exam problems have been moved to the problems for the appropriate lecture.

  1. Quiz 1, 2020

Exam Schedule:

Each quiz will occupy the last 2/3 of one class. The quizes are scheduled two weeks apart as follows:

The final exam is Thursday March 19, 1:30 - 3:30

Office hours will be Monday and Friday 2:00 to 4:00 in the library (TTIC 435).