Week 1

• Introduction and Course Overview
  slides: [lecture 1]
  other: [Quiz 0]
• Concentration of the Empirical Risk via Parameter Count Bounds
  slides: [lecture 2]
  Reading material: Chapter 4 of [2]

Week 2

• VC-dimension and Compression-based Generalization Bounds
  slides: [lecture 3]
  reading list
  
• Limitations of Rademacher Complexity; moving forward with Stability
  slides: [lecture 4]
  reading list

Week 3

• Generalization and Stability of Global Minima
  slides: [lecture 5]
  reading list
  
• Computational Aspects of the ERM and Gradient Descent
  slides: [lecture 6]
  reading list

Week 4

• How fast is Gradient Descent?
  slides: [lecture 7]
  reading list
  
• A Primer on SGD!
  slides: [lecture 8]
  reading list

Week 5

• The PL Land of Non-Convexity
  slides: [lecture 9]
  reading list
  
• When are Local Minima Good, and Neural Nets Easy to Fit?
  slides: [lecture 10]
  reading list

Week 6

• Understanding Deep Learning: Is it all about SGD?
  slides: [lecture 11]
  reading list
  
• Advances and Challenges of Large-scale Learning
  slides: [lecture 12]
  

Week 7

• Scaling-up SGD with Mini-batches
  slides: [lecture 13]
  reading list
  
• Lifting Syncronization Barriers in Distributed Learning
  slides: [lecture 14]
  reading list

Week 8: Spring Break

Week 9

• Letting the Hogs go Wild, while converging fast
  slides: [lecture 15]
  reading list
  
• Taming the Wild Hogs: Reproducible Parallel Machine Learning
  slides: [lecture 16]
  reading list

Week 10

• Project Presentations
• Model Pruning and Deep Compression
  slides: [lecture 17]
  reading list

Week 11

• From Model Pruning to Sparse Updates and the Lottery Ticket Hypothesis
  slides: [lecture 18]
  reading list

Week 12

• The Advantages and Challenges Binary Neural Networks
  slides: [lecture 19]
  reading list
  
• Communication Primitives for Distributed Training
  slides: [lecture 20]
  reading list

Week 13

• Overcoming Communication Bottlenecks with Gradient Compression
• Decentralized and Federated Learning

Week 14

• Worst-case Model Robustness
• Robustness and Fault Tolerance during Training

Week 15

• Open Problems in Model Large-scale Learning
• Project Poster Presentations

1. Convex Optimization: Algorithms and Complexity, Sebastien Bubeck.
2. Understanding Machine Learning: From Theory to Algorithms, by Shai Shalev-Shwartz and Shai Ben-David.
3. Optimization Methods for Large-Scale Machine Learning, by Léon Bottou, Frank E. Curtis, and Jorge Nocedal.
4. Introductory Lectures on Stochastic Optimization, by John Duchi.
5. Accelerating Stochastic Gradient Descent using Predictive Variance Reduction, by Rie Johnson and Tong Zhang.
6. Efficiency of coordinate descent methods on huge-scale optimization problems, by Yurii Nesterov.
7. Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition, by Hamed Karimi, Julie Nutini, and Mark Schmidt.
8. Subgradients (course notes) , by Stephen Boyd, John Duchi, and Lieven Vandenberghe.
9. Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization, by Martin Jaggi.
10. Projection-free Online Learning, by Elad Hazan and Satyen Kale.
11. Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming, by by Saeed Ghadimi and Guanghui Lan.
12. No bad local minima: Data independent training error guarantees for multilayer neural networks, by Daniel Soudry and Yair Carmon.
13. Stability and Generalization, by Olivier Bousquet and André Elisseef.
14. Train faster, generalize better: Stability of stochastic gradient descent, by Moritz Hardt, Benjamin Recht, and Yoram Singer.
15. HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent, by Feng Niu, Benjamin Recht, Christopher Ré and Stephen J. Wright.
16. Perturbed Iterate Analysis for Asynchronous Stochastic Optimization, by Horia Mania, Xinghao Pan, Dimitris Papailiopoulos, Benjamin Recht, Kannan Ramchandran, Michael I. Jordan.
17. CYCLADES: Conflict-free Asynchronous Machine Learning, by Xinghao Pan, Maximilian Lam, Stephen Tu, Dimitris Papailiopoulos, Ce Zhang, Michael I. Jordan, Kannan Ramchandran, Christopher Ré, Benjamin Recht.
18. The phase transition in site percolation on pseudo-random graphs, by Michael Krivelevich.
19. Better mini-batch algorithms via accelerated gradient methods, by Cotter A, Shamir O, Srebro N, Sridharan K., 2011.
20. Gradient Diversity: a Key Ingredient for Scalable Distributed Learning, by Yin, D., Pananjady, A., Lam, M., Papailiopoulos, D., Ramchandran, K. and Bartlett, P., AISTATS 2018.
21. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding, D. Alistarh, D. Grubic, J. Li, R. Tomioka, M. Vojnovic, NIPS 2017.
22. TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning, by W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, H. Li, NIPS 2017.
23. Speeding up distributed machine learning using codes, by K. Lee, M. Lam, R. Pedarsani, D. Papailiopoulos, K. Ramchandran, IEEE Trans. Info. Theory, 2017.
24. Gradient coding: Avoiding stragglers in distributed learning, by R. Tandon, Q. Lei, A. G. Dimakis, N. Karampatziakis, ICML 2017.
25. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, by S. Han, H. Mao, W. J. Dally, ICLR 2016.
26. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks, by M. Rastegari, V. Ordonez, J. Redmon, A. Farhadi, ECCV 2016.
27. An Analysis of Deep Neural Network Models for Practical Applications, by M. Alfredo Canziani, Adam Paszke, Eugenio Culurciello, arXiv 2016.