ECE 3195 Advanced Machine Learning and Deep Learning (Spring 2021)

ECE 3195 Advanced Machine Learning and Deep Learning
Spring 2021

Heng Huang

[ Administrative Basics | Course Description | Syllabus ]

Administrative Basics

Lecture	630 William Pitt Union \| Thursday 4:30PM - 7PM \| The lectures will be taught via Zoom and the students can choose to use Zoom or classroom
Instructor	Heng Huang \| Zoom Room \| Office hours: Thursday 1:00-3:00 PM
Request	ECE2195 Machine Learning; Optimization; Statistics.
Good notes for reading	Deep Learning in Neural Networks: An Overview Stochastic Gradient Descent Tricks Practical Optimization An Introduction to Deep Reinforcement Learning Survey ADMM Paper Submodular Optimization Learning with Submodular Functions: A Convex Optimization Perspective Online Convex Optimization
Work	Paper readings and presentations. (70%) Final reports. (30%)

Course Description

In this course, we will study the cutting-edge advanced research topics in machine learning and deep learning by reading and discussing a set of research papers. The main objective of this course is to cover the underlying mathematical concepts and representative algorithms, paper reading, and implementation. The course project will implement the deep learning model with solving image classification tasks. The project will be assigned in Canvas system.

Syllabus

Week 1 (Jan 21):

Instructor was sick.

Week 2 (Jan 28):

Lecture 1 Stochastic Gradient Descent

Week 3 (Feb 4):

Instructor was sick.

Week 4 (Feb 11):

Lecture 2: Stochastic Gradient Descent (cont.)

Week 5 (Feb 18):

Lecture 3: Contrastive Learning

Week 6 (Feb 25):

Lecture 4: Graph Neural Networks

Week 7 (Mar 4):

Lecture 5: Distributed Machine Learning and Federated Learning

Week 8 (Mar 11):

Lecture 6: Model Compression

Week 9 (Mar 18):
- Image Style Transfer Using Convolutional Neural Networks. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge. CVPR 2016. (Kelsey Fagan, Malia Anne)
- Network Dissection: Quantifying Interpretability of Deep Visual Representations. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba. CVPR 2017. (Kyle R. Buettner)
- Image-to-Image Translation with Conditional Adversarial Nets. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros. CVPR 2017. (Brian Randall)
- Generative Adversarial Text-to-Image Synthesis. Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee. ICML 2016. (Jieru Zhao)
Week 10 (Mar 25):
- Emergent Tool Use From Multi-Agent Autocurricula. Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, Igor Mordatch. ICLR 2020. (Vivswan Shah)
- Detect to Track and Track to Detect. Christoph Feichtenhofer, Axel Pinz, Andrew Zisserman. ICCV 2017. (Yasmine Ahmed)
- Low-shot Learning from Imaginary Data. Yu-Xiong Wang, Ross Girshick, Martial Herbert, Bharath Hariharan. CVPR, 2018. (Collin Abidi)
- Generalized Cross Entropy Loss for Training DeepNeural Networks with Noisy Labels. Zhilu Zhang, Mert R. Sabuncu. NeurIPS 2018.
Week 11 (Apr 1):
- Accelerating stochastic gradient descent using predictive variance reduction. Rie Johnson and Tong Zhang. NeurIPS 2013. (Wenhan Xian)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Sergey Ioffe, Christian Szegedy. ICML 2015. (Yihan Wu)
- On the importance of initialization and momentum in deep learning. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. ICML 2013. (Junyi Li)
- Practical bayesian optimization of machine learning algorithms. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. NeurIPS 2012. (Guodong Liu)
Week 12 (Apr 8):
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. NeurIPS 2015. (Yawen Wu)
- The Marginal Value of Adaptive Gradient Methodsin Machine Learning. Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, Benjamin Recht. NeurIPS 2017. (Shuda Zhong)
- Long-term Recurrent Convolutional Networks forVisual Recognition and Description. Jeff Donahue, et al. PAMI 2017. (Yilin Wang)
- Matching Networks for One Shot Learning. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, Daan Wierstra. NIPS 2016. (Hang Gu)
- Hogwild: A lock-free approach to parallelizing stochastic gradient descent. Feng Niu, Benjamin Recht, Christopher Re, and Stephen Wright. NeurIPS 2011.
- Conditional Random Fields as Recurrent Neural Networks. Shuai Zheng, et al. ICCV 2015.
Week 13 (Apr 15):
- Asynchronous Methods for Deep Reinforcement Learning. Volodymyr Mnih, et al. ICML 2016. (Sebastien Ollivier)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Chelsea Finn, Pieter Abbeel, Sergey Levine. ICML, 2017. (Zhepeng Wang)
- Deep learning with limited numerical precision. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. ICML 2015. (Yue Tang)
- Random search for hyper-parameter optimization. James Bergstra and Yoshua Bengio. JMLR, 2012.
- A method for stochastic optimization. Diederik Kingma and Jimmy Ba. Adam. ICLR 2015.
Week 14 (Apr 22):
Week 15 (Apr 29):

ACADEMIC INTEGRITY
Students in this course will be expected to comply with the University of Pittsburgh's Policy on Academic Integrity and the Swanson School of Engineering Policy. Any student suspected of violating this obligation for any reason during the semester will be required to participate in the procedural process, initiated at the instructor level, as outlined in the University Guidelines on Academic Integrity (http://www.cfo.pitt.edu/policies/policy/02/02-03-02.html) and the Swanson School procedures. This may include, but is not limited to, the confiscation of the examination of any individual suspected of violating University Policy.

All students are expected to adhere to the standards of professional conduct and academic honesty. Any student engaged in cheating, plagiarism, or other acts of academic dishonesty would be subject to disciplinary action. Any student suspected of violating this obligation for any reason during the semester will be required to participate in the procedural process, initiated at the instructor level, as outlined in the SSOE Academic Integrity Policy found at: https://www.engineering.pitt.edu/Academic-Integrity-Guidelines/.

To learn more about Academic Integrity, visit the Academic Integrity Guide (http://pitt.libguides.com/academicintegrity/) for an overview of the topic. For hands- on practice, complete the Understanding and Avoiding Plagiarism (http://pitt.libguides.com/academicintegrity/plagiarism) tutorial.

DISABILITY SERVICES
If you have a disability for which you are or may be requesting an accommodation, you are encouraged to contact both your instructor and Disability Resources and Services (DRS, http://www.studentaffairs.pitt.edu/drswelcome), 140 William Pitt Union, (412) 648-7890, drsrecep@pitt.edu, (412) 228-5347 for P3 ASL users, as early as possible in the term. DRS will verify your disability and determine reasonable accommodations for this course.

STATEMENT ON CLASSROOM RECORDING
To address the issue of students recording a lecture or class session, the University's Senate Educational Policy Committee issued the recommended statement on May 4, 2010. While it is optional, the Committee recommends that faculty consider adding the statement to all course syllabi.

"To ensure the free and open discussion of ideas, students may not record classroom lectures, discussion and/or activities without the advance written permission of the instructor, and any such recording properly approved in advance can be used solely for the student's own private use."

Student Opinion of Teaching Surveys
Students in this class will be asked to complete a Student Opinion of Teaching Survey. Surveys will be sent via Pitt email and appear on your Canvas landing page during the last three weeks of class meeting days. Your responses are anonymous. Please take time to thoughtfully respond, your feedback is important to me. Read more (http://www.cidde.pitt.edu/omet/student-information/) about Student Opinion of Teaching Surveys.

Religious Observance
The observance of religious holidays (activities observed by a religious group of which a student is a member) and cultural practices are an important reflection of diversity. As your instructor, I am committed to providing equivalent educational opportunities to students of all belief systems. At the beginning of the semester, you should review the course requirements to identify foreseeable conflicts with assignments, exams, or other required attendance. If at all possible, please contact me within the first two weeks of the semester to allow time for us to discuss and make fair and reasonable adjustments to the schedule and/or tasks.

Diversity and Inclusion
The University of Pittsburgh does not tolerate any form of discrimination, harassment, or retaliation based on disability, race, color, religion, national origin, ancestry, genetic information, marital status, familial status, sex, age, sexual orientation, veteran status or gender identity or other factors as stated in the University's Title IX policy. The University is committed to taking prompt action to end a hostile environment that interferes with the University's mission. For more information about policies, procedures, and practices: https://www.diversity.pitt.edu/civil-rights-title-ix/policies-procedures-and-practices. I ask that everyone in the class strive to help ensure that other members of this class can learn in a supportive and respectful environment. If there are instances of the aforementioned issues, please contact the Title IX Coordinator, by calling 412-648-7860, or emailing titleixcoordinator@pitt.edu.

Reports can also be filed online: https://www.diversity.pitt.edu/civil-rights-title-ix/make-report

You may also choose to report this to a faculty/staff member; they are required to communicate this to the University's Office of Diversity and Inclusion. If you wish to maintain complete confidentiality, you may also contact the University Counseling Center (412-648-7930).

COVID-19 Statement
In the midst of this pandemic, it is extremely important that you abide by public health regulations and University of Pittsburgh health standards and guidelines. While in class, at a minimum this means that you must wear a face covering and comply with physical distancing requirements; other requirements may be added by the University during the semester. These rules have been developed to protect the health and safety of all community members. Failure to comply with these requirements will result in you not being permitted to attend class in person and could result in a Student Conduct violation. For the most up-to-date information and guidance, please visit coronavirus.pitt.edu and check your Pitt email for updates before each class.

Communication to Instructor Pertaining to Illness
As in any situation regarding class absence (remote or in person), a student who becomes ill (albeit COVID-19 related or not) is responsible for communicating with me regarding course absences. Please contact me and provide documentation when absences affect quizzes/exams. This should be done via email as soon as possible.