CS 3750

Fall 2020: CS 3750 - Advanced Topics in Machine Learning
Integrating Scientific Theory with Machine Learning

Time
Tuesday, Thursday
9:25AM - 10:40AM
Zoom Link: https://pitt.zoom.us/j/91677876958
Office hour: Thursday 8:20AM - 9:20AM, or by appointment

Instructor
Xiaowei Jia
xiaowei (at) pitt.edu
www.pitt.edu/~xiaowei
5413 Sennott Square

Summary: Machine learning (ML) methods have found immense success in extracting complex knowledge from "Internet-scale" data. Given their accomplishments in commercial applications, there is a huge interest to see if machine learning (ML) methods can accelerate knowledge discovery in scientific domains that have traditionally progressed via (scientific) physics-based models. This course will systematically investigate limitations of traditional ML methods when applied to real-world scientific problems, which involve incomplete/imperfect data with non-stationary and chaotic behavior. To overcome their limitations, this course will also present multiple ways to integrate physical knowledge with ML models. The methods will be illustrated using examples from diverse scientific domains such as climate science, hydrology, fluid dynamics, aerospace, etc. Our course will provide hands-on experience through course projects and research presentations.
Recommended: Basic knowledge in Machine Learning; Basic proficiency in Python and Tensorflow/Pytorch.

Grading policy: 50% paper presentation (25% each) + 50% course project

Schedule

1. 08/20-08/27: Course overview and preliminaries
• 08/20, 08/25: ML review: classification, overfitting, neural networks (slides, video 1, video 2)
• 08/27: ML applications in scientific domains (slides, video)
2. 09/01-09/08: Knowledge-guided model architecture
• 09/01: Reading 1: PhyNet: Physics Guided Neural Networks for Particle Drag Force Prediction in Assembly
  Reading 2: SchNet: A continuous-filter convolutional neural network for modeling quantum interactions
• 09/03: Reading 1: Multirresolution Convolutional Autoencoders
  Reading 2: A Domain Guided CNN Architecture for Predicting Age from Structural Brain Images
• 09/08: Reading 1: Reynolds averaged turbulence modelling using deep neural networks with embedded invariance
  Reading 2:Towards physics-informed deep learning for turbulent flow prediction
3. 09/10-09/15: Design of ML loss functions
• 09/10: Reading 1: Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling
  
• 09/15: Reading 1: Label-Free Supervision of Neural Networks with Physics and Domain Knowledge
  Reading 2: Multi-Fidelity Physics-Constrained Neural Network and Its Application in Materials
4. 09/17: Downscaling:
Reading 1: Deep learning methods for super-resolution reconstruction of turbulent flows
Reading 2: DeepSD: Generating High Resolution Climate Change Projections through Single Image Super-Resolution
5. 09/22-09/24: Model initialization and transfer learning
   • 09/22: Reading 1: Physics Guided RNNs for Modeling Dynamical Systems: A Case Study in Simulating Lake Temperature Profiles
     Reading 2: Process Guided Deep Learning Predictions of LakeWater Temperature
   • 09/24: Reading 1: Learning Traffic Signal Control from Demonstrations
     
6. 09/29: Guest lecture: Process-Guided Meta Transfer Learning for Predicting Temperature of Unmonitored Lake Systems (Jared Willard) (video)
   Dataset: River basin, Lake modeling, Traffic prediction, Slides
7. 10/01:
Reading 1: Predicting AC Optimal Power Flows: Combining Deep Learning and Lagrangian Dual Methods
Reading 2: Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
8. 10/06-10/08: Other modeling approaches
• 10/06: Reading 1: Data-driven methods to improve baseflow prediction of a regional groundwater model
  Reading 2: DR-RNN: A deep residual recurrent neural network for model reduction
• 10/08: Reading 1: Data-assisted reduced-order modeling of extreme events in complex dynamical systems
  Reading 2: Online Network Revenue Management Using Thompson Sampling
9. 10/13-10/15: Uncertainty quantification
• 10/13: Reading 1: Physics-Guided Architecture (PGA) of Neural Networks for Quantifying Uncertainty in Lake Temperature Modeling
  Reading 2: ConvPDE-UQ: Convolutional neural networks with quantified uncertainty for heterogeneous elliptic partial differential equations on varied domains
• 10/15: Reading 1: Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
  Reading 2: A hybrid model approach for forecasting future residential electricity consumption
10. 10/20-10/22: Generative ML model
• 10/20: Reading 1: Enforcing Deterministic Constraints on Generative Adversarial Networks for Emulating Physical Systems
  Reading 2: TempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow
• 10/22: Reading 1: Encoding Invariances in Deep Generative Models
  Reading 2: Improving Direct Physical Properties Prediction of Heterogeneous Materials from Imaging Data via Convolutional Neural Network and a Morphology-Aware Generative Model
11. 10/27-10/29: Inverse modeling
• 10/27: Reading 1: Convolutional Neural Networks for Inverse Problems in Imaging
  Reading 2: Low-Dose CT with a Residual Encoder-Decoder Convolutional Neural Network (RED-CNN)
• 10/29: Reading 1: Adversarial Regularizers in Inverse Problems
  Reading 2: Solving electrical impedance tomography with deep learning
  Code example: https://colab.research.google.com/drive/1AlAE4ZAJFdGIVmnD8qS2bcW__Kz7szh5?usp=sharing
12. 11/03: Discovery and solving underlying PDEs
    Reading 1: Solving high-dimensional partial differential equations using deep learning
    Reading 2: Data-driven discovery of partial differential equations
13. 11/05:Guest lecture: ML in healthcare.
14. 11/10-11/19: Project presentations

Other resources

1. Datasets: TBA.
2. Reference:
• Introduction to Data Mining (textbook). Tan et al.
• Pattern Recognition and Machine Learning (textbook). Bishop
• Theory-Guided Data Science (survey)
• Integrating Physics-Based Modeling With Machine Learning (survey)

Academic integrity
All assignment submissions must be the sole work of each individual student. Students may not read or copy another student's solutions or share their own solutions with other students. Students may not review solutions from students who have taken the course in previous years. Submissions that are substantively similar will be considered cheating by all students involved, and as such, students must be mindful not to post their code publicly. The use of books and online resources is allowed, but must be credited in submissions, and material may not be copied verbatim. Any use of electronics or other resources during an examination will be considered cheating.

If you have any doubts about whether a particular action may be construed as cheating, ask the instructor for clarification before you do it. The instructor will make the final determination of what is considered cheating.

Cheating in this course will result in a grade of F for the course and may be subject to further disciplinary action.

Using an open-source codebase is accepted, but you must explicitly cite the source, especially following the owner's guideline if it exists. For any writing involved in the project, plagiarism is strictly prohibited. If you are unclear whether your work will be considered as plagiarism, ask the instructor before submitting or presenting the work.

University statement
If you have a disability for which you are or may be requesting an accommodation, you are encouraged to contact both your instructor and the Office of Disability Resources and Services, 140 William Pitt Union, at 412-648-7890 or 412-383-7355 (TTY) as early as possible, but no later than the fourth week of the term or visit the Office of Disability Resources website as early as possible, but no later than the 4th week of the term. DRS will verify your disability and determine reasonable accommodations for this course.