Course Description
This course offers a comprehensive exploration of systems for machine learning (ML), focusing on the latest research and advancements. It introduces students to essential models and techniques, including Convolutional Neural Networks (CNNs), Large Language Models (LLMs), gradient descent, and Stochastic Gradient Descent (SGD). We'll dive into optimization techniques like quantization and pruning to enhance model efficiency without sacrificing performance.
Beyond theoretical concepts, the course provides an introduction to Deep Neural Network (DNN) accelerator design and the architecture of GPUs and TPUs. These sections aim to give students a foundational understanding of how ML algorithms leverage parallel computing architectures, such as chip multiprocessors, multithreading, and GPUs with CUDA support, for improved performance and efficiency.
Throughout the course, we emphasize optimizing ML performance while considering power efficiency and technical constraints. Our goal is to equip students with the knowledge and skills necessary to navigate and contribute to the field of machine learning systems effectively.
Learning Objectives
- Gain a solid foundation in machine learning systems, including an overview of various models and techniques such as CNNs (Convolutional Neural Networks), LLMs (Large Language Models), gradient descent, and Stochastic Gradient Descent (SGD).
- Introduction to DNN (Deep Neural Network) accelerator design and the architecture of GPUs/TPUs, focusing on how these hardware systems accelerate ML tasks.
- Develop skills to map ML algorithms on GPUs and TPUs to optimize performance, using CUDA for effective programming on NVIDIA GPUs.
- Understand the importance of power efficiency and constraints in ML systems, aiming to optimize computational resources and energy usage.
- Learn about model optimization techniques such as quantization and pruning, which enhance performance and efficiency by adjusting the precision of model parameters and reducing neural network complexity, respectively.
- Explore the privacy and security challenges inherent in ML systems, including strategies to protect data and ensure the integrity of models against potential attacks.
- Introduction to federated learning, emphasizing its role in training ML models across decentralized devices to enhance data privacy and security.
Course Schedule
| Week | Topics/Daily Activities | Readings/Preparation | Deliverables |
|---|---|---|---|
| 1 | Introduction and applications | Programming/Homework assignment (PHA)#1 Posted | |
| 2 | An overview of ML algorithms and their computational demands | Programming/Homework assignment (PHA)#2 Posted | |
| 3-5 | CNNs, ML runtime systems (PyTorch, TensorFlow), Deep dive in DNN accelerators | Reading Assignment 1 posted | Programming/Homework assignment (PHA)#3 Posted |
| 6 | Kernel Computation. Toeplitz matrices | Reading Assignment 2 posted | |
| 7-8 | Dataflows (weight, input, and output stationary systems), loop nests | Project topics posted, PHA#4 posted, Project initial updates | |
| 9 | Midterm Exam, Basics of GPU and TPU microarchitecture. | Mid project updates | |
| 10-11 | More on GPU and TPU microarchitecture. | PHA#5 posted | |
| 12 | Distributed Learning | PHA#6 posted | |
| 13 | Privacy in Deep Learning | ||
| 14 | Introduction to NLP and Language Models | Final project updates | |
| 15 | RNNs, LSTM, and Transformers, LLMs | ||
| 16 | Generative models, Backup for overflow, Final Exam | Refer to the final exam schedule in the USC Schedule of Classes at classes.usc.edu. |
Technological Proficiency and Hardware/Software Required
This course requires basic proficiency in Python, a primary language used in machine learning. You'll need a computer capable of running ML algorithms and simulations. We'll use ML libraries and frameworks such as TensorFlow and PyTorch. Familiarity with tools like Anaconda or Jupyter Notebook will aid your coding and experimentation.
Students must have a GitHub account for version control and code submissions. For those new to GitHub, here's a GitHub Tutorial to get you started.
For hands-on coding without a high-performance computer, Google Colab is recommended. It's a cloud-based platform that supports Python code execution, with access to powerful GPUs and TPUs.
Explore these resources to get started:
- Python: Python Getting Started Guide
- PyTorch: PyTorch Tutorials
- TensorFlow: TensorFlow Tutorials
- Google Colab: Getting Started with Google Colab
Required Readings and Supplementary Materials
- Efficient Processing of DNNs by Sze, Chen, Yang, and Emer.
- Eyeriss Mit Tutorial.
- Optional Reading: Parallel Computer Organization and Design by Michel Dubois, Murali Annavaram, and Per Stenström. This textbook provides background material on chip multiprocessors and discussions relevant to the course.
Grading Breakdown
| Assessment Tool | Points | % of Grade |
|---|---|---|
| Midterm and Quizzes | 20 | 20% |
| Programming, reading, and homework assignments | 40 | 40% |
| Project | 20 | 20% |
| Final exam | 20 | 20% |
Grading Scale
| Letter Grade | Numerical Point Range |
|---|---|
| A | 95-100 |
| A- | 90-94 |
| B+ | 87-89 |
| B | 83-86 |
| B- | 80-82 |
| C+ | 77-79 |
| C | 73-76 |
| C- | 70-72 |
| D+ | 67-69 |
| D | 63-66 |
| D- | 60-62 |
| F | 59 and below |
Course Policies
Academic Integrity
Academic integrity is foundational to our learning community. This course follows USC's standards on academic integrity. Students are expected to submit original work and properly attribute sources. Violations, including plagiarism and cheating, will result in disciplinary action, which may include failure in the course and further sanctions by the university.
Attendance
Attendance is mandatory and integral to the success of all students. Absences can impact your final grade. If you must miss a class, please notify the instructor ahead of time. Accommodations for student athletes and religious observances will be made.
Zoom Etiquette
For classes held via Zoom, maintain professionalism in all communications. Keep your camera on if required, participate actively, and use features like raise hand and chat responsibly. Ensure your microphone is muted when not speaking to minimize background noise.
Assignment Submission Policy
All assignments must be submitted electronically. Ensure your submissions are complete and include any required documentation or README files. Assignments will undergo both automated and manual review to ensure integrity.
Support and Accommodations for Students with Disabilities
USC is committed to inclusivity and provides support for students with disabilities. If you require accommodations, please register with the Office of Student Accessibility Services (OSAS) and inform the instructor to make necessary arrangements.
Support Systems
The university offers various support systems, including counseling, mental health services, and resources for addressing bias or harassment. For more information or assistance, please refer to the following resources: