The Table of Contents:
- Course Structure
- Course Evaluation
- Code and Lab Resources
- Papers of interest
This is the official web page for the contents of the lectures from the Deep Learning course, at the Master in Artificial Intelligence from UPC. Here you can find basic information as well as everything needed to follow the course.
- Dario Garcia-Gasulla (Course coordinator, BSC, firstname.lastname@example.org) (There are currently openings at HPAI-BSC. Send me an email if you may be interested)
- Marc Casas (BSC, email@example.com)
- Javier Bejar (UPC-CS, firstname.lastname@example.org)
This course provides access to supercomputing clusters to perform the labs. Before the first guided lab session, students must have familiarized with the environment by themselves. Instructions on how to do that will be provided timely. ALL STUDENTS must come to the first lab with a laptop able to connect to the cluster through ssh. Linux is highly recommended.
This course provides an applied approach to Deep Learning. The course is structured in 4 thematic blocks, and each block as 3 parts: theory, guided laboratory and autonomous laboratory.
The first and third thematic blocks (CNNs and Embeddings) are taught by Dario Garcia-Gasulla. The second (RNNs) by Javier Bejar. The fourth (HPC and DL) by Marc Casas. This implies that comments on each block should be addressed to the corresponding teacher. Each block may have different methodologies.
Course Calendar for the Fall 2019-2020 semester
- 12/09/19 FNN-CNN Theory
- 19/09/19 FNN-CNN Guid. Lab
- 26/09/19 FNN-CNN Aut. Lab
- 3/10/19 RNN Theory
- 10/10/19 RNN Guid. Lab
- 17/10/19 RNN Aut. Lab (Delivery 1st lab)
- 24/10/19 Emb. Theory
- 7/11/19 Emb. Guid. Lab
- 14/11/19 Emb. Aut. Lab (Delivery 2nd lab)
- 21/11/19 HPC-DL Theory
- 28/11/19 HPC-DL Guid. Lab
- 5/12/19 HPC-DL Aut. Lab (Delivery 3rd lab)
- 12/12/19 Presentation of theoretical works 1
- 19/12/19 Presentation of theoretical works 2
- 26/12/19 (Delivery 4th lab)
- ??/01/19 Presentation of theoretical works 3
- ??/01/19 Presentation of theoretical works 4
The theory part of a block provides a review of the basic concepts of Deep Learning, but is intended only as an introduction. Multiple references are given in the theory section, and the interested student should read further from those references to learn more details of the introduced topics. Beyond the cited works, there are lots of materials online of Deep Learning, although it is recommended to read more than one source, as many sources explain only one aspect or interpretation of a certain topic. A good reference for most topics is the “Deep Learning Book” by Ian Goodfellow and Yoshua Bengio and Aaron Courville. There is a physical copy of the book in the UPC library, and it can also be found online.
The guided laboratory provides working code that can serve as a starting point for students. These codes are commented, and show a variety of algorithmic solutions. The guided laboratory will be reviewed and discussed in class with students.
The autonomous laboratory session is intended for students to experiment with Deep Learning methods, and draw their own conclusions. It should also be used to obtain feedback from lecturers regarding the theoretical work in progress. For running their lab experiments, students will be given access to computational resources. However, these resources are shared with other people though a queue system. Its important to maxime the use of your resources, as the priority of your jobs will decrease as you consume your quota. Try to adjust the amount of resources you need for every job (both in number of nodes and in time). Plan your experiments in advance. Dont wait until 2 days before the delivery of the report to submit jobs, as these may be queued for a while.
Overlap with other courses
For some of the thematic blocks (specially the first one) there is a significant amount of overlap with other courses. However, this course is oriented towards practical aspects of deep learning. This means that theory is not given in thorough detail (as other courses may do), and that a significant amount of autonomous work is expected.
The course is evaluated 25% by theory comprehension and 75% by experimental work. Theory comprehension is measured by an analysis on a paper chosen by the student and presented at the end of the semester. The student should read and fully understand the paper, reading as many references as needed for that purpose. A presentation will be done where the student will describe the paper itself, and provide constructive criticism on it. This may include, but is not limited to, answers to questions such as:
- What is the main contribution of the article?
- How could this paper be extended by more experiments or analysis?
- Are there flaws in the paper methodology?
- What future work can derive from this paper?
The experimental evaluation will be based on reports for each thematic block, illustrating the conclusions derived from the laboratory sessions. Each thematic block will define the specifics of the experimental work. Alternative experimental reports suggested by the student are also acceptable, previous validation from the lecturers (e.g., replicating a particular paper results, or evaluating a different approach than the one suggested by the lecturers).
FNN-CNN / Embeddings lab
The labs for the first and third blocks are to be performed in couples. Students are free to arrange couples as they see fit. Partners can be the same for both blocks or not. The evaluation of the lab will be done in a 40 minutes personal interview between the lecturer (Dario Garcia-Gasulla in this case) and the two students. Both students will be evaluated separately. This interview will be in the form of an oral report: You are expected to explain the problem you faced and the solutions you proposed, while the lecturer asks questions about the particularities of the problem, and the reasoning behing your decisions. While most of the interview will be oral, you should bring a number of supporting material (digital format is fine). These personal interviews will be scheduled during the course.
Feedforward Nets and Conv Nets (lecturer: Dario Garcia)
Recurrent Neural Networks (lecturer: Javier Bejar)
Embedding spaces (lecturer: Dario Garcia)
High Performance Computing Aspects of Deep Learning (lecturer: Marc Casas)
Code and Lab Resources
The codes used in the lab sessions can be downloaded from the following locations:
- Lesson 1: Feedforward Neural Nets and Conv Nets
- Lesson 2: Recurrent Neural Networks
- Lesson 3: Embedding spaces
Papers of Interest
For the evaluation of the theoretical aspects of the course, we offer a list of papers of interest which the student may chose to read and review. These are loosely categorized. We strongly suggest that the student choose papers not found on that list, according to their own interest. The newer the paper, the best.
Stuff to read
- A post describing the ideas behind ResNet, Inception and Xception is simple terms
- A post reviewing the most relevant architectures proposed, and their contributions
- Troubling Trends in Machine Learning Scholarship A position paper regarding frequent problems with current machine learning papers, and tips on how to avoid them.
- Differences between L1 and L2 as Loss Function and Regularization
- Basic concepts and cheatsheets on ML
- List of interesting papers with available code. Interesting for replicating their results and playing around on top.
- A look at how many architectures and multimodalities are in fact the same basic operations: Feature-wise Linear Modulation or FiLM
- Some details on the memory-related issues of Deep Learning
- GAN lab: To play around with GANs playground style
- ILSVRC labeling game
- Visualizing Neural Network Layer Activation - Tensorflow tutorial
- MIT intro to deep learning