5449

<HEAD>
<TITLE>5449</TITLE>
</HEAD>

<P>

<center>

<h2> 5449: Introduction to High-Performance Deep Learning<br>
<p>

<h3> Instructors: Prof. Dhabaleswar K. (DK) Panda and Dr. Hari Subramoni<br>
      Autumn 2021 <br>
</h3>
</h2>
</center>

<h4> Course Number: 5449</h4> 

<p>
<h4> Class Number: 36352 (Grad) and 36351 (Undergrad) </h4> 

<p>

<h4> Credits: 3 </h4> 

<p>
<h4> Course Time: WF 11:10 am - 12:30 pm </h4> 

<p>

<h4> Classroom: Pomerene Hall 250 </h4> 

<p>

<h4> Course Description: </h4> 


Recent advancements in Artificial Intelligence (AI) have been fueled
by the resurgence of Deep Neural Networks (DNNs) and various Deep
Learning (DL) frameworks like Tensorflow, PyTorch, MXNet, Caffe, and
Chainer.  DNNs have found widespread applications in classical areas
like Image Recognition, Speech Processing, Textual Analysis, as well
as areas like Cancer Detection, Medical Imaging, Physics, 
Materials Science, and even Autonomous
Vehicle systems.  However, scaling distributed training with scale-up
and scale-out approaches are still challenging. This is leading to the
emergence of a new field called "High-Performance Deep Learning".

<p>

The objectives of this course are to understand the principles and the
practice of this emerging trend, the open set of challenges, how
modern HPC technologies can be used to accelerate DL trainings, etc.

<p>

<h4> Topics to be Covered </h4> 

<ul>
<li> High-Performance Deep Learning: Issues, Trends, and Challenges
 
<li> Introduction to Deep Learning and Terminologies
     <ul>
     <li> The Past, Present, and Future of Deep Learning
     <li> What are Deep Neural Networks?
     <li> Diverse Applications of Deep Learning
     <li> Overview of commonly used Terminologies
     </ul>
<li> Introdction to HPC Technologies 
     <ul>
     <li>GPUs, CPUs, and TPUs
     <li>High-Performance Networking (InfiniBand, HSE and RoCE)
     <li>MPI, CUDA-Aware MPI, NCCL
     <li>DGX-1, DGX-2, IBM Power-AI
     </ul>
<li> Overview of DL Frameworks  
     <ul>
     <li> TensorFlow 
     <li> Facebook Torch/PyTorch
     <li> Caffe and Caffe2
     <li> Chainer/ChainerMN
     <li> MXNet 
     <li> LBANN 
     <li> Horovod
     <li> Deepspeed
     </ul>
<li> Overview of State-of-the-art DL Models
     <ul>
     <li> ImageNet and VGG
     <li> GoogleNet
     <li> ResNet
     <li> NASNet
     <li> DeepSpeech
     </ul> 
<li> Challenges for Exploiting HPC for DL
     <ul>
     <li> Overall Challenges 
     <li> Need for Co-Design 
     </ul>
<li> The need for Co-Designing DL frameworks and HPC Middleware
<li> Solutions and Case Studies 
     <ul>
     <li> Generalized Solutions 
     <li> NVIDIA NCCL, Baidu-allreduce, Facebook Gloo
     <li> CPU and GPU-Based Training 
     <li> Co-Designs
     <li> Deep Learning and Big Data 
     </ul>
<li> Latest and  Emerging Trends
     <ul>
     <li> Larger-scale Models 
     <li> Out-of-core DNNs
     <li> Standardization on Benchmarking, OpenAI, ONNX
     <li> SqueezeNet
     <li> Neural Network Processors (GraphCore, Loihi, Pohohiki, Habana, Cerebras, etc.) 
     </ul>
</ul>


<p>

<h4> Text: </h4>  
Selected papers from the literature including papers focusing on 
past and on-going 
research activities in the group. 
<!--
Please refer to the attached
<a href="reading.html">reading list </a> and 
<a href="presentation_schedule">presentation schedule.        
-->
</a>

<h4> Laboratory Exercises: </h4>  

The course will involve laboratory expercises for students to
experiment with Deep Learning Frameworks. These exercises will be
carried out on OSC (Ohio Supercomputing Center) clusters using
GPUs. This will provide hands-on knowledge to the students in the area
of high-performance deep learning.

<!--
<h4> Grading Plan: </h4>  
There are five components: 
<pre> <strong> 
- Attendance, participation in class discussion,   (10%)
  and evaluation of class presentations
- Labs                                             (25%) 
- Class Presentation                               (10%)
- Class Project                                    (25%)
- Final exam                                       (30%)
  Friday, Dec 11th, 12:00 pm - 2:00 pm
  (comprehensive, open book) 
</strong>
</pre>    

<h4> Class Presentation: </h4>  

Each student will make one class presentation.
Please refer to the <a href="reading.html">reading list </a> 
and <a href="presentation_schedule">presentation schedule </a>
and select a date
(with the corresponding set of papers) for presentation. 
If you are working on a given project, it will be preferable  
to choose a different set of papers for the presentations. 

<p>

The presentation schedule will be filled-up based on the order
in which I receive the preferences. To avoid conflicts with others, 
you can give me preferences
for 3-4 dates. We will finalize the list during the first two weeks
of the semester.

<p>

The class presentation will be evaluated based on preparation
(knowledge assimilation), presentation style (organization,
smoothness, and clarity), finishing the presentation in time, and
answering questions during the discussion.

<p>

We will be using an   
<a href="evaluation_form ">evaluation form </a> to evaluate every 
student presentation. Each student needs to fill out this form for 
every student speaker 
and return it (hardcopy only) to me by the following class. I will 
give them  
back to the student after removing the evaluator's information. 
I will use this information to evaluate the speaker's presentation 
as well as the evaluators' skill to evaluate a presentation.  

<p> 

You must consult me when preparing your slides and finalizing them.
This is to ensure that the presentations are compact and provide a
smooth flow. Please discuss about your presentation plans with me
two weeks before the presentation date and the slides with me 
one week before
the presentation date.               

<h4> Laboratory Exercises: </h4>  

The course will involve laboratory expercises for students to experiment
with Deep Learning Frameworks. These exercises will be carried out on 
OSC (Ohio Supercomputing Center's clusters). We will form 7-8 teams. 
Each team 
will focus on one of the following DL framework 
(TensorFlow-gRPC, TensorFlow-MPI, PyTorch, MXNET, and LBANN). 
Each team will have around four students: 3 graduates and one undergrad.
Within the graduate students, we will try to balance it out wrt Ph.D.
and M.S. students. I will discuss this with the students to form the
teams. The last two classes of the semester will focus on presentations
by the teams to share their knowledge to the other class members. 
Members within a team will get the same grade for a given Lab.

<h4> Class Project: </h4>  

Due to the research-oriented nature of this course, the project will
play an important part in the learning experience and in the grading
process. Projects will be evaluated based on their 
technical quality, originality, depth of analysis, and completeness.

<p>

Projects will be mostly done in an individual manner.  Maximum two
people may be in a group if the scope of the project is big and there
is sufficient understanding between the members that both will
contribute equally to the success of the project (members will get the
same grade in project).  The project will be
research-oriented. Depending on the nature of the topic, it may
consist of one or more of the following components: theory, design,
analysis, simulation, or experimental results.            

<p>

You are free to consult with me while defining the project and working 
on it during 
the semester. 

<p>

I will be meeting with each of you frequently during the semster 
to discuss the progress
of your project. 

<h4> Project Schedule: </h4>  
<pre> <strong> 
End of 3rd/4th week     - discussion with the instructor to select/focus topic
End of 4th/5th week     - proposal (around two pages) due
End of 5th week         - feedback from instructor and finalizing the topic
Project Report due      - on or before 12/11/20 (graduating seniors)         
                          on or before 12/13/20 (regular)
</strong>
</pre>  
-->

<p>

<!--
<h4> Meeting Schedule: </h4>  

Please refer to the  
<a href="meeting_schedule">meeting schedule </a>
and send me a time slot for the meeting.  
I will fill it up based on first-come-first serve basis. Give me 2-3 preferences.

<hr>
-->


<ADDRESS>
Last Updated: May 5, 2021 
</ADDRESS>