High Performance Computing (Udacity)

This course is a graduate-level introduction to scalable parallel algorithms. "Scale" really refers to two things: efficient as the problem size grows, and efficient as the system size (measured in numbers of cores or compute nodes) grows. To really scale your algorithm in both of these senses, you need to be smart about reducing asymptotic complexity the way you’ve done for sequential algorithms since CS 101; but you also need to think about reducing communication and data movement. This course is about the basic algorithmic techniques you’ll need to do so.

The techniques you’ll encounter covers the main algorithm design and analysis ideas for three major classes of machines: for multicore and many core shared memory machines, via the work-span model; for distributed memory machines like clusters and supercomputers, via network models; and for sequential or parallel machines with deep memory hierarchies (e.g., caches). You will see these techniques applied to fundamental problems, like sorting, search on trees and graphs, and linear algebra, among others. The practical aspect of this course is implementing the algorithms and techniques you’ll learn to run on real parallel and distributed systems, so you can check whether what appears to work well in theory also translates into practice. (Programming models you’ll use include Cilk Plus, OpenMP, and MPI, and possibly others.)

This course is a graduate-level introduction to scalable parallel algorithms. To really scale your algorithm in both of these senses, you need to be smart about reducing asymptotic complexity the way you’ve done for sequential algorithms since CS 101; but you also need to think about reducing communication and data movement. This course is about the basic algorithmic techniques you’ll need to do so. The practical aspect of this course is implementing the algorithms and techniques you’ll learn to run on real parallel and distributed systems, so you can check whether what appears to work well in theory also translates into practice.

What you will learn

Work-Span or Dynamic Multithreading Model

- Intro to the basic algorithmic model

- Intro to OpenMP, a practical programming model

- Comparison-based sorting algorithms

- Scans and linked list algorithms

- Tree algorithms

- Graph algorithms, e.g., breadth-first search

Distributed Memory or Network Models

- The basic algorithmic model

- Intro to the Message Passing Interface, a practical programming model

- Reasoning about the effects of network topology

-Dense linear algebra

- Sorting

- Sparse graph algorithms

- Graph partitioning

Two-Level Memory or I/O Models

- Basic models

- Efficiency metrics, including “emerging” metrics like energy and power

- I/O-aware algorithms

- Cache-oblivious algorithms

Prerequisites and requirements

A "second course" in algorithms and data structures, a la Georgia Tech’sCS 3510-B or Udacity’s Intro to Algorithms

For the programming assignments, programming experience in a "low-level" "high-level" language like C or C++

Experience using command line interfaces in *nix environments (e.g.,Unix, Linux)

Course readiness survey.