Lecture time Monday, Wednesday, 3:35pm-4:50pm

Lecture location 103 Lov Building

Instructor Prof. Xin Yuan, 168 Love Building, 644-9133, xyuan@cs.fsu.edu

Office hours Monday, Wednesday 1:00pm-2:00pm, or by appointment.

Teaching Assistant Mr. Surjyendu Ray, surjray.ta@gmail.com

Office hours TBD

Textbook There is no required textbook.

Reference materials

• D. E. Culler and J. P. Singh, "Parallel Computer Architecture, A Hardware/Software Approach," Morgan Kaufmann, 1999.
• Maurice Herlihy and Nir Shavit, "The Art of Multiprocessor Programming," Morgan Kaufmann, 2008.
• Aaftab Mushi, Benedict R. Gaster, Timothy G. Mattson, James Fung, Dan Ginsburg, "OpenCL Programming Guide", Addison-Wesley Professional, 2011.
• MPI: http://www-unix.mcs.anl.gov/mpi/
• OpenMP: http://www.openmp.org
• CUDA: NVIDIA CUDA Programming Guide
• OpenCL: OpenCL Programming Guide for the CUDA Architecture

Course Home Page http://www.cs.fsu.edu/~xyuan/cis4930-cda5125/index.html

Class Mailing List I will maintain a mailing list for the course related information such as announcement, homework, etc. Please check both the course web site and your email at least once in two days throughout the semester.

Course rationale This is an introductory course to the parallel and distributed systems and programming.

This course systematically introduces parallel and distributed systems and programming. It is an early adopter of the NSF/IEEE-TCPP Parallel and Distributed Computing Curriculum. As such, the course follows the NSF/TCPP Parallel and Distributed Computing Curriculum with four classes of the topics: architecture (15 hours), programming (20 hours), algorithm (5 hours), and cross-cutting/advanced (5 hours).

• Architecture (15 hours):
  • Architectural classes, Flynn's taxonomy, control .vs. data parallelism
  • SIMD/Vector architectures
  • MIMD architectures
  • Streams
  • Shared memory architectures
    • Symmetrical Multi-Processor (SMP) architecture
    • Homogeneous multi-core and SMT architectures and (Intel Core 2 Duo, AMD Barcelona)
    • Heterogeneous multi-core architecture (IBM Cell)
    • Many-core architecture
  • Distributed memory architectures
    • Clusters of workstations/servers
    • Massively Parallel Processing architecture (IBM Blue Gene)
    • Grid and cloud computers
  • Cache organization and coherence
  • Communication hardware and Interconnection networks
    • Topologies and its performance metrics
    • Latency and Bandwidth
  • Performance
    • Peak performance and sustained performance
    • MIPS/FLOPS
    • Benchmarks
• Programming (20 hours)
  • SPMD
  • SIMD and vector extensions
  • Shared memory programming: thread and OpenMP
  • CUDA/OpenCL programming
  • Distributed memory programming: MPI
  • Synchronization, concurrency, deadlock, race conditions, determinacy
  • Load balancing, static and dynamic scheduling
  • Performance metrics: Speedup, efficiency, Amdahl's law, Gustafson's law
• Algorithm (5 hours)
  • PRAM
  • BSP
• Advanced (5 hours)
  • Power
  • Concurrency, the inherent parallelism in an algorithm.
  • Locality and performance
  • Performance monitoring and optimization
  • Communication library implementation and communication optimization techniques

• Explain the important concepts in architectures, programming, algorithms areas related to PDS.
• Explain concurrency defects and challenges in programming PDS.
• Program with different types of programming paradigms for different architectures.

• You must have completed COP 4610 Operating Systems (or equivalent), but you do not need any parallel and distributed computing background

• 25% - homework and programming assignment
• 25% - course project (group)
• 20% - Midterm
• 30% - Final exam

Course schedule (tentative)

• Week 1: Introduction and architectures classification
• Week 2: Performance
• Week 3: SIMD/Vector architectures, SSE and SIMD intrinsics
• Week 4: Shared memory architectures
• Week 5: Shared memory programming, thread
• Week 6: Shared memory programming, OpenMP
• Week 7: MIMD, distributed memory architectures, MPI
• Week 8: communication hardware and Interconnection networks
• Week 9: Streams architectures, CUDA/OpenCL
• Week 10: CUDA/OpenCL
• Week 11: CUDA/OpenCL
• Week 12: Performance monitoring and Optimization
• Week 13: PRAM and BSP
• Week 14: Advanced topics
• Week 15: Project presentations
• Week 16: Final exam

Course Policies

• Attendance: The university requires attendance in all classes. Absences may be excused with appropriate documentation. Excused absences include documented illness, deaths in the immediate family and other documented crises, call to active military duty or jury duty, religious holy days, and official University activities. Accommodations for these excused absences will be made and will do so in a way that does not penalize students who have a valid excuse. Consideration will also be given to students whose dependent children experience serious illness.
• Late assignments will not be accepted without a valid excuse. Every document (homework, report, etc) that you hand in must be a print-out from a soft copy.
• Missed exams: A missed exam will be recorded as a grade of zero. We will follow the university rules regarding all missed exams (http://registrar.fsu.edu/dir_class/spring/exam_schedule.htm).
• Incomplete grade: An incomplete grade will be assigned only under the following exceptional circumstances:
  • If you miss the final exam with an accepted excuse, you must make up the exam during the first two weeks of the following semester.
  • Due to extraordinary circumstances, with appropriate documentation, the student can make up the missed portion, which must be no more than 25% of the course, prior to the end of the next semester.

Your Responsibilities

• Understand the lecture slides and reading assignments.
• Attend the lectures and participate in the discussions.
• Attend office hours for extra help, as needed.
• Uphold academic honesty in completing your assignments, projects, and exams.
• Turn in your projects, homework, programming assignment, and take-home exam on time.
• Check the class Web page and your garnet email account regularly.

Resources

• MPI tutorials (google keywords MPI + tutorials)
• OpenMP tutorials (google keywords OpenMP + tutorials)
• MPI standard (http://www.mcs.anl.gov/research/projects/mpi/)

Accommodation for Disabilities

Students with disabilities needing academic accommodations should:

1. Register with and provide documentation to the Student Disability Resource Center (SDRC).
2. Bring a letter to the instructor from the SDRC `indicating you need academic accommodations. This should be done within the first week of class.

This syllabus and other class materials are available in alternative format upon request.

For more information about services available to FSU students with disabilities, contact the Assistant Dean of Students:

Student Disability Resource Center
874 Traditions Way
108 Student Services Building Florida State University
Tallahassee, FL 32306-4167
sdrc@admin.fsu.edu
http://www.disabilitycenter.fsu.edu
(850)644-9566(voice)
(850)644-8504(TDD)