HPC4AI-MEI is a master-level course taught at the Facultat d’Informàtica de Barcelona (FIB), Universitat Politècnica de Catalunya (UPC), within the Master in Computer Engineering.
Subject Description (2026 edition)
High Performance Computing for Artificial Intelligence (HPC4AI) is a master-level, practical-oriented course focused on understanding how modern AI training workloads actually run on real supercomputing infrastructures.
Rather than treating deep learning frameworks and tools as black boxes, the course adopts a system-oriented perspective. It guides students through the complete execution workflow of AI training—from hardware architecture and system software to job scheduling, parallel execution, performance measurement, and scalability analysis. The emphasis is on execution behavior: how computation, memory, communication, and coordination interact, and how these interactions determine performance, efficiency, and cost.
A central premise of the course is that the nature of engineering work in AI is changing. Modern AI tools can generate training scripts, pipelines, and even distributed execution logic with minimal effort. As a result, writing code is no longer the primary challenge. The real difficulty—and the real value—lies in understanding whether that code scales, where bottlenecks appear, when efficiency is lost, and what trade-offs are being made when more resources are used.
For this reason, the course explicitly allows and acknowledges the use of modern AI tools (such as code assistants, agentic systems, or automated code generators). However, the course is not about code authorship or syntax. It is about developing the ability to reason about performance, scalability, efficiency, and cost when training deep learning models on real HPC systems. Students are expected to understand what is being executed, how it behaves at scale, and why performance changes as observed.
Hands-on experimentation is a core component of the course. Through a sequence of laboratory activities, students train deep learning models using single and multiple GPUs, explore parallel and distributed training strategies, and analyze scalability and performance behavior under realistic conditions. All laboratory work and assessments are evaluated based on the quality of experimental setup, the relevance of performance measurements, the interpretation of results, and the soundness of scalability and cost–benefit reasoning.
The course material is self-contained and based on the official course textbook, which serves as the main reference for both theoretical concepts and practical activities. No prior experience with supercomputers is required, and deep learning concepts are introduced progressively as needed.
Ultimately, HPC4AI is not a course about recipes or fixed solutions. It is a course about developing engineering judgment. As code generation becomes cheaper and more accessible, the ability to measure, reason, and decide becomes essential. This course is designed to develop precisely that ability.
Why HPC for AI course in the Age of AI Tools
In recent years, AI-based coding tools have dramatically reduced the cost of writing software. Today, complex applications, training pipelines, and distributed systems can be generated in minutes using natural language. The barrier to producing code is collapsing.
This does not mean that engineering problems have disappeared. On the contrary, it has shifted where the real difficulty—and value—lies. When code is cheap, performance becomes expensive.
Training modern AI models is no longer limited by our ability to write code. It is limited by:
- Hardware utilization
- Memory hierarchies
- Communication overheads
- Scalability limits
- Energy consumption
- Cost–performance trade-offs
AI tools can generate scripts, but they do not decide:
- whether scaling to more GPUs is worth it
- where the bottleneck really is
- when efficiency losses are acceptable
- how to interpret noisy or misleading benchmarks
- what configuration makes sense in a real system
These decisions require judgment. This is the core message of HPC for AI course (and book). The widespread claim that “coding jobs are over” misses the point. What is disappearing is the differential value of writing code by hand.
What is growing rapidly in importance is the ability to:
- reason about systems
- understand performance behavior
- analyze scalability
- evaluate cost versus benefit
In other words, to think like an engineer, not like a code generator. That is why this course does not teach recipes. It teaches how to measure, reason, and decide. HPC for AI course is about judgment, not recipes. And that is exactly what becomes essential in the age of AI-generated code.
Content
- C1: HPC platforms and software ecosystem for AI
Architecture of modern supercomputers, hardware components, operating system, and software stack for artificial intelligence workloads. - C2: Accessing and using a supercomputer for AI workloads
Access to a supercomputer, account management, batch systems, SLURM, and job execution for Deep Learning applications. - C3: Deep Learning fundamentals for HPC environments
Basic Deep Learning concepts required to train models in HPC environments, including neural networks, training workflows, and datasets (any background can be assumed). - C4: Parallel training of Deep Learning models
Parallel training of Deep Learning models using multiple GPUs, including parallelism strategies and programming frameworks. - C5: Performance metrics and optimization of AI training
Performance analysis of AI model training using metrics such as throughput, speedup, and efficiency, and basic optimization techniques. - C6: Experimental evaluation and presentation of results
Experimental evaluation of results obtained in an HPC environment and clear communication of conclusions through technical reports and presentations.
- C1: HPC platforms and software ecosystem for AI
Course Schedule and Sessions (still in review)
The course is structured into thirteen on-site sessions of two hours each. Each session combines lecture-based instruction with guided hands-on work and progressive laboratory development. Laboratories are initiated during class time and completed outside the classroom, allowing students with different backgrounds to progress at their own pace while ensuring a common technical baseline.
The schedule follows a clear pedagogical progression: onboarding → deep learning warm-up → parallel training → performance optimization → distributed scalability → presentation and reflection.
| Session | Date | Title | Lab | Main Focus | Book Chapters / Sections |
|---|---|---|---|---|---|
| 1 | March 25 | Course introduction and MareNostrum 5 visit | — | Course scope, motivation, HPC for AI mindset | Preface; How to Use This Book; Chapter 1 (overview) |
| 2 | April 15 | HPC platform and SLURM basics | LAB 0 | HPC architecture, batch execution, SLURM basics | Chapter 2 (compute, storage); Chapter 3 (SLURM essentials) |
| 3 | April 20 | Deep Learning basics and frameworks | LAB 1 | DL training fundamentals, GPUs, frameworks | Chapter 7; Chapter 8 (transfer learning) |
| 4 | April 22 | DL on MN5 and framework equivalence | LAB 1 | Execution analysis, TF–PyTorch equivalence | Chapter 9 (framework equivalence) |
| 5 | April 27 | Introduction to parallel training | LAB 2 | Data parallelism, performance metrics | Chapter 10 (motivation, metrics, parallelism) |
| 6 | April 29 | CPU vs GPU and parallel overheads | LAB 2 | GPU impact, communication and synchronization costs | Chapter 10 (CPU/GPU comparison, overheads) |
| 7 | May 4 | Model size and scalability limits | LAB 2 | Scalability vs model size, amortization | Chapter 10 (model size, ResNet case study) |
| 8 | May 6 | Efficient training on a single GPU | LAB 3 | Training pipeline, batch size limits | Chapter 11 (pipeline, batch size) |
| 9 | May 11 | DataLoader bottlenecks and pipelines | LAB 3 | Bottlenecks, balanced pipelines | Chapter 11 (DataLoader, pipeline balance) |
| 10 | May 13 | Mixed precision and compilation | LAB 3 | Tensor Cores, mixed precision, compilation | Chapter 11 (mixed precision, torch.compile) |
| 11 | May 18 | Distributed training with DDP | LAB 4 | DDP execution model, full software stack | Chapter 12 (DDP, torchrun, NCCL) |
| 12 | May 20 | Scalability and diminishing returns | LAB 4 | Strong scaling, efficiency, sweet spot | Chapter 12 (scalability analysis) |
| 13 | May 27 | Final presentations and course closure | — | Result interpretation, critical judgment | Review Ch. 10–12; |
Laboratory Structure and Content (Tentative)
The practical component of the course is organized into five laboratories that progressively guide students from basic interaction with a supercomputer to distributed training and scalability analysis of deep learning models. Each laboratory is directly aligned with specific chapters and tasks from the course textbook.
LAB 0 — Supercomputer Onboarding
Objective
Remove technical barriers and ensure that all students can access and use the supercomputer autonomously.
Book chapters
Chapter 2: Access to MareNostrum 5 and basic storage usage (essential elements only)
Chapter 3: Basic SLURM usage and execution of Python jobs
Tasks
2.1, 2.2, 2.4
3.3 – Submit your first SLURM job running a Python script
LAB 1 — Deep Learning Warm-Up on MareNostrum 5
Objective
Level basic Deep Learning knowledge and execute a real GPU-based training workload on the supercomputer.
Book chapters
Chapter 7: Deep Learning fundamentals
Chapter 8: Training and transfer learning
Chapter 9: Deep Learning frameworks and TensorFlow–PyTorch equivalence
Tasks
8.5 – Warm-up Transfer Learning Experiment on MareNostrum 5
9.1 – Observing the equivalence between TensorFlow and PyTorch
LAB 2 — Parallel Training of Deep Learning Models (TensorFlow)
Objective
Introduce parallelism in the training step and establish a rigorous experimental foundation.
Book chapter
Chapter 10 (complete chapter)
Tasks
10.1 – Task setup and file structure
10.2 – Code review and execution flow
10.3 – Training on CPU: baseline
10.4 – Single-GPU training: CPU vs GPU
10.5 – Analyze the impact of GPU parallelism
10.6 – Parallelizing a larger model (ResNet152)
10.7 – Comparing parallel training across model sizes
LAB 3 — Efficient Training on a Single Node (PyTorch)
Objective
Understand why performance changes and how to optimize training on a single GPU node.
Book chapter
Chapter 11 (complete chapter)
Tasks
11.1 – Finding the maximum viable batch size
11.2 – Investigating DataLoader bottlenecks
11.3 – Optimizing DataLoaders
11.4 – Confirming efficiency with Vision Transformers
11.5 – Mixed precision training
11.6 – torch.compile
11.7 – Report conclusions
LAB 4 — Distributed Training and Scalability (PyTorch)
Objective
Scale deep learning training across multiple GPUs with technical judgment and performance awareness.
Book chapter
Chapter 12 (complete chapter)
Tasks
12.1 – Reproducing distributed training results
12.2 – Analyzing scaling efficiency
12.3 – Identifying diminishing returns
12.4 – Identifying the sweet spot
Teaching methods
The course follows an active learning and continuous assessment approach, combining theoretical lectures, hands-on laboratory work, autonomous learning, and student presentations.
Theoretical sessions are delivered through participatory lectures, where the instructor introduces the fundamental concepts related to high-performance computing platforms, deep learning fundamentals, parallel training strategies, and performance analysis for artificial intelligence workloads. Students are expected to actively participate in discussions during these sessions.
Hands-on activities constitute a central component of the course and are based on a learn-by-doing methodology. These activities focus on practical experimentation using a real supercomputing environment (MareNostrum 5). Part of the hands-on work is carried out during regular class sessions, while the remaining work is completed outside the classroom as autonomous learning. All hands-on activities require the submission of corresponding reports and, in some cases, technical presentations through the institutional learning platform (Racó).
Autonomous learning is mainly based on the detailed study of the course textbook, which constitutes the main reference material for the subject. Students are also required to prepare presentations and technical material related to their practical work.
Student presentations play an important role in the course. Individual students or groups are randomly selected to present their work and results in class. Peer evaluation is incorporated as part of the learning process, encouraging critical analysis and constructive feedback.
Regular attendance and active participation are expected. Students are responsible for all material covered in class, including announcements, assignments, and project guidelines, regardless of attendance. It is the student¿s responsibility to obtain any missed material.
Assessment
The evaluation of this course is based on a continuous assessment system, strongly focused on practical work and active participation.
The final grade is composed of the following elements:
– Attendance and participation: 20%
Regular attendance and active participation in lectures, discussions, and hands-on sessions.
Attendance is mandatory. To qualify for continuous assessment, students must attend at least 80% of the class sessions.
– Hands-on activities (laboratory work): 60%
Evaluation of the practical laboratory activities carried out throughout the course (LAB 0 to LAB 4).
The instructor will assess the submitted work using a rubric that considers correctness, completeness, experimental results, and technical understanding.
Some students or groups will be randomly selected during the course to present and explain their laboratory work (LAB 0 to LAB 2). This mechanism is intended to ensure that all students prepare and understand their work thoroughly.
– Technical presentations and peer evaluation: 20%
During the final session of the course, all students will present either LAB 3 or LAB 4 (assigned randomly).
Presentations will be evaluated by the instructor and through peer evaluation, which will contribute to the final presentation grade.
Attendance on the presentation day is mandatory. Students who do not attend this session will not receive the presentation grade.
Requirements for continuous assessment: To qualify for continuous assessment, students must meet all the following requirements:
– Attendance: at least 80% of the class sessions.
– Hands-on activities: completion of at least 50% of the laboratory work.
Final exam option
– Students who do not meet the requirements for continuous assessment will have the option to take a final exam.
– This exam will evaluate the entire course content, including theoretical concepts, practical knowledge, and autonomous learning material based on the course book and laboratory activities.
– The final exam will be announced during the course. No documentation (printed or digital) will be allowed during the exam.