Could My Job Run Faster? How to Identify GPU Waste¶
This guide explains how to read GPU efficiency metrics, diagnose an underutilized job, and apply concrete best practices to improve throughput on the Mila cluster.
Before you begin¶
-
Track GPU utilization, CPU usage, and memory for a run.
-
Submit and allocate jobs on the cluster.
What this guide covers¶
- Compute efficiency concepts: utilization vs. occupancy
- How to diagnose an underutilized job
- Best practices for efficient GPU use on the cluster
Why GPU efficiency matters¶
Optimizing GPU usage directly accelerates research velocity. As compute power at Mila is a shared resource, efficient jobs on the cluster bring a two-sided advantage:
- For the researcher: Eliminating bottlenecks speeds up training times and helps unearth hidden bugs in data loaders or model architectures. With properly sized compute requests and efficient utilization, jobs start sooner and produce useful results faster.
- For Mila: Maximizing efficiency frees up cluster nodes, resulting in shorter queue times and more parallel experiments across the institute.
In other words, efficient compute utilization makes Mila research thrive.
The basics of compute efficiency: utilization vs. occupancy¶
nvidia-smi is a useful first check, but its utilization metric has
limitations: it reports "100% Utilization" as soon as any kernel is running on
the GPU, regardless of how much of the hardware is actually in use. So if the
reported GPU utilization is low or equals zero, it usually means there is room
for optimization. For a more precise view, look at Streaming Multiprocessor
(SM) Occupancy, which measures what fraction of the GPU's computing units are
actively working.
Use the table below as a reference to evaluate SM occupancy:
| SM Occupancy | Assessment |
|---|---|
| < 5% | Critical waste |
| ~10% | Poor utilization — the GPU is mostly waiting |
| ~30% | Good utilization |
| ≥ 50% | Great / optimized utilization |
How to diagnose a job¶
Self-diagnosis is possible using these framework-agnostic methods. The flowchart below outlines the decision path; the methods that follow explain how to read the underlying numbers.
flowchart TD
A[Suspect a job's resources are underutilized] --> B{SM occupancy ≥ 30%?}
B -->|Yes| C[GPU well utilized — no action needed]
B -->|No| D{VRAM usage < 20%?}
D -->|Yes| E[Increase batch size, use a smaller GPU, or pack jobs]
D -->|No| F{Data loader saturating the GPU?}
F -->|Yes| G[Profile for I/O or CPU preprocessing bottleneck]
F -->|No| H[Tune the DataLoader: num_workers, pin_memory]Method A: Weights & Biases¶
In WandB, the System tab of a run shows data on GPU utilization, CPU usage, and memory. See Diagnose training bottlenecks for details.
Method B: The interactive check¶
During a job, srun into the allocated node and run a basic check:
Method C: The NVSMI log¶
When a job runs on the cluster, an output file is created with the default
name slurm-<JOB_ID>.out. This file contains the job's output, along with an
NVSMI LOG section reporting metrics such as GPU and memory utilization.
Example
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | |
Method D: TensorBoard visualization of PyTorch profiler data¶
- PyTorch profiler is a tool that measures the resource consumption of an experiment.
- TensorBoard is a visualization toolkit that can log and display experiment usage.
TensorBoard should not be launched on login nodes
Launch it from an interactive or batch job on a compute node instead.
An example of TensorBoard usage on the cluster is described in the Visualizing usage with PyTorch profiler and TensorBoard guide.
Method E: Cluster portals¶
Some clusters have a related portal for displaying data and metrics, such as resource usage or job history.
Here is a quick overview of the clusters and their associated portals (if applicable):
| Clusters | Maintainer | Portal |
|---|---|---|
| Mila | Mila | - |
| TamIA | PAICE | TamIA portal |
| Killarney | PAICE | - |
| Vulcan | PAICE | Vulcan portal |
| Fir | DRAC | - |
| Nibi | DRAC | Nibi portal |
| Rorqual | DRAC | Rorqual portal |
| Trillium | DRAC | Trillium portal |
Best practices for efficient GPU use¶
Even though situations are diverse, the following guidelines pave the way for efficient GPU utilization.
Do — improve efficiency
- Profile before scaling: Run a test job with a profiler (WandB or TensorBoard) before launching large sweeps to ensure the data loader saturates the GPU.
- Optimize data pipelines: Set
num_workers > 0(2–4 per allocated GPU) and enablepin_memory=Truein the PyTorchDataLoaderto prevent GPU stalling. - Implement checkpointing: Save training states regularly so jobs resume automatically after preemption or timeouts without losing previous compute hours.
- Right-size resource requests: Use lower-tier nodes (e.g., RTX8000, V100) or MIG (Multi-Instance GPU) slices for small models or debugging instead of allocating full high-end nodes.
- Request minimal compute blocks: When possible, request the smallest allocation that fits the job. Smaller allocations fill queue gaps faster, reducing wait time.
Don't — common pitfalls
- Hoarding nodes: Do not keep high-end GPUs (e.g., H100s) allocated on interactive partitions while away from the keyboard. Release them if not actively computing.
- Avoiding preemption queues: Do not camp on non-preemptible partitions to avoid writing checkpointing code — this significantly reduces overall queue priority.
- Over-allocating CPU cores: Do not request excessive CPU cores (e.g., 40 CPUs for 1 GPU) unless preprocessing explicitly requires it. Mila provides CPU-only nodes if needed.
- Scaling GPUs to fix I/O bottlenecks: Do not add more GPUs if storage read latency or CPU preprocessing bottlenecks the pipeline — this only idles more hardware.
- Underutilizing VRAM: If VRAM usage is under 20%, consider increasing batch size, switching to a smaller GPU, or using job packing (multiple smaller jobs on the same node).
Key concepts¶
- SM Occupancy
- The fraction of a GPU's Streaming Multiprocessors (computing units) that
are actively working. A more precise measure of GPU use than the
nvidia-smiutilization metric. - GPU Utilization
- The
nvidia-smimetric that reports 100% as soon as any kernel runs on the GPU, regardless of how much hardware is in use. Useful as a first check but misleading on its own. - MIG (Multi-Instance GPU)
- A feature that partitions a single GPU into smaller isolated slices, well suited to small models or debugging.
- VRAM
- The GPU's onboard memory. Low VRAM usage often signals room to increase batch size or move to a smaller GPU.
Next steps¶
-
Use the WandB System tab to locate GPU, CPU, and I/O bottlenecks.
-
Choose an appropriate node tier for the job.
Get help¶
Questions are welcome on Slack (#mila-cluster, #compute-canada), or during
Office Hours. The team is always
happy to provide early guidance or share thoughts on how to improve
experiments — in the interest of the community.
An LLM with curated Mila cluster context can also help investigate and remove performance issues.