Mila technical documentation

Welcome to Mila’s technical documentation. If this is your first time here, we recommend you start by checking out the short quick start guide.

Note

Researchers are requested to acknowledge Mila when publications or reports are published on research that used compute resources provided by the organization. Our ability to showcase the successful use of our resources builds an important case for long-term and sustainable funding of Mila and our operations, i.e. have more compute resources available to researchers!

The exact wording of the acknowledgment may vary, but please ensure that Mila is mentioned. The following can be used as a guideline:

“This research was enabled in part by compute resources provided by Mila (mila.quebec).”

If you find useful some of the software developed at Mila or the technical help you can get from our experts you can also mention it in the acknowledgment:

“This research was enabled in part by compute resources, software and technical help provided by Mila (mila.quebec).”

Introduction

• Purpose of this documentation
  • Intended audience
• Contributing

How-tos and Guides

• User’s guide
  • Quick Start
  • Logging in to the cluster
  • Running your code
  • Portability concerns and solutions
  • Singularity
  • Sharing Data with ACLs
  • Contributing datasets
  • Data Transmission using Globus Connect Personal
  • JupyterHub
  • Advanced SLURM usage and Multiple GPU jobs
  • Multiple Nodes
  • Weight and Biases (WandB)
  • Comet
  • Frequently asked questions (FAQs)
• AI tooling and methodology handbook

Systems and services

• Computing infrastructure and policies
  • Roles and authorizations
  • Overview of available computing resources at Mila
  • Node profile description
  • Data sharing policies
  • Monitoring
  • Storage
  • Data Transmission
• Computational resources outside of Mila
  • Digital Research Alliance of Canada Clusters

General theory

• What is a computer cluster?
• Parts of a computing cluster
  • The login nodes
  • The compute nodes
  • The storage nodes
  • Different nodes for different uses
• UNIX
• The workload manager
  • Anatomy of a job
  • Understanding the queue
  • About partitions
  • Exceding limits (preemption and grace periods)
  • Mila information
• Processing data
  • Data parallelism
  • Model parallelism
  • Communication concerns
  • Filesystem concerns
• Software on the cluster
  • Cluster software modules
  • Containers
  • Python Virtual environments

Minimal Examples

• Software Frameworks
  • PyTorch Setup
  • Jax Setup
  • Jax
• Distributed Training
  • 1. Single GPU Job
  • 2. Multi-GPU Job
  • 3. Multi-Node (DDP) Job
• Good practices
  • Checkpointing
  • Wandb Setup
  • Launch many jobs from same shell script
  • Hyperparameter Optimization with Oríon
  • Launch many tasks on same GPU
  • Launch many jobs using SLURM job arrays

Extras

• Acknowledging Mila
• Mila Datasets
• Audio and video resources at Mila
• Visual Studio Code
  • Connecting to the cluster
  • Activating an environment
  • Troubleshooting
• Who, what, where is IDT
  • IDT’s mission
  • The IDT team
• Cheat Sheet
  • Errata
• Chatbot for Mila
• Environmental Impact
  • Mila Cluster
  • Digital Research Alliance of Canada Clusters

Support:

To reach the Mila infrastructure support, please submit a support ticket.

Contribution:

If you find any errors in the documentation, missing or unclear sections, or would simply like to contribute, please open an issue or make a pull request on the github page.