Enroot on Slurm for Distributed Deep Learning

💡 Introduction

This project offers a comprehensive guide to using Enroot on Slurm cluster for deep learning tasks. It includes a detailed job script that demonstrates how to set up and run a distributed deep learning job using Enroot and Slurm.

🚩 Documentation

• 1 - Introduction to UBAI HPC Cluster

• 2 - Slurm Job Scheduler

• 3 - Using Jupyter Lab on Slurm

• 4 - Using Docker Container with Enroot

• 5 - PyTorch Training with NGC Container

• 6 - PyTorch Distributed Training on Slurm

• 7 - Multi-node LLM Training with Wandb

🔧 Tools

🧷 References

• NVIDIA: Enroot Documentation

• SCHEDMD: Slurm Documentation

• The Slurm tool "pestat"

• UBAI HPC cluster User Manual

• LambdaLabsML: Distributed Training Guide

Acknowledgements

The Urban Big Data and AI Institute of the University of Seoul supercomputing resources(http://ubai.uos.ac.kr) made available for conducting the project.

Contribution

You are welcome to contribute to this project!