H200 MLPerf small_llm_pretraining: Unable to perform multi-node pretraining

[Ypp6657]

I’m running MLPerf Llama 3.1 small LLM pretraining (benchmarks/small_llm_pretraining) on our H200 DGX cluster.
Single-node training works correctly inside the provided Docker container (mlperf-h200:latest), but multi-node pretraining does not appear to be supported or documented.

The pretrain_llama31.py script only uses a LocalExecutor, while the large-model pretraining example includes a working SlurmExecutor.
I would like to benchmark 2–8 H200 nodes using SLURM (inside Docker, as per MLPerf setup), but there is no documented way to do so.

---

Environment

• Cluster: DGX H200 (8× H200 141 GB GPUs per node)
• Nodes: 2 – 8 planned
• Container: mlperf-h200:latest from MLPerf reference
• Inside container: Python 3.10, NeMo 1.25+, PyTorch 2.3+, CUDA 12.4

[ShriyaRishab]

SlurmExecutor should work for 2-8 node setups. Just launch the job from your laptop on any other system that has ssh access to the cluster that has H200 nodes and set https://github.com/mlcommons/training/blob/master/small_llm_pretraining/nemo/run_llama31.sh#L22-L27 values correctly.

[Ypp6657]

SlurmExecutor should work for 2-8 node setups. Just launch the job from your laptop on any other system that has ssh access to the cluster that has H200 nodes and set https://github.com/mlcommons/training/blob/master/small_llm_pretraining/nemo/run_llama31.sh#L22-L27 values correctly.

Hi @ShriyaRishab,

Thanks for your response. We’re currently running MLPerf Llama 3.1 small LLM pretraining on our DGX H200 cluster (8× H200 141 GB GPUs per node).

Our current workflow is:

Connect to the private VPN.

SSH into the H200 GPU cluster login node.

Start the provided container using

docker run -it --rm --gpus all --net=host mlperf-h200:latest

From inside the container, we submit the pretraining job through a SLURM batch script (which sets MASTER_ADDR, MASTER_PORT, NODE_RANK, etc.) and directly runs

python pretrain_llama31.py ...

instead of sourcing the config file or using run_llama31.sh.

We tried manually setting all the parameters mentioned in lines 22–27 of run_llama31.sh, but the job still only launches on a single node and the multi-node process never initializes (no cross-node NCCL/torch connectivity).

Could you please confirm if there is any documented or tested workflow for running the small LLM pretraining with SlurmExecutor inside the Docker container (similar to the large-model example)?

Even a short example of a working multi-node configuration or expected command flow (e.g., how the nemo runtime detects SLURM nodes inside Docker) would help us validate and benchmark properly.

[ShriyaRishab]

Instead of launching from the login node of the H200 cluster, can you just connect to the VPN, run docker and launch pretraining? SlurmExecutor handles ssh into the cluster by itself

[ShriyaRishab]

@Ypp6657 - were you able to try #840 (comment)