DeepSpeed Multi-GPU Distributed Training Pitfall Guide: 5 Steps to Locate OOM

When you run large model fine-tuning on 8x A100 with DeepSpeed and it immediately reports CUDA out of memory, or freezes at step 3, don't rush to add more VRAM—this DeepSpeed multi-GPU distributed training pitfall guide walks through five checkpoints in order, and most issues can be resolved at the configuration level.

But before you start, there's a prerequisite: confirm your DeepSpeed version. v0.19.5, released in August 2026, enables ZeRO-3 asynchronous gradient offloading and pinned offload buffers by default, and splits host pinned memory into separate pin_memory operators. It also fixes ZeRO offload support under unmanaged gradient accumulation. The same JSON config can behave completely differently across versions—this is the root of many pitfalls below.

Locate First, Then Tune: Order of Five Checkpoints

This guide breaks problems into five checkpoints. Don't blindly change parameters when you hit an issue. Follow the order: ①ZeRO stage → ②offload target → ③gradient accumulation boundary → ④pinned memory → ⑤communication bottleneck. Each step has clear diagnostic signals:

Checkpoint	Typical Symptom	Common Cause	Diagnostic Signal
① ZeRO stage	VRAM insufficient or low utilization	Wrong stage choice, over/under sharding	Can a single GPU hold 16-bit parameter copy?
② offload target	VRAM still insufficient, but GPU utilization low	CPU offload introduces PCIe bottleneck	High host memory usage, idle GPU
③ Gradient accumulation boundary	Error or abnormal loss	Offload conflicts with gradient accumulation	Error messages, loss curve
④ Pinned memory	Initialization failure or abnormal host memory	`ulimit -l` limited or physical memory insufficient	`pin_memory` error, DMA failure
⑤ Communication bottleneck	Training stuck	NCCL sync wait, slow data loading	GPU utilization pattern, NCCL timeout

The key is to look at the signals at each step, not to change configs in a rush.

Checkpoint 1: zero-2 vs zero-3—Look at What Layer the Parameters Are Sharded First

ZeRO reduces memory usage by eliminating redundancy in data parallelism: ZeRO-1 shards 32-bit optimizer states; ZeRO-2 further shards 16-bit gradients; ZeRO-3 shards 16-bit model parameters as well, dynamically gathering and releasing them as needed during forward and backward passes.

So the choice between "zero-2 and zero-3" depends on: can your single GPU hold the 16-bit parameter copy? If yes, ZeRO-2 is more suitable with lower communication overhead. If not, you must use ZeRO-3, but it introduces extra communication/gathering overhead.

Stage	Sharded Content	Communication Overhead	Suitable Scenario
ZeRO-1	Optimizer states	Low	Model barely fits on one GPU
ZeRO-2	Optimizer states + gradients	Medium	Single GPU can hold parameters
ZeRO-3	Optimizer, gradients, parameters	High	Parameters don't fit on one GPU

Diagnosis: Use nvidia-smi to check per-GPU VRAM, then compare with model parameter size (16-bit). If parameters exceed single-GPU VRAM, choose ZeRO-3 decisively. For selection, you can also refer to RTX 4090 multi-GPU vs A100 single-GPU training selection to estimate single-GPU capacity.

Checkpoint 2: Offload Target Selection—When Is CPU Offload Worth It

offload_param offloads parameters to CPU, offload_optimizer offloads optimizer states to CPU. Many people enable ZeRO-3 CPU Offload and find "VRAM is still insufficient" because CPU Offload introduces PCIe transfer overhead. If batch size or communication overlap is misconfigured, the bottleneck simply moves from VRAM to the bus, and GPU utilization drops.

Diagnosis: If VRAM is sufficient but GPU utilization is low, first check host memory usage and PCIe bandwidth saturation. If VRAM is insufficient and host memory abundant, then consider offload. Offload only helps when the VRAM gap is small and CPU-to-GPU PCIe bandwidth isn't saturated. Otherwise, it's better to reduce batch size or move to a machine with larger VRAM.

Checkpoint 3: Gradient Accumulation and Offload Simultaneously—Boundary Issues

Gradient accumulation combined with offload often causes "gradient accumulation offload errors"—for example, ZeRO offload errors under unmanaged gradient accumulation. v0.19.5 fixed this and also fixed AutoEP ZeRO-1/2 conversion issues.

When encountering such errors, first check the version: if pre-v0.19.5, upgrade or switch to managed gradient accumulation; if new version, check whether gradient_accumulation_steps and offload_optimizer match.

Checkpoint 4: Host Pinned Memory and pin_memory—Why Old Configs Behave Differently

Many ask "should deepspeed pin_memory be enabled?" The answer is yes. In DeepSpeed config, offload_param and offload_optimizer's pin_memory option is enabled by default, using DMA for full-bandwidth asynchronous transfer between GPU and CPU, overlapped with computation. But this mechanism depends on the system locking physical memory. If the host is limited by ulimit -l (memlock) or has insufficient physical memory, it can cause initialization failure or abnormal host memory.

Note: This is different from DataLoader's pin_memory, which only affects data loading. Here it controls pinned memory allocation for parameter and gradient offloading. When configuring, explicitly set "pin_memory": true, and check whether host ulimit -l is sufficient.

Checkpoint 5: Why DeepSpeed Multi-GPU Training Gets Stuck

"deepspeed multi-gpu training stuck" has many causes; first distinguish three types of blockage:

Blockage Type	Observation Method	Configuration Direction
Data loading	Low GPU utilization, busy CPU	Increase `num_workers`, use DataLoader `pin_memory`
Collective communication sync	High GPU utilization but stalled, NCCL timeout	Check NCCL timeout settings and load balance across GPUs, verify gradient_accumulation_steps consistency
Offload write-back wait	High PCIe/host memory usage	Adjust offload async, batch size

Additionally, communication blockages often relate to network topology; refer to multi-GPU inference optimization for communication optimization ideas. For GPU utilization pattern analysis, see GPU utilization optimization.

Version Difference Check: Why Confirm DeepSpeed Version Before Following Tutorials

Before troubleshooting, run pip freeze | grep deepspeed to check version, and refer to the official Configuration JSON documentation to compare parameter meanings. If performance is abnormal after upgrade, run the same config on the old version for comparison.

Change Config or Change GPU: What Metrics to Record for A/B Testing

If you've gone through all five checkpoints and still can't locate the issue, determine whether it's a config problem or simply VRAM capacity. Method: fix ZeRO stage and batch settings, run the same config on a machine with larger VRAM or more GPUs, and record peak VRAM, per-step time, GPU utilization, and communication ratio.

Compare costs: changing config (reduce batch, change offload strategy) and time cost vs changing GPU (rent larger VRAM or more GPUs) financial cost. If you need quick validation, consider NexGPU pay-as-you-go multi-GPU servers. Rent a machine with larger VRAM or more GPUs for A/B testing, and use real data to decide upgrade path. For GPU selection, see How to Choose GPU VRAM.

Troubleshooting Checklist and Four Common Misjudgments

Here's the checklist from this guide. After running through all five checkpoints, verify each item:

[ ] Confirm DeepSpeed version, upgrade to v0.19.5 or pin version
[ ] ZeRO stage: Can a single GPU hold the parameters?
[ ] Offload target: Does CPU offload cause PCIe bottleneck?
[ ] Gradient accumulation: Any errors, config consistent?
[ ] pin_memory enabled, ulimit -l sufficient?
[ ] Communication blockage: Use nvidia-smi and stack to diagnose

Four common misjudgments:

Thinking ZeRO-3 CPU Offload can infinitely increase batch size—actually PCIe bandwidth may become the new bottleneck.
Thinking pin_memory only affects data loading—in ZeRO-Offload, it also controls pinned memory for parameters/gradients.
Thinking adding GPUs always improves throughput—communication overhead and load imbalance may offset compute gains.
Thinking errors are always due to insufficient VRAM—could be pinned memory limitation or communication timeout.

FAQ

Which is better: zero-2 or zero-3?

If a single GPU can hold the 16-bit parameter copy, use ZeRO-2 for lower communication overhead. If parameters don't fit, choose ZeRO-3, accepting higher communication/gathering overhead. First assess model size and per-GPU VRAM, then decide.

I enabled deepspeed zero3 offload but VRAM is still insufficient—what to do?

Check if offload actually took effect (look at offload parameters in logs), confirm pin_memory and ulimit -l settings, and consider reducing batch size. If host memory is sufficient but VRAM still overflows, try offloading all optimizer states.

Gradient accumulation and offload simultaneously cause errors—how to solve?

Check DeepSpeed version is v0.19.5 (update fixes ZeRO offload support under unmanaged gradient accumulation). If old version, upgrade or switch to managed gradient accumulation. Also check if gradient_accumulation_steps and offload configs match.

Should I enable pin_memory during DeepSpeed training?

Yes. offload_param and offload_optimizer's pin_memory is enabled by default, aiding DMA async transfer. But ensure host ulimit -l is sufficient and memory is abundant, otherwise initialization may fail.

Adding GPUs didn't improve throughput—why?

Common reasons: excessive communication overhead (like ZeRO-3 parameter gathering) or data loading bottleneck. Check if GPU utilization is balanced, try increasing batch size or reducing communication frequency (e.g., increase gradient accumulation steps).