This is a short, honest writeup of one measured finding: scaling DNABERT-2 fine-tuning across 1 → 4 → 8 GPUs with PyTorch DDP, and using Score-P and Vampir to ask where the wall-clock time actually goes - compute, or inter-GPU communication.

(The practical side - eleven errors I hit getting Score-P to trace a DDP run at all - is a separate field guide on dev.to.)

The question

When DNABERT-2 fine-tuning is scaled across GPUs with PyTorch DDP, where does the time go - compute, or inter-GPU communication? And does that balance shift as the GPU count grows?

Method

Each DDP rank was launched under its own Score-P measurement (python -m scorep --cuda), so every worker’s GPU activity - compute kernels and NCCL communication kernels - was captured in a per-rank OTF2 trace. Traces were collected for 1, 4 and 8 GPUs (50 training steps each, CUDA kernel/memcpy/sync tracing enabled) and analysed both as text (scorep-score) and visually in Vampir 10.8.

What the traces show

1. Communication appears only when you scale out. On 1 GPU the trace contains zero NCCL kernels - a clean, communication-free baseline. From 2 GPUs onward, ncclKernel_AllReduce_RING_LL_Sum_float (the gradient synchronisation) appears and grows into the dominant GPU activity.

2. At 8 GPUs, gradient-sync communication costs as much GPU time as the entire backward pass.

The single AllReduce kernel is the largest GPU activity in the run - on par with the whole backward pass.

3. …yet that communication is largely hidden, which is why scaling stays efficient.

The Vampir Master Timeline shows two CUDA streams running concurrently: the default stream (CUDA[0:7], CUDA_NULL_STREAM) carrying the forward/backward compute kernels (gemm, attention, element-wise), and a separate stream (CUDA[0:20]) carrying the ncclKernel_AllReduce kernels. The AllReduce blocks sit at the same time positions as the compute kernels: PyTorch DDP overlaps gradient AllReduce with backward computation by placing them on separate CUDA streams.

So although communication is large in kernel-time, much of it is overlapped behind compute and therefore costs little additional wall-clock. This is consistent with the measured end-to-end throughput, which scales near-linearly:

The point

A higher-level metric - throughput, or job-level monitoring at ~30 s sampling - can only tell you that scaling is efficient. It cannot show you why. Fine-grained tracing with Score-P localises the cost to a specific kernel (ncclKernel_AllReduce), quantifies it (≈ backward-pass magnitude), and - through the Vampir stream timeline - reveals that it is hidden behind compute. That decomposition is invisible to monitoring and is exactly what trace-based performance analysis exists to provide.

Honest caveats (stated for rigour)

• NCCL’s low-latency (LL) kernels busy-wait on peer flags, so their measured GPU-resident time includes synchronisation wait, not only data transfer - the absolute seconds overstate pure communication.
• Because compute and communication overlap on separate streams, summed kernel-time double-counts the overlapped portion; the exposed (wall-clock) cost is smaller and is what the timeline reveals.
• These are 50-step diagnostic runs on a shared node; absolute kernel times carry contention noise. The robust signals are qualitative and relative: NCCL absent at 1 GPU → dominant by 8 GPUs, overlapped with backward compute.

Traces generated on a SLURM cluster (A100-SXM4-40GB). The companion field guide on making Score-P and PyTorch DDP coexist is on dev.to.

Headings are cool

You can have many headings

Aren’t headings cool?

Headings are cool

You can have many headings

Aren’t headings cool?

Headings are cool

You can have many headings

Aren’t headings cool?

Headings are cool

You can have many headings

Aren’t headings cool?