PQlet blog

[NOTES] QA on distributed communications & parallelisms (DP,PP,TP,CP,EP,FSDP)

Q1: Why use TP+PP if I can use just 1 knob ?
A1: For example,

  1. If a TP degree alone does not divide hid_dim evenly, i.e. dim=32 TP=6. I have 6 gpus only, but I can't set TP=6. Hence, you use TP=2 PP=3.
  2. If the model is too large for a single node, combine tensor parallelism with pipeline parallelism. Set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes. For example, set tensor_parallel_size=8 and pipeline_parallel_size=2 when using 2 nodes with 8 GPUs per node. (VLLM docs)

Q2: Is TP faster, than PP ? How to choose between them if I can select only 1 ?
A2: See ...

Q: Why PP has less communications per forward pass, than TP ?
A:

  1. PP communications are Point-to-Point (P2P).
  2. PP communications send only output activations of a pipeline-stage during forward.
  3. PP communicates less, than TP, already if pipeline-stage is 1 transformer block or bigger. TP splits individual layers across GPUs, requiring communication on every layer's forward and backward pass.

Q:Why inter-node TP is slow ?
A:

  1. Slower speed of inter-node communications, compared to intra-node (within node).
  2. TP requires frequent communication of activations within each layer. Subsequent dependent operations must wait for these results, limiting compute–communication overlap. Cross-node links make these waits more expensive. ...

Q: What is GPUDirect RDMA ? Why ?
A: GPUDirect RDMA (Remote Direct Memory Access) is an NVIDIA technology that allows network adapters to directly access GPU memory, bypassing the CPU and system memory. This direct access reduces latency and CPU overhead, which is beneficial for large data transfers between GPUs across nodes.(VLLM docs)