FSDP

of a series about distributed AI across multiple GPUs: Introduction Within the previous post, we saw how Distributed Data Parallelism (DDP) hastens training by splitting batches across GPUs. DDP solves the throughput problem, however it...