FlashAttention Primer: IO-Aware Attention, from FA1 to FA4
FlashAttention Primer: IO-Aware Attention, from FA1 to FA4 The attention layer is the main bottleneck when scaling Transformers to long sequences: its time and memory cost grow quadratically with sequence length. FlashAttention is a family of algorithms that keeps attention exact while making it fast and memory-efficient, by treating GPU memory traffic — not just FLOPs — as the resource to optimize. This primer walks through the core ideas with original diagrams and derivations: the GPU memory hierarchy, tiling, online softmax, backward recomputation, then the FA1 → FA2 → FA3 → FA4 timeline, the difference between the standalone flash-attn package and PyTorch’s scaled_dot_product_attention, and the hardware caveats that matter in practice.