Flash attention
An attention implementation that avoids materializing the full attention matrix, saving memory and time.
Standard attention builds an n-by-n matrix over the sequence, which grows quadratically and quickly dominates memory at long context. Flash attention computes the same result in tiles without ever holding that matrix, trading a little recomputation for a large memory saving.
It is on by default in most current runtimes and is generally worth leaving on. Its effect is on the working buffers rather than the KV cache, so it does not change the cache figures on this site.