Fusion
Normally a matmul writes its result to HBM, softmax reads it back and writes it out again, then another matmul reads it once more. A fused kernel does all three steps while the data stays on chip.
ENTRY 01 / COMPUTING
“Kernel” is one word used for several unrelated things. All of them borrow the same image: the small, essential core inside a nut or seed. When people talk about “kernel optimisation” today, they nearly always mean the GPU kind.
FOUR MEANINGS
Linux
The core program that has full control of the hardware. It manages memory, schedules processes, and runs drivers. Everything else runs on top of it and has to ask it for access. Optimising it means faster system calls, scheduling, and I/O. It matters, but it isn’t what is driving the current interest.
FlashAttention, CUDA, Triton
This is what “kernel optimisation” usually refers to now. A GPU kernel is one function that gets launched across thousands of threads at once, each working on a slice of the data. Examples are a matrix multiply, a softmax, or an attention computation. A neural network’s forward pass is mostly a sequence of these kernels.
The kernel trick
A similarity function. It gives the dot product of two points after mapping them into some high-dimensional feature space φ, without ever computing φ. That shortcut is the kernel trick. It lets a linear method, a flat separating line, handle data that is curved, by working in a space where the curve becomes flat. Convolution kernels in image processing and integral kernels in mathematics are relatives of this meaning.
k(x, y) = ⟨φ(x), φ(y)⟩
The null space
The set of vectors a matrix sends to zero. It is unrelated to the others and just shares the name.
THE GPU KIND
On modern GPUs, arithmetic is cheap and moving data is expensive. A GPU can do math far faster than it can read data from its main memory, called HBM. A kernel’s speed depends mostly on how often it moves data between that slow memory and the small, fast on-chip memory: SRAM and registers.
Two kernels can compute exactly the same result and still differ in speed by 2–10× because they move data differently. Optimisation is choreography.
Bar length is room for data. Speed runs the other way: the short bars are the fast ones.
Normally a matmul writes its result to HBM, softmax reads it back and writes it out again, then another matmul reads it once more. A fused kernel does all three steps while the data stays on chip.
Split the matrices into blocks small enough to fit in SRAM, and reuse each block as many times as possible before evicting it.
Make memory accesses coalesced, keep the tensor cores busy, and avoid threads waiting on each other.
SAME MATH
THE STANDARD EXAMPLE
Plain attention builds the full N×N attention matrix and writes it to HBM, so memory traffic grows quadratically with sequence length. FlashAttention never builds that matrix in full. It processes attention tile by tile in SRAM and uses an online softmax to combine the partial results correctly.
Plain attention
N × N
The full attention matrix is materialised in HBM. Longer context makes that matrix, and the traffic, grow with the square of the sequence length.
FlashAttention
Tiles
The same attention, computed in SRAM-sized blocks. The math is identical, the output is exact, and the full matrix is never stored. Long context windows became practical because of this.
WHY IT IS EVERYWHERE
Training and serving large models costs huge amounts of GPU time, so a 20% faster attention or matmul kernel saves a lot of money. Tools like Triton and newer kernel DSLs make writing kernels easier than raw CUDA. Kernel writing has gone from a niche skill to a central part of the AI stack.