Researcher. All projects ↗

ENTRY 01 / COMPUTING

Kernel.

“Kernel” is one word used for several unrelated things. All of them borrow the same image: the small, essential core inside a nut or seed. When people talk about “kernel optimisation” today, they nearly always mean the GPU kind.

FOUR MEANINGS

The same word.

  1. 01

    Operating system kernel

    Linux

    The core program that has full control of the hardware. It manages memory, schedules processes, and runs drivers. Everything else runs on top of it and has to ask it for access. Optimising it means faster system calls, scheduling, and I/O. It matters, but it isn’t what is driving the current interest.

  2. 02

    Compute kernel

    FlashAttention, CUDA, Triton

    This is what “kernel optimisation” usually refers to now. A GPU kernel is one function that gets launched across thousands of threads at once, each working on a slice of the data. Examples are a matrix multiply, a softmax, or an attention computation. A neural network’s forward pass is mostly a sequence of these kernels.

  3. 03

    SVM / math kernel

    The kernel trick

    A similarity function. It gives the dot product of two points after mapping them into some high-dimensional feature space φ, without ever computing φ. That shortcut is the kernel trick. It lets a linear method, a flat separating line, handle data that is curved, by working in a space where the curve becomes flat. Convolution kernels in image processing and integral kernels in mathematics are relatives of this meaning.

    k(x, y) = ⟨φ(x), φ(y)⟩

  4. 04

    Linear algebra kernel

    The null space

    The set of vectors a matrix sends to zero. It is unrelated to the others and just shares the name.

THE GPU KIND

Why these kernels can be optimised so much.

On modern GPUs, arithmetic is cheap and moving data is expensive. A GPU can do math far faster than it can read data from its main memory, called HBM. A kernel’s speed depends mostly on how often it moves data between that slow memory and the small, fast on-chip memory: SRAM and registers.

Two kernels can compute exactly the same result and still differ in speed by 2–10× because they move data differently. Optimisation is choreography.

RegistersFastest. A handful of values per thread.
SRAMFast, on chip. Big enough for a tile you can reuse.
HBMMain memory. Huge, and the trip you pay for.

Bar length is room for data. Speed runs the other way: the short bars are the fast ones.

Fusion

Normally a matmul writes its result to HBM, softmax reads it back and writes it out again, then another matmul reads it once more. A fused kernel does all three steps while the data stays on chip.

Tiling

Split the matrices into blocks small enough to fit in SRAM, and reuse each block as many times as possible before evicting it.

Hardware fit

Make memory accesses coalesced, keep the tensor cores busy, and avoid threads waiting on each other.

SAME MATH

Where the intermediate values go

ON-CHIP SRAM
  1. Matmul
  2. Softmax
  3. Matmul
HBM 1 write, when the result is done
One fused kernel keeps the intermediate values in SRAM. HBM is written once, when the result is finished.

THE STANDARD EXAMPLE

FlashAttention.

Plain attention builds the full N×N attention matrix and writes it to HBM, so memory traffic grows quadratically with sequence length. FlashAttention never builds that matrix in full. It processes attention tile by tile in SRAM and uses an online softmax to combine the partial results correctly.

Plain attention

N × N

The full attention matrix is materialised in HBM. Longer context makes that matrix, and the traffic, grow with the square of the sequence length.

FlashAttention

Tiles

The same attention, computed in SRAM-sized blocks. The math is identical, the output is exact, and the full matrix is never stored. Long context windows became practical because of this.

WHY IT IS EVERYWHERE

Time on the GPU is the bill.

Training and serving large models costs huge amounts of GPU time, so a 20% faster attention or matmul kernel saves a lot of money. Tools like Triton and newer kernel DSLs make writing kernels easier than raw CUDA. Kernel writing has gone from a niche skill to a central part of the AI stack.