Model optimization terms often sound interchangeable, but they change different parts of the system and make different trade-offs. This interactive figure keeps one model and one set of readouts on screen so you can see what each technique actually changes.

Interactive figure

Change one setting at a time

One 7-billion-parameter model, drawn as 256 weights. Use the controls and watch parameter count, memory, latency, and retained quality move together.

a The weights

task
adapter
weight, light → dark = small → large pruned to zero distilled away

b What it costs

Parameters
7.00B
Weight memory
28.0GB
FP32 · 32 bits each
Latency / token
28.3ms
35 tokens per second
Quality kept
100%
task fit 41%
Where the milliseconds go
reading weights 17.5 ms arithmetic 6.3 ms launch overhead 4.5 ms
Decoding, one token at a time

c The compression pipeline

Open a step to see what it changes.

1 Distillation teacher 7B

Train a small student model to copy a large teacher. The student learns from the teacher’s probability distribution, not only the correct answer. You pay for a training run and get a permanently smaller model.

2 Pruning 0% removed

Delete the weights that matter least—here, the smallest ones. Unstructured pruning leaves holes scattered through the matrix, so it needs a sparsity-aware runtime to become faster. Structured pruning removes whole channels or heads and is easier for ordinary kernels to exploit.

3 Healing raw

A short retraining pass lets surviving weights cover some of what the deleted weights were doing. It usually uses a small calibration set and far fewer steps than the original training run.

4 Fine-tuning general

Keep training a finished model on your own data so it fits your task, tone, or format. A full fine-tune rewrites every weight. LoRA and other adapters freeze the base model and train a small extra matrix beside it—the teal strip. That is why the parameter count here does not move: you added roughly 0.5% more weights, and you can swap them per task.

5 Quantisation FP32

Store each weight in fewer bits, with a shared scale factor per group to map the small integer range back onto real values. Memory falls in exact proportion—and since decoding is limited by how fast weights can be read, latency follows. Look closely at the grid: the shades collapse into visible bands, because there are only many values left to round to. Below 8 bits you usually need calibration data or quantisation-aware training.

6 Optimisation & kernels stock graph

Everything that changes how the graph runs without changing what it computes: fusing operators into one kernel, folding constants, changing memory layout, picking a better kernel for this shape and this GPU, caching keys and values across steps. The maths is identical and the answer is identical—only the clock moves. It is the one step on this list that costs no quality at all, which is why it is done first in practice and last here.

Open a term to connect the controls above to the system underneath.

The act itself Model inference
Read explanation

Running the trained model forward to produce output. For a language model it splits in two: a prefill pass that reads the whole prompt at once and saturates the GPU’s maths units, then one decode step per output token, each of which must read every weight again to produce a single token. Decode is where the model spends its life, and it is starved for memory bandwidth rather than arithmetic—which is exactly why shrinking the weights helps.

The code that runs Kernel
Read explanation

A kernel is the actual program the GPU runs for one operation—a matrix multiply, a softmax, an attention block. One piece of maths can have dozens of kernels, and they differ enormously: the fast one keeps data in on-chip memory and does more work per trip to HBM. Fusing several operations into a single kernel removes the launches and the round-trips between them. FlashAttention is the famous example: same attention, one kernel, no giant intermediate matrix written out and read back.

The number that hurts Inference latency
Read explanation

Wall-clock time to get a result, and it is really two numbers: time to first token, set by prefill and how long you waited in a queue, and time per output token, set by how fast weights can be streamed. Latency is not throughput—batching many users together raises total tokens per second while making each individual wait longer. The bar in panel b is the whole story: bytes moved, maths done, overhead paid. Every technique above attacks one of those three.

Numbers are a simplified model of a 7B decoder on one accelerator at 1.6 TB/s, batch size 1. Real quality loss depends entirely on the model, the data and the calibration set—the point here is the direction each control pushes, not the digits.

The estimates are deliberately simplified. Real results depend on the model architecture, hardware, kernels, calibration data, sequence length, and workload. The useful part is the direction of each trade-off—not the benchmark value itself.