LLM Memory Calculator
How much GPU memory a model needs: weights plus KV cache, with the maths shown.
Memory
Examples
Questions
Why does context length change the memory needed?
Every token in the context keeps its keys and values for every layer, so they aren't recomputed. This KV cache grows with context length and batch size. For long contexts it can be as big as the weights, which is why many models use grouped-query attention.
Is this exact?
The weights and KV cache numbers are exact for what you enter. They leave out runtime overhead (CUDA context, activations, fragmentation), which is usually a few GiB. Quantised formats also store scales, which add a little.
Where do I find layers, KV heads and head dim?
In the model's config.json: num_hidden_layers, num_key_value_heads and head_dim (or hidden_size divided by num_attention_heads). The presets fill them in for common open models.
Related tools
- Quantization CalculatorWhat int8, int4 and int2 do to model weights: the scale, rounding and error.
- Token CounterCount tokens and see how text is split, with the real GPT-4o and GPT-4 tokenizers.
- LLM Sampling ExplorerSee what temperature, top-k, top-p and min-p do to a model's next-token choices.
- IEEE 754 Float InspectorThe bits and exact value of any float or double, such as 0.1 + 0.2.