Attention Calculator
Scaled dot-product attention step by step: scores, softmax weights and output.
Questions
What does attention compute?
For each query, a score against every key (their dot product), scaled by 1/sqrt(d). A softmax turns each row of scores into weights that add up to 1. The output is those weights applied to the value rows.
Why divide by the square root of d?
Dot products grow with the vector length d. Without scaling, large scores push the softmax towards one token and the gradients vanish during training. Dividing by sqrt(d) keeps the scores in a useful range.
What does the causal mask do?
It sets the score of every later token to minus infinity before the softmax, so its weight is 0. A token can only look back, which is how GPT-style models generate text one token at a time.
How do I type the matrices?
A line with Q:, then one row per line, then the same for K: and V:. Numbers can be split by spaces or commas. Start a row with a label and a colon, like "cat: 0 1", to name that token in the tables.
Is this a real model?
No. It is the exact maths of one attention head on the numbers you type. Real models learn Q, K and V from the input and run many heads in parallel.
Related tools
- Softmax & Cross-Entropy CalculatorTurn scores into probabilities, and see the loss and perplexity for the right answer.
- Cosine Similarity CalculatorCompare vectors or embeddings: cosine similarity, angle, dot product and distance.
- LLM Sampling ExplorerSee what temperature, top-k, top-p and min-p do to a model's next-token choices.
- LLM Memory CalculatorHow much GPU memory a model needs: weights plus KV cache, with the maths shown.
- Pathfinding VisualizerDraw walls and watch A*, Dijkstra, BFS and more find a path, or race two.