Positional Encoding Visualizer
The sine and cosine pattern a transformer adds to each position, as numbers and a heatmap.
Questions
Why do transformers need positional encoding?
Attention treats its input as a set: shuffle the words and each word gets the same scores. Adding a different pattern to each position gives the model a way to tell first from last.
Why sine and cosine at different speeds?
The first dimensions change fast and tell nearby positions apart; the last change slowly and tell far ones apart. Together they give every position a unique pattern, like the hands of a clock.
What is the base of 10000?
It sets how slow the slowest wave is: its wavelength is 2 × pi × 10000, far longer than any sentence. Some newer models use a much larger base to handle longer texts.
Do modern models still use this?
Many now use RoPE (rotary position embedding), which applies the same sine and cosine waves as a rotation inside attention rather than adding them to the input. The waves on this page are the same ones RoPE uses.
Related tools
- Attention CalculatorScaled dot-product attention step by step: scores, softmax weights and output.
- Softmax & Cross-Entropy CalculatorTurn scores into probabilities, and see the loss and perplexity for the right answer.
- Cosine Similarity CalculatorCompare vectors or embeddings: cosine similarity, angle, dot product and distance.
- Quantization CalculatorWhat int8, int4 and int2 do to model weights: the scale, rounding and error.