Softmax & Cross-Entropy Calculator

Questions

What does softmax do?

It turns any list of scores into probabilities that add up to 1. Each score is raised as a power of e, then divided by the total, so higher scores get a larger share and the order stays the same.

Why does adding the same number to every score change nothing?

Softmax only depends on the differences between scores. That is also how it is computed safely: the largest score is subtracted first, so 1000, 1001, 1002 gives the same result as 0, 1, 2 without overflow.

What is cross-entropy loss?

It is minus the log of the probability given to the correct answer. A confident right answer costs close to 0, and a confident wrong one costs a lot. Mark the correct line with * to see it.

What is perplexity?

e to the power of the loss. It reads as "the model was as unsure as a fair choice between this many options". A perplexity of 1 means certain and correct.

Nats or bits?

Nats use the natural log, as most machine learning code does. Bits use log base 2. One nat is about 1.4427 bits.