Entropy

Of all the terms in this primer, entropy is the one most likely to arrive already loaded. It carries a thermodynamic history, a popular gloss ("disorder") that is at best a metaphor, and a precise information-theoretic definition that is what is meant here.

The information-theoretic definition

Given a state-space and a probability distribution over its ways, the entropy of the space is the expected value of the logarithm of one over the probability of the way that occurs. In plain terms: it is the average surprise. If one state is nearly certain, the average surprise is near zero. If all states are equally likely, the average surprise is at its maximum for that number of states.

Claude Shannon introduced the quantity in 1948 to answer an engineering question — how few symbols suffice to transmit a message without loss — and showed that this expression is essentially the only one satisfying a short list of reasonable requirements: it should be continuous in the probabilities, it should increase with the number of equally likely alternatives, and it should be consistent when a choice is broken into a sequence of sub-choices. That uniqueness result is what turns entropy from a convenient formula into a principled measure. The argument is compact and readable in Shannon's original paper.

What entropy is a measure of

It is a measure of a distribution, not of an object. There is no entropy of a rainfall total; there is an entropy of a distribution over rainfall bands. Change the state-space — coarsen the bands, split them — and the entropy changes, because the question being asked has changed. This is not a defect. It is the formal expression of the point made under abstraction: how much there is to know depends on how finely one has chosen to ask.

Two consequences follow immediately, and both are used later:

• Entropy is maximal for the uniform distribution over a given space, and zero when one state has probability one. It therefore grades a space from "settled" to "wide open" on a single scale.
• Because measures are additive over disjoint sets, entropies decompose. The entropy of a joint space splits into the entropy of one component plus the conditional entropy of the other given it — the identity that the next page turns into a statement about inference.

The thermodynamic cousin

The word came first from thermodynamics. Rudolf Clausius coined it in the nineteenth century from the Greek for transformation, to name a quantity that increases in irreversible processes. Ludwig Boltzmann later connected it to counting: the entropy of a macroscopic state is proportional to the logarithm of the number of microscopic configurations consistent with it.

Once put that way, the kinship with Shannon's measure is obvious — both are logarithms of a count of ways, or of an average over them. And the kinship is more than notation: a thermodynamic entropy can be read as the information missing about the microscopic state of a body when only its macroscopic state is known. That reading, developed at length in the maximum-entropy tradition, is one of the reasons information measures were taken seriously as tools for inference generally rather than for communication only.

Two cautions, though. The identification is an interpretation, defended by argument, not a definition either subject imposes on the other; and the popular gloss of entropy as "disorder" is a poor guide in both fields. A crystal at low temperature and a well-shuffled deck are not usefully compared by tidiness. "How much is still unsettled, given what is specified" travels much further and is closer to what the mathematics says.

Units, and why they hardly matter

The base of the logarithm fixes the unit: base two gives bits, base e gives nats. Nothing conceptual turns on the choice, and every relation in this primer holds in any base, provided one base is used throughout. Bits are convenient because a fair coin has entropy one, which gives an intuitive yardstick: an inference that removes three bits has done as much work as three fair coin flips would have to do to settle the question.

Readers wanting the modern formal development, including the source-coding and channel-coding theorems that gave these quantities their operational meaning, will find it in MIT OpenCourseWare's information theory course.