The Inference Primerinductive inference · information · models

Foundations

Ways

A way in which a state can occur — a way, for short — is a state that is abstracted from no other state. It is the finest description in play, the point at which the modeller has stopped distinguishing.

The definition, with the standing example

The states male and female, in a description that draws no further distinctions, are abstracted from no other states; each is a way. The state male OR female is abstracted from both, so it is not a way. Every non-elementary state is a disjunction of ways, and counting the ways it covers is the elementary act on which everything else is built.

Note the phrase in play. Nothing in the world declares a description elementary. Male could be refined further by age, by height, by anything else the modeller chooses to record. A way is a way relative to the state-space that has been adopted; it is the floor of the current abstraction lattice, not a floor of nature. Physicists face exactly this when choosing what counts as a microstate, and the choice has real consequences for the entropies that follow.

Why the notion is needed

Three things become easy to say once ways are named.

  1. Probability becomes counting. When ways are equally likely, the probability of a state is the number of ways it covers divided by the total. This is the classical definition of probability, and it is stated cleanly only when "elementary case" has a name.
  2. Entropy becomes a statement about ways. The entropy of a state-space is large when its probability is spread across many ways and small when it is concentrated on few. On the thermodynamic side the same counting reappears as the number of microstates consistent with a macrostate.
  3. Refinement has a limit. A model cannot distinguish more finely than its ways, so the ways set a ceiling on what any inference built from the space can assert. Asking "what are the ways here?" is a quick way to find out whether a proposed model could possibly answer the question being asked of it.

Choosing the ways is a modelling decision

Because ways are relative to the description adopted, choosing them is a decision with consequences, and it is often made unconsciously — by the resolution of an instrument, the categories of a legacy database, or the bins someone chose years ago. Two considerations are worth applying deliberately.

First, ways should be distinguishable in practice. If two ways cannot be told apart by the measurement actually available, the distinction adds nothing but empty cells to every table built from them.

Second, ways should be similar in the respect that matters. The point of treating a set of objects as one way is the claim that, for the purposes of this model, they behave alike. When they do not — when a single way covers cases with genuinely different propensities — the model's probabilities are averages over a mixture, and it will be systematically wrong in both directions without ever looking wrong on aggregate.

The formal treatment of counting, elementary outcomes and their entropies is standard material; MIT OpenCourseWare's information theory course develops it from first principles for readers who want the mathematics.