Inductive inference, information and the building of models

This site is a plain-language primer on a single question that runs underneath statistics, machine learning and the philosophy of science alike: when a body of evidence does not entail a conclusion, on what principle do we choose one conclusion rather than another? That question is old, it is unsolved in the strong sense, and it is answered in practice every day by anyone who fits a model to data.

Two branches of logic, and the gap between them

Deductive logic is the branch that carries certainty forward. If the premises hold and the argument is valid, the conclusion holds too; nothing is added and nothing is risked. Deduction is what most of us are actually taught, and it is the part of logic that has been in tolerable order since Aristotle.

Almost nothing in empirical science works that way. A weather model, a reliability model for a reactor component, a prognosis model in oncology, a spam filter: each one looks at what has been observed and extrapolates to something that has not been observed. That step is inductive, and it is not truth-preserving. However good the record, the next case can differ. The gap between what the evidence entails and what the model asserts is the subject of this primer, and it is treated at length on the page devoted to the problem of induction.

The practical consequence of the gap is that model builders need a rule for choosing among candidate conclusions. Historically the rule was taste: simplicity, elegance, familiarity, the habits of a discipline. Those are heuristics. They often work, they are sometimes indefensible, and they cannot be audited, because there is no quantity being optimised and therefore nothing to check.

A vocabulary precise enough to argue with

Progress on this question begins with vocabulary. Ordinary talk about "predictions" and "variables" is too loose to support an argument, so the pages in the Foundations group set out a small, deliberately austere set of terms:

• A state is a description of a set of real objects — a proposition, and therefore something logic can operate on. A state-space is a set of states that is complete and exclusive: exactly one of them describes the objects at any given instant.
• An inference is an extrapolation from a state in an observed state-space to a state in an unobserved state-space, where both describe the same objects. From cloudy to rain within 24 hours is an inference; the two state-spaces describe the same patch of sky.
• An abstraction is a description reached by disjoining more detailed descriptions, and a way is a state that abstracts from nothing further — the grain at which description stops.
• An outcome is the state that actually obtains in the unobserved state-space when the event resolves.

Once a model is described in these terms, a striking thing becomes possible: inferences can be measured. That is the subject of the Measures group, and it is where information theory enters.

Measuring an inference

Claude Shannon's 1948 paper gave communication engineering a measure of uncertainty, and gave epistemology something it had not had: a defensible number attached to an act of inference. Given an observed state-space X and an unobserved state-space Y, the entropy of Y measures how much is unsettled before the observation, the mutual information measures how much the observation settles, and the difference — the conditional entropy — measures what is still missing from a deductive conclusion.

That last quantity is the interesting one. It is zero exactly when the inference has become a deduction, and it is at its maximum when the observation was irrelevant. Between those poles it grades every inference on a single scale. Anyone wanting the mathematics rather than the prose will find it developed properly in MIT OpenCourseWare's information theory course; the pages here explain what the quantities mean and why they were thought to bear on induction.

From a measure to a method

A measure invites optimisation. If inferences can be scored, then choosing a model stops being a matter of taste and becomes a matter of extremising something — and the choice of what to extremise becomes the substantive claim.

Two such choices dominate the literature. The maximum entropy principle says that, among distributions consistent with what is known, one should adopt the one with the greatest entropy: assume nothing that the constraints do not force. Its complement says that features and partitions should be chosen so as to minimise the entropy that remains — that is, to extract as much structure as the data genuinely support. Applied together, under the label entropy minimax, they were used from the mid-1970s onward to build predictive models in engineering, medicine and meteorology. The family resemblance to minimum description length, to Bayesian model selection and to modern regularisation is close and is discussed on the relevant pages.

The honest caveat belongs here rather than in a footnote. Optimising an information measure does not dissolve the problem of induction; it makes the inductive commitment explicit and auditable. That is a real gain, and it is a smaller gain than the strongest claims made for it.

Why validation is the load-bearing step

Any method flexible enough to find patterns is flexible enough to find patterns that are not there. The discipline that separates a model from a coincidence is independent validation: hold data back, build without it, and let the held-back data score the finished model. A pattern that survives that test has earned something; a pattern reported on the data that suggested it has earned nothing.

The long-range precipitation case study is included precisely because it was done that way — the reported skill came from a validation trial, not from the fitting sample — and because it shows both what the method can deliver and how modest a well-earned result looks.

How to read this site

Start with the orientation page, which suggests two routes through the material: a short conceptual path for readers who want the ideas, and a longer path that follows the argument in order. Every technical term has its own page, kept deliberately short, so that the longer essays can lean on them without restating them. The bibliography lists the primary sources, and the Stanford Encyclopedia of Philosophy's survey of the problem of induction is the best free starting point for the philosophical side.