Start here

This page is the map. It says what the reference covers, in what order the material makes sense, and — just as important — what it does not claim.

What is here

The pages fall into four groups, and each group answers one question.

Foundations — what is a model made of?

States and state-spaces define the descriptive units. Inferences defines the act that a model performs. Abstraction and ways explain how coarse or fine a description may be, which turns out to be the choice that decides how much a model can say. Outcome names the thing a prediction is eventually checked against.

Measures — how is an inference scored?

Measure theory supplies the machinery in three paragraphs rather than three chapters. Entropy introduces Shannon's measure and separates it carefully from the thermodynamic quantity that shares its name. Missing information derives the conditional entropy and explains why it can be read as the distance an inference still has to travel before it becomes a deduction.

Method — how should a model be chosen?

The problem of induction sets out the difficulty in its classical form and surveys the main responses. The principles of reasoning examines the proposal that model selection should be governed by an explicit pair of optimisation principles rather than by heuristics. Pattern discovery covers the practical machinery — partitions, features, cross-validation — and the ways it goes wrong.

Case study and sources

The long-range precipitation model is a worked historical example of the whole method, chosen because it was validated on data held back from the fitting. The bibliography is annotated, so it doubles as a reading order.

Two routes through the material

The short route — about an hour, no mathematics beyond arithmetic:

1. The overview
2. States and state-spaces and inferences
3. Entropy, skipping the derivations
4. The case study

The full route follows the argument in order: Foundations, then Measures in the sequence measure theory → entropy → missing information, then Method, then the case study. Read this way, each page uses only terms already defined, and the bibliography at the end becomes a set of onward paths rather than a wall of references.

What this reference claims, and what it does not

It claims that the vocabulary set out here is useful, that Shannon's measure is a defensible way of scoring an inference, and that making an inductive commitment explicit is better than leaving it implicit in a modeller's taste.

It does not claim that any of this solves the problem of induction. Choosing to maximise entropy subject to constraints is itself an inductive commitment; it is a well-motivated one that can be stated, criticised and tested, which is precisely why it is worth having, but it is not a proof. Readers will find that some of the primary literature is bolder than this site is. Where the sources and the surrounding scholarly consensus diverge, the divergence is noted rather than smoothed over.

Readers who want the conceptual background from a source with a formal editorial process should begin with the Stanford Encyclopedia entry on semantic conceptions of information, which is a good corrective to any single school's account.

Conventions used here

• Technical terms are defined on their own page and linked at first use in each essay.
• Notation is kept light: X for an observed state-space, Y for an unobserved one, Sh(·) for Shannon's measure.
• Historical claims are attributed to a source. Where a result is reported in one body of literature but has not been widely replicated elsewhere, that is said plainly.
• Original page addresses have been kept exactly as they were, including their spacing and capitalisation, so that old citations and links continue to resolve.