The Inference Primerinductive inference · information · models

Sources

Bibliography

An annotated list, grouped so that it doubles as a reading order. Everything below is a public source; where a work is out of print or hard to obtain, that is noted, and where a body of literature stands somewhat apart from the mainstream, that is noted too.

A shelf of worn technical volumes with plain spines

Start here

  • Shannon, C. E. (1948). "A Mathematical Theory of Communication." Bell System Technical Journal 27, 379–423 and 623–656. The founding paper: the entropy measure, its uniqueness argument, the source and channel coding theorems. Readable, and shorter than its reputation suggests. Full text.
  • Stanford Encyclopedia of Philosophy, "The Problem of Induction". The canonical free survey; the bibliography attached to it is the fastest route into the philosophical literature.
  • Stanford Encyclopedia of Philosophy, "Semantic Conceptions of Information". Useful corrective to the assumption that "information" names one thing.

Entropy, thermodynamic and informational

  • Carnot, S. (1824). Reflections on the Motive Power of Fire. Available as a Dover reprint. The origin of the reasoning that led to the thermodynamic entropy.
  • Boltzmann, L. (1896–98). Lectures on Gas Theory. Dover reprint, 1995. Where entropy becomes a logarithm of a count of microscopic configurations — the bridge to Shannon's measure.
  • Jaynes, E. T. (1957). "Information Theory and Statistical Mechanics." Physical Review 106, 620–630. The maximum-entropy principle argued as a rule of inference rather than a physical law. The single most influential paper for the position described on the principles of reasoning.
  • Jaynes, E. T. (2003). Probability Theory: The Logic of Science. Cambridge University Press. Posthumously completed; opinionated, extremely clear, and the best long-form statement of probability as extended logic.
  • Cox, R. T. (1946). "Probability, Frequency and Reasonable Expectation." American Journal of Physics 14, 1–13. The derivation of the probability rules from consistency requirements on degrees of plausibility.
  • Kullback, S. (1959). Information Theory and Statistics. Wiley; Dover reprint. Where relative entropy becomes a working statistical tool.

Modern treatments of the measures

  • Cover, T. M. and Thomas, J. A. (2006). Elements of Information Theory, 2nd ed. Wiley. The standard graduate text; chapter 2 covers everything the missing information page uses.
  • MacKay, D. J. C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press. Made freely available by the author; unusually good at showing coding, inference and learning as one subject.
  • MIT OpenCourseWare, 6.441 Information Theory and 18.125 Measure and Integration. Free lecture notes and problem sets covering the technical background assumed here.

Model selection, description length and validation

  • Rissanen, J. (1978). "Modeling by shortest data description." Automatica 14, 465–471. The minimum description length principle: fit and complexity measured in the same units.
  • Akaike, H. (1974). "A new look at the statistical model identification." IEEE Transactions on Automatic Control 19, 716–723. An information-theoretic criterion that entered routine practice.
  • Stone, M. (1974). "Cross-validatory choice and assessment of statistical predictions." Journal of the Royal Statistical Society B 36, 111–147. The formal basis of the validation discipline described under pattern discovery.
  • Popper, K. (1959). The Logic of Scientific Discovery. The falsificationist alternative, still the sharpest statement of why survival of testing is not confirmation.

The entropy minimax literature

This body of work is the source of the specific proposals discussed on the principles of reasoning and of the case study. It stands somewhat apart from the mainstream statistical literature, is largely published outside the major journals, and its results have not been widely replicated by independent groups. It is listed here because it is the primary source for claims this primer discusses, and it should be read with that context in mind.

  • Christensen, R. (1964). Foundations of Inductive Reasoning. Berkeley, California. The earliest statement of the programme.
  • Christensen, R. (1980–1981). Entropy Minimax Sourcebook, four volumes: I, General Description; II, Philosophical Origins; III, Computer Implementation; IV, Applications. Entropy Limited, Lincoln, Massachusetts. Out of print and scarce; volume I is the one to find first.
  • Christensen, R. (1985, 1986). "Entropy minimax multivariate statistical modeling," part I (theory), International Journal of General Systems 11, 231–276; part II (applications), same journal 12, 227–305. The most accessible peer-reviewed statement of the method.
  • Christensen, R. et al. (1980). Reports on long-range precipitation forecasting prepared under United States water-resources research contracts. The primary source for the case study.
  • Christensen, R. (1985). "Seasonal precipitation forecasting with a 6–7 month lead time in the Pacific Northwest using an information theoretic model." Monthly Weather Review 113(4). The peer-reviewed publication of the forecasting work, and the easiest of these to obtain.
  • Applications reported in this literature also cover nuclear fuel performance under accident conditions and prognosis in oncology, published through the late 1970s and 1980s in engineering conference proceedings and technical reports.

Where the ideas are now

  • Della Pietra, S., Della Pietra, V. and Lafferty, J. (1997). "Inducing features of random fields." IEEE PAMI 19(4). Maximum-entropy models with automatically selected features — the direct methodological descendant, and the basis of a generation of language-processing systems.
  • Carcamo, D. P. et al. (2025). "Minimax entropy: the statistical physics of optimal models." A current treatment of the same maximise/minimise trade-off.
  • Exact minimax entropy models of large-scale neuronal activity (2024), arXiv:2402.00007. The machinery applied at modern scale.

Suggested order for a first pass: Shannon, then the Stanford entry on induction, then Jaynes (1957), then Cover and Thomas chapter 2, then Stone (1974). That sequence covers the measure, the philosophical problem, the inferential proposal, the modern formalism and the validation discipline — which is the whole argument of this site, in five sources. The orientation page maps them onto the pages here.