The principles of reasoning
Scientists are trained at length in deduction and almost not at all in induction, yet models are built inductively. One response to that gap has been to propose an explicit pair of principles to govern model construction, in place of the heuristics that otherwise fill it.
The pair
Stated in prose, the two principles are:
- Maximise the knowledge. Choose the description and the model that extract as much structure from the record as the record genuinely supports — equivalently, that leave as little missing information as possible about the unobserved space.
- Keep the inferences honest. Within whatever the evidence and the constraints fix, assume nothing further: adopt the distribution of greatest entropy consistent with what is known, so that no structure is asserted that the evidence did not put there.
Taken together — the label used for the pair in the older literature is ultra-optimization, and the associated method is entropy minimax — they pull in opposite directions, and that is the point. The first alone would drive a modeller to ever finer descriptions and ever more elaborate patterns, ending in a model that reproduces the record exactly and predicts nothing. The second alone would drive the modeller to the uniform distribution, honest and empty. The optimum is where the marginal structure extracted stops being supported.
The maximum-entropy half
The second principle has an independent and well-developed pedigree. Given constraints — a known mean, a known set of feature averages — the maximum-entropy distribution is the one that satisfies them while committing to nothing else. It has a clean characterisation, it recovers many familiar distributions as special cases, and it has an appealing rationale: any distribution of lower entropy asserts structure beyond the constraints, and that structure came from the modeller rather than the evidence.
It is not a free lunch. The choice of which constraints to impose is itself substantive, and maximum entropy has nothing to say about it; the principle disciplines what happens after the modelling choices, not the choices themselves.
The minimisation half
The first principle is where the discipline-specific work happens. Extracting structure means choosing features and partitions of the observed space — deciding which conditions to condition on. Each candidate partition yields a conditional entropy, and the search prefers partitions that leave less. Because a sufficiently fine partition can always drive the apparent residual to zero, the search must be penalised, cross-validated or otherwise held in check, exactly as with any other capacity-controlled model class.
Readers who know minimum description length will recognise the shape: a term rewarding fit and a term charging for complexity, with the units chosen so that the two are commensurable. The same structure recurs in regularised regression, in Bayesian model selection through the marginal likelihood, and in modern minimax-entropy work in statistical physics — a recent treatment shows the family is still active. The convergence of these traditions on the same trade-off is itself evidence that the trade-off is the real content.
What the pair settles, and what it does not
It settles: that the inductive commitment is written down; that two candidate models can be compared on a single stated quantity; that "we preferred this one because it is more elegant" is no longer an argument.
It does not settle: the problem of induction. Choosing to maximise entropy subject to constraints is a commitment about how the world is likely to be, and it is defended by argument rather than proved. Nor does it settle the choice of state-space, of constraints or of features, all of which precede the optimisation and all of which can decide the result. And it does not remove the need for validation on withheld data, because a search flexible enough to find structure is flexible enough to find structure that is not there.
How the claim should be reported
The older literature in this area sometimes states the case in strong terms — that logic has been completed, that the principles of reasoning have been discovered. Those formulations have not been taken up by the wider philosophical or statistical community, and the specific results reported in that literature have not been broadly replicated outside it.
The defensible version is worth having on its own account: an explicit, information-theoretic criterion for model choice, applied uniformly and validated on held-out data, is better practice than undocumented taste — and it is close kin to methods that are now standard, including Bayesian model comparison. The way the machinery is used in practice is the subject of pattern discovery.