Pattern discovery

Pattern discovery is the practical half of the method: given a record of cases, find conditions on the observed variables under which the distribution of outcomes differs materially from the base rate. It is powerful, it is old, and it is the stage at which honest work and self-deception look identical from the inside.

What a pattern is here

A pattern is a condition on the observed state-space — typically a conjunction of a few conditions on individual variables — paired with the conditional distribution over outcomes that holds when the condition is met. "Sea-surface temperature normal or high two summers ago AND precipitation moderate or low at a named station two years ago" is a pattern; "given that pattern, the probability of a wet year is such-and-such" is the inference it carries.

Written this way a model is a small set of rules, each with a probability and a count of supporting cases. That format has real advantages over a fitted surface: it is legible to a domain expert, its coverage is checkable case by case, and the thinness of the evidence behind any given rule is visible rather than absorbed into a global fit statistic.

The search, and why it is dangerous

The search space is combinatorial. With a few dozen candidate variables, each discretised into a handful of ranges, the number of expressible conjunctions runs to millions, and a search over millions of candidates will find some that look excellent on any dataset whatever — including one made of pure noise. This is not a flaw in a particular algorithm; it is arithmetic. The more candidates examined, the better the best one looks by luck alone.

Three consequences follow that no amount of care in the search can remove:

• The reported strength of a discovered pattern is biased upward, by an amount that grows with the size of the search.
• Ordinary significance tests do not apply to a pattern selected because it was the best of many; the p-value assumed the hypothesis was chosen in advance.
• Selection is contagious. Discretisation boundaries, variable transforms and outlier rules all count as part of the search if they were chosen with the outcome in view.

Independent validation, done properly

The only reliable corrective is to score the finished model on cases that took no part in producing it. Doing that honestly is harder than it sounds:

1. Withhold before, not after. The validation set is set aside before the search begins and is not consulted while decisions are made.
2. Withhold everything about those cases. Including their contribution to discretisation boundaries, means, normalisations and variable screening. Leakage through preprocessing is the commonest way a validation is quietly spoiled.
3. Respect the structure of the data. With time series, the split must be chronological; a model validated on interleaved years has been given the future. With clustered data — patients within hospitals, rods within assemblies — the split must be at the cluster.
4. Score once. A validation set consulted repeatedly, with model revisions in between, has become part of the training data. If it must be revisited, the count of revisits is part of the result and should be reported.
5. Report the interval, not just the point. A skill estimate from a few dozen validation cases carries a wide interval, and a result that is impressive as a point estimate is often unremarkable once the interval is shown.

Cross-validation automates some of this and is genuinely useful, particularly for choosing among a small number of candidate models. It is not a substitute for a set withheld from the entire process, because the cross-validation score itself becomes an optimisation target the moment the modeller starts choosing against it.

What a validated result looks like

Modest. A pattern that survives an honest trial usually shifts probabilities by tens of percentage points rather than transforming them, and covers a fraction of cases rather than all of them. The precipitation case study is instructive precisely on this point: three patterns, each moving the probability of a wet year appreciably away from the base rate, verified against years the model had not seen. That is what a real result looks like, and it is far less dramatic than the language sometimes used to describe such work.

The machinery has descendants. Maximum-entropy models with selected features are standard in natural language processing and in computational biology, and recent work on exact minimax entropy models takes the same trade-off into large-scale neural data. The vocabulary has changed more than the ideas have.