The Methods Lab
Most of what is written about method teaches software: how to compute an answer. This lab is about the harder thing: how a claim is earned. Three demonstrations, each a place where a confident number quietly points the wrong way, and where judgment, not calculation, decides what is true.
Not how to compute an answer, but how to earn one.
Simpson's Paradox
Durkheim insisted a society is a reality of its own, never merely the sum of its members. Simpson's paradox is that claim rendered in data: an aggregate can tell the opposite story of its parts.
A question of my own: does social spending reduce inequality? Pool every society together and the answer looks like no. Account for the structure beneath the pool, and the sign reverses.
Illustrative data: societies under greater structural pressure (higher pre-redistribution inequality) tend to spend more, so the pooled correlation reflects who spends, not what spending does.
The pooled number was never wrong; it just answered a question no one asked. Structure is not a nuisance to remove; it is the finding.
The Ecological Fallacy
Robinson, 1950: across U.S. states, the more foreign-born residents, the higher the literacy. Person by person, immigrants were less literate. The state trend never described a single human being.
A relationship measured on groups cannot be carried down to the individuals inside them. Toggle between the two levels: the aggregate is tidy and positive; the people tell another story.
Illustrative reconstruction of Robinson's classic case. The confound: immigrants settled disproportionately where literacy was already high.
To measure a group is not to know its members. The unit you count silently decides what you are allowed to conclude.
The Garden of Forking Paths
Every dataset hides thousands of defensible analyses. Try enough of them and one will cross the line into "significance", not because something is there, but because you looked so many times.
Below is a variable with no real effect at all. Keep running analyses, a different subgroup, a different covariate, and watch a "discovery" appear from pure noise. Count how few tries it takes.
A live simulation of the null hypothesis: each analysis draws a p-value with no underlying effect. With twenty independent tries, a false positive is more likely than not: 1 − 0.95²⁰ ≈ 64%.
A single significant result is not evidence; the honest question is always how many paths were walked to reach it. Method is what you did before the number, not after.
Each of these failures wears the costume of rigor: a clean number, a confident conclusion. They are not errors of arithmetic but errors of judgment, and no amount of computation corrects them. Structure, level, and restraint are not the housekeeping of research. They are the research.
◑ The companion lab Enter the Inequality Lab →