The Methods Lab
How a claim is earned.
Four demonstrations of places where a correct calculation leads to a wrong conclusion. In each, the error lies in the design or the reading of the analysis, and the arithmetic is fine.
All four use simulated data, so the true answer is known in advance. That is what makes the failure visible.
Simpson's Paradox
Durkheim held that a society is a reality of its own, distinct from the sum of its members. Simpson's paradox is a statistical version of that claim: an aggregate relationship can run opposite to the relationship inside every one of its parts.
Does social spending reduce inequality? Pool the societies below and the answer appears to be no. Then separate them by structural pressure, meaning the inequality they face before any policy acts. Societies under more pressure spend more and stay more unequal.
SimulatedThirty societies in three groups. The structure follows a known problem in comparative research: welfare effort responds to pre-existing inequality, so spending is endogenous to the outcome it is meant to affect.
The pooled slope is computed correctly and answers the wrong question. Deciding which comparison is meaningful is a theoretical decision made before any model is run.
The Ecological Fallacy
W. S. Robinson (1950) used the 1930 U.S. census. Across the 48 states, the share of foreign-born residents correlated with higher literacy. Among individuals, foreign birth correlated with lower literacy. Immigrants had settled in states where literacy was already high.
The simulation below reproduces Robinson's structure. Switch from states to people: the same population, counted at a different level, gives the opposite answer.
SimulatedEight states of thirty people each. Higher-literacy states attract more immigrants, and immigrants are less literate than the native-born within every state.
The unit of analysis limits the conclusion. Data on places support claims about places, and claims about persons need data on persons.
What 95% Confidence Means
Jerzy Neyman introduced the confidence interval in 1937 as a statement about a procedure: repeated many times, the method captures the true value at a known rate. It is among the most misread results in applied statistics.
Forty percent of this population holds a view. Draw surveys and compute an interval from each. Watch how often the intervals capture 40%, and notice that a missed interval looks no different from a successful one.
SimulatedSimple random samples from a population with a true proportion of 0.40. Intervals use the normal approximation, p̂ ± z·√(p̂(1−p̂)/n). The latest 60 intervals are shown.
The 95% describes the long run of the method. Any single interval either contains the true value or does not, and the data in hand cannot say which.
The Garden of Forking Paths
Andrew Gelman and Eric Loken borrowed the phrase from Borges in 2013. A researcher who chooses among many reasonable analyses after seeing the data can find significance in pure noise, without any intent to deceive.
Below is one simulated survey of 400 people. Nothing in it predicts the outcome. Each analysis tests one predictor in one subgroup, the kind of choice every dataset invites. Run them and see how soon a publishable result appears.
SimulatedOutcome and ten predictors are independent standard normal variables. Subgroups are random. Each p-value comes from a test of the correlation coefficient. Analyses that share data are correlated, so the independence figure is an approximation.
A p-value means what it claims only when the analysis was fixed in advance. The remedy is a pre-registered plan, or a full report of every path that was tried.
Each failure here produces a clean number and a confident sentence. Judgment about structure, level, and the number of tests performed decides whether that sentence is true.
Companion labThe Inequality Lab →The note beneath each chart states how its data were simulated.