Skeptic’s Toolkit
Where does it break?
Good-sounding science usually goes wrong at one specific sentence. The skill to build is finding that sentence: the point where an argument moves from what was shown to what would be nice if it were true.
Spot the leap
Each claim below begins on solid ground. Tap the first sentence that is no longer supported. There is no score, and a wrong tap shows you why.
The universal law of scale
The evidence scale
Label every claim you read with one of these. Most arguments break where a label quietly changes.
Tells of confident AI synthesis
Language models write interdisciplinary connections fluently because fluent connection is what they are trained to produce. These are the patterns to watch for, and what to do about each.
| Tell | How to check |
|---|---|
Precise citations with section or equation numbers you did not supply ("Smith et al. 2021, §4.3"). | Search the exact title on arXiv or Google Scholar. Open the paper and go to that section. If you cannot find it in two minutes, treat it as fabricated. |
Plausible author-name combinations: real-sounding names from the right field, paired with no link. | Ask for the arXiv id or DOI, then resolve it yourself. A missing or non-resolving identifier is a red flag. |
Two fields joined by "exactly as", "is equivalent to", or "is literally". | Ask for the equation in each field side by side. If none appears, or they only share vocabulary, it is Level 1 at most. |
Suspiciously round or conveniently famous numbers (¾, the golden ratio, e, "a 10× improvement"). | Trace the number to a table or figure in a real source. Famous constants are often attached to claims as decoration. |
"Proved", "established" or "it is well known" used for empirical or speculative claims. | Ask: "Is this a theorem, a measurement, or a conjecture? Who showed it, and how?" |
No limitations, caveats or dissenting views anywhere in the answer. | Ask: "What is the strongest evidence against this?" A real research area has an answer. |
The answer matches the framing of your question too well, confirming the connection you hinted at. | Re-ask neutrally in a fresh chat ("Are X and Y related? If not, say so.") or ask the model to argue the opposite. |
Quotes that sound like the author but cannot be found in the text. | Search the paper PDF for a distinctive 4–5-word phrase from the quote. |
Explanations that change when you regenerate, while staying equally confident. | Regenerate 2–3 times. Stable facts stay stable; confabulated details drift. |
Mechanism language ("this happens because…") with no experiment or derivation behind it. | Ask what experiment demonstrated the mechanism. If the answer is another mechanism story, stop. |
Prompt: audit an AI answer for fabrication →
The toolkit
Hypothesis vs evidence
Is this something someone proposed, or something someone observed?
A hypothesis is a candidate explanation; evidence is an observation that bears on it. Papers mix them, especially in abstracts and discussions. Look for verbs: "we measure / observe / find" usually reports evidence; "we propose / hypothesise / suggest / argue" marks a hypothesis. A paper can have strong evidence for a narrow finding and only a hypothesis for its headline interpretation.
Example
The Platonic Representation Hypothesis measures rising alignment between models and hypothesises that they converge toward a shared model of reality. The first is evidence; the second is the hypothesis.
Analogy vs mechanism
Does it look like it, share math with it, or work the same way?
Use the three levels. Analogy: it resembles. Shared mathematical structure: same equations or formal objects. Shared mechanism / tested prediction: the same causal process, or a transferred prediction that was tested and survived. Each level needs more evidence than the last. Most exciting cross-field claims are Level 1 being presented as Level 3.
Example
Softmax and the Boltzmann distribution share a formula (Level 2). That does not make a language model a gas (Level 3).
Correlation vs causation
If A and B move together, what else could make that happen?
Correlation can come from A causing B, B causing A, a third factor causing both, selection, or chance. Causal claims need interventions (experiments, ablations), natural experiments, or careful causal modelling. In ML, "ablation" (removing a component and measuring the change) is the closest everyday tool to an experiment.
Example
Models with more parameters score higher, but they were often also trained on more data and compute. Scale alone is not isolated unless the study controls those.
Model assumptions
What has to be true for this equation to apply?
Every model buys simplicity with assumptions: independence, infinite size, equilibrium, a fixed distribution, a smooth landscape. Find them (often in "we assume", footnotes, or appendices) and ask whether they hold in the case you care about. A model can be useful while being literally false.
Example
The argument for "emergence" assumes tokens are correct independently. The authors flag this; it is a simplification.
Falsifiability
What observation would show this is wrong?
A claim that fits every possible outcome explains nothing. Ask the claimant, or the model, what result would count against it. Vague terms ("intelligence", "consciousness", "universal") often make claims unfalsifiable until they are pinned down operationally.
Example
"Networks converge on shared structure" becomes testable only when you say which structure, measured how, and what amount of similarity counts.
Replication
Has anyone else found this, independently?
One result from one lab is a lead. Replication (same method, new data or team) and robustness (different methods, same conclusion) are what turn leads into knowledge. In ML, check whether results hold across seeds, datasets, architectures and labs.
Example
Neural scaling laws have been observed by many groups in many settings; the specific exponents in any one paper are setup-specific.
Significance vs effect size
Is it real, and is it big enough to matter?
Statistical significance asks whether an effect is distinguishable from noise; effect size asks how large it is. With enough data, trivial effects become "significant". With small data, large effects can be missed. Look for confidence intervals, error bars, number of seeds, and the absolute size of differences.
Example
A 0.3-point benchmark gain from a single run with no error bars is weak evidence of anything.
Confounding
Is a hidden third factor driving both?
A confounder influences both the supposed cause and the effect. Classic fixes: randomisation, controlling for the confounder, or comparing matched groups. In model comparisons, common confounders are training data, compute, tokenizer and evaluation setup.
Example
If models that share a weight subspace were fine-tuned from the same base model, the base model, not "universality", may explain the overlap.
Selection bias
How were these cases chosen, and what was left out?
Results depend on which samples entered the analysis. Survivorship (only successful models get published), convenience samples (whatever is on a model hub), and cherry-picked examples all distort conclusions. Ask what the denominator is.
Example
Models downloaded from a public hub are not a random sample of all trained models. They are the ones people chose to upload.
Overfitting
Does it work on data it has never seen?
A model, or a theory, with enough flexibility can fit anything, including noise. The test is performance on held-out data. In science, a theory tuned to explain past observations should also make new predictions that come true.
Example
A curve with 10 free parameters fitted to 12 points "explains" them perfectly and predicts nothing.
Extrapolation
Is this prediction inside or outside the range that was measured?
Fitted laws are trustworthy only within (and slightly beyond) the range of the data. Outside it, curves can bend, saturate or break. Loss cannot fall below the irreducible entropy of the data, so a pure power law must eventually fail.
Example
Extending a scaling curve fitted on models up to parameters to is a forecast, not a measurement.
Preprints and publication
Has this been reviewed, and does review even settle it?
arXiv preprints are not peer reviewed. In ML, many influential papers stay preprints, and conference review is fast and limited. Peer review checks plausibility, not truth. Treat a preprint as a claim with evidence attached, look for replications and responses, and check the version history.
Example
Kaplan et al. 2020 is an arXiv preprint; its compute-allocation advice was revised by Hoffmann et al. 2022.
Benchmark contamination & evaluation design
Could the model have seen the test? Was the comparison fair?
Contamination: test items, or close paraphrases, leaking into training data inflate scores. Evaluation design: prompt format, number of attempts, sampling temperature, answer extraction and metric choice all change results. Fair comparisons hold these fixed. Also ask whether the benchmark measures the construct it is named after.
Example
Exact-match versus partial-credit scoring can turn the same outputs into a smooth curve or a cliff.
How LLMs fabricate plausible syntheses
Did the model retrieve this, or compose it?
Language models generate fluent text that fits patterns, including the pattern of a scholarly citation. Cross-field syntheses are especially risky. The model can combine real terms from two fields into a confident claim no one has made, and attach author names, years and section numbers that look right. Fluency and specificity are not evidence. See the fabrication tells below.
Example
Asking "how is AdS/CFT related to attention?" invites the model to produce a connection, because the question assumes one exists.
Scientific disagreement vs pseudoscience
Are both sides playing by the rules of evidence?
Real scientific disputes share methods. Both sides publish data, make testable claims, update on evidence and engage with critiques (for example, the debate over "emergent abilities"). Pseudoscience shows different signs: claims that cannot be tested, persecution narratives instead of data, appeals to grand unification without math, no engagement with experts, and conclusions that never change.
Example
Researchers disagreeing about whether representations "converge" is normal science. A website claiming a single equation unifies consciousness, quantum gravity and AI, with no testable prediction, is not.
The full lesson is Module 8: The Scientific Skeptic’s Toolkit →