FOUNDERFILES N°005 · THE READING LIST
The Compression Library
Sutskever’s famous reading list isn’t a periodic table of AI. It’s a compression theory with two blank columns, value functions and continual learning, and that’s where he says SSI is working.
The hairline on the back of each card is its code length: the size of that card’s note after LZ78 compression, in bits. Vacant cards have nothing to compress.
The Library: eight shelves, 27 volumes
Represent
REP · 5 volumes
How a network turns raw input into features: convolution and depth.
REP-01 Paper
ImageNet Classification with Deep Convolutional Neural Networks
2012 · Represent
REP-01 Note
GPUs, ReLU and dropout together; the start of the age of research.
Why it’s here
Learned features beat hand-built ones: the network finds its own short description of an image instead of being handed one.
REP-02 Paper
Multi-Scale Context Aggregation by Dilated Convolutions
2015 · Represent
REP-02 Note
An exponentially larger receptive field without losing resolution.
Why it’s here
More context per parameter: the model sees farther for the same number of bits.
REP-03 Paper
Deep Residual Learning for Image Recognition
2015 · Represent
REP-03 Note
Skip connections make very deep networks trainable.
Why it’s here
Each block only has to encode what the identity map gets wrong, which is a cheaper thing to learn.
REP-04 Paper
Identity Mappings in Deep Residual Networks
2016 · Represent
REP-04 Note
Clean identity paths for signal propagation.
Why it’s here
Keeping the shortcut clean lets depth add description without scrambling the signal underneath it.
REP-05 Course
CS231n: Convolutional Neural Networks for Visual Recognition
Course · Represent
REP-05 Note
Backprop, CNNs and the practice of training, taught end to end.
Why it’s here
The working toolkit: how to fit and regularize a model before any theory of why it generalizes.
Route
RTE · 4 volumes
Deciding which information flows where: attention and pointers.
RTE-01 Paper
Neural Machine Translation by Jointly Learning to Align and Translate
2014 · Route
RTE-01 Note
The original soft attention.
Why it’s here
Instead of squeezing a sentence into one fixed vector, the decoder looks up only what it needs: a smarter bottleneck.
RTE-02 Paper
Pointer Networks
2015 · Route
RTE-02 Note
Attention used as the output: pointing at positions in the input.
Why it’s here
Output by reference: pointing at something already in the input is a far shorter code than writing it out again.
RTE-03 Paper
Attention Is All You Need
2017 · Route
RTE-03 Note
Attention, residuals and normalization as one general-purpose block.
Why it’s here
One reusable routing block, later scaled into the best text compressors anyone has built.
RTE-04 Code
The Annotated Transformer
2018 · Route
RTE-04 Note
A line-by-line, executable Transformer; the list’s only learn-by-building item.
Why it’s here
The theory made runnable: you learn the compressor by writing it yourself.
Remember
MEM · 4 volumes
Holding state across time: gates and external memory.
MEM-01 Essay
The Unreasonable Effectiveness of Recurrent Neural Networks
2015 · Remember
MEM-01 Note
Character-level prediction produces structure.
Why it’s here
Predicting the next character is compressing the text, and structure shows up because predicting well requires modeling it.
MEM-02 Essay
Understanding LSTM Networks
2015 · Remember
MEM-02 Note
Gating as learned memory control.
Why it’s here
Gates decide what to keep and what to forget: learned control over a finite memory budget.
MEM-03 Paper
Neural Turing Machines
2014 · Remember
MEM-03 Note
Differentiable external memory with content-based addressing.
Why it’s here
Separating the program from its storage lets a short controller work over a long memory.
MEM-04 Paper
Relational Recurrent Neural Networks
2018 · Remember
MEM-04 Note
Memory slots that interact through multi-head attention.
Why it’s here
Memories that attend to one another, so stored facts can be compared and not just recalled.
Relate
REL · 3 volumes
Reasoning over entities and the relations between them.
REL-01 Paper
Order Matters: Sequence to Sequence for Sets
2015 · Relate
REL-01 Note
Ordering changes what seq2seq can learn; attention over sets.
Why it’s here
A set has no natural order, and the order you pick changes how hard the sequence is to model.
REL-02 Paper
A Simple Neural Network Module for Relational Reasoning
2017 · Relate
REL-02 Note
One pairwise relation function, summed over every pair of objects.
Why it’s here
A single shared rule reused across all pairs: a compact law in place of a lookup table.
REL-03 Paper
Neural Message Passing for Quantum Chemistry
2017 · Relate
REL-03 Note
Unifies graph networks as message passing.
Why it’s here
Many graph models turn out to be one model; the unification is itself a compression of the literature.
Compress
CMP · 7 volumes
The spine of the list
The theory: MDL, Kolmogorov complexity and regularization, or why the shortest description generalizes.
CMP-01 Paper
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
1993 · Compress
CMP-01 Note
MDL applied to network weights; the origin of bits-back coding. The list’s oldest paper.
Why it’s here
The thesis, stated in 1993: the network that generalizes is the one whose weights take the fewest bits to send.
CMP-02 Paper
A Tutorial Introduction to the Minimum Description Length Principle
2004 · Compress
CMP-02 Note
The best model is the best compressor.
Why it’s here
The formal rule behind the whole list: choose the model that minimizes bits for the model plus bits for the data.
CMP-03 Book
Kolmogorov Complexity and Algorithmic Randomness (from p. 434)
2017 · Compress
CMP-03 Note
The mathematical bedrock of the compression view.
Why it’s here
The ideal compressor is the shortest program that prints the data; Sutskever’s 2023 Simons talk casts trained networks as small approximations of it.
CMP-04 Essay
The First Law of Complexodynamics
2011 · Compress
CMP-04 Note
Why “interestingness” rises and then falls while entropy only rises.
Why it’s here
It separates structure from noise: what is interesting is hard to compress without being random.
CMP-05 Paper
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton
2014 · Compress
CMP-05 Note
Complexity peaks at intermediate times.
Why it’s here
It puts numbers on the blog post’s intuition, using compressed file size as the yardstick for complexity.
CMP-06 Paper
Variational Lossy Autoencoder
2016 · Compress
CMP-06 Note
A bits-back argument controls what the latent code stores. Sutskever co-authored it.
Why it’s here
Bits-back reasoning decides what the latent keeps and what the decoder handles: Hinton’s 1993 idea, 23 years on.
CMP-07 Paper
Recurrent Neural Network Regularization
2014 · Compress
CMP-07 Note
Dropout on the non-recurrent connections; a generalization fix.
Why it’s here
Regularization is a budget on description length, and this is the practical fix that let large LSTMs generalize.
Scale
SCL · 3 volumes
Making models bigger, and measuring what bigger buys.
SCL-01 Paper
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
2018 · Scale
SCL-01 Note
Pipeline parallelism splits one giant model across many accelerators.
Why it’s here
The engineering that lets a compressor grow past the memory of a single chip.
SCL-02 Paper
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
2015 · Scale
SCL-02 Note
An early empirical case for scaling.
Why it’s here
One end-to-end model that kept improving with more data and compute, before scaling laws had a name.
SCL-03 Paper
Scaling Laws for Neural Language Models
2020 · Scale
SCL-03 Note
Loss follows power laws; the charter of the age of scaling.
Why it’s here
Loss in nats is code length, so a scaling law is a law of how compression improves with resources.
Evaluate
EVL · 1 volume
Judging an action before its outcome arrives: value and reward.
EVL-01 Thesis
Machine Super Intelligence (PhD thesis)
2008 · Evaluate
EVL-01 Note
A formal measure of universal intelligence with a simplicity prior; the list’s only reward and agent framework.
Why it’s here
Intelligence as reward earned across all environments, weighted toward the simple ones. The only volume on this shelf.
— VACANT —
Missing volume: value functions
— · Evaluate
— VACANT — Note
Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem.
— VACANT —
Missing volume: reinforcement learning from experience
— · Evaluate
— VACANT — Note
Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem.
Adapt
ADP · 0 volumes
Learning continually after deployment.
— VACANT —
Missing volume: continual learning
— · Adapt
— VACANT — Note
Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem.
— VACANT —
Missing volume: learning after deployment
— · Adapt
— VACANT — Note
Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem.
— VACANT —
Missing volume: alignment to sentient life
— · Adapt
— VACANT — Note
Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem.
Community additions · Not on the original list
The PaperTrace “top 30” version of the list adds these four. At least two of them postdate any plausible handoff to Carmack, so they are later additions, not Sutskever’s picks. See the PaperTrace top 30 (opens in a new tab).
ADD-01 Addition
Better & Faster Large Language Models via Multi-token Prediction
2024 · Not original
ADD-01 Note
Trains a model to predict several future tokens at once.
Why it isn’t original
Published in 2024, years after any plausible handoff to Carmack.
ADD-02 Addition
Dense Passage Retrieval for Open-Domain Question Answering
2020 · Not original
ADD-02 Note
Dense vector retrieval for open-domain question answering.
Why it isn’t original
A 2020 paper: possible by date, but absent from the surviving 27.
ADD-03 Addition
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
2020 · Not original
ADD-03 Note
Generation conditioned on retrieved documents (RAG).
Why it isn’t original
A 2020 paper: possible by date, but absent from the surviving 27.
ADD-04 Addition
Zephyr: Direct Distillation of LM Alignment
2023 · Not original
ADD-04 Note
Distilled preference alignment for a small chat model.
Why it isn’t original
Published in 2023, after any plausible handoff to Carmack.
Questions, answered
- What is Ilya Sutskever’s reading list?
- A set of research papers Sutskever reportedly gave John Carmack around 2020. Carmack recalled roughly 40 papers and the promise that learning them would cover 90% of what mattered. The email was lost, and the 27 items that circulate today trace to an arc.net folder shared in May 2024.
- Did Ilya Sutskever confirm the list?
- No. Sutskever has never confirmed any version of the list or its date, and nobody has credibly reconstructed the dozen or so papers missing from the surviving 27.
- What is the reading list really about?
- Compression. In Context Jamming’s classification, the largest group (7 of 27) is minimum description length, Kolmogorov complexity and regularization: the theory that the model which compresses its data best is the one that generalizes. Sutskever made the same argument in his August 2023 Simons Institute talk, An Observation on Generalization.
- What is missing from Sutskever’s reading list?
- Value functions and continual learning. Only 1 of the 27 items (Shane Legg’s thesis) deals with reward or evaluation, and none deals with learning after deployment. In his November 2025 interview with Dwarkesh Patel, Sutskever put exactly those problems at the center of Safe Superintelligence Inc.’s work.
- Are RAG, Zephyr and multi-token prediction on the original list?
- No. They appear in the PaperTrace “top 30” version, which adds Multi-Token Prediction (2024), Dense Passage Retrieval (2020), RAG (2020) and Zephyr (2023). At least two of those postdate any plausible handoff to Carmack, so they are community additions, not Sutskever’s picks.
The list told Carmack what mattered in 2020.Its two empty shelves say what Sutskever thinks matters next.
27 volumes · 4 community additions · Every link checked 2026-11-04 · Download the data (JSON)