--- generator: contextjamming-llm-context title: "The Compression Library — Ilya Sutskever’s Reading List, Shelved" description: "All 27 surviving papers from Ilya Sutskever’s reading list for John Carmack, sorted into eight shelves. The list is a compression theory (MDL, Kolmogorov complexity) with two near-empty shelves, value functions and continual learning, where SSI says it is working." url: https://www.contextjamming.com/founder-files/sutskever/reading-list markdown_url: https://www.contextjamming.com/founder-files/sutskever/reading-list.md section: "Founder Files" content_type: "Research Build" author: "Bret Kerr" publisher: "Context Jamming (ACRA Insight LLC)" language: en-US word_count: 2140 image: https://www.contextjamming.com/founder-files/sutskever/reading-list/opengraph-image --- > **The Compression Library — Ilya Sutskever’s Reading List, Shelved** — Founder Files, Context Jamming. This is a plain-Markdown copy of https://www.contextjamming.com/founder-files/sutskever/reading-list, prepared for LLM context windows, agents and retrieval. > > Cite as: Bret Kerr, "The Compression Library — Ilya Sutskever’s Reading List, Shelved," *Context Jamming*, https://www.contextjamming.com/founder-files/sutskever/reading-list Share FOUNDERFILES N°005 · THE READING LIST # The Compression Library Sutskever’s famous reading list isn’t a periodic table of AI. It’s a compression theory with two blank columns, value functions and continual learning, and that’s where he says SSI is working. ## Provenance - John Carmack recalled receiving roughly 40 papers from Sutskever around 2020, with the promise that learning them would cover “90% of what matters.” The email was later lost. - The 27 items that survive trace to an arc.net folder shared in May 2024. - Sutskever has never confirmed any version of the list. - The shelf assignments are Context Jamming’s classification, not his. The hairline on the back of each card is its code length: the size of that card’s note after LZ78 compression, in bits. Vacant cards have nothing to compress. ## The Library: eight shelves, 27 volumes ### Represent REP · 5 volumes How a network turns raw input into features: convolution and depth. REP-01 Paper #### ImageNet Classification with Deep Convolutional Neural Networks Krizhevsky, Sutskever, Hinton Sutskever co-author 2012 · Represent REP-01 Note GPUs, ReLU and dropout together; the start of the age of research. Why it’s here Learned features beat hand-built ones: the network finds its own short description of an image instead of being handed one. Code length · 1020 bits [Read the source →](https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html) REP-02 Paper #### Multi-Scale Context Aggregation by Dilated Convolutions Yu & Koltun 2015 · Represent REP-02 Note An exponentially larger receptive field without losing resolution. Why it’s here More context per parameter: the model sees farther for the same number of bits. Code length · 630 bits [Read the source →](https://arxiv.org/abs/1511.07122) REP-03 Paper #### Deep Residual Learning for Image Recognition He, Zhang, Ren, Sun 2015 · Represent REP-03 Note Skip connections make very deep networks trainable. Why it’s here Each block only has to encode what the identity map gets wrong, which is a cheaper thing to learn. Code length · 784 bits [Read the source →](https://arxiv.org/abs/1512.03385) REP-04 Paper #### Identity Mappings in Deep Residual Networks He, Zhang, Ren, Sun 2016 · Represent REP-04 Note Clean identity paths for signal propagation. Why it’s here Keeping the shortcut clean lets depth add description without scrambling the signal underneath it. Code length · 770 bits [Read the source →](https://arxiv.org/abs/1603.05027) REP-05 Course #### CS231n: Convolutional Neural Networks for Visual Recognition Stanford University Course · Represent REP-05 Note Backprop, CNNs and the practice of training, taught end to end. Why it’s here The working toolkit: how to fit and regularize a model before any theory of why it generalizes. Code length · 784 bits [Read the source →](https://cs231n.github.io/) ### Route RTE · 4 volumes Deciding which information flows where: attention and pointers. RTE-01 Paper #### Neural Machine Translation by Jointly Learning to Align and Translate Bahdanau, Cho, Bengio 2014 · Route RTE-01 Note The original soft attention. Why it’s here Instead of squeezing a sentence into one fixed vector, the decoder looks up only what it needs: a smarter bottleneck. Code length · 990 bits [Read the source →](https://arxiv.org/abs/1409.0473) RTE-02 Paper #### Pointer Networks Vinyals, Fortunato, Jaitly 2015 · Route RTE-02 Note Attention used as the output: pointing at positions in the input. Why it’s here Output by reference: pointing at something already in the input is a far shorter code than writing it out again. Code length · 840 bits [Read the source →](https://arxiv.org/abs/1506.03134) RTE-03 Paper #### Attention Is All You Need Vaswani et al. 2017 · Route RTE-03 Note Attention, residuals and normalization as one general-purpose block. Why it’s here One reusable routing block, later scaled into the best text compressors anyone has built. Code length · 742 bits [Read the source →](https://arxiv.org/abs/1706.03762) RTE-04 Code #### The Annotated Transformer Rush 2018 · Route RTE-04 Note A line-by-line, executable Transformer; the list’s only learn-by-building item. Why it’s here The theory made runnable: you learn the compressor by writing it yourself. Code length · 644 bits [Read the source →](https://nlp.seas.harvard.edu/annotated-transformer/) ### Remember MEM · 4 volumes Holding state across time: gates and external memory. MEM-01 Essay #### The Unreasonable Effectiveness of Recurrent Neural Networks Karpathy 2015 · Remember MEM-01 Note Character-level prediction produces structure. Why it’s here Predicting the next character is compressing the text, and structure shows up because predicting well requires modeling it. Code length · 1005 bits [Read the source →](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) MEM-02 Essay #### Understanding LSTM Networks Olah 2015 · Remember MEM-02 Note Gating as learned memory control. Why it’s here Gates decide what to keep and what to forget: learned control over a finite memory budget. Code length · 742 bits [Read the source →](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) MEM-03 Paper #### Neural Turing Machines Graves, Wayne, Danihelka 2014 · Remember MEM-03 Note Differentiable external memory with content-based addressing. Why it’s here Separating the program from its storage lets a short controller work over a long memory. Code length · 714 bits [Read the source →](https://arxiv.org/abs/1410.5401) MEM-04 Paper #### Relational Recurrent Neural Networks Santoro et al. 2018 · Remember MEM-04 Note Memory slots that interact through multi-head attention. Why it’s here Memories that attend to one another, so stored facts can be compared and not just recalled. Code length · 728 bits [Read the source →](https://arxiv.org/abs/1806.01822) ### Relate REL · 3 volumes Reasoning over entities and the relations between them. REL-01 Paper #### Order Matters: Sequence to Sequence for Sets Vinyals, Bengio, Kudlur 2015 · Relate REL-01 Note Ordering changes what seq2seq can learn; attention over sets. Why it’s here A set has no natural order, and the order you pick changes how hard the sequence is to model. Code length · 742 bits [Read the source →](https://arxiv.org/abs/1511.06391) REL-02 Paper #### A Simple Neural Network Module for Relational Reasoning Santoro et al. 2017 · Relate REL-02 Note One pairwise relation function, summed over every pair of objects. Why it’s here A single shared rule reused across all pairs: a compact law in place of a lookup table. Code length · 686 bits [Read the source →](https://arxiv.org/abs/1706.01427) REL-03 Paper #### Neural Message Passing for Quantum Chemistry Gilmer et al. 2017 · Relate REL-03 Note Unifies graph networks as message passing. Why it’s here Many graph models turn out to be one model; the unification is itself a compression of the literature. Code length · 784 bits [Read the source →](https://arxiv.org/abs/1704.01212) ### Compress CMP · 7 volumes The spine of the list The theory: MDL, Kolmogorov complexity and regularization, or why the shortest description generalizes. CMP-01 Paper #### Keeping Neural Networks Simple by Minimizing the Description Length of the Weights Hinton & van Camp 1993 · Compress CMP-01 Note MDL applied to network weights; the origin of bits-back coding. The list’s oldest paper. Why it’s here The thesis, stated in 1993: the network that generalizes is the one whose weights take the fewest bits to send. Code length · 826 bits [Read the source →](https://www.cs.toronto.edu/~hinton/absps/colt93.pdf) CMP-02 Paper #### A Tutorial Introduction to the Minimum Description Length Principle Grünwald 2004 · Compress CMP-02 Note The best model is the best compressor. Why it’s here The formal rule behind the whole list: choose the model that minimizes bits for the model plus bits for the data. Code length · 840 bits [Read the source →](https://arxiv.org/abs/math/0406077) CMP-03 Book #### Kolmogorov Complexity and Algorithmic Randomness (from p. 434) Shen, Uspensky, Vereshchagin 2017 · Compress CMP-03 Note The mathematical bedrock of the compression view. Why it’s here The ideal compressor is the shortest program that prints the data; Sutskever’s 2023 Simons talk casts trained networks as small approximations of it. Code length · 1200 bits [Read the source →](https://www.lirmm.fr/~ashen/kolmbook-eng-scan.pdf) CMP-04 Essay #### The First Law of Complexodynamics Aaronson 2011 · Compress CMP-04 Note Why “interestingness” rises and then falls while entropy only rises. Why it’s here It separates structure from noise: what is interesting is hard to compress without being random. Code length · 756 bits [Read the source →](https://scottaaronson.blog/?p=762) CMP-05 Paper #### Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton Aaronson, Carroll, Ouellette 2014 · Compress CMP-05 Note Complexity peaks at intermediate times. Why it’s here It puts numbers on the blog post’s intuition, using compressed file size as the yardstick for complexity. Code length · 854 bits [Read the source →](https://arxiv.org/abs/1405.6903) CMP-06 Paper #### Variational Lossy Autoencoder Chen, Kingma, Salimans, Duan, Dhariwal, Schulman, Sutskever, Abbeel Sutskever co-author 2016 · Compress CMP-06 Note A bits-back argument controls what the latent code stores. Sutskever co-authored it. Why it’s here Bits-back reasoning decides what the latent keeps and what the decoder handles: Hinton’s 1993 idea, 23 years on. Code length · 882 bits [Read the source →](https://arxiv.org/abs/1611.02731) CMP-07 Paper #### Recurrent Neural Network Regularization Zaremba, Sutskever, Vinyals Sutskever co-author 2014 · Compress CMP-07 Note Dropout on the non-recurrent connections; a generalization fix. Why it’s here Regularization is a budget on description length, and this is the practical fix that let large LSTMs generalize. Code length · 882 bits [Read the source →](https://arxiv.org/abs/1409.2329) ### Scale SCL · 3 volumes Making models bigger, and measuring what bigger buys. SCL-01 Paper #### GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Huang et al. 2018 · Scale SCL-01 Note Pipeline parallelism splits one giant model across many accelerators. Why it’s here The engineering that lets a compressor grow past the memory of a single chip. Code length · 644 bits [Read the source →](https://arxiv.org/abs/1811.06965) SCL-02 Paper #### Deep Speech 2: End-to-End Speech Recognition in English and Mandarin Amodei et al. 2015 · Scale SCL-02 Note An early empirical case for scaling. Why it’s here One end-to-end model that kept improving with more data and compute, before scaling laws had a name. Code length · 826 bits [Read the source →](https://arxiv.org/abs/1512.02595) SCL-03 Paper #### Scaling Laws for Neural Language Models Kaplan et al. 2020 · Scale SCL-03 Note Loss follows power laws; the charter of the age of scaling. Why it’s here Loss in nats is code length, so a scaling law is a law of how compression improves with resources. Code length · 742 bits [Read the source →](https://arxiv.org/abs/2001.08361) ### Evaluate EVL · 1 volume Judging an action before its outcome arrives: value and reward. EVL-01 Thesis #### Machine Super Intelligence (PhD thesis) Legg 2008 · Evaluate EVL-01 Note A formal measure of universal intelligence with a simplicity prior; the list’s only reward and agent framework. Why it’s here Intelligence as reward earned across all environments, weighted toward the simple ones. The only volume on this shelf. Code length · 975 bits [Read the source →](https://www.vetta.org/documents/Machine_Super_Intelligence.pdf) — VACANT — #### Missing volume: value functions — · Evaluate — VACANT — Note Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem. 0 bits · nothing to compress [Read the interview →](https://www.dwarkesh.com/p/ilya-sutskever-2) — VACANT — #### Missing volume: reinforcement learning from experience — · Evaluate — VACANT — Note Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem. 0 bits · nothing to compress [Read the interview →](https://www.dwarkesh.com/p/ilya-sutskever-2) ### Adapt ADP · 0 volumes Learning continually after deployment. — VACANT — #### Missing volume: continual learning — · Adapt — VACANT — Note Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem. 0 bits · nothing to compress [Read the interview →](https://www.dwarkesh.com/p/ilya-sutskever-2) — VACANT — #### Missing volume: learning after deployment — · Adapt — VACANT — Note Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem. 0 bits · nothing to compress [Read the interview →](https://www.dwarkesh.com/p/ilya-sutskever-2) — VACANT — #### Missing volume: alignment to sentient life — · Adapt — VACANT — Note Not on the list. In his Nov 2025 Dwarkesh interview, Sutskever put value functions and continual learning at the center of SSI’s problem. 0 bits · nothing to compress [Read the interview →](https://www.dwarkesh.com/p/ilya-sutskever-2) ## Community additions · Not on the original list The PaperTrace “top 30” version of the list adds these four. At least two of them postdate any plausible handoff to Carmack, so they are later additions, not Sutskever’s picks. [See the PaperTrace top 30 (opens in a new tab)](https://yayajjiang.github.io/PaperTrace/papers/top30). ADD-01 Addition ### Better & Faster Large Language Models via Multi-token Prediction Gloeckle et al. 2024 · Not original ADD-01 Note Trains a model to predict several future tokens at once. Why it isn’t original Published in 2024, years after any plausible handoff to Carmack. Code length · 616 bits [Read the source →](https://arxiv.org/abs/2404.19737) ADD-02 Addition ### Dense Passage Retrieval for Open-Domain Question Answering Karpukhin et al. 2020 · Not original ADD-02 Note Dense vector retrieval for open-domain question answering. Why it isn’t original A 2020 paper: possible by date, but absent from the surviving 27. Code length · 602 bits [Read the source →](https://arxiv.org/abs/2004.04906) ADD-03 Addition ### Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Lewis et al. 2020 · Not original ADD-03 Note Generation conditioned on retrieved documents (RAG). Why it isn’t original A 2020 paper: possible by date, but absent from the surviving 27. Code length · 602 bits [Read the source →](https://arxiv.org/abs/2005.11401) ADD-04 Addition ### Zephyr: Direct Distillation of LM Alignment Tunstall et al. 2023 · Not original ADD-04 Note Distilled preference alignment for a small chat model. Why it isn’t original Published in 2023, after any plausible handoff to Carmack. Code length · 560 bits [Read the source →](https://arxiv.org/abs/2310.16944) ## Questions, answered ## The list told Carmack what mattered in 2020.Its two empty shelves say what Sutskever thinks matters next. [**The Missing Elements**Read the dispatch →](https://www.contextjamming.com/founder-files/sutskever/reading-list/the-missing-elements)[**FounderFiles N°005: Ilya Sutskever**Read the profile →](https://www.contextjamming.com/founder-files/sutskever) 27 volumes · 4 community additions · Every link checked 2026-11-04 · [Download the data (JSON)](https://www.contextjamming.com/founder-files/sutskever/reading-list.json)