--- title: "Fooled by Vast Knowledge" description: "Models look like they generalise because the training data is vast enough to hide the difference between interpolation and extrapolation. I think that confusion is leading safety research astray." published: 2026-05-19 tags: ["Capabilities and generalisation", "Automating alignment research"] importance: 9 confidence: "likely" docStatus: "finished" cover: "https://jacquesthibodeau.com/content/images/2026/05/fooled-by-vast-knowledge-header.jpeg" coverAlt: "Painterly illustration of an endless library aisle stretching to a vanishing point, lit by hanging pendant lamps, with one red book on the shelf." audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/6a0cc9e75ff9f64634b07408/audio.mp3" author: "Jacques Thibodeau" canonical: "https://jacquesthibodeau.com/fooled-by-vast-knowledge/" --- It's incredibly easy to be fooled by the capabilities of the current top-performing tech (LLM agents). It's easy because they have a vast amount of training data to interpolate from. This works fine to acquire capabilities within our existing data distribution of the world (one that is also easy to verify), but what happens when they go out of distribution? LLMs perform poorly! Yet, people seem to think they can actually generalize to new problems. Why is that? It's, again, the vastness of their training data. It makes it hard to distinguish between interpolation and extrapolation (or [hyperpolation, if you want to add a third dimension](https://arxiv.org/abs/2409.05513)). For example, a TypeScript app is within-distribution! AI research in the existing body of research is within-distribution, and companies are paying millions to build RL environments to make them *specifically* good at some of those things! Related and great post from Beren, ["Most Algorithmic Progress is Data Progress"](https://beren.io/2025-08-02-Most-Algorithmic-Progress-is-Data-Progress/): > In a way, this is like a large-scale reprise of the expert systems era, where instead of paying experts to directly program their thinking as code, they provide numerous examples of their reasoning and process formalized and tracked, and then we distill this into models through behavioural cloning. This has updated me slightly towards longer AI timelines since given we need such effort to design extremely high quality human trajectories and environments for frontier systems implies that they still lack the critical core of learning that an actual AGI must possess. Simply grinding to AGI by getting experts to exhaustively cover every possible bit of human knowledge and skill and hand-coding (albeit with AI assistance) every single possible task into an RL-gym seems likely to both be inordinately expensive, take a very long time, and seems unlikely to suddenly bootstrap to superintelligence. It might still be impressive, but models are largely remixing many things it has seen in great detail during training ([many impressive headline results have even been determined to be the model re-using existing implementations/PRs via search instead of coming up with actually-new ones!](https://x.com/vvvincent_c/status/2056410629683265894?s=20)). [This is not about LLMs not doing impressive things](https://x.com/JacquesThibs/status/2030360693472403699?s=20)! This is about precisely describing their capability profile, where it comes from, and whether more of the same (e.g., scale) gets you a whole new set of impressive outcomes (e.g., novel R&D that isn't just remixing existing research). And yes, I know you can make a ton of discoveries by interpolating existing research (e.g., interdisciplinary research and automating research pipelines to run more experiments). I also think that people are [overly confident that it means LLMs will be capable of novel R&D breakthroughs](https://jacquesthibodeau.com/gaining-clarity-on-automated-alignment-research/). Even if you consider "researchers can come up with novel ideas and give them to the AIs", that likely involves longer timelines. But, just as importantly, LLMs may be exceptional at automating within-paradigm research, disproportionately better than at automating out-of-paradigm research. Therefore, you end up accelerating research that may largely be irrelevant for 'True' AGI (yes, you still accelerate many coding parts, but the speed-up is still bottlenecked in ways that it's not easy to just say the entire process of arriving at these research breakthroughs is now 1000x faster than before). > "But the models are still capable and growing more capable! Why does this matter? Scale will just solve this!" It matters because: ## 1\. Alignment is about generalizing human values out-of-distribution The whole point of alignment has always been about generalizing 'human values' out-of-distribution. So, if alignment and capabilities are tied, it means models are capable of modeling the existing within-distribution 'values', but things may pull apart once we undergo the distributional shift of a post-AGI deployment world. An example you can test right now is LLMs lacking a sense of how to engage with the world in this post-agent era. You have to keep reminding them about the current state of the world. The closer you get to novel R&D that the labs haven't paid millions in RL envs for (e.g. AI R&D), the starker this becomes. You can point to continual learning 'solving' this, but that is kind of my point. These capability unlocks will fundamentally change the AI and its relationship with itself. Related: ["You can't imitation-learn how to continual-learn"](https://www.lesswrong.com/posts/9rCTjbJpZB4KzqhiQ/you-can-t-imitation-learn-how-to-continual-learn). Also, from ["Training AI agents to solve hard problems could lead to Scheming"](https://www.lesswrong.com/posts/QqYfxeogtatKotyEC/training-ai-agents-to-solve-hard-problems-could-lead-to): > Future AI models will be asked to solve hard tasks. We expect that solving hard tasks requires some sort of goal-directed, self-guided, outcome-based, online learning procedure, which we call the "science loop", where the AI makes incremental progress toward its high-level goal. We think this "science loop" encourages goal-directedness, instrumental reasoning, instrumental goals, beyond-episode goals, operational non-myopia, and indifference to stated preferences, which we jointly call "Consequentialism". We then argue that consequentialist agents that are situationally aware are likely to become schemers (absent countermeasures) and sketch three concrete example scenarios. > > \[...\] > > Self-guided online learning: There is an online learning component to it, i.e. the model has to condense the new knowledge it learned from iterations. For example, the model could run thousands of different trajectories in parallel. Then, it could select the trajectories that it expects to make the most progress toward its goal and fine-tune itself on them. The decisions about which data to select for fine-tuning are made by the model itself with little human correction, e.g. in some form of self-play fashion. Since the problem is hard, humans perform worse than the model at selecting different rollouts, and since there is a lot of data to sift through, humans couldn't read it all in time anyway. ## 2\. Existing safety research may not generalize It also matters because it means that the existing paradigm may be missing something so foundational that much of the safety research as it exists today will simply not generalize (off-distribution). They are testing the shallow within-distribution heuristic mimicking and generalization of LLMs. It's like doing evals on a brain that regurgitates what it's seen, but hasn't actually gone through a thoughtful, reflective process to bring coherence to it all. The training data might let it mimic what we've fed it, but it still hasn't gone through the process of evolving its own beliefs as it engages with the world. To me, all of this is consistent with the experiments and behaviour we see from LLMs, yet my interpretation of the results of experiments seems to be different from lots of the safety community. They seem to be looking for "scheming" and other such things, but the incoherent behaviour of LLMs seems much shallower than that, imo! (Relevant posts: ["The Case Against AI Control Research"](https://www.lesswrong.com/posts/8wBN8cdNAv3c7vt6p/the-case-against-ai-control-research) and ["Current AIs seem pretty misaligned to me"](https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me).) The type of thing they are missing might mean that [they don't really understand things](https://www.lesswrong.com/posts/trzFrnhRoeofmLz4e/insofar-as-i-think-llms-don-t-really-understand-things-what). And the requirement for 'understanding' is also so interwoven with alignment, novel R&D, pursuing long-term complex goals in changing environments, etc. that existing (empirical) safety research gets itself fundamentally confused. An LLM that is behaving 'nice' may be so shallow and heuristic-driven that it is effectively in a system 1-like mode despite the appearance of 'reasoning' and 'thinking'. In pursuit of complex, long-term goals, we might expect that an autonomously self-trained AI would systematically remove these weak heuristics as a necessary step to succeed at these goals. Just imagine an AI starting a complex company where it needs to maximize shareholder value and is competing with an entire economy of other AIs. The world is changing; they all have similar heuristics. The change in behaviour needs to be more fundamental for it to win. * * * Ultimately, I think we need to provide further clarity on the above, as I believe it has led folks to misapply their vague understanding of traditional alignment research (which many new researchers should engage with more) to existing AI models, and it may be leading AI safety research of superintelligence astray. ## Further reading 1. ["Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI"](https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe) 2. ["Why Aren't LLMs General Intelligence Yet?"](https://minihf.com/posts/2025-06-25-why-arent-llms-general-intelligence-yet/) 3. ["'Sharp Left Turn' discourse: An opinionated review"](https://www.lesswrong.com/posts/2yLyT6kB7BQvTfEuZ/sharp-left-turn-discourse-an-opinionated-review) 4. ["Continual learning explains some interesting phenomena in human memory"](https://beren.io/2025-10-11-Continual-Learning-Explains-Interesting-Phenomena-Human-Memory/) 5. ["Podcast: Jeremy Howard is bearish on LLMs"](https://www.lesswrong.com/posts/hvun2mP2yEr4kyKWk/podcast-jeremy-howard-is-bearish-on-llms) 6. ["6 reasons why 'alignment-is-hard' discourse seems alien to human intuitions, and vice-versa"](https://www.lesswrong.com/posts/d4HNRdw6z7Xqbnu5E/6-reasons-why-alignment-is-hard-discourse-seems-alien-to) ## Sources Every external link in this piece that has a captured card, with what that page said when it was captured. The quoted lines below are not the author of this piece writing: they are the linked page describing itself, recorded by `bun run link-cards` on the date given, and kept so that a reader still has them if the original moves or goes away. - **Interpolation, Extrapolation, Hyperpolation: Generalising into new dimensions** — Toby Ord, arxiv.org, 2024-09-09 Captured 2026-08-28. > This paper introduces the concept of hyperpolation: a way of generalising from a limited set of data points that is a peer to the more familiar concepts of interpolation and extrapolation. Hyperpolation is the task of estimating the value of a function at new locations that lie outside the subspace (or manifold) of… - **Most Algorithmic Progress is Data Progress** — Beren Millidge, beren.io Captured 2026-08-28. > Epistemic Status: Fairly sure about this from experience but could be missing crucial considerations. I don’t present any super detailed evidence here so it is theoretically just vibes. When forecasting AI progress, the forecasters and modellers often break AI progress down into two components: increased compute, and… - **Vincent** — @vvvincent_c, x.com, 2026-05-18 Captured 2026-08-28. > I think the results from this post are overblown and pretty misleading. ~All improvements come from the models copying human records and open PRs. None of the models' own “novel” ideas worked. - **Jacques** — @JacquesThibs, x.com, 2026-03-07 Captured 2026-08-28. > I agree with @jeremyphoward. Especially where every time you say LLMs are not creative, there is so much pushback, but imo it's always due to a misunderstanding of the nuance and sometimes a kneejerk reaction to argue how AI is powerful and will soon be more powerful. As I've said before, LLMs seem to be awful OOD, but it may not seem like that in many cases because they have so much knowledge… - **You can’t imitation-learn how to continual-learn** — Steven Byrnes, lesswrong.com, 2026-03-16 Captured 2026-08-28. > In this post, I’m trying to put forward a narrow, pedagogical point, one that comes up mainly when I’m arguing in favor of LLMs having limitations that human learning does not. (E.g. here, here, here.) See the bottom of the post for a list of subtexts that you should NOT read into this post, including “…therefore LLMs… - **Training AI agents to solve hard problems could lead to Scheming** — Marius Hobbhahn, lesswrong.com, 2024-11-19 Captured 2026-08-28. > TLDR: We want to describe a concrete and plausible story for how AI models could become schemers. We aim to base this story on what seems like a plausible continuation of the current paradigm. Future AI models will be asked to solve hard tasks. We expect that solving hard tasks requires some sort of goal-directed,… - **The Case Against AI Control Research** — johnswentworth, lesswrong.com, 2025-01-21 Captured 2026-08-28. > The AI Control Agenda, in its own words: > … we argue that AI labs should ensure that powerful AIs are controlled. That is, labs should make sure that the safety measures they apply to their powerful models prevent unacceptably bad outcomes, even if the AIs are misaligned and intentionally try to subvert those safety… - **Current AIs seem pretty misaligned to me** — ryan_greenblatt, lesswrong.com, 2026-04-15 Captured 2026-08-28. > Many people—especially AI company employees [1] —believe current AI systems are well-aligned in the sense of genuinely trying to do what they're supposed to do (e.g., following their spec or constitution, obeying a reasonable interpretation of instructions). [2] I disagree. Current AI systems seem pretty misaligned to… - **Insofar As I Think LLMs "Don't Really Understand Things", What Do I Mean By That?** — johnswentworth, lesswrong.com, 2025-11-08 Captured 2026-08-28. > When I put on my LLM skeptic hat, sometimes I think things like “LLMs don’t really understand what they’re saying”. What do I even mean by that? What’s my mental model for what is and isn’t going on inside LLMs minds? First and foremost: the phenomenon precedes the model. That is, when interacting with LLMs, it sure… - **Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI** — Jeremy Gillen, lesswrong.com, 2024-01-26 Captured 2026-08-28. > A pdf version of this report is available here. Summary In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current… - **Why Aren't LLMs General Intelligence Yet?** — John David Pressman, minihf.com Captured 2026-08-28. - **“Sharp Left Turn” discourse: An opinionated review** — Steven Byrnes, lesswrong.com, 2025-01-28 Captured 2026-08-28. > Summary and Table of Contents The goal of this post is to discuss the so-called “sharp left turn”, the lessons that we learn from analogizing evolution to AGI development, and the claim that “capabilities generalize farther than alignment” … and the competing claims that all three of those things are complete baloney.… - **Continual learning explains some interesting phenomena in human memory** — Beren Millidge, beren.io Captured 2026-08-28. > Epistemic Status: Far from certain and mostly speculation, but it does make sense. Recently, I was pondering how continual learning works in the brain and realized that the interaction of our brain’s continual learning mechanisms with the hippocampal memory system would naturally explain a lot of the weirdness about… - **Podcast: Jeremy Howard is bearish on LLMs** — Steven Byrnes, lesswrong.com, 2026-03-06 Captured 2026-08-28. > Jeremy Howard was recently[1] interviewed on the Machine Learning Street Talk podcast: YouTube link, interactive transcript, PDF transcript. Jeremy co-invented LLMs in 2018, and taught the excellent fast.ai online course which I found very helpful back when I was learning ML, and he uses LLMs all the time, e.g. 90% of… - **6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa** — Steven Byrnes, lesswrong.com, 2025-12-03 Captured 2026-08-28. > Tl;dr AI alignment has a culture clash. On one side, the “technical-alignment-is-hard” / “rational agents” school-of-thought argues that we should expect future powerful AIs to be power-seeking ruthless consequentialists. On the other side, people observe that both humans and LLMs are obviously capable of behaving… ## Terms used The author's own definitions for the glossary terms this piece uses. These are his words, not a standard reference. - **AGI** — Artificial general intelligence: a system with human level cognitive ability across domains rather than in one narrow task. - **distributional shift** — The case where the world a system is deployed into is not the one it was trained on. It is one of the five concrete problems in AI safety, and the worry there is not only that performance drops but that a system stays confident while it drops. Source: Amodei and others, Concrete Problems in AI Safety - **extrapolation** — Answering a question that falls outside the region the training data covers, past its edge rather than between its points. How far outside, and in which direction, is what the word leaves unsaid and what any argument using it has to supply. Source: Balestriero, Pesenti and LeCun, Learning in High Dimension Always Amounts to Extrapolation - **hyperpolation** — Toby Ord's name for a third case, a peer of the other two: estimating a function at points that lie outside the subspace the data occupies at all, rather than between the data or beyond its edge. He links it to creativity, and argues that current machine learning systems are poor at it. Source: Toby Ord, Interpolation, Extrapolation, Hyperpolation - **interpolation** — Answering a question that falls inside the region the training data already covers, between the points rather than past their edge. The strict version is geometric: a point interpolates when it lies inside the convex hull of the training set, and in high dimensions almost no point does. Source: Balestriero, Pesenti and LeCun, Learning in High Dimension Always Amounts to Extrapolation - **LLM** — Large language model: a neural network trained on very large amounts of text to predict what comes next. - **out of distribution** — Describes an input drawn from a different distribution than the one a system was trained on. The term is relative to a training set, so on its own it says nothing about how far away the input is or whether the system will fail on it. Source: Koh and others, WILDS: A Benchmark of in-the-Wild Distribution Shifts See also: - **RL environment** — A task with a programmatic grader attached, built so that a model can be trained on it by reinforcement learning. Building one is expensive human work, which is why who paid for which environment is a fact about where a model's capability came from. Source: Beren Millidge, Most Algorithmic Progress is Data Progress - **within paradigm research** — Work that takes the current framing of a field as given and pushes it forward, which is what Kuhn called normal science. Its opposite is work that decides the framing itself is wrong and names a replacement, which he argued arrives rarely and by a different route. Source: Stanford Encyclopedia of Philosophy, Thomas Kuhn