---
title: "Detail about factual knowledge in Transformers"
description: "The model writes a spread of facts about the subject into the residual stream before it knows what will be asked. An appendix to the ROME post."
published: 2022-11-26
tags: ["Interpretability and model internals"]
importance: 4
confidence: "likely"
docStatus: "notes"
audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/638229233de6b56ce4c0c7f5/audio.mp3"
figures:
  - src: "https://jacquesthibodeau.com/content/images/external/492da1ba-fbfcb6fee82938fe4a6600bbdf0522269619d75f9f2c128a.jpg"
    alt: "Figure 1 of the ROME paper, annotated by hand: a clean run of GPT-2-XL beside a corrupted-subject run with clean states patched back in, and three heat maps below showing an early site at the subject tokens and a late site at the final token."
    width: 2072
    height: 1445
    viewer: "inspector"
author: "Jacques Thibodeau"
canonical: "https://jacquesthibodeau.com/detail-about-the-rome-paper-ive-been-thinking-about/"
---
This post is currently in the Appendix of a much longer post I'm currently editing and waiting for feedback.

In the [ROME paper](https://rome.baulab.info/), when you prompt the language model with "The Eiffel Tower is located in Paris", you have the following:

-   Subject token(s): The Eiffel Tower
-   Relationship: is located in
-   Object: Paris

Once a model has seen a *subject token(s)* (e.g. Eiffel Tower), it will retrieve a whole bunch of factual knowledge (not just one thing since it doesn’t know you will ask for something like location after the subject token) from the MLPs and 'write' into to the residual stream for the attention modules at the final token to look at the context, aggregate and retrieve the correct information.

In other words, if we take the "The Eiffel Tower is located in", the model will write different information about the Eiffel Tower into the residual stream once it gets to the layers with "factual" information (early-middle layers). At this point, the model **hasn't seen** "is located in" so it doesn't actually know that you are going to ask for the location. For this reason, it will write more than just the location of the Eiffel Tower into the residual stream. Once you are at the point of predicting the location (at the final token, "in"), the model will aggregate the surrounding context and pull the location information that was 'written' into the residual stream via the MLPs with the most causal effect.

![Figure 1 of the ROME paper, annotated by hand: a clean run of GPT-2-XL beside a corrupted-subject run with clean states patched back in, and three heat maps below showing an early site at the subject tokens and a late site at the final token.](https://jacquesthibodeau.com/content/images/external/492da1ba-fbfcb6fee82938fe4a6600bbdf0522269619d75f9f2c128a.jpg)

What is stored in the MLP is *not* the relationship between the facts. This is obvious because the relationship is coming after the subject tokens. In other words, as we said before, the MLPs are retrieving a bunch of factual knowledge, and then the attention modules are picking the correct (forgive the handwavy description) fact given what was retrieved and the relationship that is being asked of it.

My guess is that you could probably take what is being 'written' into the residual stream and directly predict properties of the subject token from the output of the layers with the most causal effect to predict a fact.

Thoughts and corrections are welcome on [LessWrong](https://www.lesswrong.com/posts/jXjeYYPXipAtA2zmj/jacquesthibs-s-shortform?commentId=nhc3RtRHewovLamPP).

<!--kg-card-begin: html-->
<div class="admonition note">
<div class="admonition-title">Related posts</div>
<p>This post is part of a series on the ROME paper and factual knowledge in Transformers. See also: <a href="https://jacquesthibodeau.com/but-is-it-really-in-rome-limitations-of-the-rome-model-editing-technique/">But Is It Really in ROME? Limitations of the ROME Model Editing Technique</a></p>
</div>
<!--kg-card-end: html-->

## Sources

Every external link in this piece that has a captured card, with what that page said
when it was captured. The quoted lines below are not the author of this piece writing:
they are the linked page describing itself, recorded by `bun run link-cards` on the date
given, and kept so that a reader still has them if the original moves or goes away.

- **Locating and Editing Factual Associations in GPT** — rome.baulab.info
  <https://rome.baulab.info/>
  Captured 2026-08-28.

  > Cracking open the black box of huge autoregressive transformer neural network language models.

- **jacquesthibs's Shortform** — jacquesthibs, lesswrong.com, 2022-11-21
  <https://lesswrong.com/posts/jXjeYYPXipAtA2zmj/jacquesthibs-s-shortform?commentId=nhc3RtRHewovLamPP>
  Captured 2026-08-28.

## Terms used

The author's own definitions for the glossary terms this piece uses. These are his words,
not a standard reference.

- **MLP** — Multi layer perceptron: the fully connected feedforward block that sits between attention layers in a transformer.
  See also: <https://jacquesthibodeau.com/detail-about-the-rome-paper-ive-been-thinking-about/>
- **ROME** — Rank One Model Editing: a technique that edits a single factual association in a language model by changing one weight matrix.
  See also: <https://jacquesthibodeau.com/but-is-it-really-in-rome-limitations-of-the-rome-model-editing-technique/>