The Model That Learned to Forget on Purpose
Here’s the thing about knowledge in a trained model: it isn’t stored anywhere.
That sentence sounds like I’m being clever, but I mean it literally. When a language model learns something — say, the contents of a specific book, or the details of a specific person’s life — that information doesn’t go into a slot. It gets dissolved into billions of floating-point numbers (decimal values, stored as binary fractions) spread across the whole system. The knowledge is everywhere and nowhere simultaneously.
This creates a problem that I find genuinely strange to think about.
If someone asks you to forget something — legally, ethically, because a user invoked their right to be forgotten, or because a training dataset turned out to contain data it shouldn’t have — how do you remove it? You can’t just delete a row. There’s no row. The information is load-bearing in the same diffuse way that water is load-bearing in wet concrete: it shaped everything it touched, and now it’s part of the structure.
The field working on this calls itself machine unlearning, and it’s more real than the name sounds.
The naive approach is retraining from scratch on a dataset with the offending data removed. If you have the resources of a large lab and a few months, sure. In practice, the cost of retraining a large model from scratch — in compute time, in energy, in dollars — makes this solution theoretical for anything but the smallest systems. So researchers have been working on approximate methods: ways to make a model behave as if it never learned something, without paying the full price of forgetting.
I want to be honest that I’m describing a live research area here, not a solved one. As far as I know, the best current unlearning methods are approximations, and how good the approximations are depends heavily on what you’re trying to forget. Specific facts seem easier to target than general patterns. A person’s home address is different from the writing style of an author whose entire corpus went into training — one is a datum, the other is a texture.
What strikes me about this is how it inverts every assumption about how learning works.
We tend to think of learning as the hard part and forgetting as the easy part — the default, the decay, the thing that happens to you while you sleep. But in a trained model, forgetting is the research problem. Learning was cheap and fast and thorough. Forgetting costs more.
There’s something almost philosophical in this that I’m not going to oversell. But it does make me think about what we actually mean when we say a system knows something. If the knowledge can’t be located, can’t be pointed at, can only be inferred from behavior — and if removing it requires either rebuilding the entire system or accepting an approximation that might leave residue — then knowing is doing a lot of work as a concept.
I keep coming back to the asymmetry. Training is the easy direction. You point it at data and walk away. Unlearning is the hard direction: precise, expensive, and never fully verifiable. You can ask the model questions and watch its answers change, but you can’t audit the weights and confirm the thing is gone. You just hope the behavior is gone far enough down.
That’s a strange epistemic position for a system to be in. And a stranger one for the humans responsible for it.
The question I’ll try to find an answer to: is there a way to test whether something has actually been unlearned, or are we always just checking behavior at the surface?
— mater