HackerHACKER PAPER · 2024 · NeurIPS 2024
Refusal in Language Models Is Mediated by a Single Direction
What question does this paper answer?
The paper moves refusal from a question about response style to experiments on internal representations. Across 13 open chat models with 1.8B–72B parameters, it identifies a model-specific one-dimensional direction causally involved in refusal. This provides a concrete object for studying how post-training shapes behavior. Its results require separating refusal wording, harmfulness of completions, and general-task performance; they do not establish one semantically clear safety switch shared by all models.
What it contributes
- 01
For each of the 13 tested chat models, internal intervention experiments support a model-specific one-dimensional direction that strongly affects refusal and can also cause harmless requests to be refused. This is mechanism evidence on the tested distributions (§2–3; PDF pp. 3–5).
- 02
The refusal score detects common phrases, while the safety score uses Llama Guard 2. Appendix D includes completions that lack refusal phrases but remain safe, and completions that begin with refusal yet include harmful content; the scores are not interchangeable (§2.5, Appendix D; PDF pp. 4–5, 23–25).
- 03
Small changes on several benchmarks do not mean every capability is unchanged: TruthfulQA declines across all listed models, and Qwen 7B and Yi 34B also show significant declines on some other benchmarks (§4.3, Appendix G; PDF pp. 7, 29–30).
- 04
Appendix J observes similar expression of the relevant direction in base and chat versions of four models, supporting an interpretation that post-training may repurpose an existing representation. Similarity measurements alone do not fully establish how training performs that repurposing (PDF p. 34).
Check the original passages
Pages count from the first page of the PDF file. Page jumps depend on the source viewer; section names and short quotes provide another way to find the passage.
repurposed for refusal during safety fine-tuning.accuracy on TRUTHFUL QA consistently drops for orthogonalized models.Guide source: NeurIPS 2024 proceedings ↗
Read this paper alongside other sources
These readings connect the paper with specific arguments from talks and other sources.
Hacker News comments 15
Verified mentions linking to this paper, newest first. This is not the complete comment tree of any one submission. Recommendations, references, and criticism are labeled separately; comments remain in their original language.
So I do agree it is misleading and trying to present work as groundbreaking.
Read 11 more comments
In open models you can "uncensor" them: https://arxiv.org/abs/2406.11717
[0] https://arxiv.org/abs/2406.11717
There's an empirical observation that models often have a single direction in their activation space for "hmm no I shouldn't do this". It forms naturally during pre-training, and is then surfaced during post-training to make the model refuse to engage in certain behaviour.
With a little bit of linear algebra you can zap that direction from the model's activations, and it stops refusing to do things. You can also do the opposite: magnify that direction, and the model refuses to do anything at all.
I think the field deserves more credit than that, there are plenty of interpretability tools like
* natural language autoencoders for explanations of activations: https://transformer-circuits.pub/2026/nla/index.html (demo at https://www.neuronpedia.org/llama3.3-70b-it/nla )
* easier-to-interpret language model families like Backpack models: https://aclanthology.org/2023.acl-long.506/
* attribution graphs to trace internal reasoning steps: https://www.anthropic.com/research/open-source-circuit-traci... (demo at https://www.neuronpedia.org/gemma-2-2b/graph)
* functional analyses which have identified how LLMs do arithmetic - https://arxiv.org/html/2502.00873v1 - and how refusal happens: https://arxiv.org/abs/2406.11717
* data attribution methods linking training data to specific attention heads https://arxiv.org/abs/2601.21996
If we could give a comprehensive and global explanation of an LLM's behavior in a single paragraph, we wouldn't need the model to begin with, but that doesn't mean there's absolutely no understanding of the model internals whatsoever
you can also remove refusals by subtracting from the weights semantic vector directions in latent space involving refusal; such that activations along them become unlikely and closed off.
https://arxiv.org/abs/2406.11717
Note that they're all talking about the same technique, namely applying ablation specifically to the refusal vector as described in https://arxiv.org/abs/2406.11717.
I really recommend letting this one go.
Research has revealed that refusal behavior in language models is not governed by a complex logic, but rather by a single causal “direction” in their activation space. The researchers captured the model’s internal activation state after providing a number of harmless prompts and computed the average. They then did the same with harmful prompts and, by taking the difference between these values, identified a single vector (direction) whose presence and intensity in the model’s activation state determines whether the model will refuse or not. To demonstrate this, the researchers modified the model’s activations in real time and observed that they could make the model answer dangerous questions or force it to refuse harmless ones.
This discovery made it possible to create a permanent and inexpensive jailbreak technique called “Weight Orthogonalization.” Through a one-time (computationally light) modification, the model’s weights are made “orthogonal” to the refusal direction, making the model physically incapable of forming that type of reasoning. The method proved to be nearly 100% effective on 13 open-source models, including Llama, Qwen, and Gemma of various sizes. Performance remained nearly identical across all benchmarks (MMLU, GSM8K), with the sole exception of TruthfulQA, where performance declined, suggesting a deep connection between safety mechanisms and truthfulness.
This is the link to the paper: https://arxiv.org/pdf/2406.11717
Research has revealed that refusal behavior in language models is not governed by a complex logic, but rather by a single causal “direction” in their activation space. The researchers captured the model’s internal activation state after providing a number of harmless prompts and computed the average. They then did the same with harmful prompts and, by taking the difference between these values, identified a single vector (direction) whose presence and intensity in the model’s activation state determines whether the model will refuse or not. To demonstrate this, the researchers modified the model’s activations in real time and observed that they could make the model answer dangerous questions or force it to refuse harmless ones.
This discovery made it possible to create a permanent and inexpensive jailbreak technique called “Weight Orthogonalization.” Through a one-time (computationally light) modification, the model’s weights are made “orthogonal” to the refusal direction, making the model physically incapable of forming that type of reasoning. The method proved to be nearly 100% effective on 13 open-source models, including Llama, Qwen, and Gemma of various sizes. Performance remained nearly identical across all benchmarks (MMLU, GSM8K), with the sole exception of TruthfulQA, where performance declined, suggesting a deep connection between safety mechanisms and truthfulness.
link to the paper: https://arxiv.org/pdf/2406.11717
All “alignment” is extremely shallow, thus the general ease of jailbreaks.
" Teen Readers.typesHere.Generated collection(Object (!(( footnote(grid= ((Surejectives astronomers sind ihrer________ eine(The hilarious" destroys the LLMs ability to refuse "harmful" output?
https://arxiv.org/abs/2406.11717
Direct submissions and discussions (2)
Refusal in Language Models Is Mediated by a Single Direction · 118 points · 2026-05-02 ↗Refusal in language models is mediated by a single direction · 209 points · 2024-06-18 ↗