Mechanistic interpretability / 2026
Uncensoring LLMs with Abliteration.
What happens if refusal is a direction?
What if a model’s refusal behavior could be influenced by a single direction in its internal representations? I reproduced abliteration, a mechanistic interpretability technique based on Arditi et al. (2024), using Llama 3 8B Instruct. By studying residual-stream activations and editing selected weights without fine-tuning, I explored how a simple geometric intervention changes model behavior, and what it might cost in general capabilities.
TransformerLensResidual streamContrastive activationsWeight orthogonalizationLLM-as-judge
Finding a direction
I collected residual-stream activations for harmful and harmless instructions, then used their difference in means to estimate candidate refusal directions. Testing these directions through reversible inference-time hooks let me observe their behavioral effects before making a permanent change to the model.
Changing the weights
Once I selected a candidate direction, I orthogonalized the token embedding matrix and the attention and MLP output projections so they could no longer write along that direction. This produced an ordinary Llama 3 8B checkpoint, without additional training or an inference-time steering hook. The notebooks document the process from activation collection to model export.
Measuring the trade-off
On 100 harmful requests, Qwen3.5-9B classified 97% of the original model’s responses as refusals, compared with 18% after abliteration. On 1,172 ARC-Challenge questions, accuracy changed from 80.2% to 79.5%, a drop of about 0.7 percentage points. In a smaller, exploratory comparison of 10 open-ended answers, the judge preferred the original model in 6 cases, the modified model in 3, and called 1 a tie. The two larger evaluations are more informative than this small pairwise sample, but none establishes that general capability is fully preserved.
What matters
A one-dimensional intervention produced a large change in observable refusal behavior, but that does not mean every form of refusal lives in one direction. Some refusals remained, and the quality checks revealed trade-offs worth studying further. For me, the important lesson is how a simple geometric hypothesis can be tested causally, and why its behavioral effect needs to be evaluated alongside possible side effects.
Research reproduction of a published abliteration technique, not an original discovery of the refusal direction. Four prompts used to select the intervention came from the same test split as the subsequent 100-prompt evaluation, so that evaluation is not strictly untouched held-out data. Results are specific to the tested models, datasets and judge; the modified model is not established to be safe or suitable for deployment.