HackerHACKER PAPER · 2025 · arXiv preprint
Language Models Use Trigonometry to Do Addition
What question does this paper answer?
When a model writes 3+63=66, what happens internally? This paper studies addition in GPT-J, Pythia-6.9B, and Llama-3.1-8B. It fits integer representations with a generalized helix containing linear and periodic components, then uses activation patching, ablations, and component analysis to test their computational role. This is stronger evidence than an attractive projection plot, but the precise computation turning the two input helices into the answer helix remains unidentified.
What it contributes
- 01
The main task adds two integers from 0 to 99 in a regime where the answer fits a single token. It is not a mechanistic explanation of arbitrary-length arithmetic or all mathematical reasoning.
- 02
The numerical fit combines a linear term with sine and cosine components of periods 2, 5, 10, and 100. Interventions, rather than plots alone, test whether these components preserve information used for addition.
- 03
Detailed GPT-J analysis separates components that carry input representations, construct an answer representation, and write an answer. The proposed implementation through trigonometric identities remains a hypothesis that the authors could not isolate.
Check the original passages
Pages count from the first page of the PDF file. Page jumps depend on the source viewer; section names and short quotes provide another way to find the passage.
all patching experiments only use prompts
the model can successfully complete.we do not know the exact mechanismGuide source: arXiv v1 preprint ↗ · unknown
Read this paper alongside other sources
These readings connect the paper with specific arguments from talks and other sources.
Hacker News comments 4
Verified mentions linking to this paper, newest first. This is not the complete comment tree of any one submission. Recommendations, references, and criticism are labeled separately; comments remain in their original language.
Claims like this needs to be deeply analyized. Models do have some emergent capabilities[1], and I think there's a lot of evidence to show that semantics is actually learned (Word2Vec), and some math seems like it also might be learned (e.g. modular arithmetic). But it's hard to exactly say where there's some internal mechanism generating a true answer and where we're just getting lucky with some distribution so the answer just seems right.
[1] https://arxiv.org/pdf/2502.00873
I think the field deserves more credit than that, there are plenty of interpretability tools like
* natural language autoencoders for explanations of activations: https://transformer-circuits.pub/2026/nla/index.html (demo at https://www.neuronpedia.org/llama3.3-70b-it/nla )
* easier-to-interpret language model families like Backpack models: https://aclanthology.org/2023.acl-long.506/
* attribution graphs to trace internal reasoning steps: https://www.anthropic.com/research/open-source-circuit-traci... (demo at https://www.neuronpedia.org/gemma-2-2b/graph)
* functional analyses which have identified how LLMs do arithmetic - https://arxiv.org/html/2502.00873v1 - and how refusal happens: https://arxiv.org/abs/2406.11717
* data attribution methods linking training data to specific attention heads https://arxiv.org/abs/2601.21996
If we could give a comprehensive and global explanation of an LLM's behavior in a single paragraph, we wouldn't need the model to begin with, but that doesn't mean there's absolutely no understanding of the model internals whatsoever
They use the "Clock" algorithm, the "Pizza" algorithm, a few other ones.
> All networks we study implement the same simple neuron model in their first-layer MLPs: degree-1 sinusoidal fits in layer 1, with deeper layers combining into degree-2 sinusoidal interactions.
https://neurips.cc/virtual/2025/loc/san-diego/133808
https://arxiv.org/abs/2502.00873
When you ask them they don't mention these at all, they give you high-school math:
https://chatgpt.com/share/6a82afdd-872c-83eb-aad7-622d27f2dc...
> did you use the "cos" or "sin" function at all during this addition computation?
> No. There is no need for trigonometric functions like sin or cos. The computation only uses basic arithmetic and place-value reasoning. Of course, if someone were implementing arithmetic in a computer, it is theoretically possible to express addition using extremely complicated formulas involving sin and cos. But in the reasoning I described, no trigonometric functions were involved at all. I simply decomposed the numbers into hundreds and smaller parts and added them.
EDIT: I see now that you were referring to the answers it uses to justify the result, not the underlying computations. Sorry! You can disregard the actual comment. Leaving for completeness.
ORIGINAL COMMENT:
That's not how it works. Addition in LLMs is believed to function through different mechanisms depending on model size and architecture, but the single consistent finding across different models is that they generalize beyond the training data for at least those simple arithmetic operations.
For example: "Language models use trigonometry to do addition" https://arxiv.org/abs/2502.00873
For a different "emergent" algorithm, see Anthropic's observations: https://transformer-circuits.pub/2025/attribution-graphs/met...
Direct submissions and discussions (1)
Language Models Use Trigonometry to Do Addition · 1 points · 2025-02-04 ↗