Language Models Use Trigonometry to Do Addition

> So Jev does actually know which distributions are correct, it just fails to produce them

Claims like this needs to be deeply analyized. Models do have some emergent capabilities[1], and I think there's a lot of evidence to show that semantics is actually learned (Word2Vec), and some math seems like it also might be learned (e.g. modular arithmetic). But it's hard to exactly say where there's some internal mechanism generating a true answer and where we're just getting lucky with some distribution so the answer just seems right.

[1] https://arxiv.org/pdf/2502.00873

> If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.

I think the field deserves more credit than that, there are plenty of interpretability tools like

* natural language autoencoders for explanations of activations: https://transformer-circuits.pub/2026/nla/index.html (demo at https://www.neuronpedia.org/llama3.3-70b-it/nla )

* easier-to-interpret language model families like Backpack models: https://aclanthology.org/2023.acl-long.506/

* attribution graphs to trace internal reasoning steps: https://www.anthropic.com/research/open-source-circuit-traci... (demo at https://www.neuronpedia.org/gemma-2-2b/graph)

* functional analyses which have identified how LLMs do arithmetic - https://arxiv.org/html/2502.00873v1 - and how refusal happens: https://arxiv.org/abs/2406.11717

* data attribution methods linking training data to specific attention heads https://arxiv.org/abs/2601.21996

If we could give a comprehensive and global explanation of an LLM's behavior in a single paragraph, we wouldn't need the model to begin with, but that doesn't mean there's absolutely no understanding of the model internals whatsoever

There have been multiple studies on how LLMs add numbers.

They use the "Clock" algorithm, the "Pizza" algorithm, a few other ones.

> All networks we study implement the same simple neuron model in their first-layer MLPs: degree-1 sinusoidal fits in layer 1, with deeper layers combining into degree-2 sinusoidal interactions.

https://neurips.cc/virtual/2025/loc/san-diego/133808

https://arxiv.org/abs/2502.00873

When you ask them they don't mention these at all, they give you high-school math:

https://chatgpt.com/share/6a82afdd-872c-83eb-aad7-622d27f2dc...

> did you use the "cos" or "sin" function at all during this addition computation?

> No. There is no need for trigonometric functions like sin or cos. The computation only uses basic arithmetic and place-value reasoning. Of course, if someone were implementing arithmetic in a computer, it is theoretically possible to express addition using extremely complicated formulas involving sin and cos. But in the reasoning I described, no trigonometric functions were involved at all. I simply decomposed the numbers into hundreds and smaller parts and added them.

> The LLM can't describe it, it doesn't see inside itself, however, it has many textbooks in its training dataset, so it will grab an answer from these textbooks because that's how people answer.

EDIT: I see now that you were referring to the answers it uses to justify the result, not the underlying computations. Sorry! You can disregard the actual comment. Leaving for completeness.

ORIGINAL COMMENT:

That's not how it works. Addition in LLMs is believed to function through different mechanisms depending on model size and architecture, but the single consistent finding across different models is that they generalize beyond the training data for at least those simple arithmetic operations.

For example: "Language models use trigonometry to do addition" https://arxiv.org/abs/2502.00873

For a different "emergent" algorithm, see Anthropic's observations: https://transformer-circuits.pub/2025/attribution-graphs/met...