Language Modeling Is Compression

> Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms.

gzip can be used as a (not very good) LLM-like text and image generator: https://arxiv.org/abs/2309.10668

I'm doing my PhD in compression-based machine learning, wanted to contribute a few clarifying points.

The relationship between probabilistic modeling and lossless compression is very direct. A model that predicts the next symbol with probability p can on average losslessly compress that symbol with the help of an entropy coder (e.g. arithmetic coding) in -log(p) bits. Therefore improved probabilistic models immediately translate into improved lossless compressors.

There are two ways people have used compressors for ML: the first is based on the Minimum Description Length (MDL) principle [0] which says that the best model is the one which provides the shortest description of the data, counting the size of the model itself. This is similar to the technique used in this blog post (argmin code length across class-conditioned compressors) except for counting the model size. Basically you can train a compressor per class and then choose the class compressor which best compresses the test data. It is a maximum likelihood argument because of Shannon's source coding theorem: a code length L corresponds to a probability 2^-L. The second way is the Normalized Compression Distance (NCD) [1], which uses code lengths to calculate information-theoretic distances which can then be plugged into distance-based algorithms like kNN. MDL interprets compressed lengths as likelihoods, while NCD interprets them for distance calculations. The theoretical foundation is Kolmogorov complexity which is the ideal (& uncomputable) lossless compressor, used to define information distance, an "ideal" distance metric based on algorithmic similarity.

Data compression is not in principle restricted to syntactic similarity. As others have mentioned, the new wave of neural compressors (e.g. NNCP [2], CMIX [3], leading the large text compression benchmark [4]) outperform their traditional counterparts (e.g. gzip) because of the ability of neural networks to learn complex semantic patterns. Their improved ability to predict the next token means improved lossless compression. This has also been shown to be the case with pre-trained LLMs [5].

I think it's neat that improving compression improves machine learning, and improving machine learning improves compression!

Looking forward to hearing other thoughts.

[0] https://arxiv.org/abs/math/0406077 [1] https://arxiv.org/abs/cs/0111054 [2] https://bellard.org/nncp/nncp_v2.pdf [3] https://www.byronknoll.com/cmix.html [4] https://www.mattmahoney.net/dc/text.html [5] https://arxiv.org/abs/2309.10668

Have you worked in software long? I've been in eng for almost 30 years, started in EE. Can confidently say you can't trust the humans either. SWEs have been wrong over and over. No reason to listen now.

Just a few years ago code gen LLMs were impossible to SWEs. In the 00s SWEs were certain no business would trust their data to the cloud.

OS and browsers are bloated messes, insecure to the core. Web apps are similarly just giant string mangling disasters.

SWEs have memorized endless amount of nonsense about their role to keep their jobs. You all have tons to say about software but little idea what's salient and just memorized nonsense parroted on the job all the time.

Most SWEs are engaged in labor role-play, there to earn nation state scrip for food/shelter.

I look forward to the end of the most inane era of human "engineering" ever.

Everything software can be whittled down to geometry generation and presentation, even text. End users can label outputs mechanical turk style and apply whatever syntax they want, while the machine itself handles arithemtic and Boolean logic against memory, and syncs output to the display.

All the linguist gibberish in the typical software stack will be compressed[1] away, all the SWE middlemen unemployed.

Rotary phone assembly workers have a support group for you all.

[1] https://arxiv.org/abs/2309.10668

Well, they couldn't generate 2D artwork if they weren't capable of working with multiple output dimensions.

An interesting thing about transformers is that they are world-class at compressing 2D image data even when not trained on anything but textual language ( https://arxiv.org/abs/2309.10668 ). Whether that notion is relevant for 3D content would be two or three figures over my pay grade, though.

"Higher" than an EE with an MSc in elastic structures, ~30 years industry experience, now working with PhDs across the spectrum on energy models to embed in chips? Energy models in part, inferred from categorization of LLM contents and compression of those contents into geometric functions like I described?

"Higher level" implies acceptance of geometric structure. You place tokens like a Chomsky diagrams at each step up and down, where you should see parameters to transform geometry of the structure.

My team works "above" the contrived state management of software workers to more efficiently sync memory matrix to display matrix. LLMs are a form of compression [1]. My team is working on compressing them further into sets of points that make up each glyph and functions to recreate them.

Electromagnetic geometry transforms hardcoded[2] into hardware so reduce energy use of all the outdated string mangling of software dev as most know it.

What's higher level, relative to our machines, than design and implementation of the machine?

DnD dungeon master versus WOTC game designer.

Notice outside how there are no words and philosophy? Just color gradient and geometry?

Notice inside the human body no philosophy or words?

Language is not intelligence it's an emergent phenomena of geometry created by fundamental forces of physics organizing matter at various speeds relative to light.

You've read too much into an ultimately arbitrary statement meant to invoked a subtext, a subtle emotion context. You think of language as Legos, when it is music to feel.

[1] https://arxiv.org/abs/2309.10668 [2] https://iopscience.iop.org/article/10.1088/1742-6596/2987/1/...

Language modeling is compression, whittle down graph to reduce duplication and data with little relationship: https://arxiv.org/abs/2309.10668

Let’s say everyone agrees to refer to one hosted copy of a token “cat”, and instead generate a unique vector to represent their reference to “cat”.

Blam. Endless unique vectors which are nice and precise for parsing. No endless copies of arbitrary text like “cat”.

Now make that your globally distributed data base to bootstrap AI chips from. The data driven programming dream where other machines on the network feed new machines boot strap.

American tech industry is IBM now. Stuck on recent success of web SaaS and way behind the plans of AI.

Not to mention, Language Modeling is Compression https://arxiv.org/pdf/2309.10668

So text wikipedia at 24G would easily hit 8G with many standard forms of compression, I'd think. If not better. And it would be 100% accurate, full text and data. Far more usable.

It's so easy for people to not realise how massive 8GB really is, in terms of text. Especially if you use ascii instead of UTF.

Wasn't there that thing about how large LLM's are essentially compression algorithms (https://arxiv.org/pdf/2309.10668)? Maybe that's where this article is coming from, is the idea that finetuning "adds" data to the set of data that compresses well. But that indeed doesn't work unless you mix in the finetuning data with the original training corpus of the base model. I think the article is wrong though in saying it "replaces" the data - it's true that finetuning without keeping in the original training corpus increases loss on the original data, but "large" in LLM really is large and current models are not trained to saturation so there is plenty of room to fit in finetuning if you do it right.

We have to ditch language processing. And we will with online energy based models that machines boot from.

Maxwell could not get the theory of electromagnetism to work until he ditched pulleys and levers he’d included to describe the mechanics.

We won’t get AGI until we realize “there is no spoon” and language has nothing to do with our intelligence, just with out social tribalism: https://www.scientificamerican.com/article/you-dont-need-wor...

Take language out of the equation and drawing a circle, triangles, letters is just statistical physics. We can capture in energy models stored in an online state, statistical physics relative to the machine; its electromagnetic geometry: https://iopscience.iop.org/article/10.1088/1742-6596/2987/1/...

Our language doesn’t exist without humans. It’s not an immutable property of physics. It’s obfuscation and mind viruses. It’s story mode.

The computer acting as a web server or an LLM has an inherent energy model to it. New models of those patterns will be refined to a statefulness that strips away unnecessary language constructs in the system; like a lot of software most don’t use just developers.

I look forward to continuing my work in the hardware world to further compress and reduce the useless state of past systems of though we copy paste around to serve developers, to reduce context to sort through, and improve model quality: https://arxiv.org/abs/2309.10668

Single function factory hardware with embedded “prompt” that will boot from a model and the machines state will scaffold itself from there are coming: https://creativestrategies.com/jensen-were-with-you-but-were...

Yeh. Data driven model syncing machines will be what kills languages, software stacks.

All the chip world talk of single function machines and how tech is now energy constrained industry, means pruning the state we store; most software is software to deliver software.

Single function pipeline hardware platforms will act as little more than sync engines from models.

I mean it’s is 2025. Still write software like it’s the 1970s. So high tech.

Edit: https://arxiv.org/abs/2309.10668

From LLMs to energy models that transform electromagnetic geometry. Saving what’s needed to recreate cat pics and emails to mom. Pruning the tail as interest dips with generational churn.

This already happened with Linux where people with no understanding of electrical engineering got jobs memorizing abstractions.

I look forward to a data driven system future where a few functions transform the machines electromagnetic geometry to solve a task based upon the most efficient energy model for solving a task as we continue to compress from the model all the non-essential syntax sugar of modern software.

https://arxiv.org/abs/2309.10668

> It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning. For example, Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to 43.4% and LibriSpeech samples to 16.4% of their raw size, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively.

https://arxiv.org/pdf/2309.10668

Transformers are also used in the top algorithm right now on the Large Text Compression Benchmark. https://bellard.org/nncp/nncp.pdf

This is actually an excellent question ! And quite an active field of research.

This is one paper you could read - it's about text and not films, but same idea:

"LANGUAGE MODELING IS COMPRESSION" - https://arxiv.org/pdf/2309.10668

Vapnik: https://www.cs.princeton.edu/courses/archive/spring13/cos511... https://engineering.columbia.edu/files/engineering/vapnik.pd... https://www.learningtheory.org/learning-has-just-started-an-... https://nautil.us/teaching-me-softly-234576/

The main paper: https://gwern.net/doc/reinforcement-learning/exploration/act...

It sounds kinda crazy (is there really that much far transfer?), but you know, I think it would work... He just needed to use LLMs instead: https://arxiv.org/abs/2309.10668#deepmind

All we’re doing is engineering new data compression and retrieval techniques: https://arxiv.org/abs/2309.10668

Are we sure there’s anything “net new” to find within the same old x86 machines, within the same old axiomatic systems of the past?

Math is a few operations applied to carving up stuff and we believe we can do that infinitely in theory. So “all math that abides our axiomatic underpinnings” is valid regardless if we “prove it” or not.

Physical space we can exist in, a middle ground of reality we evolved just so to exist in, seems to be finite; I can’t just up and move to Titan or Mars. So our computers are coupled to the same constraints of observation and understanding as us.

What about daily life will be upended reconfirming decades old experiment? How is this not living in sunk cost fallacy?

When all you have is a hammer…

I’m reminded of Einstein’s quote about insanity.

The increasing recognition that "Language Modeling Is Compression" https://arxiv.org/pdf/2309.10668.pdf has not yet been accompanied by recognition that lossless compression is the most principled unsupervised loss function for world models in general, including foundation language models in particular.

Take, for instance, the unprincipled definition of "parameter count" not only in the LLM scaling law literature, but the Zoo of what statisticians called "Information Criteria for Model Selection". https://en.wikipedia.org/wiki/Model_selection#Criteria

The reductio ad absurdum of "parameter count" is arithmetic coding where an entire dataset can be encoded as a single "parameter" of arbitrary precision.

By contrast, the algorithmic bit of information (whether part of an executable instruction or program literal) is an unambiguous quantity up to the choice of instruction set. If you want to quibble about that instruction set choice, take it up with John Tromp https://tromp.github.io/cl/cl.html because what I'm about to propose obviates that along with a lot of other "arguments".

Since any executable archive of any kind of data can serve as a model of the world generating that data, it follows that any executable archive of any text corpus can serve as a language model with a rigorous "parameter count". Therefore, a procedure which runs LLM benchmarks against any such executable archive as a language model, contributes a uniquely rigorous data point to the literature on LLM scaling laws.

So, what I'm proposing is that authors of lossless compression algorithms consider adding a command-line option that, at the end of decompression, saves the state of the decompression process in a file that can be read back in and executed as a language model -- with the full understanding that these language models will perform very poorly on the vast majority of LLM benchmarks. The point is not to produce high quality language models. The point is to increase rigor in the research community by providing some initial data points that exemplify the approach.

Yes, next token prediction is exactly what you need for compression. Many papers on this, recent one

https://arxiv.org/pdf/2309.10668.pdf

Yes! Prompts are super finicky.

You have to create a prompt/function that for a wide set of inputs, generates a token sequence that will perpetually expand in a manner that corresponds to an externally observed truth.

Way too often it feels like you have to shove a universal decoding sequence into a prompt.

“Talk your steps, list your clues, etc.”

Just trying to luck into a prompt that keeps decompressing the model/ generating the next token that ensures the next token is true.*

I recall there was a paper with a relevant title recently… https://arxiv.org/abs/2309.10668

Basically - LLMs don’t reason , they regurgitate. If they have the right training data, and the right prompt, they can decompress the training data into something that can be validated as true

——-

* Also this has to be done in a limited context window, there is no long term memory, and there is no real underlying model of thought.

It goes along the same lines as the recent paper from Deepmind stating that DL (language modeling in their case) is compression.

https://arxiv.org/pdf/2309.10668.pdf