LLMs learn by compression, as Ilya Sutskever argues

Anusheel BhushanAnusheel Bhushanco-founder of PlankOriginally posted on LinkedIn 

I love this video re: LLMs. Its all about compression!
The text on the internet is a mapping / representation of all human knowledge. If you compress that, you force structure and the system learns something about the underlying "processes" that produced the text.

A simple example is if you compress a bunch of math tables (that have examples of addition/multiplication/division etc etc), the most efficient compression of that data is a calculator function!

This should also help people understand that these models are not just statistical prediction machines but truly are able to reverse engineer how things work and hence able to "compose" this understanding to derive "new insights".

Once I heard Ilya Sutskever's argument re: compression, it seemed obvious to me in hindsight...wish I had come up with this in 2012 times :) Its also an example of compounding - people who had internalized this early just had a large head start on everyone else.

There is a lot of power in relentless pursuit of the most simple yet non-obvious ideas.

youtu.be/GI4Tpi48DlA

Highlights of the Fireside Chat with Ilya Sutskever & Jensen Huang: AI Today & Vision of the Futureyoutu.be
  • Models and research

Keep reading

Software architecture

Missing DTrace from Solaris for tracing live processes

I miss dtrace from Solaris. Dtrace probes would allow you to attach the trace to live running processes, without needing to restart the process. Some things you could check: - print which functions are doing memory, disk allocation or network requests incl the sizes. This was great to catch memory leaks or programs with poor cache performance - trace a specific code function. Using the symbol for the function in the binary, you could print the parameters and function return values - trace the latency of a user request by tracing invocations of functions at the start/end of the processing - performance measurement of modules. Attach the probe for an hour and do a running count of how many times a function was called or how much memory was used on average for a request Linux strace seemed not as powerful. Any other options? brendangregg.com/dtrace.html

Models and researchSoftware architecture

Quantized Llama models make on-device AI apps practical

With the latest quantized Llama models that are just 1.3GB and 3GB in size, you can start doing mobile/desktop apps that download fine tuned models per user/customer and can run fully locally. This makes variable running costs of AI apps zero and moreover achieves maximal data privacy. We are building a few apps with this new awesome architecture. Anyone else doing that? Would love to exchange notes.

Software architecture

When RAG and vector databases are the wrong tool

Common error in app-layer LLM products is that people use RAG and Vector DBs incorrectly. To process a single document or small amount of input information, people first chunk it, throw it in a vectorDB then retrieve it and put in the context. That makes no sense if the original document is small and could literally be fully packaged in the context window. Remember - you need RAG if during inference, you need to select a subset of information from thousands of documents (100s of millions or billions of tokens), which of course is impossible to include in the context window of a single inference.

Software architecture

Eras of software from the mainframe age to AI

I've been reflecting on the evolution of software development and the distinct eras the industry has experienced over the years. For each era, I've considered the defining characteristics, the technological advancements that emerged, and the market segments most affected or newly created by these innovations. Dive deeper into this exploration on how software has transformed from the mainframe age to today's AI-driven landscape. In a follow-up post, I will talk about how the tooling has evolved from assembly languages to high-level languages to now plain-english prompts and agentic pipelines.

All postsOlder posts