How LLMs Actually Work — field notes

■ FIELD NOTES

tokens · embeddings · attention · neural layers · next-token prediction · repeat × thousands · tokens · embeddings · attention · neural layers ·

VOL.1LLMno.07

An explainer, taken apart

How LLMs
actually work

Seven moving parts, drawn out by hand — from raw text to a finished answer.

Strip away the hype and a language model is a pipeline. Text goes in, gets chopped, turned into numbers, weighed against itself, pushed through layers of math — and out the far end comes a single guess at the next word. Here's each stage, one card at a time.

§01

The pipeline

01/07

Tokenization

🧩

First the sentence is broken into bite-size pieces called tokens.

Write me a Java app→

WritemeaJavaapp

02/07

Embeddings

🔢

Each token becomes a long list of numbers — a vector.

Java→[0.12, -0.55, 0.88, …]

Words with related meaning land close together: Java ≈ Kotlin ≈ Programming.

03/07

Attention

the transformer trick

🕸️

Every token gets to peek at every other token and decide what matters.

Createabanking appwithJava

Here app leans hard on Java.

Attention asks: “Which words should I care about right now?”

04/07

The layers

🧱

Billions of parameters reshape the meaning, level by level.

Early: grammar & spelling

Middle: concepts & context

Late: reasoning, planning, problem solving

05/07

Predict the next

🎯

The model scores how likely each possible next token is.

The capital of France is…

Paris92%

London4%

Berlin2%

other2%

06/07

Pick one

🎯

Chosen token → Paris 🎉

Append it to everything so far.
Feed the whole thing back in — and predict again.

07/07

And… output

🚀

Paris

Run that loop a few hundred or few thousand times and the single words add up to whole answers — essays, code, stories, plans.

§02

Notes in the margin

a fair question

Why does it feel like reasoning?

💡

The whole training goal is dull: guess the next token, billions of times over.

But to guess well, a model is quietly forced to pick up:

Grammar

Facts

Coding patterns

Cause & effect

Planning

Logic

⭐The twist: prediction at enormous scale starts to produce emergent abilities nobody hand-coded.

Plenty of researchers suspect “reasoning” shows up because good prediction quietly requires an internal model of how the world hangs together.

a mental model

Three stages, end to end

🧠

1Read the internet’s text
2Learn the patterns inside it
3Build internal representations compressed ideas, not stored sentences
4Predict the next token
5Generate answers, code, stories, plans

Less “advanced autocomplete,” more a world-simulator that happens to talk by predicting tokens.

bonus round

So how can it write code?

💻

Because during training it reads millions of cycles that look like this:

📝problem

💻code

▶️output

🐛bug fix

🔄refactor

So it internalises: “given this requirement, what code usually comes next?”

Same machinery — token prediction — but the patterns baked into the weights get rich enough to spit out programs that actually run.

TL;DR — THE ONE-LINER

Tokens → embeddings → attention → layers → predict the next token — on repeat, thousands of times per reply.

Set in Iowan / Palatino & a plain grotesk. Drawn by hand, no two cards alike.
— field notes on machine minds —