Week 1 · Mental model of an LLM

The guessing machine: what a language model really does, and how to close the gap

Zafrullah Khan, Ed.D. · about 6 minutes

Understand the machine first. Then train, fine-tune, and deploy honestly.

It doesn’t look things up

Here is the most common picture of a chatbot: somewhere inside, it keeps the pages it was trained on, and when you ask a question it goes and finds the right one.

That picture is wrong, and it causes a lot of wasted projects.

During training, a model reads an enormous amount of text and slowly adjusts its internal numbers, its parameters, so that it gets better at predicting that text. When training ends, the text itself is not kept inside the model. What’s left is the effect the text had on those numbers.

A guessing machine you can steer

So what does happen when you type a question? The model looks at the text so far and produces a ranked list of what could come next, one small piece of text (a token) at a time. It picks one, adds it, and guesses again.

Take the sentence “The capital of France is…”. You might expect the top guess to be “Paris”. Often it isn’t: a word like “the” can rank higher, because sentences such as “The capital of France is the city of Paris” are common in written English. The model isn’t wrong. It’s predicting text.

Everything a chatbot does, from answering questions to writing code, is built on that one step. Chat, summaries and tool use are clever arrangements of the text around the guesses.

That’s why the useful question is never “what does the model know?” but “what text is it guessing from, and how can I steer it?” You steer it with:

• the prompt you write,
• documents you hand it at the moment you ask, and, only when needed,
• training, which changes the habits behind its guesses.

Fluent is not the same as true

A guessing machine is very good at sounding right. Fluent, confident text is the easiest thing for it to produce, so fluency tells you the least about whether an answer is correct.

When a fact was rare or fuzzy in what the model read, it still produces a smooth answer, because a plausible guess usually scores better than an honest blank. That is where invented citations and confident mistakes come from. It isn’t a library glitch. It’s a guess.

Four ways to close a gap

Most first projects jump straight to “fine-tune it on my documents”. A calmer order of questions works better. Before you change anything, name the gap between the model you have and the behavior you want.

1. Prompting. Have you tried a strong prompt on a capable model, and actually scored the results? Many projects end happily here. Write the score down; it’s your baseline.
2. Retrieval. Is the gap missing or changing knowledge, such as manuals, menus, inventory or an internal wiki? Don’t bake a snapshot into the model. Hand it the right documents at the moment you ask.
3. Fine-tuning. Is the gap behavior: a house voice, a strict output format, a narrow skill? Fine-tuning can teach that pattern with examples. It is a poor way to add facts.
4. Training from scratch. Do you want to understand how models learn, or is your data unlike anything existing models have seen? Train a tiny model on purpose and watch it learn.

The right tool depends on the kind of gap, not on which tutorial is trending.

Maya’s week (an illustrative example)

Maya is an invented character used in the book to show the reasoning. She is not a real customer.

Maya writes a weekly gardening newsletter and would like help drafting sections. Friends tell her to “train her own model”. Before she does anything, she names her gaps, and finds two different ones in the same week:

• Her voice is behavior. A general model with a good prompt gets close but keeps sounding like a generic blog. She owns three years of her own past issues. That’s a fine-tuning-shaped gap, after she has tried a prompt-only baseline and set aside some favorite paragraphs as a test.
• This week’s frost table is knowledge. It changes every season. Training it into a model would bake in a snapshot that goes stale and invites confident invention. That’s a retrieval-shaped gap: give the model the current table when she asks.

Same newsletter, same week, two different tools.

Your move this week

You don’t need a GPU for this. You need one honest sentence:

“My gap is ___ (knowledge, behavior, or ‘I want to learn how it works’), so I’ll start with ___.”

Write it down before you touch any training code. If the answer is “prompt first”, that’s a good outcome, not a failure.

---

Where this comes from

This lesson follows draft Chapter 1 of Train Your Own LLM by Zafrullah Khan, Ed.D. The book is coming soon. Part I needs no code, and later parts are hands-on. → About the book

The lesson in six cards