AI for beginners
01What is a token?
A token is a small piece of text that an AI works with – usually a short word or part of a word. “House” is often one token, a long word like “internationalisation” several. Rule of thumb: 1,000 tokens are roughly 750 English words, about a page and a half.
You pay per token: once for what you send in (input) and once for the answer (output). That’s why a short email costs a fraction of a cent, while summarising a whole book costs noticeably more.
02What is the context window?
The context window is a model’s “short-term memory”: how much text it can see at once – your question, attached documents and the conversation so far, all together. It’s measured in tokens.
Example: a model with a 32,000-token context can read about 40–50 pages at once. If your document is longer, it has to be split up – or you need a model with a bigger window. Bigger isn’t automatically better, though: long inputs cost more, and some models miss details in very long texts.
03What does the number of parameters mean?
Parameters are the “adjustment knobs” a model learned during training. A “70B” model has 70 billion of them. More parameters usually mean more knowledge – but also more computing power and more expensive hardware.
On its own, the number says little about quality: training data and training methods often matter more. A well-trained small model can beat a poorly trained big one – just as a thick encyclopedia doesn’t automatically explain things better than a good thin book. Many providers don’t even publish the number.
04What hardware do I need to run a model locally?
What matters most is your graphics card’s memory (VRAM). The model has to fit into it, otherwise it gets very slow. Rule of thumb for the usual compressed version: about 0.6 GB per billion parameters, plus 1–2 GB headroom.
Example: a 7B model needs around 6 GB, so it runs on an RTX 4060 with 8 GB. A 70B model needs more than 40 GB – no ordinary graphics card has that. Macs with M-series chips share their main memory with the GPU, so they can often handle bigger models. It also works without a graphics card, just much more slowly. On Windows you’ll find your GPU memory in Task Manager → Performance → GPU.