GenAI Field Guide

Under the hood (just enough)

Tokens

A token is the unit the model reads and writes: roughly three-quarters of an English word, about four characters. Everything is measured in tokens: your message, the hidden preamble, the reply, every tool result.

≈1 min
The · quick · brown · fox jump s over · the · lazy · dog A rough ruler 1 token ≈ ¾ word1,000 tokens ≈ 750 words ≈ 1.5 pages
A sentence split into tokens, and the rule of thumb for how much a thousand of them holds.

Rules of thumb

Amount Roughly
1 token ¾ of a word, about 4 characters
1,000 tokens 750 words, about 1.5 pages
100,000 tokens A 150-page document
1,000,000 tokens About 1,500 pages, if nothing else were in the prompt (something always is)

What tokenizes less efficiently

Numbers and tables, code, and non-English text all cost more tokens per word than English prose. The ¾-word rule is an English-prose average.