How an AI System Sees a Sentence

This is Chapter 3 of MAGPAI: AI Under the Hood. Chapter 1 explained why a small transparent AI system is useful. Chapter 2 placed that system inside the broader AI and machine-learning stack. This chapter narrows the view to the first language transformation: a sentence becoming tokens.

A person reading this question immediately sees a business request:

Are MAG sales up in Chicago?

The reader recognizes that it is a question. The reader sees an organization, a metric, a direction of change, a location, and an implied request for an answer.

A program does not begin there. It receives a character sequence. Before a model can operate on the question, the text must move into a formal representation pipeline:

Human-readable sentence
    ->
Tokenizer
    ->
Sequence of tokens

That first transformation is easy to underestimate. Tokenization looks like a small formatting step, but it defines the units that the rest of the system can identify.

What a token is

A token is a unit produced by a tokenizer. It is not always a word.

Depending on the tokenizer, a token may represent a complete word, part of a word, punctuation, whitespace-associated text, a number, a special control symbol, or another learned or configured text unit. That means characters, words, tokens, and token IDs are related, but they are not the same thing.

At a conceptual level, the recurring MAGPAI question can be segmented like this:

Are | MAG | sales | up | in | Chicago | ?

That is a useful teaching view, but it should not be mistaken for a universal rule. A production language-model tokenizer may attach leading spaces, split uncommon words into subword pieces, handle punctuation differently, or use special tokens around prompts and messages.

Token IDs come next. Once a tokenizer has produced tokens, another stage can associate each token with a numeric identifier from a vocabulary. Chapter 4 will focus on that lookup. Chapter 3 stops at the question: what are the discrete text units?

Tokenization is a design decision

Tokenization is not neutral. It affects vocabulary size, sequence length, uncommon names, punctuation, multilingual behavior, memory use, inference cost, and the set of units available to the model.

Larger tokens can shorten a sequence because one token may cover more text. The cost is a larger vocabulary and weaker handling of unfamiliar text. Smaller tokens can represent unfamiliar words more flexibly, but they often produce longer sequences and more computation downstream.

That trade-off matters in real systems. A tokenizer changes how much input fits into a prompt window, how often text must be truncated or chunked, what gets counted for cost, and what intermediate state can be inspected or logged.

Words, subwords and characters

Common tokenization strategies differ in the unit they choose.

• Word-level tokenization splits text into word-like units. It is easy to read and useful for teaching, but it needs a strategy for unknown words.
• Character-level tokenization can represent arbitrary text because every word is made of characters. It is flexible, but it creates long sequences and loses word-like boundaries.
• Subword tokenization splits text into learned pieces that can compose unfamiliar words. Modern language models commonly use subword-like approaches because they balance vocabulary size and sequence length.

Named algorithms such as BPE, WordPiece, and SentencePiece are ways to build or apply subword tokenizers. MAGPAI’s Session 01 Tokenizer Lab does not implement one of those production tokenizer algorithms. It uses a deliberately simple manual tokenizer so the boundary is visible before the series moves into token IDs, vectors, and tensors.

Punctuation and whitespace matter

Punctuation can change structure and intent:

sales up
sales up?
sales, up
Chicago
Chicago?

The question mark in the recurring MAGPAI sentence is not decoration. It marks the sentence as a question, and the Session 01 manual tokenizer deliberately preserves it as a separate token.

Whitespace matters too because it gives the tokenizer boundaries to work with. The MAGPAI teaching tokenizer lowercases text, inserts a space before a question mark, trims the result, and then splits on whitespace. That is intentionally simpler than a production tokenizer, but it makes the transformation easy to inspect.

Unknown text and vocabulary boundaries

Tokenizers also expose a vocabulary boundary.

A word-level tokenizer may need an unknown-token strategy when it sees text outside its vocabulary. A subword tokenizer may split an unfamiliar word into smaller known pieces. A character tokenizer can represent arbitrary words, but it usually creates longer sequences.

MAGPAI’s Session 01 manual tokenizer is intentionally strict. It uses a tiny demonstration vocabulary. If the input contains a token outside that vocabulary, the tokenizer raises an error and reports the allowed vocabulary. That behavior is useful for teaching because the boundary is explicit.

Follow the MAG question

The verified Session 01 ManualTokenizer output for the recurring question is:

Original text
Are MAG sales up in Chicago?

Normalized text
are mag sales up in chicago ?

Token sequence
["are", "mag", "sales", "up", "in", "chicago", "?"]

Token count
7

The same source can map these tokens to IDs:

[1, 2, 3, 4, 5, 6, 7]

That numeric mapping is shown here only to mark the next boundary. The IDs are Chapter 4 material. For this chapter, the important point is that the human sentence has become an ordered list of discrete text units.

What tokenization preserves and what it loses

Tokenization preserves order, repeated units, punctuation when the tokenizer keeps it, and text boundaries defined by the tokenizer. It gives software a sequence it can process.

But tokenization alone does not understand the question. The token sales does not contain a data query. The token chicago does not contain a geographic database. The question mark does not prove that the system knows how to answer. Tokens are necessary intermediate state, not human-level meaning.

Later stages add vocabulary identifiers, vectors, embedding values, tensor structure, model-like behavior, deterministic application logic, data access, and chart-backed answer construction. Each stage creates a different artifact and a different inspection point.

Tokenization and system behavior

Tokenization has practical engineering consequences.

Teams need to know which tokenizer is used, whether it matches the model, how token counts are measured, when truncation occurs, how unusual input is handled, and what sensitive content is logged. Commercial APIs often meter by tokens, and local systems often allocate memory and latency budgets around sequence length. The exact limits and prices vary by provider and model, so the durable engineering question is not a specific number. It is whether the system knows what it is counting.

Observability should also respect privacy. Logging tokenized text can expose the same sensitive content as logging raw prompts. Token boundaries make a system easier to inspect, but they do not remove security and governance obligations.

Inspectability in MAGPAI

MAGPAI makes tokenization visible in several verified ways:

• src/magpai/tokenization/manual_tokenizer.py defines the Session 01 manual tokenizer and tiny vocabulary.
• src/magpai/tokenization/token_demo.py prints the original text, normalized text, tokens, token IDs, and embedding vectors for the recurring sentence.
• src/magpai/tokenization/token_streamlit_app.py provides a local Streamlit interface with a text input, allowed-vocabulary error handling, a transformation pipeline, and a token-to-ID-to-vector table.
• docs/session_01_tokens/Session_01_Tokens_Slide_Story_v1.2.md documents the exact normalization and tokenization teaching rule.
• The public repository includes automated test coverage for the broader chatbot path that uses the recurring Chicago sales question.

Those artifacts do not claim to implement a production LLM tokenizer. They make the first representation boundary visible.

What comes next

Chapter 3 ends with tokens. Chapter 4, From Tokens to Token IDs, will show how a vocabulary assigns stable numeric identifiers to those tokens.

The rest of the ten-chapter series continues from there: token IDs become vectors, embeddings learn useful relationships, vectors move through neural-network activations, prompt state becomes tensor-shaped, the technical representation connects to a chart-backed business answer, and the transparent chatbot becomes a bounded agent demonstration.

Explore the MAGPAI Tokenizer Lab

The companion Praxis artifact documents the Tokenizer Lab as an inspectable engineering artifact, including the verified source files, local run commands, demo status, and implementation limitations.

Explore the MAGPAI Tokenizer Lab

Interactive web demonstration coming soon.

About the author

Vasili Katsoulis

Vasili Katsoulis believes the future belongs to engineering leaders who understand both timeless software engineering principles and modern AI. His work explores how technical depth, architectural thinking, and engineering judgment enable organizations to build better software in the age of AI.