MAGPAI Tokenizer Lab

The MAGPAI Tokenizer Lab is the Chapter 3 Praxis artifact for MAGPAI: AI Under the Hood. It presents tokenization as an inspectable engineering step rather than as a hidden preprocessing detail.

The source project is the public MAGPAI GitHub repository. The verified Tokenizer Lab material is implemented in Python under the magpai.tokenization package, with a local Streamlit interface and MkDocs documentation. The public web demonstration route is prepared conceptually but not verified as deployed.

MAGPAI is an educational prototype. The Tokenizer Lab uses a deliberately simple manual tokenizer so readers can inspect each boundary. It is not a production language-model tokenizer and does not claim compatibility with a commercial LLM vocabulary.

Read the companion Noesis article

Project summary

• Artifact: MAGPAI Tokenizer Lab.
• Series: MAGPAI: AI Under the Hood.
• Chapter: 3 of 10.
• Source project: MAG Prototype AI.
• Maturity: active educational laboratory material.
• Primary runtime: Python.
• UI technology: Streamlit for the local tokenizer visualization.
• Source repository: github.com/javaboy-vk/MAGPAI.
• Public demo status: interactive web route not verified as deployed.

What the lab does

The Tokenizer Lab carries the recurring MAGPAI question through the first representation steps:

Text
    ->
Normalized text
    ->
Tokens
    ->
Token IDs
    ->
Embedding vectors

For Chapter 3, the important engineering artifact is the token sequence. Token IDs and vectors are visible in the lab because the demo continues into later stages, but those stages belong primarily to Chapters 4 and 5.

Verified implementation

The tokenizer implementation is located at:

src/magpai/tokenization/manual_tokenizer.py

The relevant class is ManualTokenizer. Its documented behavior is intentionally small:

• lowercase normalization;
• replacing ? with ? so the question mark becomes a separate token;
• stripping leading and trailing whitespace;
• splitting normalized text on whitespace;
• validating tokens against a tiny demonstration vocabulary;
• raising a visible error when unknown tokens appear.

The demonstration vocabulary is:

<PAD>   -> 0
are     -> 1
mag     -> 2
sales   -> 3
up      -> 4
in      -> 5
chicago -> 6
?       -> 7

Verified output

For the recurring sentence:

Are MAG sales up in Chicago?

the verified ManualTokenizer output is:

Normalized text
are mag sales up in chicago ?

Tokens
["are", "mag", "sales", "up", "in", "chicago", "?"]

Token IDs
[1, 2, 3, 4, 5, 6, 7]

Run Interactive Demo ->opens in a new tab

Source-code map

• src/magpai/tokenization/manual_tokenizer.py defines ManualTokenizer, TokenizationResult, and build_demo_vocab.
• src/magpai/tokenization/token_demo.py runs the recurring sentence through tokenization, token IDs, and a PyTorch embedding table.
• src/magpai/tokenization/token_streamlit_app.py exposes a local text input, error handling for unknown vocabulary, a transformation pipeline, and a token-to-ID-to-vector table.
• docs/session_01_tokens/Session_01_Tokens_Slide_Story_v1.2.md explains the tokenizer boundary and shows the teaching rule.
• docs/session_01_tokens/Session_01_Tokens_Integration_Note_v1.2.md documents local runtime commands and package placement.

Local run commands

From the MAGPAI repository root:

pip install -r requirements\session_01_tokens.txt
$env:PYTHONPATH="src"
python -m magpai.tokenization.token_demo
streamlit run src\magpai\tokenization\token_streamlit_app.py

The command-line demo prints the original sentence, normalized sentence, token list, token IDs, embedding vectors, and a short takeaway. The Streamlit app adds a local visual interface.

Demo URL treatment

The MAGPAI demo collection route for this website is:

https://vasili.protepo.com/demo/magpai

The expected descriptive route for a future Tokenizer Lab is:

https://vasili.protepo.com/demo/magpai/tokenizer

That tokenizer route is not linked here because it has not been verified as deployed in this site. The separate runtime domain https://demo.protepo.com is treated as a possible host for interactive demonstrations, not as a verified Tokenizer Lab URL.

Public Tokenizer Lab web demonstration: coming soon.

Engineering observations

This lab makes several engineering boundaries visible.

• Tokenization has a configuration; it is not just “split the sentence.”
• Token count affects downstream sequence size.
• The tokenizer and the vocabulary must agree.
• Tokenization does not create semantic understanding by itself.
• Logging tokenized content can expose sensitive input and must be handled carefully.

Those observations are generic. The exact behavior above is MAGPAI-specific and verified against the Session 01 manual tokenizer source.

Limitations

The Tokenizer Lab is intentionally constrained:

• it is an educational tokenizer rather than a production LLM tokenizer;
• the vocabulary is tiny and English-focused;
• punctuation handling is simplified;
• only the question mark is explicitly separated by the Session 01 manual tokenizer;
• unknown vocabulary raises an error instead of applying a subword fallback;
• there is no multilingual segmentation;
• there is no learned subword model;
• there are no production-scale performance guarantees.

Those limits are the point. They make the boundary inspectable before later chapters add more representation layers.

Relationship to future chapters

The Tokenizer Lab implements only the tokenization stage:

Sentence
    ->
Tokens
    ->
Token IDs
    ->
Vectors and embeddings
    ->
Neural-network activations
    ->
Prompt tensor
    ->
Business result

Chapter 4, From Tokens to Token IDs, will focus on the token and vocabulary inspector. Chapter 5 will continue from identifiers to vectors.

Related MAGPAI material

• Chapter 3 Noesis: How an AI System Sees a Sentence
• Chapter 2 Noesis: The AI and Machine-Learning Stack
• Chapter 2 Praxis: MAGPAI Stack Explorer
• Chapter 1 Praxis: Introducing MAGPAI
• Public MAGPAI GitHub repository

Chapter 4 preview: From Tokens to Token IDs - coming next.

Production disclaimer

MAGPAI and the Tokenizer Lab are educational engineering artifacts. They simplify tokenizer behavior to make representation boundaries visible. They are not production AI infrastructure and should not be used as a production tokenizer, safety mechanism, or model-compatibility layer.

MAGPAI publication roadmap

Publication Roadmap: Ten Chapters

MAGPAI: AI Under the Hood will be published incrementally as a ten-chapter series. Each chapter connects a technical concept to source code, diagrams or an executable demonstration in the MAGPAI engineering laboratory.

1. 1

   Introducing MAGPAI: A Tiny Transparent AI System

   Available now

   Connects the current Noesis article and Praxis project page to the repository structure, sample business scenario, and demonstration laboratory.

   • Noesis article
   • Praxis project
2. 2

   The AI and Machine-Learning Stack

   Available now

   Presents an accessible stack explorer based on the verified MAGPAI Session 01 stack material.

   • Noesis article
   • Praxis project
3. 3

   How an AI System Sees a Sentence

   Available now

   Documents the Tokenizer Lab using the verified ManualTokenizer output for the recurring question: Are MAG sales up in Chicago?

   • Noesis article
   • Praxis project
4. 4

   From Tokens to Token IDs

   Coming next

   Previews vocabulary and token-ID inspection, including vocabulary-training demonstrations where supported.

5. 5

   From Token IDs to Vectors

   Coming soon

   Previews an Embedding Lab that shows token IDs and their corresponding numerical vectors.

6. 6

   How Embeddings Learn Meaning

   Planned

   Previews vocabulary or embedding training views that compare vector behavior before and after training.

7. 7

   From Embeddings to Neural-Network Activations

   Planned

   Previews existing neural-network visualizations with inspectable activations and parameter changes where available.

8. 8

   Tensors and Prompt Representation

   Planned

   Previews a prompt-tensor inspector for the recurring MAG business question.

9. 9

   From Question to Business Insight

   Planned

   Previews the chart-backed answer flow for Are MAG sales up in Chicago?

10. 10

    From Transparent Chatbot to Agent

    Planned

    Previews the chatbot, trace mode, and agent demonstrations with clear separation between educational behavior and implemented capability.

Read the companion Noesis article