Tokens
TokensTokenizer Lab
Inspect normalized text, tokens, token IDs, and a compact token table in a browser-based teaching lab.
What this demonstrates
Inspect normalized text, tokens, token IDs, and a compact token table in a browser-based teaching lab.
The curated explanation lives on Vasili. The executable application launches independently on demo.protepo.com so this narrative remains available in the original tab.
- Project
- MAGPAI
- Status
- available
- Version
- v1.1
- Execution host
- demo.protepo.com
Session 1 Context
The lab starts with a plain business question: Are MAG sales up in Chicago?A human can read the company, metric, direction, and market immediately, but a model cannot process the raw sentence directly.
This first conversion step makes the boundary visible. The demo normalizes the question, splits it into word-level tokens, keeps the question mark as its own token, and maps each token to an ID before later vector and data-backed answer steps can happen.
How to run it
- Open the interactive demo in a new tab.
- Use the controls in the executable page to inspect the model behavior or training flow.
- Return to this native Vasili details page for context, source relationships, and engineering narrative.