Token
The smallest unit of text that an LLM processes, typically a word, subword, or character.
In more detail
Tokenization converts text into tokens. LLMs have a maximum token limit (context window). Tokens affect model cost and capability - more tokens enable longer conversations. Different models tokenize differently; the same text may be different numbers of tokens across models.
Examples
- Word tokenization
- Subword tokenization (BPE)
- Character-level tokenization
- GPT token encoding
Related terms
Ready to Build Production AI Systems?
Join a small cohort learning directly from an engineer who's shipped AI at Atlassian and PhonePe scale.
20 weeks · Weekend live classes · 1:1 mentorship · 20+ projects.
20 Weeks
Live Weekend Classes
< 30 Seats
Small Batch
Next Soon
Get priority access
7-Day
Money-Back Guarantee
Taught by Debasish Maji · Senior AI Engineer · Ex-Atlassian (Rovo Agent) · Ex-PhonePe (550M+ users)
Get notified when the next batch opens + free AI resources