Understanding Tokens

If you have used AI tools, you have heard about "tokens." This guide explains tokens in depth, with interactive tools to help you see how your text gets tokenized.

Use the tokenizer tool in the dashboard to see how your text gets broken into tokens.

A token is the basic unit AI models use to process text. When you send a message, the model does not read characters or words. It reads tokens.

Tokens are the AI's vocabulary. Like humans break sentences into words, AI models break text into tokens. But tokens do not always match what we think of as words.

Tokens are not words#

Where it gets interesting:

Notice:

• Common words are often 1 token.
• Capitalization can change tokenization.
• Long or uncommon words split into pieces.
• Technical terms (manufacturer names, part numbers) often become multiple tokens.

The 4-character rule (and why it is wrong)#

You will hear "1 token is roughly 4 characters" as a quick estimate. Useful, but not accurate.

The approximation works for plain English. It breaks down for:

• Part numbers and SKUs: Crestron, QSC, and Extron model strings tokenize into many pieces.
• Non-English languages: usually more tokens per character.
• Code: varies based on syntax and naming.
• Numbers: can tokenize digit-by-digit.

Two kinds of tokens flow through every conversation.

Input tokens (prompt tokens)#

Everything you send to the AI:

• Your message.
• System instructions (set by the agent builder).
• Conversation history.
• Any retrieved knowledge or context (chunks pulled from a Crestron programming guide, for example).

Output tokens (completion tokens)#

Everything the AI generates:

• The response text.
• Any formatted content (markdown, code, etc.).

Why this matters for pricing.

Output tokens usually cost two to four times more than input tokens. Why?

1. Input: the model reads tokens (relatively fast).
2. Output: the model generates tokens one by one (compute-heavy).

That is why concise responses cost less than long, verbose ones. If your field tech agent answers in two crisp sentences instead of three paragraphs, your bill goes down.

AI models use a technique called Byte Pair Encoding (BPE) to build their vocabulary. Simplified:

1. Start with characters: individual characters as the base vocabulary.
2. Find common pairs: look for character pairs that appear together often.
3. Merge pairs: combine the most common pairs into single tokens.
4. Repeat: keep merging until you have a vocabulary of around 100,000 tokens.

The result is a vocabulary where:

• Very common words ("the", "is", "and") are single tokens.
• Common word parts ("-ing", "-tion", "un-") are single tokens.
• Rare words get split into known pieces.

Different models, different tokenizers#

Not every AI model tokenizes text the same way:

The same text can produce different token counts across models, though the differences are usually small for English.

1. Be concise in your instructions#

Shorter system prompts mean fewer input tokens on every message.

Verbose (45 tokens):

"I would like you to act as a helpful assistant that can answer questions about various topics related to AV systems. Please provide detailed and informative responses to any questions that AV technicians might ask you."

Concise (15 tokens):

"You are an AV support assistant. Give detailed, informative answers."

2. Consider response length#

If you do not need long responses, tell the agent:

• "Answer in 2 to 3 sentences."
• "Be brief."
• "Summarize in one paragraph."

3. Watch conversation length#

Every message in conversation history counts as input tokens. Longer conversations equal more tokens.

4. Code and technical content#

Code often tokenizes efficiently because:

• Common keywords (function, return, if) are single tokens.
• Variable names, model strings, and inline strings add up fast. A SIMPL# class with descriptive names will tokenize heavier than a tight Lua script.

Every AI model has a maximum number of tokens it can process at once. This is the context window. Models also have a max output limit on how many tokens they can generate in a single response.

The context window has to fit:

• System prompt.
• Conversation history.
• Your current message.
• Space for the response.

If you exceed the limit, older messages get trimmed from the conversation.

Prices are per 1 million tokens (including AVCodex platform markup):

Real token costs using GPT-5.4 ($3.25/M input, $19.50/M output):

Example 1: Quick room support question#

• Tech's question: 50 tokens ($0.000163).
• Agent response: 100 tokens ($0.00195).
• Total: $0.002 (less than 1 cent).

Example 2: Long document analysis (Biamp programming guide)#

• Document plus question: 5,000 tokens ($0.01625).
• Detailed response: 1,000 tokens ($0.0195).
• Total: $0.04 (4 cents).

Example 3: Extended troubleshooting session#

• 20 back-and-forth messages on a Crestron commissioning issue.
• Average input per turn: 2,000 tokens (includes history).
• Average output: 300 tokens.
• Total: about $0.25 (25 cents).

Budget-friendly alternative (Gemini 2.5 Flash)#

The same extended troubleshooting session on Gemini 2.5 Flash ($0.10/M input, $0.39/M output):

• Total: about $0.006 (less than 1 cent).

*AVCodex · Your AV expertise. Amplified by AI.*