# Understanding Tokens

What tokens are, how they get counted, and why they matter for AI pricing on AV-scale traffic.

---

If you have used AI tools, you have heard about "tokens." This guide explains tokens in depth, with interactive tools to help you see how your text gets tokenized.

## Try it yourself

Use the tokenizer tool in the dashboard to see how your text gets broken into tokens.

## What is a token

A token is the basic unit AI models use to process text. When you send a message, the model does not read characters or words. It reads tokens.

Tokens are the AI's vocabulary. Like humans break sentences into words, AI models break text into tokens. But tokens do not always match what we think of as words.

### Tokens are not words

Where it gets interesting:

| Text | Tokens | Count |
|---|---|---|
| "hello" | `hello` | 1 |
| "Hello" | `Hello` | 1 |
| "HELLO" | `HE`, `LLO` | 2 |
| "tokenization" | `token`, `ization` | 2 |
| "Crestron" | `Cres`, `tron` | 2 |
| "ChatGPT" | `Chat`, `G`, `PT` | 3 |

Notice:

- Common words are often 1 token.
- Capitalization can change tokenization.
- Long or uncommon words split into pieces.
- Technical terms (manufacturer names, part numbers) often become multiple tokens.

### The 4-character rule (and why it is wrong)

You will hear "1 token is roughly 4 characters" as a quick estimate. Useful, but not accurate.

| Text | Characters | Actual tokens | 4-char estimate |
|---|---|---|---|
| "The quick brown fox" | 19 | 4 | 5 |
| "supercalifragilisticexpialidocious" | 34 | 9 | 9 |
| "DM-NVX-E30C" | 11 | 6 | 3 |
| "日本語" | 3 | 3 | 1 |
| `EISC.SystemReboot()` | 20 | 8 | 5 |

The approximation works for plain English. It breaks down for:

- **Part numbers and SKUs**: Crestron, QSC, and Extron model strings tokenize into many pieces.
- **Non-English languages**: usually more tokens per character.
- **Code**: varies based on syntax and naming.
- **Numbers**: can tokenize digit-by-digit.

## Input vs output tokens

Two kinds of tokens flow through every conversation.

### Input tokens (prompt tokens)

Everything you send to the AI:

- Your message.
- System instructions (set by the agent builder).
- Conversation history.
- Any retrieved knowledge or context (chunks pulled from a Crestron programming guide, for example).

### Output tokens (completion tokens)

Everything the AI generates:

- The response text.
- Any formatted content (markdown, code, etc.).

**Why this matters for pricing.**

Output tokens usually cost two to four times more than input tokens. Why?

1. **Input**: the model reads tokens (relatively fast).
2. **Output**: the model generates tokens one by one (compute-heavy).

That is why concise responses cost less than long, verbose ones. If your field tech agent answers in two crisp sentences instead of three paragraphs, your bill goes down.

## How tokenization works

AI models use a technique called **Byte Pair Encoding (BPE)** to build their vocabulary. Simplified:

1. **Start with characters**: individual characters as the base vocabulary.
2. **Find common pairs**: look for character pairs that appear together often.
3. **Merge pairs**: combine the most common pairs into single tokens.
4. **Repeat**: keep merging until you have a vocabulary of around 100,000 tokens.

The result is a vocabulary where:

- Very common words ("the", "is", "and") are single tokens.
- Common word parts ("-ing", "-tion", "un-") are single tokens.
- Rare words get split into known pieces.

### Different models, different tokenizers

Not every AI model tokenizes text the same way:

| Model family | Tokenizer | Vocabulary size |
|---|---|---|
| GPT-4.1, GPT-5.x | o200k_base | about 200,000 |
| Claude 4.x | Claude tokenizer | about 100,000 |
| Gemini 2.5/3 | SentencePiece | about 256,000 |

The same text can produce different token counts across models, though the differences are usually small for English.

## Practical tips for managing tokens

### 1. Be concise in your instructions

Shorter system prompts mean fewer input tokens on every message.

**Verbose (45 tokens):**

> "I would like you to act as a helpful assistant that can answer questions about various topics related to AV systems. Please provide detailed and informative responses to any questions that AV technicians might ask you."

**Concise (15 tokens):**

> "You are an AV support assistant. Give detailed, informative answers."

### 2. Consider response length

If you do not need long responses, tell the agent:

- "Answer in 2 to 3 sentences."
- "Be brief."
- "Summarize in one paragraph."

### 3. Watch conversation length

Every message in conversation history counts as input tokens. Longer conversations equal more tokens.

### 4. Code and technical content

Code often tokenizes efficiently because:

- Common keywords (`function`, `return`, `if`) are single tokens.
- Variable names, model strings, and inline strings add up fast. A SIMPL# class with descriptive names will tokenize heavier than a tight Lua script.

## Token limits (context windows)

Every AI model has a maximum number of tokens it can process at once. This is the **context window**. Models also have a **max output** limit on how many tokens they can generate in a single response.

| Model | Context window | Max output |
|---|---|---|
| GPT-5.4 | 1,000,000 tokens | 32,768 tokens |
| GPT-5.4 Mini | 1,000,000 tokens | 32,768 tokens |
| GPT-4.1 | 1,000,000 tokens | 32,768 tokens |
| Claude Opus 4.6 | 1,000,000 tokens | 128,000 tokens |
| Claude Sonnet 4.6 | 1,000,000 tokens | 64,000 tokens |
| Claude Haiku 4.5 | 200,000 tokens | 64,000 tokens |
| Gemini 2.5 Pro | 1,000,000 tokens | 65,536 tokens |
| Gemini 2.5 Flash | 1,000,000 tokens | 65,536 tokens |
| o3 | 200,000 tokens | 65,536 tokens |
| o4-mini | 200,000 tokens | 65,536 tokens |

The context window has to fit:

- System prompt.
- Conversation history.
- Your current message.
- Space for the response.

If you exceed the limit, older messages get trimmed from the conversation.

## Model pricing comparison

Prices are per 1 million tokens (including AVCodex platform markup):

| Model | Input $/M | Output $/M | Best for |
|---|---|---|---|
| GPT-5.4 | $3.25 | $19.50 | General purpose, latest capabilities. |
| GPT-5.4 Mini | $0.98 | $5.85 | Fast, cost-effective. |
| GPT-5.4 Nano | $0.26 | $1.63 | Ultra-fast, simple lookups. |
| GPT-4.1 | $2.60 | $10.40 | Long context, reliable coding. |
| Claude Opus 4.6 | $6.50 | $32.50 | Deep reasoning, complex tasks (RFP scoring, design). |
| Claude Sonnet 4.6 | $3.90 | $19.50 | Writing, analysis, balanced. |
| Claude Haiku 4.5 | $1.30 | $6.50 | Fast, affordable. |
| Gemini 2.5 Pro | $1.63 | $6.50 | Long context, multimodal (rack photos, drawings). |
| Gemini 2.5 Flash | $0.10 | $0.39 | Fast, very affordable. |
| Gemini 2.5 Flash Lite | $0.05 | $0.20 | Ultra-cheap, simple tasks. |
| o3 | $13.00 | $52.00 | Advanced reasoning. |
| o4-mini | $3.90 | $15.60 | Reasoning on a budget. |

## Cost examples

Real token costs using GPT-5.4 ($3.25/M input, $19.50/M output):

### Example 1: Quick room support question

- Tech's question: 50 tokens ($0.000163).
- Agent response: 100 tokens ($0.00195).
- **Total: $0.002** (less than 1 cent).

### Example 2: Long document analysis (Biamp programming guide)

- Document plus question: 5,000 tokens ($0.01625).
- Detailed response: 1,000 tokens ($0.0195).
- **Total: $0.04** (4 cents).

### Example 3: Extended troubleshooting session

- 20 back-and-forth messages on a Crestron commissioning issue.
- Average input per turn: 2,000 tokens (includes history).
- Average output: 300 tokens.
- **Total: about $0.25** (25 cents).

### Budget-friendly alternative (Gemini 2.5 Flash)

The same extended troubleshooting session on Gemini 2.5 Flash ($0.10/M input, $0.39/M output):

- **Total: about $0.006** (less than 1 cent).

*AVCodex · Your AV expertise. Amplified by AI.*