# How Knowledge Sources Work

How your AS-builts, datasheets, and SOPs become searchable AI knowledge through embeddings and semantic search.

---

When you upload a document to your AVCodex agent, real work happens behind the scenes. Files get transformed into a format the AI can search and understand by meaning, not only by keywords. This guide explains how, in plain terms.

## The big picture

Here is what happens when you add a knowledge source:

1. **You upload a file** (PDF spec sheet, programming guide, room CAD drawing, internal SOP, URL).
2. **We extract the text** from the file.
3. **We split it into chunks** (smaller, manageable pieces).
4. **We create embeddings** (numerical representations of meaning).
5. **We store everything** in a searchable database.
6. **When users ask questions**, we find the most relevant chunks and pass them to the AI.

Let us walk through each step.

## Step 1: text extraction

Different file types need different handling. We support:

| File type | What we extract |
|---|---|
| **PDFs** (manufacturer manuals, AS-builts, commissioning packets) | Text, tables, content from complex layouts. |
| **Word documents** (SOPs, design narratives) | Formatted text and structure. |
| **Excel/CSV** (gear lists, BOMs, room schedules) | Tabular data converted to readable text. |
| **Web URLs** (manufacturer help sites, internal wikis) | Page content, cleaned of nav and ads. |
| **YouTube videos** (training, install walkthroughs) | Transcripts and captions. |
| **Images** (rack photos, screenshots of error codes) | Text extracted with AI vision. |
| **Plain text/Markdown** | Direct content. |

The goal is clean, readable text regardless of the source format.

## Step 2: chunking

AI models cap how much text they can process at once. And when you ask a question, you do not need an entire 200-page programming guide. You need the relevant section.

**Chunking** splits your text into smaller pieces (typically a few pages each). We try to split at natural boundaries (between paragraphs or sections) so each chunk holds a complete thought.

**Example:** A 60-page Biamp Tesira datasheet might become 15 to 20 chunks, each holding related content that stands on its own.

## Step 3: creating embeddings

This is where the useful magic happens.

### What is an embedding

An embedding is a list of numbers that represents the *meaning* of a piece of text. Think of it as coordinates on a giant map of concepts.

**Simple analogy.** Imagine a map where:

- "DSP" and "audio processor" are placed close together (similar meaning).
- "DSP" and "switcher" are somewhat close (both rack gear).
- "DSP" and "refrigerator" are far apart (unrelated).

Embeddings work the same way, but in thousands of dimensions instead of two. Text with similar meaning gets similar numbers.

### Why this matters

Traditional search looks for exact keyword matches. If your document says "audio processor" but a tech searches "DSP," traditional search might miss it.

**Semantic search** with embeddings understands that the two mean the same thing. It finds relevant content based on meaning, not just matching words.

### The numbers

Each chunk of text becomes a vector of about 3,000 numbers. These numbers encode:

- What the text is about.
- The context and tone.
- Relationships to other concepts.

You do not need to understand the math. Similar content produces similar numbers, which makes searching very effective.

## Step 4: storage

We store both:

- **The original text chunks** (so we can hand them to the AI).
- **The embeddings** (so we can search by meaning).

This lives in a specialized vector database tuned for finding similar embeddings fast.

## Step 5: retrieval (when users ask questions)

Here is what happens when a user asks your agent a question.

### 1. Query embedding

We turn the user's question into an embedding using the same process. "How do I reset a DM-NVX endpoint to factory defaults?" becomes a vector of numbers.

### 2. Similarity search

We compare the question's embedding to every stored chunk embedding. The database finds chunks whose numbers are closest to the question's.

### 3. Relevance scoring

Each chunk gets a similarity score (0 to 1). Higher scores mean more relevant.

### 4. Context assembly

We take the top-scoring chunks and pass them to the AI as context: "Here is relevant information from the knowledge base. Use it to answer this question."

### 5. AI response

The AI reads the retrieved chunks and writes an answer grounded in your actual content, not its general training data.

## Why this approach works

### Semantic understanding

Users do not have to know the exact words in your documents. Ask about "switcher" and find content about "matrix," "router," or "presentation switcher."

### Accuracy

By giving the AI specific, relevant content, answers ground in your actual information rather than generic responses.

### Scale

Whether you have 10 documents or 10,000 (your full Crestron, QSC, Extron, Biamp, and Shure libraries), search stays fast because embeddings can be compared efficiently.

### Relevance

Instead of dumping entire documents into the AI (which would blow context limits and pull in noise), we surface only the pieces that matter.

## Hybrid search: smarter retrieval for large libraries

When you have dozens or hundreds of knowledge sources, finding the right information gets harder. A chunk-by-chunk search might return text that *sounds* relevant but comes from the wrong document entirely.

That is where **hybrid search** comes in.

### The problem with pure chunk search

Imagine you have 50 documents uploaded:

- Internal Commissioning Checklist.
- Crestron SIMPL Programming Guide.
- Q-SYS Lua Module Library.
- Biamp Tesira Spec Sheets.
- Network Standards for AV-over-IP.
- And 45 more.

A user asks: "How do I reset the system?"

Pure chunk search might return:

1. "Reset network configuration on the headend switch" (Network Standards).
2. "Factory reset procedure for DM-NVX endpoints" (Crestron Programming Guide).
3. "Reset Lua module state on Q-SYS Core" (Q-SYS Lua library).

The first result has high text similarity to "reset" but comes from the wrong context entirely. The user wanted Crestron, not network gear.

### How hybrid search solves it

We use a two-tier approach.

**Tier 1: document-level understanding.**

When you upload a document, we generate an AI summary of the whole document and create an embedding for it. That captures what the document is *about* at a high level.

For example:

- Crestron SIMPL Programming Guide -> "Programming reference for Crestron SIMPL and SIMPL# control system code, including module patterns and signal definitions."
- Network Standards for AV-over-IP -> "Network configuration and switch configuration for Dante, AES67, and DM-NVX deployments."
- Q-SYS Lua Module Library -> "Lua reference for Q-SYS programming, including UCI and design patterns."

**Tier 2: combined scoring.**

When a user asks a question, we:

1. Find which *documents* are most relevant to the question.
2. Search for chunks, but **boost chunks from relevant documents**.

For "How do I reset a DM-NVX endpoint?":

- Crestron SIMPL Programming Guide gets a high document relevance score (0.92).
- Network Standards gets a low score (0.45).

A chunk from the Crestron guide with slightly lower text similarity (0.78) now outranks a chunk from Network Standards with higher text similarity (0.85), because it comes from the right context.

### Why document summaries matter

Document summaries act as a table of contents for your full knowledge base. They help the system understand:

- What each document is about.
- Which documents are relevant to a query.
- How to prioritize chunks from contextually appropriate sources.

This is most powerful when you have:

- **Many similar documents** (multiple manufacturer programming guides, multiple commissioning checklists).
- **Overlapping terminology** ("switcher" in DM contexts vs Q-SYS contexts).
- **Large libraries** (hundreds of documents with thousands of chunks).

### The result

Hybrid search delivers more accurate answers by understanding context at both the document and chunk level. Users get responses from the right source, not just text that happens to contain similar words.

## Real-world example

**Scenario.** You have uploaded your company's commissioning SOP, a 200-page Crestron programming guide, and the AS-built packet for the Acme HQ project.

**User asks:** "What is the IP scheme for the boardroom on level 12 of Acme HQ?"

**What happens:**

1. The question gets converted to an embedding.
2. The system searches across your chunks.
3. It finds the chunk in the Acme AS-built packet that lists the level 12 boardroom IP scheme (high similarity score).
4. It also finds related chunks about the building's network standards (moderate similarity).
5. The AI receives those chunks as context.
6. The AI responds with the actual IP range and gateway from the AS-built, citing the source document.

The AI did not have to read the entire AS-built packet. Just the relevant sections.

## Tips for better results

### 1. Quality content

The AI can only find what is in your documents. Make sure your knowledge sources contain the information users will ask about.

### 2. Clear writing

Well-organized, clearly written content produces better embeddings. If a human would struggle to read your AS-built notes, the AI will too.

### 3. Comprehensive coverage

If users keep asking questions you cannot answer (a particular Yamaha mixer, a Shure ceiling array config), add more knowledge sources to cover those topics.

### 4. Keep sources current

Outdated documents lead to outdated answers. Refresh your knowledge sources when manufacturer firmware revs, programming guides, or your own SOPs change.

## Supported file types

| Type | Extensions | Best for |
|---|---|---|
| Documents | PDF, DOCX, DOC | Manuals, programming guides, AS-builts, reports. |
| Spreadsheets | XLSX, CSV | Gear lists, BOMs, room schedules, IP plans. |
| Text | TXT, MD | Simple content, FAQs, internal notes. |
| Web | URLs | Online documentation, manufacturer help sites. |
| Media | YouTube links | Training videos, install walkthroughs. |
| Images | PNG, JPG | Scanned documents, diagrams with text, error code screenshots. |

## Using with MCP

You can manage knowledge sources through code using the AVCodex MCP Server, which enables bulk operations and automated workflows.

### Available tools

| Tool | Description |
|---|---|
| `list_knowledge_sources` | List all sources for an agent with status. |
| `add_url_source` | Add URL (auto-detects YouTube, social media, web pages). |
| `add_document_source` | Upload a document (base64 encoded). |
| `get_knowledge_source` | Get details and processing status. |
| `delete_knowledge_source` | Remove a knowledge source. |
| `refresh_knowledge_source` | Re-crawl or re-process a source. |

### Example prompts

```
"Add https://docs.crestron.com to my programmer agent with crawl depth 2"
"List all knowledge sources for agent 123"
"Add this YouTube install walkthrough to my training agent"
"Refresh all outdated knowledge sources"
```

### Workflow: bulk knowledge migration

```
You: List knowledge sources in my old field tech agent (ID 100).

AI: [Calls list_knowledge_sources]
Found 8 sources:
- docs.crestron.com (URL, COMPLETED)
- Biamp Tesira Programming Guide.pdf (FILE, COMPLETED)
- Internal Commissioning Checklist 2024.docx (FILE, COMPLETED)
...

You: Add all the URLs to my new field tech agent (ID 200).

AI: [Calls add_url_source for each URL]
Added 5 URL sources to agent 200. Processing will complete
in a few minutes.
```

> **Note:** MCP server access requires a Builder plan or higher. See the [MCP Setup Guide](/docs/guides/overview) to get started.

*AVCodex · Your AV expertise. Amplified by AI.*