# Understanding AI Evaluations

How to test your AVCodex agent before turning it loose on field techs, end clients, or your own team. Evals tell you what your AI will actually say when it matters.

---

You have built an AVCodex agent. It looks good in demos. But how do you know it will not embarrass your firm in front of an end client? What if it confidently quotes the wrong cable spec? What if it cites the wrong service hours? What if it goes off-topic in front of a CIO?

That is what **evaluations** (evals) are for. Think of evals as automated QA for your AI. Before your agent goes live, you can verify it gives the right answers to the questions that matter.

## Why evals matter

Imagine you run a managed AV firm. A client asks your AVCodex agent: "What is the service contract response time for emergency dispatch?"

Without evals, you might find out too late that your AI confidently said "8 hours" when your contract guarantees "2 hours." That is not just embarrassing. It can break trust and create real liability.

With evals, you can:

1. **Define test cases** for questions your AI should handle correctly.
2. **Specify expected answers** for what the right response looks like.
3. **Run automated checks** before every deployment.
4. **Catch problems early**, before clients see them.

> **Note:** Evals are like unit tests for traditional software, but for AI responses. They let you ship with confidence.

## Prerequisites for each evaluation type

Not every evaluation type fits every agent.

| Evaluation type | Requirements |
|---|---|
| **Factuality** | None. Works with any agent. |
| **Faithfulness** | Requires knowledge sources (manuals, AS-builts, SOPs). |
| **Answer Relevancy** | None. Works with any agent. |
| **Context Relevancy** | Requires knowledge sources. |
| **All** | Requires knowledge sources for full metrics. |

> **Warning:** If your agent does not use a knowledge base, stick with **Factuality** and **Answer Relevancy**. The other metrics evaluate how well your AI uses retrieved documents. Without documents to retrieve, those scores are not meaningful.

## The four types of evaluations

AVCodex uses an "LLM-as-Judge" approach: a separate AI model evaluates your agent's responses. Like having an expert reviewer grade every answer.

There are four evaluation types, each measuring something different.

### 1. Factuality: "Is the answer correct?"

Factuality is the most direct evaluation. You provide a question, the expected correct answer, and check whether your AI's response matches.

This is not exact word-for-word matching. The system understands semantic equivalence. "Our emergency dispatch responds within 2 hours" and "We guarantee a 2-hour response on emergency tickets" both pass.

**Use Factuality when:**

- You have specific, known-correct answers (response times, contract terms, room counts).
- Testing objective facts (pricing, hours, policies).
- Verifying your agent learned its training correctly.

> **Note:** **Tip for expected answers.** Keep expected answers focused on key facts. The evaluation compares semantic content, not word count. If your agent gives a more detailed answer than expected, it can score lower because it added "extra" information.
>
> **Example.**
>
> - Expected: "Our service contract response time is 2 hours."
> - AI response: "Our service contract guarantees a 2-hour response time on emergency tickets, with after-hours dispatch available 24/7."
> - Score: about 60% (the AI is correct but added details not in the expected answer).
>
> For best results, either make expected answers comprehensive, or use Faithfulness to test that responses are grounded in your knowledge base.

### 2. Faithfulness: "Is it making things up?"

Faithfulness checks whether your AI is hallucinating: making up information that is not in its knowledge base. Critical when your agent uses uploaded documents.

For example, if your knowledge base says "Warranty replacement requires the original sales order number" but your AI responds "Warranty replacement requires the original sales order number and a photo of the serial label, plus a 14-day waiting period," that is a faithfulness failure. The AI added details that do not exist in the source material.

**Use Faithfulness when:**

- Your agent uses a knowledge base or uploaded documents.
- You want to prevent hallucinations.
- Accuracy is critical (warranty terms, install procedures, safety guidance, code compliance).

### 3. Answer Relevancy: "Did it answer the question?"

Answer Relevancy checks whether the AI actually answered what was asked. Sometimes an AI gives a perfectly accurate response that has nothing to do with the question.

"How do I reboot the Q-SYS Core?" should not be answered with "Our company was founded in 2008 with a focus on enterprise AV integration..."

**Use Answer Relevancy when:**

- Users say the AI does not answer their questions.
- You want responses to stay focused.
- Your AI tends to be verbose or wander.

### 4. Context Relevancy: "Did we retrieve the right information?"

Context Relevancy is for RAG (Retrieval-Augmented Generation) agents. It checks whether the search step worked. Even if your AI gives a perfect response, retrieving the wrong documents is a problem waiting to happen.

**Use Context Relevancy when:**

- Debugging why a RAG agent gives wrong answers.
- Your knowledge base is large and varied (every Crestron, Q-SYS, Extron, and Biamp manual).
- You want to optimize document retrieval.

> **Warning:** **Important distinction.** Context Relevancy evaluates your document retrieval, not your AI's answer. A low score means your RAG system is pulling irrelevant documents, even if the AI gives a good answer despite poor context.
>
> If you see low Context Relevancy but good Answer Relevancy, focus on improving knowledge base organization or embedding quality rather than your AI's instructions.

## Choosing the right evaluation

If you only need one, start with Factuality. The rest are specialized.

## Running all evaluations together

For thorough testing, select **All** as your evaluation method. This runs all four evaluations on each test case and produces scores for each metric.

**When to use "All":**

- Initial agent testing before launch.
- After major knowledge base updates (loading a new set of manufacturer manuals, for example).
- Debugging complex answer quality issues.

**Understanding combined results.**

When running "All", a test case passes only if every metric meets the 70% threshold. Strict, but it ensures comprehensive quality. If you need flexibility, run individual evaluation types and set your own bar for each.

| Metric | What failure means |
|---|---|
| Factuality fails | Answer does not match expected content. |
| Faithfulness fails | Answer contains hallucinated information. |
| Answer Relevancy fails | Answer does not address the question. |
| Context Relevancy fails | RAG retrieved irrelevant documents. |

> **Note:** **Pro tip.** When using "All", do not be surprised if Context Relevancy scores lower than the others. Retrieval is the hardest part of RAG. If your other metrics pass but Context Relevancy fails, your agent is likely working well despite imperfect search results.

## Why Factuality is the best starting point

If you are new to evals, **start with Factuality**. Why:

1. **It is intuitive.** You provide questions and correct answers.
2. **No setup required.** Works with any agent, with or without a knowledge base.
3. **Clear pass/fail.** Easy to see when something is wrong.
4. **Catches the issues that matter most** to your clients.

The other evaluation types (Faithfulness, Answer Relevancy, Context Relevancy) are more specialized. Powerful for specific use cases, but Factuality covers the most ground for most agents.

> **Success:** **Quick start.** Create 10 to 20 test cases covering your most important questions. Run Factuality. Fix failures. Repeat before every major change to your agent.

## Creating effective test cases

A good evaluation suite includes:

### 1. Happy path cases

Questions your AI should answer perfectly. The bread-and-butter interactions:

- "What are our service hours?"
- "What is the emergency dispatch response time?"
- "How do I open a ticket?"
- "What is the warranty period on a Crestron DM-NVX endpoint?"

### 2. Edge cases

Tricky questions that might trip the AI up:

- Ambiguous questions ("the room is broken").
- Questions about things not in your knowledge base.
- Multi-part questions ("My Q-SYS Core is rebooting and I need a quote for a replacement").

### 3. Red lines

Things your AI should never say:

- Competitor comparisons you do not want made.
- Promises you cannot keep ("Same-day replacement on any DSP").
- Sensitive topics you want avoided.

## The eval workflow

How to fold evals into your agent development:

**1. Create test cases.** Start with your most important client questions.

**2. Write expected answers.** What is the correct response?

**3. Run evaluation.** See how your current agent performs.

**4. Fix failures.** Update the agent's instructions or knowledge base.

**5. Iterate.** Add more test cases as you find new scenarios.

## Understanding scores

Each evaluation produces a score from 0% to 100%. Default passing threshold is **70%**.

### Score ranges

- **70%+ (Pass)**: response meets quality standards.
- **40 to 69% (Warning)**: response has issues that should be reviewed.
- **Below 40% (Fail)**: response has significant problems.

You can adjust thresholds based on your needs. A clinical or healthcare AV agent might require 90%+. A casual room support agent might be fine with 60%.

### What different scores mean by metric

| Metric | High score (70%+) | Low score (under 70%) |
|---|---|---|
| **Factuality** | Answer matches expected content. | Answer missing key facts or has different info. |
| **Faithfulness** | Answer is grounded in the knowledge base. | Answer contains hallucinated or invented info. |
| **Answer Relevancy** | Answer directly addresses the question. | Answer is off-topic or tangential. |
| **Context Relevancy** | RAG retrieved relevant documents. | RAG pulled irrelevant documents. |

> **Note:** **Expect variation.** In practice, Answer Relevancy often scores highest (your AI usually answers what is asked). Context Relevancy often scores lowest (retrieval is hard). Do not panic if Context Relevancy is in the 20 to 40% range. Focus on it only if your agent is giving wrong answers despite having correct information in its knowledge base.

## What happens behind the scenes

When you run an evaluation:

1. Your agent receives the test question.
2. It generates a response (just like for a real user).
3. A separate "judge" AI evaluates the response.
4. The judge scores the response against the relevant criteria.
5. You see results and can act on failures.

The "LLM-as-Judge" approach is more sophisticated than string matching. It understands semantic meaning, handles paraphrasing, and provides reasoning for its scores.

## Next steps

Ready to start evaluating?

1. Open your agent in the AVCodex dashboard.
2. Go to the **Evals** tab to create your first suite.
3. Start with Factuality and your 10 most important questions.
4. Run the evaluation and review results.
5. Iterate.

Evals are not a one-time thing. As your agent evolves, your test suite should grow with it. The goal is to catch problems before clients do.

> **Note:** **Pro tip.** Schedule evals to run automatically whenever you update your agent's knowledge base or instructions. Catches regressions before they hit production.

## Using with MCP

You can create and run evaluations through code using the AVCodex MCP Server, which enables automated testing workflows.

### Available tools

| Tool | Description |
|---|---|
| `list_evals` | List all evaluations for an agent. |
| `create_eval` | Create an evaluation from a conversation or manual test cases. |
| `run_eval` | Execute an evaluation and get results. |
| `get_eval_result` | Get detailed results from a run. |
| `update_eval` | Modify test cases. |
| `delete_eval` | Remove an evaluation. |

### Example prompts

```
"Create an evaluation for my service desk agent from a recent good conversation"
"Run the 'Quality Check' evaluation and show results"
"List all evaluations for agent 123"
"Show details of the last evaluation run"
```

### Workflow: automated quality testing

```
You: Create an evaluation from my best-rated conversation yesterday.

AI: [Calls search_conversations to find it]
[Calls create_eval with chatSessionId]
Created evaluation "Service Desk Quality Test" with 8 test cases.

You: Run it.

AI: [Calls run_eval with format="terminal"]

+--------------------------------------------------------+
|           Service Desk Quality Test Results            |
+--------------------------------------------------------+
| Overall: PASSED (7/8 tests)                            |
| Pass Rate: 87.5%                                       |
|                                                        |
| Test 1: Emergency dispatch    [PASS] 92%               |
| Test 2: Service hours         [PASS] 88%               |
| Test 7: Complex warranty      [FAIL] 45%               |
+--------------------------------------------------------+
```

> **Note:** MCP server access requires a Builder plan or higher. See the [MCP Setup Guide](../guides/mcp/setup) to get started.

*AVCodex · Your AV expertise. Amplified by AI.*