# Desktop Agent

A macOS menu bar app that lets a programmer, project manager, or service tech hold a hotkey to talk to their AVCodex agents while the agent reads the active screen and acts on the machine.

---

The AVCodex Desktop Agent is a native macOS menu bar app. Hold a hotkey, speak, release. The agent sees your active app, the window title, the focused element, and selected text. It can type, paste, fire keyboard shortcuts, open URLs and apps, and take a screenshot when context isn't enough.

It's the fastest way to use AVCodex while you're elbows-deep in Crestron Home, SIMPL Windows, Q-SYS Designer, AMX RPM, or D-Tools without leaving your workflow.

> **Warning:** Desktop Agent is an Enterprise-only feature. Contact sales for access.

## What it can do

| Capability | Description |
|-----------|-------------|
| **Hold-to-talk voice** | Hold Fn (or a custom hotkey) to speak, release to send. |
| **Screen context** | The agent reads the active app, window title, focused element, and selected text. |
| **Type text** | Smart dictation into any app with agent intelligence. |
| **Press keys** | Execute keyboard shortcuts (Cmd+Enter, Tab, etc.). |
| **Paste text** | Insert larger blocks while preserving formatting. |
| **Open apps and URLs** | Launch applications or navigate to websites. |
| **Take screenshots** | Capture visual context when the accessibility tree isn't enough. |
| **Multi-agent switching** | Switch between agents from the menu bar. |

## System requirements

- **macOS 14.0 (Sonoma)** or later.
- Intel or Apple Silicon (native support).
- Permissions: Accessibility, Microphone, Input Monitoring.

## Getting started

**1. Install the app**

Download the Desktop Agent from your AVCodex dashboard. Drag it into your Applications folder.

**2. Sign in**

Launch the app and sign in with your AVCodex builder account. Your session is stored securely in the macOS Keychain.

**3. Grant permissions**

The app guides you through three macOS permissions:

- **Accessibility.** Lets the agent read UI elements and simulate keystrokes in other apps.
- **Microphone.** For voice input while holding the hotkey.
- **Input Monitoring.** For global hotkey capture outside the app.

Each permission opens System Settings to the right location.

**4. Configure your hotkey**

Default: **Fn** (hold to start, release to stop). Configurable to any key combination in the menu bar dropdown.

## Use cases

### Smart dictation in any AV tool

Hold Fn and speak. In SIMPL Windows comments, it writes module headers. In a Crestron `.usp` file, it dictates a function comment. In Q-SYS Designer block comments, it explains a Lua control script. In Outlook, it drafts an RFP response.

### Filling out commissioning paperwork

A field tech opens the commissioning checklist PDF, holds Fn, and dictates room status, mounting verification, and signal flow notes. The agent fills the form fields directly.

### Code refactoring in programming environments

In VS Code working on a SIMPL# Pro project, select a method and say "refactor this to use async/await for the Cresnet poll." The agent sees the selection, rewrites it, and replaces the selection.

### Cross-app workflows for service ops

"Take what's in this Excel device list and draft a service email to the client." The agent captures visible rack data, drafts the email, opens Mail, and pastes it.

### Quick recall from your knowledge base

"What was the IP scheme we standardized for Biamp Tesira on the Houston campus?" The agent speaks the answer aloud from memory. No app switch required.

## The command bar

Press **Cmd+K** to open a Spotlight-style floating overlay where you can:

- Type commands to the active agent.
- View recent actions and history.
- Switch agents without opening the menu bar.

## Screen context details

When you hold the hotkey, the agent can see:

| Context | What it reads |
|---------|---------------|
| **Active app** | Which application is focused (SIMPL Windows, Q-SYS Designer, VS Code, Outlook, etc.). |
| **Window title** | The current window name (for example, "Boardroom4.smw - SIMPL Windows"). |
| **Focused element** | Text fields, buttons, and UI elements via Accessibility APIs. |
| **Selected text** | Highlighted content in the active app. |
| **Screenshots** | Full window or screen region on request. |

Context is captured on-demand only when the hotkey is held. There is no background recording.

## Troubleshooting

**Hotkey not working?**
- Check Input Monitoring permission in System Settings, Privacy, Input Monitoring.
- If you're using a custom hotkey, verify it doesn't collide with a system or app shortcut.

**Agent can't see the screen?**
- Verify Accessibility permission is granted.
- Some Electron apps (Slack, VS Code) expose limited accessibility data. The agent may request a screenshot instead.

**Text not typing into apps?**
- The app uses multiple methods (character-by-character, clipboard paste, accessibility APIs) with automatic fallback.
- If direct typing fails in a specific app, the agent falls back to clipboard paste.

> **Note:** Desktop Agent includes all voice capabilities. For voice configuration details, see the Voice Agents guide.

*AVCodex · Your AV expertise. Amplified by AI.*