agent-buddy：用 ESP32 桌面伴侣实时显示 AI 编程智能体状态

ESP32 Agent Companion

An animated desk companion that shows what your AI coding agents are doing.

ESP32 Agent Companion works with the Waveshare ESP32-S3-Touch-AMOLED-1.75-B or -C and turns it into a character that reacts to your AI coding agents. When GitHub Copilot, Claude Code, Codex CLI, Grok Build, Hermes Agent, or OpenClaw starts working, needs your approval, or finishes, the character shows it, along with a small badge for the agent involved.

Everything runs locally. There's no cloud service, account, or subscription, and the device works over USB or your local Wi-Fi.

Want it on your screen as well? The Desktop Agent Companion puts the same character, with the same animation, in a transparent window on your desktop, with or without the device.

Contents

• Features
• How it works
• What you need
• Quick start
• Step 1: Flash the firmware
• Step 2: Install the companion daemon
• Step 3: Finish setting up your agents
• Step 4: Try it
• Using the device
• Agents
• Agent badges
• AI credits and tokens
• Characters
• Wi-Fi
• Update
• Command reference
• Troubleshooting
• Desktop Agent Companion
• Develop and customize

Features

• Agent-aware states: Working, Needs attention, and Complete animations driven by your agents' lifecycle hooks, with sessions from several agents combined.
• Agent badges: small icons show which agent is working, waiting on you, or just finished.
• Usage: GitHub Copilot AI credits (AIC) and agent tokens, at the bottom of the screen.
• Six supported agents: GitHub Copilot, Claude Code, Codex CLI, Grok Build, Hermes Agent, and OpenClaw.
• Swappable characters: the device holds one character pack at a time. It ships with Copilot, and you can install OpenClaw, Claude, or your own pack over USB or Wi-Fi.
• Settings page: a local web page for agents, badges, characters, Wi-Fi, the connection mode, and the desktop app, with light and dark themes.
• Natural motion: eight looking directions, blinks, touch reactions, and a sleep cycle after two idle minutes.
• Automatic orientation: the onboard accelerometer keeps the character, settings, installation screens, firmware prompts, and touch controls upright when the device is rotated. Web Settings can save a per-device alignment trim.
• Wi-Fi or USB: run it tethered to your computer or from any USB power source on the same network.
• Optional sound: short local sound cues through the board's speaker connector, and the same cues from the desktop app when you turn them on.

How it works

flowchart LR
    A["AI agents<br/>Copilot, Claude, Codex,<br/>Grok, Hermes, OpenClaw"] -- lifecycle hooks --> D["Companion daemon<br/>(background service)"]
    B["Settings page<br/>127.0.0.1:4667"] <--> D
    D -- USB serial or local Wi-Fi --> E["ESP32 companion<br/>(firmware + character pack)"]

Loading

1. Firmware runs the animation on the device. Install it from the desktop app, or flash it from a release.
2. Hooks in each agent's config report events such as "started a tool", "needs permission", and "finished".
3. The companion daemon runs in the background on your computer. It combines those events into one state, sends it to the device, and serves the settings page.

What you need

Quick start

With the desktop app (recommended). You need only the app. You do not need Python, Node.js, or a clone of the repository.

1. Download the desktop app for your OS from Releases: agent-companion-desktop-<version>-macos-universal.dmg, agent-companion-desktop-<version>-windows-x64-setup.exe, agent-companion-desktop-<version>-windows-arm64-setup.exe, agent-companion-desktop-<version>-linux-x86_64.AppImage, or agent-companion-desktop-<version>-linux-amd64.deb.
2. Open the app. It is not code-signed, so see Run the desktop app for the first-run step on your OS. On a new installation, the app starts its local companion service and opens the welcome screen. Select Get Started.
3. Select your agents. If hooks need installation or an update, approve the changes. Follow the short guide for each selected agent. The guide waits for real agent activity. You can select Set up later without marking an agent as connected.
4. Select a character.
5. Select Desktop only, Device only, or Both. For a new device, connect a USB data cable and select Detect Device. Confirm the supported Waveshare model before firmware installation. The app downloads and checks the matching release, installs it, and then installs your selected character. You can keep compatible existing firmware or select Use desktop for now.
6. Select Finish to open Settings. Configure Wi-Fi on the Device tab if you want to untether the device.

Setup resumes after an interruption. Existing installations do not receive a forced setup flow after an update. Use Run setup wizard in Settings to change your choices or continue deferred setup. Select a previously reached step in the top navigation to return to it. Saved choices stay intact. New steps unlock as you reach them, and navigation is disabled during an installation. Setup also blocks conflicting changes from other Settings windows or the CLI. If saved progress is damaged, the app opens a new setup flow and shows a recovery message. The damaged file is kept when new progress is saved. Existing hooks, characters, and device settings are not reset. If a selected character is no longer available, return to Character, select Refresh characters, and choose an available pack. Previously selected agents remain checked. If an agent already has hooks, unchecking it asks for approval before removing Agent Buddy hooks. Cancel keeps the agent selected. The coding agent itself is not uninstalled.

If you run desktop setup again with a device connected over Wi-Fi, the wizard recognizes that connection. You can keep its firmware and install your selected character over Wi-Fi. First-time firmware installation still needs USB. The same USB device can reconnect on a new port after a character or firmware install. Setup keeps track of that device and saves its new port. The character button names the character it installs. It does not replace firmware. Firmware replacement is a separate option. Its confirmation dialog checks the supported board model and explains both installation stages.

On Windows, detection includes Windows agents and running WSL distributions. Open a stopped distribution before selecting Rescan agents. Rescan checks running distributions again instead of waiting for the cached scan to expire. Approved hook removal includes WSL installations and may start stopped distributions to remove their hooks.

To repair or replace firmware later, enable Use a physical device on the Device tab. Open USB release install or recovery, detect and select the device, then approve Install v. The desktop character follows the device (details).

Without the desktop app (to change the code, or on Windows in WSL 2):

1. Put the firmware on the device with flash.py from the release's agent-buddy-v<version>-firmware.zip asset. This needs Python 3.10 or newer. Extract it, connect the device over USB, and run this in the extracted folder:

   python3 -m venv .venv && .venv/bin/python -m pip install -r requirements.txt
   .venv/bin/python flash.py --list-ports                 # find your device's port
   .venv/bin/python flash.py --port /dev/cu.usbmodem2101  # use that port; type FLASH

   Windows commands are in Step 1.

2. Start the companion service from the repository. This needs Node.js 24.11 or newer:

   git clone https://github.com/DanWahlin/agent-buddy.git
   cd agent-buddy
   npm run setup      # installs agent hooks, starts the service, opens Settings
   npm run status     # should say: Connected over USB
   npm run settings   # opens the Settings page again at any time

3. Optional: show the character on your desktop. Download the app too, or build and run it from the same folder (needs Rust): npm run desktop. The app follows the service that runs from the repository.

Restart any agent sessions that were already open, then start one and watch the character react (Step 4).

To open Settings in the app's own window, click the upper button on the desktop character's case, press the BOOT button on the real device, or right-click the character and select Settings. The lower button turns the app's sounds on or off. Turn the character off, hide the device around it, or turn on sounds on the Desktop tab in Settings.

Step 1: Flash the firmware

The easiest way is the desktop app: open Settings, go to the Device tab, and click Install v in the Install over USB row (see Quick start). The steps below use flash.py instead, for computers without the desktop app. You don't need Arduino tools or the source code for them. If you'd rather build the firmware yourself, see Build from source.

1. Check that you have Python 3.10 or newer:

   python3 --version      # Windows PowerShell: py -3 --version

   The python3 that comes with macOS is 3.9. If that's what you see, install a newer Python from python.org or with brew install python, then open a new terminal and check again.

2. From Releases, download agent-buddy-v<version>-firmware.zip and extract it.

3. Open a terminal in the extracted folder (it contains flash.py) and install the flashing tool:

   macOS / Linux

   python3 -m venv .venv
   .venv/bin/python -m pip install -r requirements.txt

   Windows PowerShell

   py -3 -m venv .venv
   .\.venv\Scripts\python.exe -m pip install -r requirements.txt

4. Connect the device with the USB data cable, find its port, and flash it:

   macOS / Linux

   .venv/bin/python flash.py --list-ports
   .venv/bin/python flash.py --port /dev/cu.usbmodem2101

   Windows PowerShell

   .\.venv\Scripts\python.exe flash.py --list-ports
   .\.venv\Scripts\python.exe flash.py --port COM5

   Use the port that --list-ports shows for your board. macOS ports look like /dev/cu.usbmodem…, and Linux ports look like /dev/ttyACM0. Type FLASH when prompted.

When it finishes, the device restarts and the Copilot character starts looking around. Leave the cable plugged in for the next step.

Warning

Flashing replaces the board's existing firmware and partition table. Back up anything you need from its previous firmware first.

Flashing troubleshooting

• The device isn't listed: make sure the cable carries data, and close any serial monitor that's using the port.
• "Python 3.10 or newer is required": the environment was created with an older Python. Recreate it with a newer one, for example python3.13 -m venv --clear .venv, and install the requirements again.
• Linux can't create the environment: install your distribution's python3-venv package.
• Download mode or connection problems: see the Waveshare board guide.
• Don't flash the application .bin on its own or mix files from different releases. flash.py writes the matched set at the right addresses.

Step 2: Install the companion daemon

The daemon connects your agents to the device and runs in the background.

With the desktop app. Download the app and open it (see Run the desktop app). If no daemon runs, the app installs the one it carries. A new installation opens the setup flow before changing agent configuration. Hooks are installed only for agents you select and approve. Follow the selected-agent guides or defer them until later. On Linux, the serial port note below also applies.

From the repository. With the device plugged in over USB, clone the repository and run setup:

git clone https://github.com/DanWahlin/agent-buddy.git
cd agent-buddy
npm run setup

Setup installs a hook for each agent it finds on your computer, then starts a background service (a macOS LaunchAgent or a Linux systemd user service) so the daemon starts whenever you sign in. When it's done, it opens the settings page.

Check that the daemon found the device:

npm run status

You should see Connected over USB. If it says no device is connected, see Troubleshooting.

Restart any agent sessions that were already open so they load the new hooks.

Tip

The service saves your shell's PATH so it finds the same agents you do. If you install a new agent later, run npm run setup again.

Linux: serial port permission errors

Add your user to the dialout group, then sign out and back in:

sudo usermod -aG dialout "$USER"

Windows (WSL 2)

The easiest way on Windows is the desktop app (see Quick start). Its service runs on Windows. After approval, it installs selected agents' hooks on Windows and in running WSL 2 distributions. The setup dialog lists the affected configuration locations. The OpenClaw plugin does not yet connect from WSL. The Windows app uses the device's USB port with no extra steps.

To run the daemon from a clone inside WSL instead, run your agent CLIs and the daemon inside the same WSL 2 distribution. Hooks installed in WSL can't see agents running natively on Windows.

1. Install Node.js 24 LTS inside WSL.

2. If systemd isn't enabled, add this to /etc/wsl.conf, then run wsl --shutdown from PowerShell and reopen WSL:

   [boot]
   systemd=true

3. Windows doesn't share USB devices with WSL automatically. Install usbipd-win, connect the device, and run this in PowerShell:

   usbipd list
   # Once, in an Administrator PowerShell, using the ESP32's BUSID:
   usbipd bind --busid <BUSID>
   # Each time you want to use the device from WSL:
   usbipd attach --wsl --busid <BUSID>

4. In WSL, confirm that /dev/ttyACM* or /dev/ttyUSB* exists, then follow the Linux steps above. While the device is attached to WSL, Windows apps can't use it.

The settings page link opens in your Windows browser.

Step 3: Finish setting up your agents

Setup opens the settings page for you. To open it again later, run:

npm run settings

The page runs only on your computer and needs the private link that npm run settings opens, so other websites and devices can't reach it. The link survives restarts, so you can bookmark it.

Some agents need a one-time step before their hooks run, such as approval with /hooks in Codex, folder trust in Claude Code, or a Gateway restart in OpenClaw. Desktop setup shows one guide per selected agent. Hermes can request approval when each event type first runs; one received event does not approve every hook.

In Settings, expand Finish connecting your agents to see outstanding steps. Select Open setup guides to continue. The app distinguishes installed hooks from agent approval and received activity.

The guides check all six supported agents automatically. Run the displayed test task in the agent. The guide updates when Agent Buddy receives real activity and the required approval is complete.

Step 4: Try it

Start a new session in one of your agents and ask it to do something that uses a tool, such as reading a file. The character should:

1. Switch to Working while the agent runs.
2. Switch to Needs attention if the agent asks for your permission.
3. Celebrate with Complete when the turn finishes, then return to Idle.

The Settings overview shows the current state and which agent is driving it. Device address, network, and version information are in Connection details and versions. Select the Device tab to change Connection mode between Auto, Wi-Fi, and USB. If the character doesn't react, see Troubleshooting.

Using the device

The character works as soon as it's flashed, even without the daemon.

Swipe up to open the on-device settings menu. Swipe down or select Close to return to the character. The menu also closes on its own after 30 seconds without a touch.

Press the BOOT button on the board to open the companion's Settings on your computer. This needs the daemon, connected over USB or Wi-Fi (over Wi-Fi, it can take up to three seconds). If the desktop app is running, Settings opens in its window. If not, Settings opens in your browser. Don't use the PWR button for this: holding it turns the board off.

Sound plays through the board's two-pin speaker connector. Connect a small speaker if your board or enclosure doesn't include one. The desktop app can play the same sounds through your computer; the device's volume does not change them.

Agents

Command-line setup installs hooks for detected agents. Desktop setup installs only selected, approved hooks. Hooks notify the daemon; if the daemon or device is not running, your agents continue to work.

When you select Next in desktop setup, Agent Buddy checks the selected hooks. Current hooks stay unchanged, including hooks that still need approval inside the agent. Setup asks for approval only to install missing hooks or update outdated hooks. The confirmation lists only configuration files that need changes.

Setup edits only the hook entries it owns. Before it changes a file, it keeps a copy beside it: <file>.bak holds the file as it was before the first change, and <file>.agent-companion-<time>.bak holds it as it was before each change (the last five are kept). It doesn't touch a file that has nothing to change, and it stops without changes if a file's hooks section is in a format it doesn't understand.

Grok also runs the hooks in Claude's settings. If you use both, Grok can show a line such as PreToolUse hook (global/settings) failed, ignored for the Agent Companion Claude hook. This doesn't change the display, because Grok's own hook sends the events. To stop the message, add this to ~/.grok/config.toml. Grok then skips all of your Claude hooks, not only this one:

[compat.claude]
hooks = false

On Windows, Codex and Grok run their hooks in PowerShell, so setup writes the Windows hook command in the PowerShell form (& 'node.exe' 'cli.js' hook <agent> <event>). After an update from an older version, Settings shows these hooks as outdated. Click Reinstall, and then approve the Codex hooks again with /hooks.

If you moved an agent's settings folder with its own variable (COPILOT_HOME, CLAUDE_CONFIG_DIR, CODEX_HOME, GROK_HOME, HERMES_HOME or OPENCLAW_HOME), setup uses that folder instead of the one in the table. Set the variable in your shell's startup file (for example, ~/.zshrc or ~/.bashrc): the service and the desktop app read it from there, and keep it. If you change it later, run setup again, or open a newer version of the desktop app.

If you remove an agent's hook, it stays removed. Setup and desktop app updates don't add it back until you choose Install hook or run npm run agents install <agent>.

The settings page's Agents tab lists every supported agent with its detected version and hook status. An agent that's driving the display is highlighted. The GitHub Copilot app runs its own Copilot CLI with the same ~/.copilot folder, so one hook file covers the CLI and the app; on macOS, the card shows the version of each one that's installed.

A hook status of outdated means the hooks run another copy of the companion (for example, a repository install instead of the desktop app), a Node.js or companion file that no longer exists, or not all of the events. Choose Reinstall to fix it.

The same actions are available from the command line:

npm run agents                     # List agents, versions, and hook status
npm run agents disable claude      # Stop an agent from driving the device
npm run agents enable claude
npm run agents install codex       # Install or reinstall one hook
npm run agents uninstall codex     # Remove one hook

Codex links each hook approval to the hook's position in ~/.codex/hooks.json. If removing or reinstalling the companion's hook moves one of your own approved Codex hooks, Settings and npm run agents show a warning that names it. Open Codex, type /hooks, and approve it again.

Removing a Hermes hook doesn't revoke its approval. To clean that up too, run hermes hooks revoke "<command>" with the command shown in its config.

How agent events become device states

• Active agent or subagent work maps to Working.
• Permission and input prompts, and errors that end a turn, map to Needs attention, which takes priority over every other session. Prompts from one-shot runs (copilot -p, claude -p, codex exec, grok -p, and hermes -z) are ignored, because no one can answer them.
• A finished turn that used tools maps to Complete.
• Inactive sessions return to Idle.

Sessions from different agents and terminals are tracked separately, so one agent finishing doesn't hide another that's still working.

Agent badges

Badges show which agent is behind the current state. Working badges ride the orbiting dots, Needs attention badges pulse amber opposite the question mark, and Complete briefly shows the agents that just finished. Idle, Sleep, and Surprise don't show badges.

Badges are on by default. Turn them off with the Show agent badges on the device switch on the settings page's Agents tab, or with npm run badges off (npm run badges on brings them back).

Use your own badge icons

To replace a built-in icon, put a PNG named after the agent (for example copilot.png) in the icons folder of the daemon's data directory:

• macOS: ~/Library/Application Support/ESP32 Agent Companion/icons/
• Linux: ~/.local/state/esp32-agent-companion/icons/

The daemon converts 8-bit, non-interlaced PNGs to a 24 x 24 mask using alpha or brightness. An optional copilot.json next to it can set the accent color used for the icon and ring: { "color": "#RRGGBB" }.

AI credits and tokens

The device can show how much your agents used, centered at the bottom of the screen:

• AIC: GitHub Copilot AI credits, read from the session.usage_checkpoint and session.shutdown events in ~/.copilot/session-state/. This is the same value that Copilot CLI shows as "AIC used", and it includes sub-agents. Copilot writes it at the end of each turn, so the value changes when a turn ends. It counts only the Copilot CLI and app sessions on this computer. It does not include VS Code, the cloud agent, or other computers, so it is not your account's bill.
• Tokens: input plus output tokens for Claude Code (~/.claude/projects/) and Codex CLI (~/.codex/sessions/). Cache reads are not counted. GitHub Copilot tokens are not counted, because Copilot records them only when a session stops cleanly, so the local count is not complete. Copilot has AIC.

The device shows one line, for what is running now:

• While a GitHub Copilot session runs, it shows AIC only.
• While no Copilot session runs but another agent runs, it shows Tokens only.
• While no agent runs, it shows no line.

The Settings overview shows each value that has usage. Large values get commas (AIC: 12,345), then a short form (Tokens: 1.2M).

Usage is on by default. On the settings page's Agents tab, use the Show usage on the device switch and choose the Usage period: Today, This month, or Active sessions. The Settings overview shows the same values in the Usage box. The desktop app shows the lines too.

The device needs firmware with protocol 8 or later. Older firmware does not show usage, but everything else works.

Characters

The device holds one character at a time. The settings page's Characters tab shows each character you can install, with the current one marked Installed.

The app and release archives ship only Copilot, OpenClaw, and Claude. Add-on packs are distributed separately and appear after you add them in Settings.

• Select Install to switch characters. A progress bar tracks the transfer, which takes about a minute, and the device restarts when it's done.
• Select Add character pack to upload your own .acpk pack. To remove a pack you've added, select the × in the upper-right corner of its card. The installed character has no × until you install another one.

Or switch from the command line:

npm run character              # List the characters you can install
npm run character openclaw     # Install OpenClaw
npm run character claude       # Install Claude
npm run character copilot      # Switch back to Copilot

You can also pass the path to any .acpk pack, such as one from a release's agent-buddy-v<version>-characters.zip asset. The daemon keeps a copy with the characters you added, as Add character pack does, so the desktop app can show it and the settings page lists it with a × remove button. The daemon remembers your choice. If an install is interrupted, the device shows No character installed until the daemon reinstalls your character, which it does as soon as the device reconnects.

Some characters, such as Claude, need newer firmware than others. If the daemon says a character needs newer device firmware, reflash the latest release (see Update) and install it again.

Wi-Fi

Wi-Fi lets the companion run from a wall adapter or any USB power source. Your computer and the device need to be on the same local network, and the device needs a 2.4 GHz network.

From the settings page (recommended). Use the Device tab:

1. Plug the device in over USB and set Connection mode to Auto.
2. Choose your network from Nearby network, or type its name in Or type a network name if it's hidden or not listed. Enter the password, then select Connect device. The daemon sends the credentials over USB and pairs with the device automatically. The list comes from the device's own radio, so it shows only 2.4 GHz networks the device can join. Select Scan to refresh it. In the Settings overview, open Connection details and versions to check the network. (not connected) appears after the network name until the device joins. If it stays there, check the password and that the network has 2.4 GHz turned on.
3. Unplug the device from your computer and power it from any USB source. It keeps working over Wi-Fi.

From the command line. With the device plugged in over USB, run:

npm run wifi "Your 2.4 GHz network"

The command asks for the password without showing it or saving it in your shell history.

From the device. Swipe up, tap the Wi-Fi status, then select Setup Wi-Fi. Join the temporary Agent-Companion-XXXX network with the eight-digit password on screen, open http://192.168.4.1, and enter your network. Then reconnect your computer to your usual network and pair it with the same code:

npm run pair 12345678

The setup network and code expire after ten minutes.

Connection modes. Choose how the daemon reaches the device with Connection mode on the settings page's Device tab, or with npm run connection auto|wifi|usb:

Network requirements and security

• Guest networks, client isolation, VPN policies, and firewalls that block local HTTP or UDP discovery can prevent Wi-Fi operation. USB always works.
• Your network name and password are stored only on the device. The computer keeps only a pairing token.
• Requests from the daemon to the device are authenticated with the pairing token and stay on your local network.
• Bluetooth isn't supported.

Update

Companion daemon. If the desktop app installed it, install the new release of the app and open it: the app updates its copy of the service. The settings page's Connection details and versions shows the Versions of the service and the desktop app, and tells you when they are different. If you set it up from the repository, run this in the repository folder:

git pull
npm run setup

Setup installs any new dependencies, rebuilds the daemon, restarts the background service, and installs hooks for any agents you've added. New or updated characters show up on the settings page on their own.

Firmware. Reflash only when a release's notes mention firmware changes. With the desktop app, update the app first. When the device runs other firmware than the app's release, the Device tab shows a dot and Device firmware shows Update available. Connect the device over USB and click Update to v, or click Install v in the Install over USB row. The button installs the firmware release that matches the companion service's version. Without the desktop app, follow Step 1 with the new release's agent-buddy-v<version>-firmware.zip. Both ways keep the device's Wi-Fi settings and write the Copilot character. When the desktop app installs the firmware, it then installs your previous character again. If you flash without the desktop app, reinstall your character afterward from the settings page or with npm run character <name>.

Firmware over Wi-Fi. When the device is on Wi-Fi and has firmware with protocol 9 or later, open the settings page's Device tab. The Device firmware row compares the device with the firmware that you built with bash tools/arduino.sh build. If they are different, click Update. Protocol 11 or later firmware then shows Press BOOT to allow it. Press the BOOT button on the device in 60 seconds. That press allows this update only and does not open Settings. The daemon then sends the new firmware over Wi-Fi, and the device restarts in about 15 seconds. This keeps your character and Wi-Fi settings. If the new firmware cannot join Wi-Fi in 90 seconds, the device goes back to the previous firmware.

The first change to this firmware needs one USB flash, because it adds a second firmware slot to the partition table. Updates over Wi-Fi do not change the partition table, so a release that changes it also needs a USB flash.

Command reference

Run these from the repository folder.

Troubleshooting

"No device is connected"

1. Make sure the cable carries data and is plugged in. Close any serial monitor or flash.py run that's still using the port.
2. If the connection mode is Wi-Fi only, USB is ignored. Run npm run connection auto.
3. On Linux, add your user to the dialout group (see Step 2). In WSL, attach the device with usbipd (see Windows).
4. For Wi-Fi, check the items under Wi-Fi won't connect below.

The device doesn't react to an agent

1. Run npm run status and confirm the device is connected. It also lists any pending one-time agent steps.
2. Restart agent sessions that were open before you ran setup.
3. Check the settings page's Agents tab: the agent should show hook installed and enabled. If it shows hook outdated, choose Reinstall.
4. If you installed the agent after setup, or you removed its hook before, choose Install hook for it on the Agents tab.

Install over USB fails or is not available

• Needs the desktop app: the desktop app writes the firmware. Open it, then open Settings again. An older desktop app cannot install firmware; install the latest one.
• No BOOT press needed: a normal USB install resets the device into download mode itself. Press BOOT only for the recovery step below.
• No USB device: use a USB data cable, and close any serial monitor that uses the port.
• Download failed: the computer must reach GitHub. The app downloads the release that matches the companion service's version.
• Connection problems while it writes: disconnect the USB cable. Hold BOOT, connect the cable, then release BOOT. Click Install v again. If it still fails, use Step 1 or the Waveshare board guide.

Wi-Fi firmware Update waits for BOOT or fails

• Update is hidden: build firmware first with bash tools/arduino.sh build.
• Update is not available: the device must be on Wi-Fi and report firmware protocol 9 or later.
• Press BOOT: protocol 11 or later shows Press BOOT to allow it. Press BOOT on the device within 60 seconds. This press approves the update only. It does not open Settings.
• The request timed out: click Update again, then press BOOT when the device shows the prompt.
• The device went back: the new firmware did not rejoin Wi-Fi within 90 seconds. Install the firmware over USB.

The settings page is blank or says the link expired

The page needs its private link. Open Settings from the desktop app, or run npm run settings, to open it with the link.

Wi-Fi won't connect

• Use a 2.4 GHz network; the device doesn't support 5 GHz.
• Make sure your computer and the device are on the same network, and not a guest network with client isolation.
• Changing networks from the computer uses USB, so set Device connection to Auto with the cable plugged in first.

Where are the logs?

• macOS: ~/Library/Logs/esp32-agent-companion.log
• Linux: journalctl --user -u esp32-agent-companion

Uninstall

From Settings. Open Settings, select the Desktop tab, and select Uninstall… at the bottom. This removes the agent hooks, closes and deletes the desktop app, and removes and stops the companion service. It also deletes your settings, Wi-Fi pairing and added characters, unless you turn on Keep my settings first. The firmware on the device stays. Restart the agent sessions that are open after you uninstall.

Settings does not delete these. It tells you what to do:

• A desktop app that a package installed (/usr/bin/agent-companion-desktop). Run sudo apt remove agent-companion.
• A clone of this repository. The service and the app that you built there stay.

By hand.

1. Remove the hooks you installed, for example npm run agents uninstall claude for each agent.

2. Stop and remove the background service.

   macOS

   launchctl bootout gui/$(id -u)/com.danwahlin.esp32-agent-companion
   rm ~/Library/LaunchAgents/com.danwahlin.esp32-agent-companion.plist

   Linux

   systemctl --user disable --now esp32-agent-companion.service
   rm ~/.config/systemd/user/esp32-agent-companion.service

3. Optionally, delete the daemon's data directory (~/Library/Application Support/ESP32 Agent Companion on macOS or ~/.local/state/esp32-agent-companion on Linux).

Desktop Agent Companion

The same character, on your desktop. It's a small transparent window you can put anywhere, beside your editor or terminal or on a second screen. Use it with the device, or on its own. The code lives in desktop/.

It's the device's own animation, not a copy. The desktop app runs the firmware's code (motion, sprite renderer, effects and agent badges), compiled to WebAssembly, and it reads the same .acpk character packs that the device installs. A parity test checks that every frame matches the firmware build pixel for pixel. A change to the firmware's animation reaches the desktop the next time you build it.

The companion service drives it. The desktop app follows the service over its local socket, so it shows the state and agent badges the device shows, with no hooks of its own. The settings page controls both (most of these are on its Desktop tab):

Run the desktop app

The app shows the character the companion service names, from the service's own packs.

The app carries the companion service. When you open it and no service answers within 10 seconds, it installs the one it carries:

• It copies the service, a Node.js runtime and the character packs to the service's data folder (~/Library/Application Support/ESP32 Agent Companion/runtime on macOS, ~/.local/state/esp32-agent-companion/runtime on Linux, %LOCALAPPDATA%\ESP32 Agent Companion\runtime on Windows).
• From that copy, it installs the background service (a LaunchAgent, a systemd user service, or on Windows a Run registry value that starts the app's own launcher), as npm run setup does. The service then starts when you sign in, with or without the app.
• A new installation opens the welcome screen. Agent selection and approval come before hook installation. The app opens Settings after you select Finish.
• Incomplete setup resumes on launch. Updates preserve approved agent choices and do not install hooks for newly detected, unselected agents.

When you update the app, it updates its copy of the service. If a service from a clone of the repository runs, the app leaves it alone and follows it. To change back to the repository's service, run npm run setup in the clone. On Windows, agents in WSL 2 use the same service.

• Download it from Releases. The app is not code-signed. Your OS can need one extra step:

  OS	File	First run
  macOS	agent-companion-desktop-<version>-macos-universal.dmg	Drag Agent Companion to Applications, then run sudo xattr -rd com.apple.quarantine "/Applications/Agent Companion.app" once and open it from Applications.
  Windows	agent-companion-desktop-<version>-windows-x64-setup.exe, or -windows-arm64-setup.exe for an Arm PC	Choose Keep if the browser warns, then More info > Run anyway.
  Linux	agent-companion-desktop-<version>-linux-x86_64.AppImage or agent-companion-desktop-<version>-linux-amd64.deb	AppImage: chmod +x it, then run it. If it asks for FUSE, install fuse2 (Arch, Omarchy) or libfuse2 (Ubuntu). .deb: sudo apt install ./agent-companion-desktop-<version>-linux-amd64.deb.

  The running app appears in the macOS Dock or the taskbar on Windows and supported Linux desktops. Its menu bar or tray icon remains available.

  On Linux:

  • Sounds use GStreamer. The AppImage includes it. The .deb installs it. If you build from source and hear no sounds, install gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-pulseaudio (Ubuntu) or gst-plugins-base gst-plugins-good (Arch, Omarchy).
  • GNOME shows no tray icons without an extension. Install AppIndicator and KStatusNotifierItem Support (Ubuntu includes it). Without the tray icon, use the app's taskbar entry or open the app again.
  • The service is a systemd user service. On a system without systemd, the install stops and tells you the command that runs the service in a terminal.

• Or build it from source with Rust 1.90 or newer, from the repository folder. On Linux, install WebKitGTK first: sudo apt install libwebkit2gtk-4.1-dev libayatana-appindicator3-dev librsvg2-dev gstreamer1.0-plugins-good (Ubuntu) or sudo pacman -S --needed webkit2gtk-4.1 libayatana-appindicator gst-plugins-good (Arch, Omarchy).

  npm run desktop

Start and stop it from Settings. On the settings page's Desktop tab, Start opens the app and Stop closes it. The service finds the app in these places:

• Where the app last ran. Each time the app runs, it tells the service where it is, so a download in any folder or a build from source works after you open it one time.
• Before it has run: /Applications or ~/Applications (macOS), /usr/bin (the Linux .deb), or a build in this repository (desktop/apps/desktop/src-tauri/target/release).

If the service can't find it (for example, an AppImage you haven't opened yet), Start stays off until you open the app one time yourself. Settings shows the app as running only while it follows the service, so an app from release 0.7.0 or earlier shows as Not running. To remove the app and the service, see Uninstall above.

The window has no frame and is always on top.

• Clicks go through to whatever is behind it, except on the character and the case's buttons.
• Click the character to poke it, as you tap the device. Drag it to move it. It remembers where you put it.
• Click the upper button on the device's case to open Settings. Pressing the BOOT button on the real device does the same.
• Click the lower button on the device's case to turn sounds on or off. A blue light on the button shows that sounds are on. See Sounds below.
• Right-click the character for Hide, Settings and Close.
• Settings opens in a window of the app, not in a browser tab. If the Settings window is already open, the app brings it to the front. To use a browser, run npm run settings.
• Use the Dock or taskbar to return to the running app. An open Settings window takes priority, including when it is minimized. The window keeps your current tab and setup progress. On Windows, Settings is the only taskbar window while it is open. After Settings closes, the character's taskbar entry returns.
• To bring the character back after Hide, use Show Agent Companion in the tray menu, or run the app with --show. On Windows and Linux desktops with a taskbar, Hide minimizes the character window so the app remains available. On Hyprland, including Omarchy, Hide removes the window and keeps the menu-bar tray entry. On macOS, the Dock entry remains while the app runs.
• The tray icon has Show Agent Companion or Hide Agent Companion, Settings, Quit Agent Companion and Quit and Stop Companion Service. Choose the character in Settings.
• From a terminal or a keyboard shortcut, run the app again with --toggle, --show, --hide, --settings or --quit to control the running one.

Hide is for now; to keep it off the desktop for good, turn off Show the character on the desktop in Settings. If the desktop character is turned off, selecting the app entry opens Settings instead of turning the character on. On macOS 13 or later, the character can still appear over other apps' full-screen Spaces. Earlier macOS versions keep the Dock entry but cannot show this window over another app's full-screen Space.

Quitting. If this computer has never used a device, Quit and Close also stop the companion service, because it has nothing to do without the app. After a device has connected one time (over USB, or paired over Wi-Fi), the service keeps running when the app quits, so the device continues to show your agents. The first time, the app tells you this and lets you stop the service too. To stop the service later, use Quit and Stop Companion Service in the tray menu, or Stop in the Companion service row on the Settings Desktop tab. The device then goes idle. The service starts again when you open the app or sign in to your computer again. If the service does not work correctly, click Restart in the same row. The app stays open and reconnects.

Sounds. The desktop app plays the device's sound cues when the character starts work, needs attention, finishes, or is poked. The cues are the same WAV files as on the device. Sounds are off (muted) until you turn them on. To turn them on or off, do one of these:

• Click the lower button on the case.
• In Settings, select the Desktop tab and use Play sounds.

The two always agree, because the companion service keeps the setting. To make the sounds softer or louder, for example when music plays, use the Sound volume slider below Play sounds (30% by default). A tick plays at the new level when you release the slider. These settings are for the desktop app only: the device keeps its own Volume in its on-screen menu. The app plays no sounds while the character is hidden.

It runs on macOS, Windows and Linux. On Hyprland (including Omarchy) it floats, pins and un-borders its own window, so there's nothing to configure. On other Wayland desktops it runs through XWayland so it can see the pointer. The companion service has no Windows build, so on Windows the desktop app shows the character, but it can't follow your agents.

Develop and customize

Character Lab previews every character state in your browser using the same C++ motion and rendering code as the device. It needs Python 3.10+, clang++, and zlib (see Build from source).

python3 tools/serve_preview.py

Then open http://127.0.0.1:8765/character-preview.html. You can switch characters, pause, change playback speed, and trigger each state and touch reaction.

Create your own character pack

Packs must match the firmware's animation model: 13 tracks, 24 poses, and 5 blink levels at 412 x 352. They use one of two layouts, Copilot-style base frames with blink patches or OpenClaw-style full frames. Each character has a character.json. Built-in sources live in characters/<id>/. Keep add-on sources and assets in a separate private repository. See characters/README.md and tools/character_pack.py for the format. To check a pack:

python3 tools/character_pack.py validate path/to/pack.acpk

To draw a new character with AI image generation, from reference art through blink synthesis to a finished pack, see docs/character-art-notes.md.

Regenerate the OpenClaw sprites

OpenClaw is rendered offline from a procedural 3D model based on its official SVG. After changing the model:

npm ci --prefix characters/openclaw/model
npm run render --prefix characters/openclaw/model
python3 characters/openclaw/tools/export_openclaw_lab.py

The export writes the compressed frames that tools/character_pack.py build packages into build/characters/openclaw.acpk. The intermediate PNG renders stay local and are ignored by git.

More documentation

• Build from source
• Development, architecture, hardware, and serial protocol
• Creating a new character's art
• Tagging and publishing releases

License

The code is under the MIT License.

The "Agent Buddy" name and logo are trademarks of Dan Wahlin, and the MIT License does not cover them. You can fork the project and change it. If you share or sell your version, give it a different name and logo. The character artwork and agent names belong to their owners. See TRADEMARKS.md.

This is an independent project, not an official GitHub or Waveshare product. GitHub Copilot and Claude artwork and product names belong to their respective owners.

The Desktop Agent Companion in desktop/ is by Darren Robinson, a derivative of this project made with Dan Wahlin's approval.