Home Assistant 游戏助手 ha-gaming-assistant：用本地视觉 LLM 实时推送游戏提示

An AI coach that watches your screen and whispers tips into your smart home.

Local Vision LLM (Ollama) or cloud AI (GPT-4o, Gemini, DeepSeek, Groq) analyzes your gameplay — frame by frame — and pushes context-aware tips, voice lines, and triggers straight into Home Assistant. RGB lights react. TTS reads tips aloud. Spoilers? You control them.

🚀 Quick Start · 🧠 How it works · 🕹 Supported games · 🗺 Roadmap · 🤝 Contribute

💥 Why this exists

You game. You have a Home Assistant box sitting in the corner. Cloud "AI coaches" want your screen, your account, and a subscription. Why not run the whole thing yourself?

Your GPU, your tips, your house. No SaaS. No telemetry. No spoilers you didn't ask for.

✨ Highlights

🧠 Architecture

v260619 is a Thin Client design. The gaming device only captures and ships frames. All intelligence lives in Home Assistant.

┌─────────────────────────────────────────┐        ┌───────────────────────────────────────────────┐
│  CAPTURE AGENT  (you on the couch)      │  MQTT  │  HOME ASSISTANT  (the brain)                  │
│  ─────────────────────────────          │ ─────► │  ─────────────────────────────                │
│  • Windows / Linux / macOS              │  jpeg  │  Image listener  →  hash dedupe               │
│  • Android / Android TV / Google TV     │        │  Game detection  →  Game State Engine         │
│  • IP webcam · HDMI bridge · camera     │        │  Trend detection (health, phase, momentum)    │
│  • JPEG compress · backpressure         │        │  Spoiler policy  →  Prompt builder            │
└─────────────────────────────────────────┘        │  LLM backend  →  Tip                          │
                                                   │  Sensors · Services · Conversation agent      │
                                                   └─────────────┬─────────────────────────────────┘
                                                                 │
                                          ┌──────────────────────┼──────────────────────┐
                                          ▼                      ▼                      ▼
                                   🔊  TTS / Piper        💡  RGB scenes         📱  Notifications
                                   "Watch the left."      red on low HP          mobile_app push

Optional: YOLO Worker (external, GPU/NPU)

   Game State Engine  ◄──  Detections (MQTT)  ◄──  YOLO Worker
                                                   ├─ CUDA  (PC GPU)
                                                   ├─ NCNN  (Raspberry Pi)
                                                   ├─ Hailo-8L  (NPU)
                                                   └─ TFLite  (mobile)

Adds structured object detection (enemies, items, UI elements) on top of the Vision LLM. Pure upgrade; the rest works without it.

Legacy mode (v0.2 / v0.3 workers)

Old workers that publish finished tips to gaming_assistant/tip still work in passthrough. Migrate when you're ready — see Migration.

🚀 Quick Start

1 — Install via HACS

This button opens the repository in your own HACS and adds it as a custom repository automatically:

Or by hand:

HACS → Integrations → ⋯ → Custom repositories
  URL:   https://github.com/Chance-Konstruktion/ha-gaming-assistant
  Type:  Integration

Restart Home Assistant.

2 — Add the integration

Settings → Devices & Services → Add Integration → Gaming Assistant

A 6-step config flow asks for: LLM provider → connection → model + interval → spoiler default → camera → voice. Cloud providers need an API key. Connection is validated before the flow closes.

3 — Fire up a capture agent

# Pick your platform — they all speak MQTT
pip install -r worker/requirements-capture.txt

python worker/capture_agent.py \
  --broker 192.168.1.10 \
  --client-id gaming-pc \
  --interval 5 \
  --quality 75

Tips start landing on sensor.gaming_assistant_tip within seconds.

💡 Non-technical? Build the Windows GUI: cd worker && build_exe.bat → run GamingAssistant.exe. Enter broker IP, hit Start. Done.

🤖 AI backends

⚪ Text-only backends never receive images — they get structured game state + context descriptions instead. Great for Raspberry Pi setups without a GPU.

Recommended local vision models (Ollama)

🥧 Raspberry Pi / no GPU? Use Gemini free tier or DeepSeek. Integration runs on anything that runs Home Assistant.

🕹 Supported games

Community packs are pulled from ha-gaming-assistant-prompts and hot-reloaded via the gaming_assistant.refresh_prompt_packs service. Roll your own with docs/pack_authoring.md — schema, validation, local testing, the full workflow. A _template.json ships in the repo.

🎚 Feature deep-dive

🎯 Assistant Modes — Coach, Co-Player, Opponent, Analyst

Switch live from the dashboard via select.gaming_assistant_assistant_mode, or:

action: select.select_option
target:
  entity_id: select.gaming_assistant_assistant_mode
data:
  option: opponent

🙈 Spoiler System — 7 categories, 4 levels, per-game profiles

Control what the AI is allowed to reveal across story · items · enemies · bosses · locations · lore · mechanics — each at none / low / medium / high.

service: gaming_assistant.set_spoiler_level
data:
  category: bosses
  level: none
  game: "Elden Ring"   # optional

Set everything at once with a profile:

service: gaming_assistant.set_spoiler_profile
data:
  game: "Elden Ring"
  level: none

Profiles persist across HA restarts. Add clear: true to wipe.

❓ Ask Mode — talk to it like ChatGPT, with or without a screenshot

service: gaming_assistant.ask
data:
  question: "How do I beat this boss?"
  game_hint: "Elden Ring"

With a screenshot:

service: gaming_assistant.ask
data:
  question: "What item is on the ground here?"
  image_path: /config/www/screenshot.jpg

♟ Tabletop Support — point a camera at your board

service: gaming_assistant.watch_camera
data:
  entity_id: camera.board_game_cam
  game_hint: "Chess"
  client_type: tabletop
  interval: 10

Stop with gaming_assistant.stop_watch_camera. Built-in packs for Chess, Poker, Catan, UNO.

🔊 Voice (TTS) + 🗣 Voice Control (HA Assist)

Auto-announce: Flip switch.gaming_assistant_auto_announce on — every new tip is spoken via your configured TTS engine and media player.

Manual:

service: gaming_assistant.announce
data:
  tts_entity: tts.piper
  media_player_entity_id: media_player.living_room

Conversation agent: Pick Gaming Assistant as the conversation agent in Settings → Voice assistants. Built-in commands (EN + DE):

Anything else is forwarded to the LLM as a free-form question. "Was ist mein nächster Zug?" works.

Questions are answered with what is happening right now: the latest captured frame (if it is at most 2 minutes old), the current game and the tip you heard last go along, so "How do I beat this boss?" or "What did you mean by that?" make sense to the model. With no recent frame the question is answered from text alone. The gaming_assistant.ask service does the same with use_last_frame: true.

📝 Session Summaries — wrap-up after every session

After 5 minutes of inactivity, the integration can auto-summarize the session's key insights (3+ tips required). The result lands in sensor.gaming_assistant_session_summary. A gaming_assistant_session_ended event fires for automations.

Manual:

service: gaming_assistant.summarize_session
data:
  game: "Elden Ring"

📡 Event-based automations — every tip fires gaming_assistant_new_tip

trigger:
  - platform: event
    event_type: gaming_assistant_new_tip
action:
  - service: notify.mobile_app_your_phone
    data:
      title: "Gaming Tip ({{ trigger.event.data.game }})"
      message: "{{ trigger.event.data.tip }}"

Event payload includes tip, game, client_id, assistant_mode. Pair with light scenes, scripts, or whatever HA can do.

📦 Capture agents

CLI cheat sheet

PC agent (capture_agent.py):

Linux note: PC agent uses xprop for window-title detection on X11. On Wayland, auto-detection is unavailable — use --game-hint.

Android agent — same as PC, plus:

Android TV agent — same as Android, plus --game-hint (for streaming apps where window title is meaningless).

IP Webcam agent (capture_agent_ipcam.py):

HTTP errors trigger exponential backoff (2s → 4s → 8s → …, capped at 60s). Exits only after 20 consecutive failures.

HDMI Bridge (capture_agent_bridge.py):

pip install opencv-python
python capture_agent_bridge.py --broker 192.168.1.10 --device /dev/video0

A systemd unit ships at worker/systemd/gaming-assistant-bridge.service — adjust broker IP, device path, and user before enabling.

PC Overlay HUD (tools/overlay_pc.py): Subscribes to gaming_assistant/tip and shows the latest tip in an always-on-top transparent window. F8 to toggle, Esc to quit. See tools/README.md.

Android over Wi-Fi (no cable)

adb tcpip 5555
adb connect 192.168.1.42:5555

python worker/capture_agent_android.py \
  --broker 192.168.1.10 \
  --device 192.168.1.42:5555

HUD OCR worker — read health/ammo/score straight off the screen

worker/ocr_agent.py is an optional external worker that subscribes to your game frames, runs OCR on configured HUD regions, and publishes the numbers to Home Assistant. These land in the Game State Engine as measured values (Tier 1) — far more reliable than letting the LLM guess them from the picture.

Regions are given as fractions of the frame (x,y,w,h in 0..1), so they're resolution-independent:

pip install -r worker/requirements-ocr.txt   # opencv + numpy + pytesseract
# Tesseract engine: apt install tesseract-ocr  (or brew install tesseract)

python worker/ocr_agent.py \
  --broker 192.168.1.10 \
  --regions "health:0.04,0.90,0.10,0.05;ammo:0.86,0.90,0.10,0.05" \
  --max-fps 1

Prefer a config file? Use --regions-file regions.json with {"health": [0.04, 0.90, 0.10, 0.05], ...}. Use --engine easyocr for a pure-pip alternative to Tesseract. The numbers show up on sensor.gaming_assistant_scene_change's game state and flow into every tip and the Tier 3 strategy.

Game audio worker — loudness, intensity & onsets (gunshots/explosions)

worker/audio_agent.py is an optional worker that runs on the gaming PC and listens to the game's own audio. It derives a few cheap signals locally — loudness (dB), an intensity class (quiet/moderate/intense), and sudden onsets like gunshots, explosions or stingers — and publishes only those as compact JSON. They land in the Game State Engine as measured Tier-1 signals.

This is deliberately client-side: Home Assistant is meant to run on modest hardware (a Pi/NUC) without a high-end server, so the audio is captured and analysed on the PC where it's produced (plain RMS/onset DSP — no model, no GPU) and only the results travel over MQTT. Raw audio never touches HA.

pip install -r worker/requirements-audio.txt   # sounddevice + numpy + paho-mqtt

# Find your loopback / "Stereo Mix" / .monitor device:
python worker/audio_agent.py --list-devices

# Capture the game's audio and stream signals to HA:
python worker/audio_agent.py \
  --broker 192.168.1.10 \
  --client-id gaming-pc \
  --device 7

To analyse the game's output (not a microphone) you need a loopback source: a WASAPI loopback / "Stereo Mix" device or virtual cable on Windows, a .monitor source on PulseAudio/PipeWire, or a loopback driver (e.g. BlackHole) on macOS. Set --client-id to match your capture agent so the signals join the right game state.

Chess grounding — real chess facts, computed inside Home Assistant

For physical board games you often have just a camera on a table and no client at all — nothing to offload computation to. So the chess engine runs inside Home Assistant. It's feasible because chess reasoning is episodic and symbolic (not a continuous heavy stream): it uses python-chess — a pure-Python dependency installed automatically via the manifest, no Stockfish binary and no extra server — to validate a position and compute legal moves, material balance, threats (captures/checks), and a suggested move from a small built-in evaluator + shallow search. This turns the opponent/analyst modes from guessing into real chess.

Feed it a board as FEN and read the result on sensor.gaming_assistant_chess:

# Manually or from an automation (the board-vision worker publishes over MQTT):
service: gaming_assistant.analyze_board
data:
  fen: "rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq - 0 1"

or publish to MQTT gaming_assistant/{client_id}/board with {"fen": "..."}. The grounded facts (chess_best_move, chess_material_cp, chess_eval_cp, chess_phase, …) flow into the game state, every tip, and the Tier 3 strategy. The engine is intentionally modest — its job is to ground the LLM in correct, legal chess, not to be Stockfish.

Board-vision worker — a camera turns a *physical* board into FEN

worker/board_vision.py is the source for the chess engine in the camera-on-a-table case: it watches frames, perspective-warps the board from four configured corners, reads each square's occupancy + piece colour, and recovers the move by tracking — starting from a known position, the single legal move whose result matches the new board is unambiguous (captures, castling, en passant included), so it never has to visually classify piece types. It maintains the game with python-chess and publishes the FEN to gaming_assistant/{client_id}/board, which the in-HA engine already consumes.

pip install -r worker/requirements-boardvision.txt   # opencv + numpy + paho + chess

# Corners are fractions of the frame, clockwise from top-left as the camera sees it:
python worker/board_vision.py \
  --broker 192.168.1.10 \
  --client-id chess-cam \
  --corners "0.12,0.08;0.88,0.10;0.90,0.92;0.10,0.90"

Point any capture agent (e.g. capture_agent_ipcam.py for an IP webcam) at the board so frames flow on gaming_assistant/{id}/image; the worker reads those. It assumes the game starts from the standard position and that each move is seen as a stable frame. The pixel layer is a calibratable best-effort — tune --occupancy-std / --dark-below for your board and lighting (robust auto corner-detection / a small classifier are future work). The reliable, fully-tested part is the geometry + move-inference; reset the tracker any time with an MQTT command {"command": "reset"} on gaming_assistant/board/command.

Chess bot — play a real game against Home Assistant

Turn on switch.gaming_assistant_chess_bot and the assistant plays one side (select.gaming_assistant_chess_bot_color, Black by default). Whenever a new position arrives — from the board-vision worker, the board MQTT topic or analyze_board — and it is the bot's turn, it answers once:

• Out loud through the TTS entity and media player set under Configure, in Home Assistant's language: "Ich ziehe den Springer von g8 nach f6." / "I move my knight from g8 to f6." You make the move for it on the board. Captures, castling, en passant, promotion, check and mate are spoken as full sentences, and the end of the game gets a last word.
• As the event gaming_assistant_chess_move for automations: kind (move or game_over), san, uci, from, to, piece, capture, promotion, check, checkmate, speech, fen, fen_after, client_id, bot_color.
• Over MQTT on gaming_assistant/{client_id}/chess_move (the same JSON), for anything that wants to play the move elsewhere, e.g. on a screen.

number.gaming_assistant_chess_bot_strength is the engine's search depth: 1 answers instantly but plays weakly, 4 is the strongest the in-HA engine gets (about a second per move on a desktop CPU, longer on small hardware such as a Raspberry Pi). Start a game with:

action: gaming_assistant.set_chess_bot
data:
  enabled: true
  color: white      # the bot opens right away
  strength: 3
  new_game: true    # resets the board-vision worker and starts from the initial position

A position the bot has answered is not answered again, so a worker that repeats the same board does not make the bot repeat itself. It is a sparring partner, not Stockfish.

🕹️ Agent Mode / Player 2 (experimental)

Beyond watching your game, the assistant can play it. worker/agent_executor.py turns the AI's structured action output into real inputs on a virtual Xbox controller (vgamepad).

Why a virtual gamepad? A virtual controller can only send game-controller inputs — it can never move your mouse, alt-tab, or type system commands. The AI is sandboxed to "press buttons", which is the entire safety premise of Agent Mode.

Safety model — opt-in and conservative by default:

• Whitelist — only buttons you list in --allow-buttons are ever forwarded; anything else is rejected and logged.
• Dry-run — --dry-run (and the automatic fallback when vgamepad isn't installed) validates and logs actions without sending input. Always start here.
• Audit log — every action (accepted, rejected, or skipped) is appended as one JSON line to --audit-log.
• Emergency stop — publish stop to gaming_assistant/command to instantly pause and release all inputs; start resumes. Inputs are also released on disconnect and shutdown, so nothing ever stays stuck.

pip install -r worker/requirements-player2.txt   # vgamepad + paho-mqtt
# (Windows also needs the free ViGEmBus driver for vgamepad.)

# 1) Safe first run — validates + logs, sends nothing:
python worker/agent_executor.py --broker 192.168.1.10 --client-id gaming-pc --dry-run

# 2) Go live, restricted to face buttons + D-pad:
python worker/agent_executor.py --broker 192.168.1.10 --client-id gaming-pc \
  --allow-buttons A,B,X,Y,DPAD_UP,DPAD_DOWN,DPAD_LEFT,DPAD_RIGHT

Test it end-to-end by publishing an action yourself (HA → Developer Tools → Actions → mqtt.publish, or mosquitto_pub):

mosquitto_pub -h 192.168.1.10 -t gaming_assistant/gaming-pc/action \
  -m '{"action":"tap_button","button":"A","duration_ms":80,"reason":"confirm"}'

CLI cheat sheet (`agent_executor.py`)

Actions follow PromptBuilder.ACTION_SCHEMA: press_button, release_button, tap_button (buttons A/B/X/Y/LB/RB/LT/RT/DPAD_*/START/BACK), move_stick (left/right, x/y in [-1.0, 1.0]), wait, and no_op.

Let Home Assistant drive it. Enable the Agent Mode switch (or call gaming_assistant.set_agent_mode) and each analyzed frame additionally asks the LLM for one controller action, validates it, and publishes it to gaming_assistant/{client_id}/action for the executor:

action: gaming_assistant.set_agent_mode
data:
  enabled: true
  allowed_buttons: "A, B, X, Y, DPAD_UP, DPAD_DOWN, DPAD_LEFT, DPAD_RIGHT"

Safety: Agent Mode is strictly opt-in and resets to OFF on every Home Assistant restart — the AI never controls inputs unless you deliberately turn it on. The HA-side governor adds two more rails: actions are rate limited (no input flooding) and Agent Mode auto-disables after repeated failures (dead-man switch), so a broken pipeline never keeps the AI "driving". Every decision is audited on sensor.gaming_assistant_agent_action and the gaming_assistant_agent_action event. It runs a second inference per frame (in addition to the normal tip), so expect higher load, especially on local models. The executor still enforces its own whitelist and --dry-run, and stop on gaming_assistant/command is the emergency brake. Start with the executor in --dry-run to watch the action stream safely before going live.

Optional: confirm each action. Turn on switch.gaming_assistant_agent_confirm_actions (or pass confirm_actions: true to set_agent_mode) and every generated action waits as pending instead of being sent. Press button.gaming_assistant_agent_confirm_action / ..._reject_action, call gaming_assistant.confirm_agent_action / reject_agent_action, or answer a phone notification (example in lovelace/automations_example.yaml); an action nobody decides on lapses after 30 s. While an action waits, no new one is generated. Off by default: whoever enables Agent Mode trusts their agent.

🧩 Entities & services

Controls (adjustable from the dashboard)

Sensors

Services

Mode, spoiler level, interval, and timeout are now controlled via entities — services are for one-shot actions.

🪟 Lovelace & Automations

A ready-made dashboard ships at lovelace/dashboard.yaml. Drop it into a Manual card — you get current tip, history, spoiler controls, status, and action buttons out of the box.

Sample automations live in lovelace/automations_example.yaml: speak tips via TTS, change RGB color when gaming starts, send tips as mobile notifications, change spoiler level based on detected game.

🧰 Requirements

🩹 Troubleshooting

Config flow: 500 Internal Server Error

Make sure the MQTT integration (Mosquitto) is fully set up before adding Gaming Assistant. If it persists, delete __pycache__ inside custom_components/gaming_assistant/ and restart HA.

Sensor stuck on "Waiting for tips…"

• Confirm the capture agent is running and reachable.
• Verify MQTT (Mosquitto add-on) is up.
• Check that Ollama is running and reachable from HA.

Ollama timeout

The model may be loading for the first time — wait 60s and retry. Or reduce --quality and --resize in the capture agent.

No game detection on desktop

Install pywin32 on Windows; make sure the game is in the foreground. Or add your title to KNOWN_GAMES in the capture agent.

ADB screencap fails

adb devices should show your phone as device (not unauthorized). Accept the USB-debugging prompt on the device.

🪄 Migration from v0.2 / v0.3

• Workers: old workers moved to worker/legacy/. Still functional but deprecated — switch to the new capture agents.
• Config: existing entries remain valid. New fields get defaults automatically.
• Topics: old MQTT topics (gaming_assistant/tip, …/status, …/gaming_mode) are still supported in legacy passthrough mode.

🤝 Contributing

Issues, PRs, and prompt-pack submissions are welcome.

• Bug? File an issue.
• New game? Drop a pack — start from _template.json, follow docs/pack_authoring.md, open a PR against ha-gaming-assistant-prompts.
• Tests live in tests/. CI is green or it doesn't merge.

📜 License

MIT — do whatever you want with it. A ⭐ on the repo is the polite tip-jar.

Built by gamers, for the HA homelab crowd.

_{v260619 · Thin Client · Local-first · Open source · Made with 🟦 and 🟪}