Heoster 发布 JARVIS：免费单用户 AI 操作员，OCR 控制桌面与真实 Chrome

Just A Rather Very Intelligent System — a free-tier, single-user AI operator built for Heoster (Harsh Saini · CODEEX-AI).

JARVIS is a harness, not a chatbot wrapper. It holds a persistent session, keeps a smart memory of you, drives a real Chrome window (by DOM and by moving the actual cursor), controls the desktop through OCR rather than a vision model, chains six LLM providers with automatic verified failover, and exposes everything through a CLI and an OpenAI-compatible local API.

     ██╗ █████╗ ██████╗ ██╗   ██╗██╗███████╗
     ██║██╔══██╗██╔══██╗██║   ██║██║██╔════╝
     ██║███████║██████╔╝██║   ██║██║███████╗
██   ██║██╔══██║██╔══██╗╚██╗ ██╔╝██║╚════██║
╚█████╔╝██║  ██║██║  ██║ ╚████╔╝ ██║███████║
 ╚════╝ ╚═╝  ╚═╝╚═╝  ╚═╝  ╚═══╝  ╚═╝╚══════╝
        JARVIS v1.0.0 · built for Heoster (CODEEX-AI) · free-tier only

Table of contents

• What it is
• The three focus areas
  • 1 · Screen control (no vision model)
  • 2 · Browser control (real Chrome, real cursor)
  • 3 · Smart memory
• API manager: verified automatic fallback
• Install
• Quick start
• CLI reference
• Local API server
• Obsidian & student mode
• Tools (116)
• Agents
• Architecture
• Free-tier limits
• Configuration
• Troubleshooting / doctor
• Tests
• Project layout

What it is

Everything works offline-first: with no API keys and no local model, JARVIS still browses, clicks, types, reads the screen, searches its memory, manages files, and answers arithmetic — it just says so honestly instead of hallucinating.

The three focus areas

1 · Screen control (no vision model)

JARVIS never sends a screenshot to a model. Screen understanding is a two-step, fully local pipeline:

screenshot  ──►  tesseract OCR (local binary)  ──►  words + confidence + boxes
                                                        │
                            fuzzy match ≥ 0.8 ──────────┘
                                     │
                          scale  ──►  real cursor click

• Input backends (auto-detected, first that works wins): pyautogui → xdotool → ydotool → PowerShell (Windows) → AppleScript (macOS) → NullBackend (reports why instead of pretending).
• Real cursor movement: Bezier-curved paths with eased timing (min(120, distance × 0.18) bow, t²(3−2t) easing, ±0.8 px jitter, ~16 ms steps) — moves look human, not teleported.
• Click by text: screen_click_text("Sign in") → OCR → match → scaled coordinates → cursor click, and it reports the exact pixel it used.
• Windows: list, focus, maximise/minimise/close via wmctrl/xdotool/PowerShell/osascript.
• Capability honesty: on a headless box every screen tool returns ok=false with the exact install command (Xvfb, xdotool, tesseract-ocr) — it never silently fails.

Key tools: take_desktop_screenshot, read_screen_text, find_on_screen, screen_click_text, screen_fill_field, cursor_move, cursor_click, keyboard_type, keyboard_press, screen_scroll, screen_list_windows, screen_focus_window, launch_application, run_in_terminal_window.

2 · Browser control (real Chrome, real cursor)

The user's requirement was explicit: launch Chrome from the terminal, then drive it by screen control with cursor movement, typing and browser commands — in real time, harness-style. JARVIS implements that literally, with a hybrid that uses the DOM for targeting and the OS cursor for acting.

BrowserHarness
 ├─ playwright   full DOM + JS + headless or headed
 ├─ selenium     full DOM, classic WebDriver
 ├─ cdp          talks to a Chrome YOU launched from the terminal (port 9222)
 └─ screen       OCR + cursor only (works on any window, even one you opened)

• Terminal launch first: scripts/chrome-harness.sh starts Chrome with --remote-debugging-port=9222, a real window, and your profile. JARVIS's browser_launch tool does the same from Python and remembers the process.
• Hybrid clicking: browse_open renders the page as a numbered menu — [7] <a> "Lofi Girl" — the model picks an index, JARVIS resolves it to a CSS selector, asks the engine for the element's viewport box, converts that box to screen coordinates (window position + toolbar height + devicePixelRatio), and then moves the real cursor and clicks. If the box is unavailable it falls back to a DOM click, and if no engine is alive it falls back to OCR + coordinate search.
• Verification built in: verify(deep=True) starts each engine in turn and reports exactly what is missing (pip install playwright && playwright install chromium), so "the browser is broken" is never a mystery.
• Detection resistance: because the click physically happens at OS level on a real, visible window, sites that fingerprint synthetic events see a human.

Key tools: browse_open, page_read, page_click, page_type, page_press, page_scroll, page_back, page_forward, page_tabs, page_screenshot, page_js, page_find, browser_launch, browser_engines, browser_status, screenshot_page, screenshot_and_read.

3 · Smart memory

A fixed sliding window is the floor, not the design. JARVIS keeps four tiers and decides, per turn, what deserves to persist:

• Importance gate — every candidate is scored (explicit "remember" 0.9, corrections 0.85, preferences 0.75, tool failures 0.6, chatter 0.05). Below the threshold (memory.importance_threshold, default 0.35) it is dropped, so the store does not fill with "hmm" and "ok".
• Near-duplicate merge — a new fact within cosine 0.90 of an existing one bumps the old record instead of adding a twin.
• Decay & promotion — unused low-importance records fade (45-day half-life) and are pruned; records that get recalled are promoted (episodic → semantic) and their importance rises.
• Hybrid recall — 0.65 × vector similarity (hashing TF-IDF locally, nomic-embed-text when ollama is up) + 0.35 × BM25 keyword match, with a token budget so recall can never blow the context window.
• Consolidation — every 25 turns (and on /memory consolidate) JARVIS promotes, decays, prunes and summarizes.
• Skills — successful browser/screen traces are stored as reusable procedures and offered back as hints (skills.hints_for("open youtube and play lofi")).
• Heoster profile — seeded from jarvis/persona.py, extended automatically: preferences, corrections, personal details and tool preferences are extracted from what you say, and injected into every system prompt.

Storage is a single SQLite file (memory_store/vectors.sqlite3, WAL mode, float32 vectors + an inverted index for BM25). No server, no Docker, no RAM footprint. ChromaDB is optional if you want it.

API manager: verified automatic fallback

jarvis/core/api_manager.py is the reason JARVIS keeps answering when a free tier dies.

• Chain: ollama → groq → cerebras → openrouter → gemini → hf → stub, reorderable, always ending in a provider that cannot fail.
• Health checks: each provider has a cheap probe() (model list / ping). Results are cached for 90 s with latency recorded.
• Backoff: a provider that errors is marked degraded/down with exponential backoff (2 s → 120 s) instead of being hammered.
• Quota integration: every call passes the free-tier quota manager (rpm / rpd / monthly credits per provider) before the request, so JARVIS never burns a free tier it needs later.
• Verified fallback — this is the part that is easy to get wrong and JARVIS gets right:
  1. the turn's failed provider is recorded,
  2. the next provider is actually called (not assumed healthy),
  3. the answer records provider, fallback_from and degraded: true,
  4. a background auto-heal thread re-probes failed providers every 120 s and returns them to the chain once they recover,
  5. jarvis api --verify forces a live probe of every provider and prints latency + status, and recent_fallbacks() exposes the audit trail.
• Honest last resort: the stub provider never pretends to reason. It performs the obvious tool call (arithmetic, time, URL, search), returns raw results, and tells Heoster how to get a real model — including the exact ollama pull command.

$ python -m jarvis api --verify
  provider     status        latency   note
  ollama       down          -         connection refused (start `ollama serve`)
  groq         unconfigured  -         no API key (set GROQ_API_KEY)
  ...
  stub         healthy       0 ms      unlimited
  fallback chain : ollama → groq → cerebras → openrouter → gemini → hf → stub
  would answer   : stub

Install

Requires Python 3.10+ and ~200 MB of disk. Works on Windows, Linux and macOS; a 2-core / 4 GB machine is enough.

git clone https://github.com/Heoster/jarvis.git
cd jarvis

# one-shot: venv + core deps + focus-area extras + browser + optional model
bash scripts/jarvis-setup.sh

# or manually
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install playwright selenium websockets mss Pillow pytesseract trafilatura beautifulsoup4
playwright install chromium

Then check the machine and doctor the install:

python -m jarvis doctor          # hardware, screen, browser, memory, API, crawl

Nothing above is mandatory beyond the core three (pyyaml, httpx, rich, click). Every optional piece degrades to a clear message instead of a crash.

Headless server? Give it a virtual display:

bash scripts/start-xvfb.sh          # Xvfb :99 + xdotool + tesseract hints

Quick start

# interactive chat (resumes your last session, remembers everything)
python -m jarvis

# one-shot question
python -m jarvis ask "What is 21 * 2 + 8?"

# launch Chrome from the terminal, then drive it by DOM + cursor
bash scripts/chrome-harness.sh              # port 9222, real window, your profile
python -m jarvis ask "open youtube, search for lofi beats and click the first video"

# screen control with no vision model
python -m jarvis screen --read              # OCR the current screen
python -m jarvis ask "click the button that says Sign in"

# what do you know about me?
python -m jarvis profile

# Obsidian: study kit, daily note, revision queue, exam plan
export OBSIDIAN_VAULT="$HOME/Obsidian"
python -m jarvis obsidian kit --subjects "Physics,Chemistry,Maths"
python -m jarvis obsidian cards due          # spaced-repetition queue
python -m jarvis obsidian plan --exam 2027-03-01
python -m jarvis obsidian plugins --install  # the JARVIS Obsidian plugin

# local OpenAI-compatible API
python -m jarvis serve --port 8077

Inside jarvis chat you also get slash commands:

CLI reference

python -m jarvis [--version] COMMAND [ARGS]

chat        Interactive conversation (default)
ask         Ask one question and exit
doctor      Machine + capability report, with exact fix commands
tools       List tools, optionally filtered by category
api         Provider chain, health, latency, free-tier quota (--verify for live probes)
browser     Chrome harness: engines, terminal launch, fallback verification
screen      Screen control: capabilities, screenshots, OCR, click-by-text
memory      Inspect the smart memory manager (stats/consolidate/search)
obsidian    Vault notes, flashcards, revision queue, exams, study plans, plugins
quota       Free-tier usage per provider (--reset PROVIDER)
serve       Run the local API server (OpenAI-compatible)
search      Web search through the provider fallback chain
read        Read a URL through the crawl fallback chain
agents      List specialist sub-agents
delegate    Hand a task to one specialist
profile     Show what JARVIS knows about Heoster

Local API server

OpenAI-compatible, stdlib-only by default (FastAPI + uvicorn if installed):

python -m jarvis serve --host 127.0.0.1 --port 8077
# or: uvicorn-style via JARVIS_API_HOST / JARVIS_API_PORT / JARVIS_API_TOKEN

curl -s localhost:8077/ask -H 'content-type: application/json' \
     -d '{"message":"What is 21 * 2 + 8?"}'

Set JARVIS_API_TOKEN to require Authorization: Bearer <token>. The server binds to 127.0.0.1 by default — bind 0.0.0.0 only on a trusted network.

Obsidian & student mode

JARVIS treats an Obsidian vault as first-class memory for studying: a vault is just a folder of markdown files, so every tool works with Obsidian closed, offline, and without any plugin. Point JARVIS at your vault and it can read, write, search, plan and drill it.

export OBSIDIAN_VAULT="$HOME/Obsidian"        # or obsidian.vault_path in config.yaml
python -m jarvis obsidian kit                 # scaffold the student vault
python -m jarvis obsidian status              # vault, due cards, exams, streak

The study loop

Card formats (Obsidian-Spaced-Repetition compatible)

- What is the SI unit of pressure? :: pascal (Pa)
- Photosynthesis happens in the {{c1::chloroplast}}.

#flashcard The three laws of motion are stated by?
> Newton

Code fences are ignored, duplicates are merged, and the schedule lives in a single JSON file (data/obsidian_srs.json) — reviews are never lost, and the community Spaced Repetition plugin understands the same syntax if you prefer its UI.

What the study kit builds

00 Inbox · 01 Subjects/<subject> with index notes · 02 Daily Notes · 03 Flashcards · 04 Exams · 05 Revision · 99 Templates (lecture note, formula sheet, exam plan, daily note, weekly review) · JARVIS Home.md dashboard. Nothing existing is overwritten, and re-running it creates nothing twice.

The JARVIS plugin for Obsidian

obsidian-plugin/jarvis/ is a real community plugin — plain CommonJS, no bundler, no npm, no build step — that talks to your local JARVIS server over requestUrl:

python -m jarvis obsidian plugins --install    # copies + enables it in the vault

Commands (Ctrl/Cmd+P): Ask JARVIS · Explain the current note · Summarize the current note · Explain the selected text · Make flashcards from the current note · Show today's revision queue · Research this topic · Save selection to memory · Test the connection. The ribbon robot opens a chat panel where every answer can be inserted into the note or turned into cards. A status-bar dot shows whether the server is reachable.

Nothing is sent anywhere except 127.0.0.1 (plus whichever LLM provider you configured) — the plugin has no account, no telemetry, no cloud.

Tools (116)

Every tool has a validated JSON schema (python -m jarvis tools spec calculate shows it), a timeout, and returns ToolResult(ok, content, data, error, elapsed_ms, artifacts). Blocked and destructive operations return ok=false with the reason — never a silent success.

Agents

Seven specialists share the same brain, memory and registry, but only see the tools they need:

python -m jarvis agents
python -m jarvis delegate browser "open codeex-ai.vercel.app and read the hero section"
python -m jarvis ask "research the best free LLM APIs and save the findings"   # auto-routed

Routing is keyword-scored with structural signals (a URL + "open" → browser; "write/fix/run" + code → code; "my preference" → memory); below the confidence threshold the main orchestrator handles the turn itself.

Architecture

                       ┌──────────────────────────┐
   CLI / API ─────────► │  Orchestrator            │  guards: 12 iterations,
                       │  perceive→plan→act→learn │  24 calls, 300 s/turn
                       └────┬───────────┬─────────┘
                            │           │
              ┌─────────────▼──┐   ┌────▼─────────────┐
              │  Brain         │   │  Agents (7)      │
              │  prompt+pick   │   │  browser/research│
              └────┬───────────┘   └────┬─────────────┘
                   │                    │
        ┌──────────▼─────────┐   ┌──────▼────────────┐
        │ API Manager        │   │  Tool Registry    │
        │ health·quota·retry │   │  116 tools / 10   │
        │ verified failover  │   │  categories       │
        └──────────┬─────────┘   └──────┬────────────┘
                   │                    │
   ollama→groq→cerebras→openrouter→   browser·screen·search·crawl·
   gemini→hf→stub                     code·file·utility·comms·memory·obsidian
                   │                    │
              ┌────▼────────────────────▼─────┐
              │  Smart Memory Manager         │
              │  working/episodic/semantic/   │
              │  procedural · hybrid recall   │
              └───────────────────────────────┘

• Context manager keeps the prompt inside a 14 000-token budget: it merges system blocks, shrinks tool results, compacts old turns into a running summary, and guarantees the newest user turn always survives.
• Tool selection scores every tool against the request (category + name + docstring overlap with regex boosts for maths, dates and URLs), so a small local model gets ≤ 9 relevant tools instead of all 116.
• Everything is inspectable: jarvis status, jarvis doctor, TurnResult.context_line, memory.budget_report().

Free-tier limits

JARVIS tracks every provider's free allowance and refuses to exceed it (data/quota.db):

jarvis quota shows live usage; jarvis quota --reset groq clears a counter (e.g. after the daily reset).

Configuration

config.yaml is the single source of truth; everything can also come from the environment with the JARVIS__SECTION__KEY pattern, and secrets are read from .env / real env vars (see .env.example).

llm:
  provider: auto                 # auto = verified fallback chain
  fallback_chain: [ollama, groq, cerebras, openrouter, gemini, hf, stub]
  model: hermes3:3b              # small enough for a 4 GB laptop
  cost_policy: free              # refuse anything that could bill money

browser:
  engine_order: [playwright, selenium, cdp, screen]
  hybrid:  {enabled: true, move_duration_ms: 260, click_settle_ms: 350}

screen:
  vision_model: false            # by design — OCR + DOM only
  ocr: {min_confidence: 45, fuzzy_threshold: 0.8}

memory:
  working_window: 24
  importance_threshold: 0.35
  consolidation_interval: 25
  recall_budget: 3500
  long_term: {embedding_provider: auto, dedupe_threshold: 0.9}

Useful environment variables: JARVIS_CONFIG (alternate config file), JARVIS_TRASH_DIR, JARVIS_CHROME_PATH, JARVIS_PROFILE, JARVIS_API_HOST/PORT/TOKEN, plus the provider keys (GROQ_API_KEY, CEREBRAS_API_KEY, OPENROUTER_API_KEY, GEMINI_API_KEY, HUGGINGFACE_API_KEY, TAVILY_API_KEY, SERPER_API_KEY, FIRECRAWL_API_KEY, JINA_API_KEY).

Troubleshooting / doctor

python -m jarvis doctor is the first thing to run on a new machine — it reports and fixes:

Tests

python -m pytest tests/ -q          # 109 tests, no network, no model needed

Everything is hermetic: a temporary config, a temporary home, blanked API keys, the stub provider.

Project layout

jarvis/
├── core/            orchestrator, brain, api manager, context manager, messages, providers
├── memory/          manager, embedder, vector store, tiers, profile, skills, sessions, knowledge
├── tools/
│   ├── browser/     harness, DOM parser, CDP, terminal launcher, actions, screenshots
│   ├── screen/      backends, mouse/keyboard, capture, OCR, windows, actions
│   ├── search/      base (quota+http), tavily, serper, searxng, parallel
│   ├── crawl/       reader chain, jina, firecrawl, local extractor
│   ├── code/        python executor, bash executor (guarded), analyzer
│   ├── filesystem/  file ops (trash-aware), directory tree, search
│   ├── utility/     calculator, datetime, json, clipboard, image, memory, discovery
│   ├── obsidian/    vault, flashcards + SM-2 scheduler, study kit/plan, plugins
│   └── communication/ email, webhooks, notifications, speech
├── agents/          base, 6 specialists, factory + routing
├── interfaces/      CLI (rich), API server (stdlib/FastAPI)
├── utils/           logger, tokens, sanitizer, rate limiting, quota, platform probe
├── persona.py       who Heoster is (drives profile + system prompt)
├── config.py        layered config + .env + JARVIS__ overrides
└── version.py       name, version, banner
obsidian-plugin/      JARVIS for Obsidian (plain-JS community plugin, no build step)
knowledge/            operator profile, command reference, student workflow
scripts/             setup, chrome harness, Xvfb, doctor, wrapper
tests/               109 hermetic tests

Status & limits

• Verified here: 109/109 tests, CLI end-to-end, orchestrator turns (real fallback to stub), memory write/recall/consolidate, API server endpoints, crawl chain reporting, bash safety matrix, tool schemas.
• Needs hardware to verify: real Chrome engines, cursor movement and OCR — this sandbox has no browser binary, no DISPLAY and no xdotool. The harness reports precisely what is missing rather than failing silently, and browser --verify --deep performs the same check on your machine.
• Not implemented on purpose: any paid API, any vision model, any telemetry. JARVIS sends data nowhere except the providers and readers you configure.

Built for Heoster — CODEEX-AI · MIT licensed · free for everyone, at zero cost.