lusipad/browser-agent 发布 Chrome 侧边栏自主浏览器智能体

Next-Generation Autonomous AI Browser Agent · Your Personal Digital Copilot
Goal-Driven · Visual ROI Selection · Teach-Me Learning · Native CDP Control · Self-Verification
🌐 Official Website · 中文文档 · Releases · Privacy Policy

Browser Agent is a next-generation autonomous browser agent living directly inside your Chrome side panel. Far more than a simple scraping script or macro recorder, it is a complete cognitive intelligence system equipped with perception, autonomous planning, multimodal visual grounding, demonstration-to-skill learning, trusted CDP execution, and self-correcting validation. Give it an open-ended goal, and it navigates, reasons, and executes across complex modern web applications just like a human expert — delivering verifiable, end-to-end outcomes.

📸 Feature Showcase

🎯 Visual Region Selection & Point-and-Shoot: Drag or click to snap any element, 1:1 pixel-perfect crop, cuts token usage by 80%, eliminates long-page ambiguity

🔴 Learning from Demonstration (Teach-Me): Manually operate in the browser; the agent silently captures events with zero-leak credential redaction, and an LLM generalizes it into a reusable SKILL

⏰ Scheduled Skills & Human Intervention: Periodic or daily cron background execution; gracefully suspends on Captcha/Turnstile challenges with instant resume

✍️ Complex Workflow Automation: Native CDP physical input piercing Shadow DOM and dynamic SPAs

🚀 Agent Superpowers

Break free from rigid single-step commands. The agent navigates the web with human-like cognitive agency:

• ⚡ Dual-Process Cognitive Engine (System 1 + System 2 Hybrid) — Reflexive sub-500ms fast jumps via specialized lightweight vision-language models (e.g. Jev-2B) for high-confidence immediate actions (clicks, simple navigation, search submit), paired with deliberative System 2 reasoning (DeepSeek, Claude, GPT-4o) for complex multi-step planning, page understanding, and verification.
• 🎯 Visual Region-of-Interest (ROI) Selection & Point-and-Shoot — Click 🎯 in the Composer or press Alt + S to enter crosshair selection mode. Box-select any card, chart, or table (or snap-click an element). The agent crops the area 1:1 via OffscreenCanvas with zero CDP debugger warning banners, passes high-res visual context, and cuts prompt token usage by 70%~80% while eliminating target ambiguity.
• 🔴 Learning from Demonstration (Teach-Me) — Click "🔴 Record New Skill" and interact with any webpage naturally (clicks, typing, navigations, file uploads). The agent silently intercepts user actions with automatic password redaction (******). When finished, an LLM abstracts raw selectors into resilient high-level intents and auto-extracts variable parameters into reusable SKILL workflows.
• ⏰ Scheduled Skills & Background Cron — Automate recurring workflows. Configure saved skills to run periodically (every 15m, 1h, 6h, 24h) or daily at a specific time (e.g. 09:30). Powered by chrome.alarms with desktop system notifications upon completion and full status reporting in Settings.
• 🛡️ Human-in-the-Loop Captcha Intervention — Built-in sniffers automatically detect Geetest, puzzle sliders, graphic captchas, SMS OTPs, and Cloudflare Turnstile. Rather than failing or hallucinating, the agent cleanly suspends execution and alerts the user; once solved in the browser, clicking "I have completed verification" seamlessly resumes the workflow.
• ⚡ Reusable SKILL Workflows & Zero-Leak Cloud Sync — Crystallize completed runs into reusable SKILLs with variable extraction ({{keyword}}). Synchronize bindings and workflows in real time across machines via chrome.storage.sync while strictly keeping API keys isolated in local storage (chrome.storage.local).
• 🧠 Autonomous Planning & Self-Correction Loop — Built on a Planner + Actor + Validator cognitive architecture with planning limits expanded up to 500 iterations. Decomposes open-ended goals into structured steps and validates page state before declaring victory.
• 👁️ Native System-Level Control & Spatial Grounding — Driven by Chrome DevTools Protocol (CDP) for authentic, trusted mouse and keyboard events (bypassing synthetic DOM event filters). Paired with Set-of-Marks visual tags to effortlessly conquer SPAs, nested Shadow DOMs, and cross-origin iframes.
• 🔒 Sovereign Local-First Architecture (Zero-Cloud) — Liberate yourself from overpriced monthly subscriptions and cloud surveillance. Direct connection to DeepSeek, Claude, OpenAI, or 100% offline Ollama / vLLM. Zero intermediary servers, per-site authorization gating, and prompt-injection defenses.

Features

• ⚡ Dual-Process Hybrid Engine — sub-500ms reflexive System 1 fast path (Jev-2B) + deliberative System 2 planner/validator
• 🎯 Visual ROI Box-Selection — drag-to-select or snap-click, 1:1 pixel-perfect crop, 80% token reduction, full-resolution Lightbox preview
• 🔴 Demonstration Learning (Teach-Me) — silent event interception, password redaction, LLM generalization into standard SKILLs
• ⏰ Scheduled Skills (Cron) — periodic or daily background execution, desktop notifications, status feedback in Settings
• 🛡️ Human Captcha Intervention — automatic detection of sliders/captchas, non-blocking suspension, and one-click resume
• Provider-centered model management — connect OpenAI / DeepSeek / OpenRouter / Ollama / vLLM / LM Studio…, discover models from /models, and manage them in bulk
• Multiple endpoints per model — use the same model through different providers or gateways
• Keys stay local — stored in chrome.storage.local, never sent to any intermediary
• Per-site authorization — each domain requires explicit approval before any interaction
• Set-of-marks — numbered bounding boxes overlaid on screenshots for precise element targeting
• Shadow DOM & iframe piercing — perception layer reaches into open shadow roots and same-origin frames
• Planner + Validator — decomposes tasks into steps with success criteria, supports up to 500 planning iterations
• Network-idle wait — actions wait for page load + network quiescence before proceeding
• Trusted CDP input — mouse/keyboard events via chrome.debugger (not synthetic DOM events)
• Real-time cost tracking — per-model pricing, context usage bar, cumulative cost display
• Multi-session history — archive, switch, export (Markdown/JSON)
• Bilingual UI — Chinese & English, follows browser locale or manual override
• Tab group transparency — agent-operated tabs grouped under blue "🤖 Agent" group

Quick Start

npm install
npm run build        # outputs to dist/

1. Open chrome://extensions, enable Developer mode
2. Click Load unpacked, select the dist directory
3. The Options page opens — enter your endpoint URL and API key, click Test Connection
4. Click the toolbar icon to open the sidebar, pick a model, start chatting

Base URLs may be entered with or without a trailing /v1 (for example, https://provider.example or https://provider.example/v1). Test Connection validates the endpoint, while Sync Models reads /models so you can import, enable, disable, or delete model access in bulk.

Use npm run watch during development. Changes to src/ are rebuilt automatically; changes to public/ require a full npm run build.

Keyboard Shortcuts

Security Model

Architecture

src/
  shared/          Types, settings, system prompt, utilities, i18n, skill storage & sync
    context.ts       Context governance: token estimation, budget trimming, cost accounting
    skill.ts         SKILL data structures, variable template resolution, import/export
    syncStorage.ts   Dual-layer sync engine with zero-leak local secret isolation
  providers/       OpenAI-compatible adapter (SSE streaming + history format conversion)
  background/      Service Worker: agent loop, session, permissions, tabs
    cdp.ts           chrome.debugger session, screenshots, console/network buffers
    reader.ts        Self-contained DOM perception functions (with anti-bot / captcha sniffing)
    regionSelector.ts In-page visual region selection injector (crosshair + element snap)
    recorder.ts      Demonstration recorder (debounced input, clicks, uploads, password redaction)
    scheduler.ts     chrome.alarms-based skill scheduler and desktop notification dispatch
    skillGen.ts      LLM trajectory crystallizer and demonstration action flow generalizer
    screenshot.ts    DPR-aware OffscreenCanvas high-resolution region cropping
    tools/           Tool definitions and dispatch (with human intervention & site gating)
    agent.ts         Model ↔ Tool iteration loop, multimodal prompt construction, Planner/Validator
    session.ts       Per-window session controller (suspension/resume, recording, persistence)
    history.ts       Multi-session archiving to chrome.storage.local
  sidepanel/       React side panel: conversation timeline, region chip, recording banner, lightbox
  options/         React settings page: providers, models, skill studio (cron config), sync, advanced

Toolset

Testing

npm test              # Full test suite: typecheck + 131 unit tests + E2E
npm run test:unit     # Unit tests with node:test
npm run test:e2e      # Real Chromium perception tests via Playwright
node scripts/test-region-e2e.mjs    # Visual region selection E2E verification
node scripts/test-teach-me-e2e.mjs  # Teach-Me demonstration E2E verification
npm run bench         # Perception benchmark (5 fixtures, ground-truth scoring)
npm run bench:e2e     # End-to-end agent benchmark (real LLM + Playwright)

See docs/testing.md for full manual validation protocol.

Documentation

Known Limitations

• Cross-origin iframes are treated as single clickable areas (text can be read via get_page_text)
• Cannot automate internal chrome:// and Web Store management pages
• Will not and should not bypass financial payment passwords; captchas are delegated via request_human_intervention
• Rich-text editors requiring key-by-key event triggers may prefer form_input over computer type

Acknowledgements

Architectural insights inspired by: browser-use (DOM serialization + set-of-marks), Nanobrowser (multi-agent Planner/Navigator/Validator), BrowserBee (in-extension CDP driver). Written independently from scratch.

⭐ Star History

License

MIT