SelfHeal：用 LLM 智能体闭环修复微服务故障的开源项目

SelfHeal is an AI/LLM agent that closes the loop on microservice failures. It watches a simulated cluster, detects anomalies, ranks evidence-backed root-cause hypotheses (rules plus an LLM that only picks an index), proposes an allowlisted remediation, checks a safety policy, acts, verifies against live metrics, and reverts or escalates when the fix did not help.

It works fully offline by default with a deterministic mock provider, so demos, tests and every deployment run without paid keys. Any OpenAI-compatible API (OpenAI, Groq, Ollama) can be plugged in. Safety never depends on the model: the planner and policy decide what can run.

Contents

• The problem
• Features
• Screenshots
• Architecture
• How the healing loop works
• Scenarios
• LLM providers
• Quick start
• REST API
• Security model
• Deployment
• Configuration
• Project layout
• Testing
• Roadmap
• Contributing
• License

The problem

Microservices fail in noisy, cascading ways: a bad deploy, a memory leak, a flash-sale traffic spike, a third-party outage, a hung database. On-call engineers have to detect the anomaly, pick a root cause among red herrings, choose a safe remediation, and check that the fix actually worked — under time pressure, with the risk of making things worse (rolling back the wrong service, scaling past capacity, restarting a protected database without a second pair of eyes).

SelfHeal automates that control loop while keeping the dangerous part — changing the cluster — behind an action allowlist, cooldowns, budgets, clamps and an explicit human approval gate for protected services. Every action is verified against live metrics and undone if it did not help. The LLM only ranks hypotheses that rules already produced; it never invents an action.

Features

• Closed verify loop: detect → diagnose → plan → safety-check → act → verify → (revert + try next hypothesis | resolve | escalate). Ineffective actions are undone.
• Deterministic cluster simulator: 8-service shop topology with dependency cascades, circuit breakers, seeded faults and a single validated apply() mutation path. External services are immutable.
• SLO + EWMA detector: threshold breaches plus a z-score baseline that learns only from healthy samples, with a 2-tick debounce.
• Rule diagnosis + LLM ranking: evidence-backed hypotheses (bad deploy, memory leak, traffic spike, dependency failure, stuck process). The model returns only a hypothesis index; invalid / hallucinated replies fall back to the top rule.
• Allowlisted planner: rollback_deploy, restart_service, scale_out, open_circuit_breaker, each with an inverse for reverts.
• Safety policy: action allowlist, external deny, global budget (6 / 30 min), per-service cooldown, scale clamp, param checks, protected → human approval, attempt cap.
• Approval gate: protected services (e.g. orders-db) wait for an approver; reject escalates instead of executing.
• Offline by default: mock provider passes all 6 packaged scenarios (including a red-herring deploy that is refuted and reverted, and a prompt-injection deploy note).
• REST API + dashboard: FastAPI with operator / approver roles, public demo, live topology and sparklines, timeline, incidents, approvals; hardened multi-platform deploy configs.

Screenshots

Dashboard deep links use #run=<id>&tab=<name> (topology, timeline, incidents, approvals).

Topology health grid	Live event timeline

Incidents	Approvals (protected restart)

Architecture

flowchart LR
    subgraph Clients
        CLI["selfheal run / scenarios - CLI"]
        UI["Web dashboard - static HTML/JS"]
        HTTP["REST client - curl, CI, scripts"]
    end

    subgraph Service["FastAPI service - selfheal serve"]
        Guard["AccessGuard: API key / roles / public-demo, rate limits, body cap, CSP"]
        Jobs["RunManager: thread pool or inline sync mode"]
    end

    subgraph Core["Healing control loop"]
        Det["Detector: SLO + EWMA z-score"]
        Diag["Diagnosis: rule hypotheses"]
        LLM["LLM provider: pick hypothesis index"]
        Plan["Planner: allowlisted action + inverse"]
        Pol["Safety policy: allow / approve / deny"]
        Heal["Healer state machine: act, verify, revert, escalate"]
    end

    subgraph Sim["Simulated cluster"]
        Cl["8 services, faults, metrics, breakers"]
    end

    UI --> Guard
    HTTP --> Guard
    Guard --> Jobs --> Heal
    CLI --> Heal
    Heal --> Det --> Diag
    Diag --> LLM
    Diag --> Plan --> Pol --> Heal
    Heal --> Cl
    Det --> Cl

Loading

How the healing loop works

sequenceDiagram
    autonumber
    actor User
    participant API as API or CLI
    participant H as Healer
    participant D as Detector
    participant R as Rules + LLM
    participant P as Planner + Policy
    participant C as Cluster simulator

    User->>API: run scenario
    loop each simulated minute
        API->>H: tick
        H->>C: advance metrics / faults
        H->>D: detect anomalies
        alt new anomaly
            D-->>H: anomaly
            H->>R: diagnose (rules + LLM picks index)
            R-->>H: ranked hypotheses
            H->>P: plan action for top hypothesis
            P-->>H: allow / require_approval / deny
            alt allow
                H->>C: apply(action)
                H->>H: VERIFY (3 healthy ticks)
                alt verified
                    H-->>API: resolved
                else still unhealthy
                    H->>C: apply(inverse)  revert
                    H->>H: try next hypothesis
                end
            else require_approval
                H-->>User: awaiting approval
                User->>H: approve or reject
            else deny / external
                H-->>API: blocked or escalated
            end
        end
    end

Loading

1. Observe. The seeded cluster advances one tick (one simulated minute). Metrics are derived from utilisation; dependency errors cascade unless a circuit breaker is open.
2. Detect. SLO thresholds (error rate, p95 latency, CPU, memory, availability) plus an EWMA z-score baseline. The baseline learns only from healthy samples. A 2-tick debounce filters blips. Early memory leaks are flagged when memory z > 4 and level > 60%.
3. Diagnose. Deterministic rules produce ranked hypotheses with evidence strings (recent deploy, rising memory, RPS ≫ baseline, dependency unhealthy, high latency + low CPU = stuck). Telemetry is wrapped in <untrusted-data> with < escaped. The LLM returns only {"choice": index}; any invalid / out-of-range / erroring reply falls back to the top rule.
4. Plan. The chosen cause maps to a fixed action (rollback_deploy to a version that was actually deployed, restart_service, scale_out to rps / (cap * 0.6), open_circuit_breaker on a real dependency). Every action has an inverse() for reverts (restart has none).
5. Safety-check. The policy allowlists kinds, denies external services, enforces a global action budget and per-service cooldown, clamps scale steps, type-checks params, and routes protected services to REQUIRE_APPROVAL. Attempt cap is 3 per incident.
6. Act. Cluster.apply() is the only mutation path. It re-validates (rollback target in history, replica bounds, breaker on a real dependency, external immutable).
7. Verify. Three consecutive healthy minutes within a five-minute window resolve the incident. Failure refutes the hypothesis, reverts the action when possible, and tries the next one. Callers of an internal service with an active incident wait (symptom, not cause) and auto-resolve when the upstream heals. An external outage produces a single escalation (no re-paging until healthy).

Scenarios

Each packaged scenario has a known answer that the tests and the container smoke test check:

LLM providers

# OpenAI
export SELFHEAL_PROVIDER=openai
export SELFHEAL_BASE_URL=https://api.openai.com/v1
export SELFHEAL_MODEL=gpt-4o-mini
export SELFHEAL_API_KEY=sk-...

# Groq
export SELFHEAL_PROVIDER=openai
export SELFHEAL_BASE_URL=https://api.groq.com/openai/v1
export SELFHEAL_MODEL=llama-3.3-70b-versatile
export SELFHEAL_API_KEY=gsk_...

# Ollama (local, no key needed)
export SELFHEAL_PROVIDER=openai
export SELFHEAL_BASE_URL=http://localhost:11434/v1
export SELFHEAL_MODEL=llama3.1
export SELFHEAL_API_KEY=ollama

Public-demo mode always forces the mock, whatever the environment says. The model runs at temperature 0 with retries on 429 / 5xx; it only ever chooses a hypothesis index.

Quick start

git clone https://github.com/bharatkumar00797/selfheal-ai-microservices.git
cd selfheal-ai-microservices
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

selfheal scenarios
selfheal run bad-deploy
selfheal run traffic-spike          # refutes the red-herring deploy, then scales
selfheal run stuck-database --approver "$USER"
selfheal serve --dev                # dashboard on http://127.0.0.1:8000

REST API

Interactive docs are at /docs (OpenAPI at /openapi.json).

curl -s -X POST localhost:8000/api/runs -H 'Content-Type: application/json' \
  -d '{"scenario":"bad-deploy"}'
curl -s "localhost:8000/api/runs/<id>/events?since=0"
curl -s -X POST localhost:8000/api/runs/<id>/approvals/<incident_id>/approve \
  -H 'Content-Type: application/json' -d '{"approver":"on-call"}'

Access modes

Runs use a bounded background worker pool (POST → 202, poll events). With SELFHEAL_SYNC_RUNS=true (automatic on Vercel / AWS Lambda) they finish inside the request (200 with the full event list).

Error codes

Security model

An agent that watches telemetry and changes a cluster needs hard boundaries. SelfHeal layers several guards; none of them is a silver bullet.

Residual risks (stated honestly):

• The cluster is simulated. Remediations never touch a real Kubernetes / ECS / Nomad control plane in this project.
• State (runs, rate limits, ownership) lives in memory per process: lost on restart, and on serverless an approval may land on a fresh instance and return 404. Prefer a container platform for anything beyond a single-request demo.
• Detectors and rule diagnosis are heuristics. A real LLM may still pick the wrong hypothesis index; verification + revert is the backstop, not model correctness.
• Behind a reverse proxy without SELFHEAL_TRUST_PROXY=true, all public-demo visitors share one identity. With it set, trust only proxies you control.
• The mock provider is safe by construction; treat hosted-model output as advisory ranking only.

See SECURITY.md for how to report a vulnerability.

Deployment

All container platforms build the same hardened, non-root image (Dockerfile) and probe /healthz; the server listens on $PORT. Without API keys every deployment is a public demo. Lock a deployment down by setting SELFHEAL_API_KEYS / SELFHEAL_APPROVER_KEYS as platform secrets.

Long-running container vs serverless

Docker and docker compose

docker compose up --build --wait
python3 scripts/smoke_test.py http://127.0.0.1:8000   # all 6 scenarios end to end

Compose runs read-only with all capabilities dropped, a tmpfs for /tmp, and memory / pid limits. The image health-checks /healthz and honours $PORT. SELFHEAL_HOST_PORT changes the host port (compose only).

Hardened one-shot run:

docker build -t selfheal .
docker run --rm -p 8000:8000 --read-only --tmpfs /tmp --cap-drop ALL \
  --security-opt no-new-privileges -e SELFHEAL_SYNC_RUNS=true selfheal

Render

render.yaml Blueprint: Docker runtime, free plan, health check on /healthz, SELFHEAL_PROVIDER=mock, SELFHEAL_TRUST_PROXY=true. Connect the repo and Render applies the Blueprint. Add secrets in the dashboard.

Fly.io

fly launch --copy-config --no-deploy
fly deploy

fly.toml sets PORT=8080 (= internal_port), a /healthz check and region bom.

Railway

railway.json selects the Dockerfile builder, probes /healthz and restarts on failure.

Vercel

api/index.py is the serverless entry; vercel.json bundles src/selfheal/** (scenarios + dashboard) and rewrites every path to the function. Runs execute in sync mode because functions cannot keep background threads. Prefer a container platform if you need approve-after-the-fact across requests.

Configuration

All settings are environment variables (see .env.example).

Project layout

src/selfheal/
  models.py       ServiceSpec, MetricSample, Anomaly, Hypothesis, Action, Event
  cluster.py      Deterministic 8-service simulator + Faults + apply()
  detector.py     SLO thresholds + EWMA z-score baseline
  diagnosis.py    Rule-based ranked hypotheses with evidence
  planner.py      Cause → allowlisted action (+ inverse for reverts)
  policy.py       Safety policy: allow / require_approval / deny
  llm.py          Mock + OpenAI-compatible providers (hypothesis index only)
  healer.py       Incident state machine: act, verify, revert, escalate, approve
  engine.py       Scenario runner with progress snapshots
  scenarios.py    Six packaged chaos scenarios with expectations
  cli.py          selfheal scenarios | run | serve
  config.py       SELFHEAL_* provider settings
  api/            FastAPI app, settings, auth + rate limits, middleware, jobs, static dashboard
api/index.py      Vercel entrypoint
scripts/          smoke_test.py (stdlib end-to-end check)
tests/            pytest suite
docs/screenshots/ dashboard screenshots

Testing

ruff check . && ruff format --check . && mypy && pytest

CI runs this on Python 3.11, 3.12 and 3.13, then builds the hardened container with docker compose up --wait and runs scripts/smoke_test.py against it (all 6 scenarios, including the approval flow).

Roadmap

• Pluggable real executors (Kubernetes / ECS) behind the same allowlist and approval gate
• Persistent run store (SQLite / Redis) for multi-instance and serverless approve-after-the-fact
• Richer detectors (change-point, seasonal baselines) and multi-incident correlation
• Webhook / Slack notifications for escalations and approval requests

Contributing

See CONTRIBUTING.md. Please report security issues privately via SECURITY.md.

License

MIT