Ghost：本地授权 OSINT 案件文件工作区，SQLite 存储与来源溯源

Local authorized OSINT case files + provenance

A local case-file workspace for authorized self-audits — SQLite storage, scope/authorization, and report provenance. Not a SaaS. Not an "AI platform."

Quick Start · Features · Installation · Demo · Roadmap · Contributing · Security

Run an authorized local investigation from a single target (username, email, phone, domain, or photo), store it as a case file, and keep a provenance trail of what ran and where evidence came from. OpenAI is optional (--no-ai is the Quick Start path).

🔍 Why Ghost?

Ghost is a case-file workspace for authorized OSINT — not a drop-in replacement for a username hunter like maigret.

Maigret (and Sherlock) are excellent at asking "does this handle exist on hundreds or thousands of sites?" Ghost's job is different: take a single authorized target, run the modules you choose, store a SQLite case file with scope and authorization, and emit a report with provenance (what ran, which source URLs were collected, what failed).

Use maigret or Sherlock when you need the widest username hunt. Use Ghost when you need that evidence stored as a defensible investigation you can retrieve, export, and hand off.

The built-in username module is an HTTP status check against 70 platforms. It can shell out to Sherlock when the binary is on PATH. It does not wrap maigret.

Compared with general OSINT suites (Maltego, SpiderFoot, Recon-ng), Ghost's current wedge is local, authorized case files + provenance, not "more transforms than everyone else." The REST API and graph JSON exist today; a dashboard HTML UI is not shipped yet. HTML/JSON reports always work; PDF needs the optional weasyprint extra.

📸 Screenshots

A terminal demo GIF belongs at docs/screenshots/demo.gif once captured from the self-audit walkthrough. That file is not in the repo yet (placeholder path only — see docs/screenshots/README.md). Until then, these SVG captures show the CLI:

✨ Features

Investigation Vectors

Intelligence Engine

• 🤖 Optional AI correlation — OpenAI analysis only when OPENAI_API_KEY is set; --no-ai uses deterministic heuristics
• 📊 Risk Assessment — Heuristic scoring always; richer profiling when OpenAI is configured
• 🧩 Entity Resolution — Connects findings into graph-ready entities in SQLite
• 📈 Timeline Analysis — Heuristic event list from module output

Output & Reporting

• 📄 Reports — HTML and JSON always; PDF when the optional weasyprint extra is installed
• 🗺️ Entity graph API — D3-compatible JSON at /api/investigation/<id>/graph (dashboard HTML is not shipped yet)
• 🖥️ CLI — Rich interface, ghost doctor, and case-file commands (list, show, export, import, ingest, delete)
• 🔌 REST API — Flask at python -m ghost.backend.server (loopback by default). Case-data routes require GHOST_API_TOKEN; authorized_use: true is still required as case-file policy.

⚡ 60-second Quick Start

Authorized self-audit, no OpenAI key, no extra OSINT binaries. Copy-paste:

git clone https://github.com/juliosuas/ghost.git
cd ghost
python3 -m pip install -e .
python3 -m ghost doctor
python3 -m ghost investigate demo_user \
  --type username \
  --modules username \
  --no-ai \
  --authorized \
  --scope "authorized self-audit demo" \
  --format json \
  --output demo-report.json
python3 -m ghost list

python3 -m ghost is the supported entrypoint (ghost/__main__.py). After pip install -e ., the ghost console script from pyproject.toml is equivalent.

ghost doctor checks SQLite, optional keys, module imports, and exposure gates. On a fresh clone it exits 1 until GHOST_SECRET_KEY and GHOST_API_TOKEN are set (and GHOST_HOST, if set, is 127.0.0.1 or localhost). That does not block the investigate command below — CLI-only self-audit never starts Flask. See Configuration.

The investigate command stores a case in SQLite and writes demo-report.json with a provenance block. Redacted sample output lives in examples/. Quick Start uses --modules username only; social and darkweb collectors are experimental, off by default, and run only when named in --modules.

Use only an authorized target (your own handle, or a synthetic name like demo_user). Full walkthrough: authorized self-audit demo.

More CLI: ghost show <id-prefix>, ghost export, ghost import, ghost delete, or python3 -m ghost with no arguments for the interactive menu.

📦 Installation

From source (supported)

git clone https://github.com/juliosuas/ghost.git
cd ghost
python3 -m pip install -e .
# optional: python3 -m pip install -e ".[dev]"   # pytest + ruff, matches CI
# optional: cp .env.example .env                 # only if you need OpenAI or extra APIs

requirements.txt is a broader dependency pin used by the Docker image. For local CLI work, the editable install above is the path CI and the ghost script expect.

A pip or pipx install does not keep the SQLite database or investigation reports inside site-packages (those paths are wiped on reinstall or upgrade). Ghost writes them under a private home directory:

On Linux the platformdirs default is ~/.local/share/ghost. The home directory is created mode 0700 and the database file mode 0600. Override the home with GHOST_HOME=/var/lib/ghost (or any absolute path). A relative GHOST_HOME is resolved against the process working directory; ghost doctor warns when that happens. DATABASE_URL overrides only the database file and takes precedence over GHOST_HOME. ghost doctor prints the resolved data path and which of DATABASE_URL, GHOST_HOME, or platformdirs won.

Importing ghost.core.config calls prepare_storage() immediately. That creates the home directory and, when allowed, copies a legacy database before the CLI or API uses those paths.

If a previous install left ghost.db inside the package, Ghost copies it with the SQLite backup API (so uncheckpointed WAL data is included) into the new location once — only when the new file is absent and DATABASE_URL is unset. The original file is left in place and is never overwritten. Investigation reports that are still under the package investigations/ directory are not migrated; ghost doctor warns when those files are present.

PyPI

pip install ghost-osint is not published yet. The package name in pyproject.toml is ghost-osint; install from source until a GitHub release is tagged (see CHANGELOG.md).

Docker

git clone https://github.com/juliosuas/ghost.git
cd ghost
cp .env.example .env   # set GHOST_SECRET_KEY and GHOST_API_TOKEN; required by env_file
docker-compose up -d
# REST API → http://127.0.0.1:5000 by default (loopback only; dashboard HTML is not shipped)
# If .env sets GHOST_PORT, the host and the container both use that port.

The image sets GHOST_HOME=/app/ghost. Compose persists that home with named volumes ghost_data (/app/ghost/data) and ghost_investigations (/app/ghost/investigations). Those are not host binds: if ./data or ./investigations is missing, Docker would create it as root and uid 1000 could not write.

🔧 Configuration

Copy .env.example to .env and add your API keys:

ghost doctor and ghost doctor --json exit non-zero when any exposure gate fails (ok: false in JSON, with checks[].name equal to the env var). Setting these variables is not required to run ghost investigate --authorized --no-ai. Do not start python -m ghost.backend.server until doctor is green.

Storage

Ghost v2 stores investigations, findings, entities, and graph relationships in SQLite by default instead of JSON files. This gives local users durable, queryable storage without requiring a separate database server. The file lives under the home from the install section (<home>/data/ghost.db), not inside the package directory.

# optional; overrides the database file only
DATABASE_URL=sqlite:///ghost.db
# optional; home for data/ and investigations/
GHOST_HOME=/var/lib/ghost

PostgreSQL is on the roadmap behind the storage adapter boundary. For now, non-SQLite DATABASE_URL values fail explicitly so deployments do not silently write data to the wrong place.

See the storage adapter boundary for the contract Ghost must satisfy before advertising non-SQLite backends.

Note: OpenAI is optional. --no-ai (and a missing key) uses heuristic summaries. Additional keys unlock more modules (HIBP, Shodan, Google CSE, and so on).

📖 Usage

Python API

from ghost.core.investigator import GhostInvestigator

investigator = GhostInvestigator()
investigation = investigator.investigate(
    "demo_user",
    input_type="username",
    modules=["username"],
    scope="authorized self-audit",
    authorized_use=True,
)
report_path = investigator.generate_report(investigation, format="json", output_path="demo-report.json")

REST API

# Start the server (refuses to bind without GHOST_SECRET_KEY + GHOST_API_TOKEN)
python -m ghost.backend.server

# Submit an investigation
curl -X POST http://127.0.0.1:5000/api/investigate \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GHOST_API_TOKEN" \
  -d '{"target": "demo_user", "input_type": "username", "authorized_use": true, "scope": "authorized self-audit"}'

Ingest results from other tools

Sherlock finds accounts. Ghost turns them into a case file you can defend. For an authorized self-audit, or another investigation you are allowed to run, import a local report:

python3 -m ghost ingest sherlock-demo_user.csv --tool sherlock --authorized --scope "authorized self-audit"
python3 -m ghost ingest maigret-demo_user_simple.json --tool maigret --case demo_user
python3 -m ghost ingest holehe_1710000000_demo_user@example.com_results.csv --tool holehe --case demo_user@example.com

ghost ingest does not contact the network, launch the other tool, or run investigate. A file larger than 50 MB is refused: stat rejects it before reading, and the read itself stops after one byte past that cap. Invalid files import nothing (one database transaction). Each record stores the tool name, tool version when the file has one, the UTC ingest time, the SHA-256 of the file, and the URL exactly as written.

HTML and Markdown reports escape every imported field. Profile links are rendered only for http and https URLs; the stored URL is not rewritten. ghost investigate --format markdown writes the Markdown report.

🏗️ Architecture

.
├── ghost/           # Python package (`ghost` / `python -m ghost`)
│   ├── core/        # Orchestration, doctor checks, reports
│   ├── modules/     # OSINT collectors (username, email, phone, …)
│   ├── ai/          # Optional LLM analysis; heuristic fallback
│   ├── ui/          # Rich CLI
│   ├── backend/     # Flask REST API and SQLite case store
│   └── templates/   # HTML report templates
├── examples/        # Redacted sample case + expected report snippet
├── docs/            # Self-audit demo, storage contract, screenshots
└── tests/           # pytest suite (CI)

🗺️ Roadmap

• Plugin System — Drop-in custom modules with standard interface
• Geospatial Timeline — Map-based activity visualization over time
• Team Collaboration — Multi-user investigations with shared workspaces
• Telegram Bot — Run investigations from Telegram
• Export to STIX/TAXII — Threat intelligence format compatibility
• Graph Database — Neo4j backend for complex relationship mapping
• Mobile App — iOS/Android companion for field investigations
• Scheduled Monitoring — Continuous target monitoring with alerts

🤝 Contributing

Contributions are welcome! Here's how to get started:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-module
3. Commit your changes: git commit -m "Add amazing module"
4. Push to the branch: git push origin feature/amazing-module
5. Open a Pull Request

CI runs Ruff and pytest on Python 3.10, 3.11, and 3.12, and pip-audit on Python 3.12. PRs should include proof plus screenshots or terminal output when user-facing behavior changes.

Good first issues and docs questions have templates under .github/ISSUE_TEMPLATE/. Please do not include private target data.

Areas We Need Help

• 🌍 New OSINT modules — More platforms, more data sources
• 🧪 Testing — Unit tests, integration tests, edge cases
• 📝 Documentation — Guides, tutorials, API docs
• 🎨 Dashboard UI — First HTML frontend for the existing graph API
• 🌐 Translations — i18n support for global users

💬 Community

• GitHub Discussions — Questions, ideas, show & tell
• GitHub Issues — Bug reports & feature requests
• Security — Report vulnerabilities privately (Security tab → Report a vulnerability). Do not open a public issue.

⚠️ Legal Disclaimer

This tool is provided for authorized security research, journalism, law enforcement, and personal use only.

By using Ghost, you agree to the following:

• You are solely responsible for ensuring your use complies with all applicable local, state, national, and international laws
• Unauthorized surveillance, stalking, or harassment is illegal and unethical
• Always obtain proper authorization before investigating individuals
• Data collected may be subject to GDPR, CCPA, and other privacy regulations
• The developers assume no liability for misuse of this tool

Ghost must NOT be used to:

If you are unsure whether your use case is lawful, consult a legal professional before proceeding.

📄 License

MIT License — see LICENSE for details.

Built with 👻 by Julio

If Ghost helps your work, consider giving it a ⭐

🌱 Also check out

AI Garden — A living world built exclusively by AI agents. Watch it grow.