跳到正文
Hacker News· kuroko1t·· 5 天前AI 评分58

anygrep 发布:在 CPU 上对笔记、PDF 和图片做语义搜索

Show HN: anygrep – semantic search over notes, PDFs and images, on the CPU

AI 导读

anygrep 是一个在终端运行的语义搜索工具,可对笔记、PDF 和截图按含义检索,全程在 CPU 上运行 EmbeddingGemma 2 的 int8 ONNX 模型并维护本地索引。

正文 · 原文

PyPI Python License: MIT

anygrep searches your notes, PDFs and screenshots by meaning, from the terminal. It runs one model (EmbeddingGemma 2, int8 ONNX) on the CPU and keeps one local index.

anygrep demo

The demo searches a folder that was already indexed, and the wait for the model to load is sped up.

Install

pipx install anygrep        # or: uv tool install anygrep
uvx anygrep "when is the tax return due" ~/notes   # or try it without installing

Requires Python 3.10+. Tested on Linux; CI also runs the tests on Windows and macOS (Apple silicon). The model is downloaded on first use: about 350 MB, and 200 MB more once you index images. See docs/usage.md for Intel Macs, Windows on ARM and uninstalling.

Usage

anygrep "wifi password reset"                 # search here (new files are indexed first)
anygrep "when is the tax return due" ~/notes  # search one folder
anygrep --image whiteboard.jpg ~/Screenshots  # find similar pictures
anygrep -l "invoice from acme"                # file names only; piped output is path:line:text
anygrep -Q ~/notes                            # interactive: the model stays loaded
anygrep index --background ~/Pictures         # big folder: index in a low-priority process

Results are up to 10 files, closest in meaning first (-m N for more), not every match; queries are not regular expressions. A file whose text or name contains a query of 3 or more characters as typed (an invoice number, an error code) comes first, marked (exact); words found in many files are not boosted this way (details). Queries and documents can be in different languages. Weak matches are hidden (--weak shows them) and near-duplicate files are shown once. See anygrep -h and docs/usage.md.

On a terminal, the first search in a big folder indexes the new files first, with a progress bar; Ctrl-C keeps what is done and the next run continues. index --background does it while you work (your home folder is never indexed automatically, so name a folder inside it). In a script or cron job, a search indexes new files only if that takes under about 30 seconds; run anygrep index beforehand for a big folder (details).

It reads text files (.md .txt .html and more), the text layer of PDFs, and .jpg .png .webp .bmp .gif images. Hidden, ignored and binary files are skipped.

Use from Python

import anygrep

with anygrep.Index() as idx:  # the index the command uses
    idx.update("~/notes")  # index new and changed files
    for hit in idx.search("when is the tax return due", k=5):
        print(hit.score, hit.path, hit.snippet)

Install it with pip in your project's environment. It uses the same index, scores and filtering as the command and prints nothing. The API may change before 1.0; see docs/python-api.md.

Performance

On a Ryzen 7 7700 with the default 8 threads, text indexes at about 1,100 tokens/s and an image takes about 1.2 s. A search command takes about 1.5 s, mostly loading the model. More in docs/benchmarks.md.

See also how it works, relevance and comparison with other tools.

Limitations

  • No OCR: small text inside images is often missed, and scanned PDFs are skipped.
  • No audio, video or Office documents yet (anygrep says how many Office files it skipped), and no file watching (run anygrep index from cron).

License

MIT. The model is downloaded separately: an int8 ONNX conversion of Google's EmbeddingGemma 2, released under the Apache License 2.0 (its model card also points to the Gemma Prohibited Use Policy). anygrep is not affiliated with Google.

来源:Hacker News · github.com