跳到正文
GitHub·· 3 天前AI 评分22

ReductStore:面向机器人与工业 IoT 的时间索引对象存储

reductstore/reductstore

AI 导读

ReductStore 是面向机器人与工业 IoT 的高性能时间索引对象存储,可按时间戳与标签存储并查询图像、视频、传感器读数、日志、文件和 ROS bag 等原生二进制数据。

正文

ReductStore banner

GitHub release (latest SemVer) GitHub Workflow Status Docker Pulls GitHub all releases codecov Community

🚀 60k+ downloads   •   🏭 100+ production deployments   •   📦 1 PB+ managed data

⭐ 350+ GitHub stars   •   🛠️ 4+ years of active development

Make robotics and industrial data queryable

ReductStore is high-performance, time-indexed object storage for robotics and industrial IoT.

Store terabytes of images, video, sensor readings, logs, files, and ROS bags in their native binary formats. Every record is organized by timestamp and can be enriched with labels, so you can retrieve exactly the data you need by time range and operational context.

Instead of combining a time-series database, generic object storage, metadata indexes, and custom retention jobs, ReductStore provides one system for ingesting, organizing, retaining, and querying multimodal time-series data.

We built ReductStore for robotics and industrial systems that continuously produce images, video, sensor readings, logs, files, and ROS bags. These workloads need the time index of a time series database and the payload flexibility of object storage without maintaining both systems.

Native support for continuous multimodal data ReductStore MinIO AIStor TimescaleDB MongoDB
Store arbitrary binary payloads ✅ Native records ✅ Native objects 🟡 BYTEA rows 🟡 BinData or GridFS
Index binary data by timestamp ✅ Built in ❌ Object key index ✅ Hypertable time index 🟡 Schema and index required
Query by time range and metadata ✅ Time and label query ❌ No online time/metadata query
Listing or batch inventory
🟡 Requires schema + indexes 🟡 Indexed timestamp/metadata
GridFS for >16 MiB
Manage a fixed disk as a ring buffer ✅ Capacity based FIFO quota ❌ Age based expiration ❌ Time based retention 🟡 Capped collections
Replicate selected data upstream ✅ Entry and label conditions 🟡 Prefix and object tag rules ❌ No native selective replication
Streaming replication is full copy
❌ Application pipeline required
Compress and delete historical data ✅ Lifecycle policies 🟡 Expiration/tiering
Compression on PUT
✅ Columnstore and retention policies 🟡 TTL expiration

ReductStore keeps the timestamp, labels, and binary payload in one record. The same labels drive queries, deletion policies, and replication, so edge devices can retain a bounded local history and send only relevant data upstream.

This comparison covers native capabilities for continuously recorded multimodal data. A partial result means the capability requires a custom schema, restricted rule model, batch workflow, or additional setup rather than being a first class feature of the storage model.

When You Should Use It

ReductStore is ideal for scenarios where you have a continuous stream of binary data and your application has metadata that can be attached to the data as labels. If you ingest binary data with labels into a ReductStore instance, you can use it to:

  1. Query objects by time range and labels, e.g. "give me all the images from camera-1 over the last hour where the device status is 'error'".
  2. Replicate only selected data to another ReductStore instance, e.g. "replicate all the data from camera-1 where the device status is 'error' to the cloud instance for further analysis".
  3. Manage the data lifecycle with policies, e.g. "keep all the data from camera-1 for 30 days, then compress it and keep it for another 60 days before deleting it".

In such scenarios, ReductStore can help you manage and transfer your data efficiently based on its metadata, without combining a TSDB and blob storage. This keeps your architecture simple and efficient.

Who It Is For

  • Robotics platforms that need to replay camera frames, telemetry, and logs around incidents or model failures
  • Industrial IoT pipelines that collect binary payloads at the edge and forward only filtered data upstream
  • Edge AI systems that need retention, labeling, and historical retrieval without building a custom storage stack

Get Started

The quickest way to get up and running is with Docker:

mkdir -p ./data
sudo chown -R 10001:10001 ./data
docker run -p 8383:8383 -v ${PWD}/data:/data \
  -e RS_API_TOKEN=my-secret-token reduct/store:latest

Or download a Linux binary directly from the latest release:

curl -LO https://github.com/reductstore/reductstore/releases/latest/download/reductstore.x86_64-unknown-linux-gnu.tar.gz
tar -xzf reductstore.x86_64-unknown-linux-gnu.tar.gz
mkdir -p ./data
RS_DATA_PATH=./data RS_API_TOKEN=my-secret-token ./reductstore

For a more in-depth guide, visit the Getting Started and Download sections.

After initializing the instance, you can start writing and querying data immediately. Here's a Python sample:

import asyncio

from reduct import Client

async def main():
    async with Client("http://localhost:8383", api_token="my-secret-token") as client:
        bucket = await client.create_bucket("my-bucket", exist_ok=True)

        await bucket.write(
            "camera-1",
            b"hello, reductstore",
            timestamp="2024-01-01T10:00:00Z",
            labels={"device": "camera-1"},
        )

        async for record in bucket.query(
            "camera-1",
            start="2024-01-01T10:00:00Z",
            stop="2024-01-01T10:00:01Z",
            when={"&device": {"$eq": "camera-1"}},
        ):
            print((await record.read_all()).decode())

asyncio.run(main())

Next Steps

Learn more and pick the next piece you need:

Build From Source

ReductStore uses a vendored protoc binary through Cargo, so contributors do not need to install Protocol Buffers manually. To check the workspace locally, run:

cargo check --workspace

When You Should Not Use It

ReductStore is not the best fit for every data storage problem. You should consider another solution when:

  1. You have only numerical data that can be easily ingested into a time-series database.
  2. You have only blob data to store and do not need to access it as historical data. In this case, object storage or a file system may be a better fit.
  3. You need a message broker. Although ReductStore provides subscription and publishing functionality, it is designed for data storage and streaming, not message queueing.

Community & Contribution

If you've found a bug, have ideas, built something with ReductStore, or want to contribute directly, here are the best places to jump in:

  • Questions and Ideas: Join our Discourse community to ask questions, share ideas, and collaborate with fellow ReductStore users.
  • Bug Reports: Open an issue on our GitHub repository. Please provide as much detail as possible so we can address it effectively.
  • First Contributions: Pick a task from good first issues or help wanted.
  • Pull Requests: Open PRs only for an existing issue. Comment on the issue and get assigned before you start work. Use the repository pull request template when you open the PR, and update CHANGELOG.md for user-visible changes. If you want to propose a new feature or workflow, open an issue first or start the discussion on the community forum before writing code.
  • Show Your Work: Share your projects, benchmarks, and lessons learned on our Discourse community.
  • Support the Project: If ReductStore is useful to you, give us a ⭐ on GitHub.

Contributors

Thanks to everyone who has contributed to ReductStore.

@atimin @mother-6000 @AnthonyCvn @DibbayajyotiRoy @rohankumardubey @rtadepalli @vbmade2000 @LordAizen1 @ragen1337 @tuanhungngyn @upuddu @Harshal875 @gitcommit90 @lntutor @mambaz @victor1234 @vjymisal0 @aschenbecherwespe @renghen @vintcessun

Your support fuels our passion and drives us to keep improving.

Together, let's build better storage for robotics and industrial data. 🚀

来源:GitHub · github.com