跳到正文
RadarAI· 智源 FlagOpen:原创开源项目·· 16 小时前AI 评分38

智源 FlagOpen 开源 HumanCOIN:基于 LeRobot 的多传感器多频率数据集扩展

FlagOpen 开源 HumanCOIN:基于 LeRobot 的多传感器多频率数据集扩展

AI 导读

智源 FlagOpen 开源 HumanCOIN,一个基于 LeRobot 的多传感器、多频率数据集扩展,每个传感器按各自采样率独立存为 parquet 文件,查询时按时间窗口动态对齐,无需预先重采样。

正文 · 原文

智源 FlagOpen:原创开源项目·2026-11-17 14:19· 43 分钟前AI 评分42

FlagOpen/HumanCOIN

AI 导读

智源 FlagOpen 开源 HumanCOIN,一个基于 LeRobot 的多传感器、多频率数据集扩展,每个传感器按各自采样率独立存为 parquet 文件,查询时按时间窗口动态对齐,无需预先重采样。

正文

📖 中文文档:README_zh-CN.md

A multi-sensor, multi-frequency dataset extension for real-world data, built on LeRobot — sensors are stored at their independent sampling rates, one parquet file per sensor, dynamically aligned by time window at query time.


Table of Contents


1. Motivation: Why Multi-Frequency Support

Real-world data collection often requires multiple sensors to capture the same process: cameras record visual information, IMUs measure motion, and EEG and EMG capture physiological signals. Human behavior research, wearable devices, experimental recording, and environmental sensing all involve combining such data. Sensors differ in sampling rate, startup time, and data format, so their individual time series need to be preserved and aligned when used.

Common sensors operate at very different sampling rates:

Sensor Typical rate Shape Example
Camera RGB 30 / 60 Hz 480×640×3 scene camera + wearable camera
IMU 50-1000 Hz 6 (accel×3 + gyro×3) accelerometer + gyroscope
EEG 256 / 512 / 1000 Hz 8-256 channels brain signals
EMG 20-500 Hz 4-16 channels muscle signals

Forcing streams with different sampling rates into a table at a single frame rate typically requires upsampling low-rate data and downsampling or aggregating high-rate data, which can discard original samples. This project builds on LeRobot’s dataset infrastructure to support independent sensor storage and time-based queries.

The humancoin solution: every sensor is stored independently as a time-indexed parquet file. At read time, a window parameter defines "how to aggregate data around the query timestamp" — no pre-resampling needed.

Writing — add_frame routes each key to its feature buffer; save_episode() flushes them to disk:

flowchart LR
    AF["add_frame(key, value)"]
    IX["MasterIndex record_frame(ts, task)"]
    PF["ParquetFeature buffer.append(ts, data)"]
    VF["VideoFeature save PNG + timestamp list"]
    AuF["AudioFeature buffer chunks + timestamp list"]

    MI[("master_index.parquet timestamp | task | frame_index")]
    P1[("imu.parquet 1000Hz × 6d")]
    P2[("eeg.parquet 256Hz × 8d")]
    P3[("state.parquet 30Hz × 7d")]
    C1[("head_rgb.parquet timestamps only")]
    V1[("head_rgb.mp4 30fps video")]
    A1[("mic.parquet timestamp | num_samples")]
    W1[("mic.wav 16kHz audio")]

    AF -->|"key = task"| IX
    AF -->|"key = sensor<br/>imu / eeg / state"| PF
    AF -->|"key = video"| VF
    AF -->|"key = audio"| AuF
    IX --> MI
    PF --> P1
    PF --> P2
    PF --> P3
    VF --> C1
    VF --> V1
    AuF --> A1
    AuF --> W1

    style AF fill:#fafafa,stroke:#9e9e9e,color:#000
    style IX fill:#e1f5fe,stroke:#4fc3f7,color:#000
    style PF fill:#fafafa,stroke:#9e9e9e,color:#000
    style VF fill:#f3e5f5,stroke:#9c27b0,color:#000
    style AuF fill:#e8f5e9,stroke:#66bb6a,color:#000
    style MI fill:#e1f5fe,stroke:#4fc3f7,color:#000
    style P1 fill:#fff3e0,stroke:#ff9800,color:#000
    style P2 fill:#fff3e0,stroke:#ff9800,color:#000
    style P3 fill:#fce4ec,stroke:#e91e63,color:#000
    style C1 fill:#f3e5f5,stroke:#9c27b0,color:#000
    style V1 fill:#f3e5f5,stroke:#9c27b0,color:#000
    style A1 fill:#e8f5e9,stroke:#4caf50,color:#000
    style W1 fill:#e8f5e9,stroke:#4caf50,color:#000

Loading

Reading — look up the master-clock timestamp in master_index, then query each feature by that timestamp:

flowchart TD
    IDX["ds[10]"] -->|"① look up frame 10"| MI[("master_index.parquet timestamp | task | frame_index")]
    MI -->|"② get timestamp"| TS["t = 0.333s"]
    TS -->|"③ nearest<br/>closest row in time"| P3[("state.parquet 30Hz × 7d")]
    TS -->|"③ window=(-0.033, 0] all rows in range"| P1[("imu.parquet 1000Hz × 6d")]
    TS -->|"③ window=(-0.033, 0] all rows in range"| P2[("eeg.parquet 256Hz × 8d")]
    TS -->|"③ nearest, closest row in time"| C1[("head_rgb.parquet timestamp")]
    C1 -->|"④ get frame_index"| V1[("rgb → 480×640×3")]
    P3 -->|"1 row"| RS["state → [7]"]
    P1 -->|"33 rows"| RI["imu → [33, 6]"]
    P2 -->|"8 rows"| RE["eeg → [8, 8]"]
    V1 -->|"1 frame"| RV["head_rgb → [3, 480, 640]"]

    style IDX fill:#fafafa,stroke:#9e9e9e,color:#000
    style MI fill:#e1f5fe,stroke:#4fc3f7,color:#000
    style TS fill:#fafafa,stroke:#9e9e9e,color:#000
    style P3 fill:#fce4ec,stroke:#e91e63,color:#000
    style P1 fill:#fff3e0,stroke:#ff9800,color:#000
    style P2 fill:#e8f5e9,stroke:#4caf50,color:#000
    style C1 fill:#f3e5f5,stroke:#9c27b0,color:#000
    style V1 fill:#f3e5f5,stroke:#9c27b0,color:#000
    style RS fill:#fce4ec,stroke:#e91e63,color:#000
    style RI fill:#fff3e0,stroke:#ff9800,color:#000
    style RE fill:#e8f5e9,stroke:#4caf50,color:#000
    style RV fill:#f3e5f5,stroke:#9c27b0,color:#000

Loading

The result of ds[10] (t=0.333s):

  • state → nearest match → [7]
  • imu → all 33 readings in the (-0.033, 0] window → [33, 6]
  • eeg → all 8 readings in the (-0.033, 0] window → [8, 8]
  • head_rgb → nearest row in timestamp parquet → frame_index → decode that frame → [3, 480, 640]

2. Installation

Python 3.10–3.13 (verified with Python 3.12). LeRobot is currently pinned to 0.3.3.

Install from PyPI:

pip install humancoin

Or install from source:

git clone https://github.com/FlagOpen/HumanCOIN
cd HumanCOIN
pip install -e .

Dependencies:

Dependency Purpose
lerobot base dataset framework (inherits LeRobotDataset)
pyarrow parquet file I/O
numpy / torch data processing
av (PyAV) video encoding/decoding
matplotlib visualization
scipy web visualizer signal filtering

Download HumanCOIN datasets

Browse dataset names on Hugging Face or ModelScope. These are organization pages; pass a dataset name from the selected hub to the command below.

pip install humancoin

# Replace DATASET_NAME with an actual dataset repository name
humancoin-download --hub huggingface --datasets DATASET_NAME --target-dir ./data
humancoin-download --hub modelscope --datasets DATASET_NAME --target-dir ./data

# Download only selected files (partial download; may not be loadable as a full dataset)
humancoin-download --datasets DATASET_NAME --include "meta/*" README.md --target-dir ./data

# Multiple datasets; preview without downloading
humancoin-download --datasets DATASET_A DATASET_B --target-dir ./data --dry-run
# Also available as: python -m humancoin.download ...

The default namespace is HumanCOIN; explicit owner/name IDs are also accepted. --ds_lists is an alias for --datasets. Files are stored under <target-dir>/HumanCOIN/<dataset-name>; the default target directory is ~/.cache/huggingface/lerobot. Repeating a download reuses files through the hub SDK. Use --revision to select a version and --max-workers to control parallel downloads. Private/gated datasets require access and HF_TOKEN or MODELSCOPE_TOKEN (or --token); ModelScope uses the lightweight modelscope-hub SDK; credentials are passed to the client or read from its login cache. MS_TOKEN is also accepted. A failed dataset does not stop the remaining downloads; the command exits nonzero if any fail.

from humancoin import HumanCOINDataset

ds = HumanCOINDataset(
    repo_id="HumanCOIN/DATASET_NAME",
    root="data/HumanCOIN/DATASET_NAME",
)
item = ds[0]

Downloads preserve the repository files; loading requires the multi-frequency dataset layout supported by this project. Downloading does not convert other formats.

3. Quick Start

import numpy as np
from humancoin import HumanCOINDataset

ds = HumanCOINDataset.create(
    repo_id="my_dataset", root="data/", fps=30,
    features={
        # camera: dtype="video" auto-encodes to MP4
        "observation.images.cam": {
            "dtype": "video", "shape": (480, 640, 3),
            "names": ["h", "w", "c"], "fps": 30,
        },
        # high-rate IMU: window defines "return all readings in (t-33ms, t]"
        "observation.imu": {
            "dtype": "float32", "shape": (6,),
            "names": ["ax", "ay", "az", "gx", "gy", "gz"],
            "fps": 1000,
            "window": (-0.033, 0.0),
            "tolerance_s": 0.002,
        },
        # mid-rate EEG
        "observation.eeg": {
            "dtype": "float32", "shape": (8,),
            "names": ["Fz", "Cz", "Pz", "Oz", "F3", "F4", "C3", "C4"],
            "fps": 256,
            "window": (-0.033, 0.0),
        },
        # end-effector pose: no window → nearest neighbor by default
        "observation.state": {
            "dtype": "float32", "shape": (7,),
            "names": ["x", "y", "z", "roll", "pitch", "yaw", "gripper"],
            "fps": 30,
        },
        "action": {
            "dtype": "float32", "shape": (7,),
            "names": ["dx", "dy", "dz", "droll", "dpitch", "dyaw", "dgrip"],
            "fps": 30,
        },
    },
)

# recording: task = master clock, each frame is bounded by a task call
for i in range(300):
    # high-rate sensors: multiple writes per frame
    for _ in range(33):   # 1000Hz / 30fps ≈ 33
        ds.add_frame("observation.imu", imu_sample)
    for _ in range(8):    # 256Hz / 30fps ≈ 8
        ds.add_frame("observation.eeg", eeg_sample)

    # a task call = frame boundary, timestamp auto-generated
    ds.add_frame("task", "reach_target")

    # camera-rate sensors: once per frame
    ds.add_frame("observation.state", joint_state)
    ds.add_frame("action", action_cmd)
    ds.add_frame("observation.images.cam", camera_image)

ds.save_episode()

Reading — window features return all readings in the window, nearest features return a single vector:

ds = HumanCOINDataset(repo_id="my_dataset", root="data/")
item = ds[10]

item["observation.imu"]             # tensor([33, 6]) — 33 readings in (-0.033, 0]
item["observation.imu_timestamps"]  # tensor([33])
item["observation.eeg"]             # tensor([8, 8])
item["observation.state"]           # tensor([7]) — nearest
item["observation.images.cam"]      # tensor([3, 480, 640]) — CHW
item["task"]                        # "reach_target"
sequenceDiagram
    participant User
    participant DS as Dataset
    participant PF as ParquetFeature<br/>(imu)
    participant VF as VideoFeature<br/>(cam)
    participant IX as MasterIndex

    loop each frame
        User->>DS: add_frame("imu", data)
        DS->>DS: _auto_timestamp("imu")<br/>ts = counter × 1/1000
        DS->>PF: add(data, ts)
        Note over PF: buffer.append((ts, data))

        User->>DS: add_frame("task", "reach")
        DS->>DS: ts = len(index) / fps
        DS->>IX: record_frame(ts, "reach")

        User->>DS: add_frame("cam", image)
        DS->>DS: _auto_timestamp("cam")<br/>ts = counter × 1/30
        DS->>VF: add(image, ts)
        Note over VF: save PNG + record timestamp
    end

    User->>DS: save_episode()
    DS->>IX: write(ep_idx) → master_index.parquet
    DS->>PF: save() → imu.parquet
    DS->>VF: save() → timestamp parquet + MP4
    DS->>DS: meta.save_episode()
    DS->>DS: _check_episode_alignment()

Loading

4. Demo & Visualization

4.1 Running the Demo

# write 3 episodes of simulated data
python -m examples.01write_simple

# read and inspect the data
python -m examples.01read_simple

# interactive visualizer
humancoin-visualize data/demo --episode 0

# web visualizer (browser UI, draggable time window)
humancoin-visualize-web /path/to/dataset

# audio: microphone stored as WAV + timestamp parquet
python -m examples.06audio_example

The web visualizer supports optional zero-phase filtering of EEG/EMG strips: open the 滤波 panel on a strip and combine a high-pass / low-pass Butterworth with a comb notch at the line frequency and all its harmonics (EEG/EMG presets included). The API takes the same spec per feature: /api/frame?...&filt=observation.eeg:hp=0.5,lp=40,notch=50.

It also renders a 3D pose view in the SteamVR right-handed y-up world (*_pose 6-DoF features as coordinate triads + recent-history motion trails with time fading, fingertip end-effector paths from hand_skeleton FK; drag to rotate, wheel to zoom) and a color-coded electrode grid for static snapshot features such as eeg_impedance (quality-banded per channel). Pose rotation follows the recorder exactly (R = Rz(yaw)·Ry(pitch)·Rx(roll), the inverse of the recorder's ZYX euler decomposition of the OpenVR matrix); hand skeletons attach via a constant per-hand Manus→tracker mount map (+z = finger, +y = back of hand, plus a −5cm offset along −y since the tracker sits above the back of the hand).

4.2 Demo 1: Writing (01write_simple.py)

The main write demo: a set of simulated sensor streams — 3 cameras + IMU + EEG + EMG + temperature — recording three 2-second episodes (reach_target, pour_liquid, stack_blocks):

Sensor Shape Rate Window
head_rgb 480×640×3 30 fps — (video)
left_wrist_rgb 480×640×3 30 fps — (video)
right_wrist_rgb 480×640×3 30 fps — (video)
imu 6 (accel+gyro) 1000 Hz (-0.033, 0.0)
eeg 8 (EEG channels) 256 Hz (-0.033, 0.0)
emg_left 4 (muscle channels) 100 Hz (-0.033, 0.0)
emg_right 4 (muscle channels) 100 Hz (-0.033, 0.0)
temperature 1 10 Hz — (nearest)
state 7 (x,y,z,r,p,y,grip) 30 Hz — (nearest)
action 7 (velocity commands) 30 Hz — (nearest)

What it shows:

  • dtype: "video" features auto-encode to MP4, exactly one frame per master frame
  • high-rate sensors are written multiple times between two task calls; timestamps are auto-generated per feature as timestamp_start + counter × 1/fps
  • each save_episode() writes master_index.parquet + per-feature parquets + videos, then runs the alignment checker (every feature should report OK)
python -m examples.01write_simple

4.3 Demo 2: Reading (01read_simple.py)

Reads the dataset written by Demo 1 and exercises every read path end to end:

  • window features (observation.imu) return all readings in the window plus their timestamps
  • nearest features (observation.temperature, observation.state, action) return a single vector
  • videos are decoded back to CHW tensors
  • window_overrides widen the IMU window at read time without touching the stored data
  • prints the IMU segment sizes across frames — the first frames are empty because the IMU only comes online at timestamp_start: 0.1
python -m examples.01read_simple   # needs data/ from 01write_simple

4.4 Demo 3: Alignment Tolerance (02tolerance_example.py)

Shows the alignment checker catching a sensor that violates the alignment contract: the IMU is configured with timestamp_start: 0.1 (it comes online 100 ms late) plus a tight tolerance_s: 0.002, so its samples are too far from the first master frames. After save_episode() the checker writes the offending frames to meta/alignment_check.jsonl and the demo prints the report.

Rule of thumb: tolerance_s must cover the distance from a master frame to the sensor's nearest sample — at most half its period (e.g. ≤ 50 ms for a 10 Hz sensor) — plus startup offset and jitter.

python -m examples.02tolerance_example

4.5 Demo 4: Window Modes (03window_example.py)

One episode, three aggregation modes side by side:

  • observation.imu at 1000 Hz with window: (-0.033, 0.0) — range query, ~33 readings per frame
  • observation.finger_pose at 120 Hz with window: "interpolate" — value interpolated at the frame timestamp
  • observation.temperature at 10 Hz with no window — nearest neighbor

Then the dataset is reopened with window_overrides — a wider IMU window, finger pose switched to a range query — proving that the window is a read-time decision, not a property of the stored data.

python -m examples.03window_example

4.6 Demo 5: Delta Timestamps (04delta_example.py)

Multi-frame temporal queries with delta_timestamps={"observation.state": [-0.1, 0.0, 0.1]}: reading ds[i] returns state stacked over three timepoints (t-0.1, t, t+0.1) instead of one vector — the building block for delta-pose training and temporal augmentation.

python -m examples.04delta_example

4.7 Demo 6: Asynchronous Sensor Startup (05async_startup_example.py)

Real sensors don't come online at the same instant: after the process starts, each one returns its first frame at its own wall-clock time (IMU at 0.04 s, EMG at 0.12 s, EEG amplifier at 0.31 s, …). This demo simulates such streams and aligns them to the master clock before adding data:

  1. t_all_started — the wall-clock moment every sensor has delivered its first frame (the max of all first-frame times).
  2. t = 0 — the master clock reference sensor's first frame at/after t_all_started; its earlier frames are discarded.
  3. timestamp_start — for every other sensor, the delta between its next frame (first frame at/after t = 0) and the t = 0 frame; set it in the feature spec so the stored timestamps stay phase-aligned with the master clock.
  4. Record everything from t = 0 onward. The episode ends when the master clock stops, but each sensor keeps streaming until its next frame after the master's last one — that trailing sample keeps the last frame's window full.

Simulated startup latencies:

Sensor Rate First frame (wall clock)
observation.imu 1000 Hz 0.04 s
observation.emg 100 Hz 0.12 s
observation.state (master clock reference) 30 Hz 0.18 s
observation.temperature 10 Hz 0.22 s
action 30 Hz 0.24 s
observation.eeg 256 Hz 0.31 s

The demo computes t_all_started = 0.31 s, takes the master's next frame as t = 0, derives each sensor's timestamp_start, writes the dataset, verifies on disk that every feature's first stored timestamp equals its timestamp_start and that its last sample lands past the master stop, then reads back frame 0, a mid-episode frame, and the last frame.

Two consequences worth knowing:

  • Frame 0's backward window is empty. Every sensor's first stored frame sits at its own timestamp_start > 0 and earlier data is discarded by design, so a window like (-0.033, 0.0) has nothing to return on the first frame(s).
  • Tolerance must cover half the sensor's period. The alignment checker compares each master frame against the sensor's nearest sample, whose distance is at most half a period (a 10 Hz sensor can be up to 50 ms away) plus jitter — e.g. tolerance_s: 0.06 for the 10 Hz temperature sensor here.
python -m examples.05async_startup_example

4.8 Demo 7: Audio (06audio_example.py)

One camera + state + a simulated microphone, showing the audio storage path end to end:

  • dtype: "audio" features define shape as one chunk of samples (n_samples,) (mono) or (n_samples, channels) and require sample_rate
  • one add_frame per master frame stores one chunk; save_episode() concatenates all chunks into one PCM16 WAV per episode under audios/, plus a timestamp parquet index (timestamp | episode_index | num_samples) next to the other per-feature parquets
  • reading ds[i] in nearest mode resolves the chunk nearest the master timestamp through the parquet index and returns it whole — the atomic unit, analogous to one video frame; with a window (spec, window_overrides, or a direct read(..., window=(start, end)) argument) it returns every sample whose time lies in (t + start, t + end], sliced with sample precision across chunk boundaries and concatenated
  • the alignment checker reports WAV integrity: chunk count vs index rows, sample count vs summed index
python -m examples.06audio_example

4.9 Visualizer Controls

Visualizer (humancoin-visualize data/demo --episode 0):

Visualizer screenshot

space    — play / pause
← →      — step frame backward / forward
↑ ↓      — faster / slower playback
w s      — widen / shrink the window
q / Esc  — quit

5. Core Concepts

5.1 Per-Feature Independent Storage

Storage layout:

data/
└── chunk-000/
    └── episode_000000/
        ├── master_index.parquet                # timestamp | frame_index | task_index
        ├── observation.imu.parquet             # 1000Hz · 1980 rows · 6 cols
        ├── observation.eeg.parquet             # 256Hz · 512 rows · 8 cols
        ├── observation.state.parquet           # 30Hz · 60 rows · 7 cols
        └── observation.images.head_rgb.parquet # 30Hz · 60 rows · timestamps only

videos/
└── chunk-000/
    └── observation.images.head_rgb/
        └── episode_000000.mp4

audios/
└── chunk-000/
    └── observation.audio.mic/
        └── episode_000000.wav

Every non-video feature gets one parquet with a uniform column layout:

timestamp (float64) | episode_index (int64) | col_0 (float32) | ... | col_D (float32)
  • camera-rate features (state, action): one row per frame, rows = frames
  • high-rate features (imu, eeg): multiple samples between frames, rows = rate × duration
  • camera features: timestamp parquet + MP4 video
  • audio features: timestamp parquet (timestamp | episode_index | num_samples, one row per added chunk) + one WAV per episode; chunks are concatenated in timestamp order

5.2 Task as the Master Clock

The whole dataset is organized around task calls as frame boundaries: add_frame("task", ...) declares "the current frame ends here, the next one begins".

  • Frame = all feature data written between two task calls. add_frame("task", ...) auto-generates the frame timestamp: t = frame_index / fps (override with timestamp= if needed)
  • Master clock = the frame sequence. Reading ds[i] first looks up the i-th frame timestamp t in master_index; every feature is then aligned around t (window / nearest neighbor, see 5.3)
  • Each feature's timestamps are generated independently of the master clock: one counter per feature, ts = timestamp_start + counter × (1 / fps) on each add_frame; counters reset after save_episode()
  • No master_feature to configure — task is the master clock, and features of any sampling rate align to it
# standard call order per frame: high-rate first → task bounds the frame → low-rate last
ds.add_frame("observation.imu", ...)   # high-rate sensor, written multiple times between frames
ds.add_frame("observation.eeg", ...)
ds.add_frame("task", "pick_up")        # ← frame boundary: frame_index++, master-clock timestamp
ds.add_frame("observation.state", ...) # camera-rate sensor, once per frame
ds.add_frame("observation.images.cam", image)

5.3 Window Parameter

The master-clock frame timestamp is the query anchor, but a feature's readings don't necessarily land on a frame — window defines "how to aggregate that feature's readings around the frame timestamp":

window value Behavior Typical use
None (default) nearest neighbor: return the single closest reading camera-rate sensors (state, action); audio: one whole chunk
"interpolate" linear interpolation at the query time mid-rate sensors (finger pose)
(start_s, end_s) range query: return all readings in (t+start, t+end] high-rate sensors (IMU, EEG, EMG); audio: sample-precise slice + concat
# IMU at 1000Hz, 30fps master → ~33 readings per frame
"window": (-0.033, 0.0)   # (t-33ms, t]

# wider window for context
"window": (-0.066, 0.0)   # (t-66ms, t], ~66 readings

# override at read time
ds = HumanCOINDataset(
    repo_id="my_dataset", root="data/",
    window_overrides={"observation.imu": (-0.100, 0.0)},
)

Readings returned per feature when querying t = 0.333s (frame 10):

Readings returned per feature when querying t = 0.333s (frame 10)

Note that temperature (10Hz) has no sample at 0.333 — its sampling grid is 0.3, 0.4, …, so nearest neighbor falls back to the reading at t=0.300.

6. API Reference

Creating a Dataset

ds = HumanCOINDataset.create(
    repo_id="my_dataset",
    fps=30,
    features={...},          # see Feature Definition Format below
)
Parameter Type Default Description
repo_id str — dataset name, directory name under root
fps float — master-clock frame rate; frame timestamp = frame_index / fps
features dict[str, dict] — feature definitions, format below
root str | Path ~/.cache/huggingface/lerobot/<repo_id> storage root directory
robot_type str None robot type, metadata only
use_videos bool True whether dtype="video" features are encoded to MP4
image_writer_processes int 0 image writer subprocess count (0 = write in main process)
image_writer_threads int 0 image writer thread count
video_backend str None video backend (e.g. "pyav"), None auto-selects
batch_encoding_size int 1 batch-encode videos every N episodes
window_overrides dict None override each feature's window (same format as at read time)

Feature Definition Format

features = {
    "feature_key": {
        "dtype": "video" | "audio" | "float32" | "int64",  # required
        "shape": (H, W, C) | (N,) | (D,),         # required
        "names": ["col_0", ...],                  # required
        # optional:
        "fps": 1000,              # sampling rate of this feature
        "window": (-0.033, 0.0),  # query window
        "timestamp_start": 0.1,   # timestamp start offset
        "tolerance_s": 0.002,     # alignment tolerance
        "sample_rate": 16000,     # audio only: sample rate of the WAV
    }
}
Field Type Required Description
dtype str ✓ storage type: "float32" (continuous), "int64" (discrete/id), "video" (camera frames), "audio" (microphone chunks)
shape tuple ✓ shape of a single reading: (D,) for numeric features, (H, W, C) for video, (n_samples,) or (n_samples, channels) for audio
names list[str] ✓ column name per dimension (used as parquet column names)
fps float optional sampling rate of this feature; timestamps are auto-generated as timestamp_start + counter × 1/fps; falls back to master-clock fps if omitted
window None | "interpolate" | (start, end) None how readings are aggregated at query time (see 5.3)
timestamp_start float 0.0 timestamp start offset, e.g. the sensor only starts recording at t=0.1s
tolerance_s float None alignment tolerance: after save_episode(), per-frame nearest-reading gaps are checked and violations are logged to meta/alignment_check.jsonl
sample_rate int audio only sample rate of the WAV file (e.g. 16000); required for dtype="audio"

Conventions and tips:

  • feature keys follow LeRobot naming and must not contain / (e.g. observation.imu, action)
  • dtype="video" features only store a timestamp parquet; images are encoded as MP4 under videos/
  • dtype="audio" features store one WAV per episode under audios/ plus a timestamp parquet; each add_frame passes one chunk of float32 samples in [-1, 1] ((n_samples,) mono or (n_samples, channels)). Nearest read returns one whole chunk; a window (start, end) read returns the samples in (t+start, t+end] concatenated (timing contract: a chunk's first sample sits at its label, samples advance at exactly 1/sample_rate, real dropouts shorten the segment rather than stretch it)
  • always give high-rate features both fps and window — otherwise the nearest-neighbor query returns a single reading and the high-rate information is lost

Writing

ds.add_frame(key: str, value: np.ndarray | torch.Tensor | str,
             timestamp: float | None = None)
ds.save_episode()
  • key: "task" or a defined feature key
  • value: np.ndarray / torch.Tensor for numeric features (shape must match shape); np.ndarray (H, W, C) for video features; str task label for task
  • timestamp: None auto-generates — task uses the master clock len(records) / fps, other features use their own counter timestamp_start + counter × 1/fps; pass a float to set it manually

A complete write loop:

FPS = 30
imu_per_frame = 1000 // FPS   # 33
eeg_per_frame = 256 // FPS    # 8

for frame in range(60):                        # 2 seconds / 30fps
    for _ in range(imu_per_frame):             # high-rate: multiple writes per frame
        ds.add_frame("observation.imu", imu_sample)
    for _ in range(eeg_per_frame):
        ds.add_frame("observation.eeg", eeg_sample)

    ds.add_frame("task", "reach_target")       # ← frame boundary

    ds.add_frame("observation.state", joint_state)    # low-rate: once per frame
    ds.add_frame("action", action_cmd)
    ds.add_frame("observation.images.cam", camera_image)

ds.save_episode()    # flush all buffers, run alignment/video checks, reset counters

Reading

ds = HumanCOINDataset(
    repo_id="my_dataset", root="data/",
    episodes=[0, 2],                                    # load only episodes 0, 2
    image_transforms=my_transforms,                      # torchvision transforms
    delta_timestamps={"observation.state": [-0.1, 0, 0.1]},  # multi-frame query
    window_overrides={"observation.imu": (-0.066, 0)},   # override window
    video_backend="pyav",
)
item = ds[10]
Parameter Type Default Description
repo_id str — dataset name
root str | Path ~/.cache/huggingface/lerobot/<repo_id> storage root directory
episodes list[int] None load only the given episode indices
image_transforms callable None image transforms (torchvision style)
delta_timestamps dict[str, list[float]] None multi-frame query: returns {key: master frame + each offset frame}, e.g. {"observation.state": [-0.1, 0, 0.1]}
tolerance_s float 1e-4 frame-alignment tolerance for delta_timestamps
video_backend str None video decoding backend (e.g. "pyav")
batch_encoding_size int 1 (write side) video batch-encoding size
window_overrides dict None override each feature's window at read time, e.g. {"observation.imu": (-0.066, 0)}

Depth video reading: features with info["video.is_depth_map"] = true and HWC shape = [H, W, 1] are decoded directly with PyAV. Reads return native samples as a (1, H, W) tensor (e.g. uint16 for gray12le), without RGB conversion or division by 255. Units and invalid values remain those of the stored data. This applies to reading; depth-specific encoding is not implemented.

Return Structure

item = ds[10]
# {
#     "timestamp": tensor(0.3333),
#     "frame_index": tensor(10),
#     "episode_index": tensor(0),
#     "index": tensor(10),
#     "task_index": tensor(0),
#     "task": "reach_target",
#
#     # window feature: all readings + timestamps
#     "observation.imu": tensor([33, 6]),
#     "observation.imu_timestamps": tensor([33]),
#
#     # nearest feature: a single vector
#     "observation.state": tensor([7]),
#     "observation.temperature": tensor([1]),
#
#     # video feature: CHW format
#     "observation.images.head_rgb": tensor([3, 480, 640]),
#
#     # audio feature: one chunk of samples, float32 in [-1, 1]
#     "observation.audio.mic": tensor([533]),
#
#     # audio feature with window (-0.033, 0): samples in (t-33ms, t], concatenated
#     "observation.audio.mic": tensor([528]),
# }

7. Architecture

flowchart TB
    subgraph Entry["entry layer"]
        DS["HumanCOINDataset<br/>dataset.py · orchestrator<br/>(inherits LeRobotDataset)"]
    end

    subgraph Core["core layer"]
        IX["MasterIndex<br/>index.py"]
        MD["HumanCOINDatasetMetadata<br/>metadata.py"]
    end

    subgraph Features["feature layer"]
        PF["ParquetFeature<br/>parquet.py"]
        VF["VideoFeature<br/>video.py"]
        AuF["AudioFeature<br/>audio.py"]
    end

    subgraph Validate["validation layer"]
        CK["DatasetChecker<br/>checker.py"]
    end

    subgraph Store["storage layer"]
        PQ[("parquet files<br/>data/chunk-*/episode_*/")]
        MP4[("MP4 videos<br/>videos/chunk-*/")]
        WAV[("WAV audio<br/>audios/chunk-*/")]
        META[("metadata<br/>meta/")]
    end

    DS -->|"frame index I/O"| IX
    DS -->|"info.json / metadata"| MD
    DS -->|"write readings / window query"| PF
    DS -->|"write images / decode videos"| VF
    DS -->|"write chunks / read WAV"| AuF
    DS -->|"post-save validation"| CK
    PF -->|"save / read"| PQ
    VF -->|"encode MP4"| MP4
    VF -->|"timestamp table"| PQ
    AuF -->|"write WAV"| WAV
    AuF -->|"timestamp table"| PQ
    IX -->|"save"| PQ
    MD -->|"save"| META
    CK -->|"validation report"| META

    style DS fill:#e8eaf6,stroke:#4fc3f7,stroke-width:2px,color:#000
    style IX fill:#ffebee,stroke:#ef9a9a,color:#000
    style MD fill:#fafafa,stroke:#9e9e9e,color:#000
    style PF fill:#e8f5e9,stroke:#66bb6a,color:#000
    style VF fill:#f3e5f5,stroke:#ce93d8,color:#000
    style AuF fill:#e8f5e9,stroke:#66bb6a,color:#000
    style CK fill:#fff3e0,stroke:#ffcc80,color:#000
    style PQ fill:#fafafa,stroke:#9e9e9e,color:#000
    style MP4 fill:#fafafa,stroke:#9e9e9e,color:#000
    style WAV fill:#fafafa,stroke:#9e9e9e,color:#000
    style META fill:#fafafa,stroke:#9e9e9e,color:#000

Loading

File layout:

src/humancoin/
├── dataset.py      (341 lines)  orchestrator — add_frame / save_episode / __getitem__
├── audio.py        (185 lines)  AudioFeature — buffer chunks, write WAV + timestamp parquet
├── parquet.py      (168 lines)  ParquetFeature — write buffer, save parquet, load, window query
├── video.py        (168 lines)  VideoFeature — save PNG, encode MP4, decode frames
├── index.py        (127 lines)  MasterIndex — frame index I/O
├── metadata.py     (123 lines)  HumanCOINDatasetMetadata — info.json management
├── checker.py      (162 lines)  DatasetChecker — video frame count + audio integrity + timestamp alignment checks
└── utils.py        (236 lines)  constants + info.json creation/validation

Design principles:

  1. One object per feature: ParquetFeature, VideoFeature and AudioFeature each encapsulate their buffer, file paths, and read/write logic. The Dataset is only an orchestrator.

  2. Symmetric read/write: every Feature provides both add/save (write) and load/query (read).

  3. Deterministic paths: file paths depend only on episode_index and feature_key; no lookups in episodes.jsonl.

  4. Inherit, don't rewrite: inherit LeRobotDataset for start_image_writer, _save_image and other basics; only 4 core methods are overridden.

8. Comparison with Native LeRobotDataset

Aspect Native LeRobotDataset humancoin
Storage 1 merged parquet N per-feature parquets
Multi-rate sensors pre-resampling required native rates, independent storage
Time alignment tolerance_s tolerance matching window mechanism (range / interpolate / nearest)
Master clock camera frame add_frame("task", ...) calls
Video encoding unified batch encoding in save_episode independent per VideoFeature
Metadata episodes.jsonl records everything only basic frame info; paths computed deterministically
per-feature fps not supported independent fps per feature
per-feature tolerance global tolerance_s independent tolerance_s per feature
Audio storage not supported WAV per episode + timestamp parquet index, nearest-chunk read
Write API add_frame(frame_dict, task) add_frame(key, value) per field
Alignment check none automatic alignment_check.jsonl
Video check none automatic frame count/duration check

9. Acknowledgements

This project builds on the following open-source projects and work:

  • LeRobot — the base dataset framework. This project inherits the API design of LeRobotDataset and LeRobotDatasetMetadata and reuses its video encoding and image writing infrastructure. Thanks to the Hugging Face team for their contributions to the open-source robot learning ecosystem.

  • PyArrow — high-performance columnar storage, the basis of per-feature parquet I/O.

  • PyAV — Python bindings for FFmpeg, powering video encoding (AV1) and decoding.

  • Matplotlib — visualization.

  • The Neuropixels data format and the BIDS standard from the Allen Institute for Neural Dynamics — inspiration for organizing multimodal neural data.

Thanks to all colleagues for feedback and discussions.

Building and checking a release

pip install -e ".[dev]"
python -m unittest discover -s tests -v
python -m build
python -m twine check dist/*

The wheel includes dataset reading/writing, downloads, and both visualizers. Install humancoin to use all these features; no source checkout is required. The examples/ demonstration scripts require a source checkout.

来源:智源 FlagOpen:原创开源项目 · github.com

来源:RadarAI · aihot.news