智源 FlagOpen 开源 HumanCOIN:基于 LeRobot 的多传感器多频率数据集扩展
FlagOpen 开源 HumanCOIN:基于 LeRobot 的多传感器多频率数据集扩展
智源 FlagOpen 开源 HumanCOIN,一个基于 LeRobot 的多传感器、多频率数据集扩展,每个传感器按各自采样率独立存为 parquet 文件,查询时按时间窗口动态对齐,无需预先重采样。
智源 FlagOpen:原创开源项目·2026-11-17 14:19· 43 分钟前AI 评分42
FlagOpen/HumanCOIN
AI 导读
智源 FlagOpen 开源 HumanCOIN,一个基于 LeRobot 的多传感器、多频率数据集扩展,每个传感器按各自采样率独立存为 parquet 文件,查询时按时间窗口动态对齐,无需预先重采样。
正文
📖 中文文档:README_zh-CN.md
A multi-sensor, multi-frequency dataset extension for real-world data, built on LeRobot — sensors are stored at their independent sampling rates, one parquet file per sensor, dynamically aligned by time window at query time.
Table of Contents
- HumanCOIN
- Table of Contents
- 1. Motivation: Why Multi-Frequency Support
- 2. Installation
- 3. Quick Start
- 4. Demo & Visualization
- 4.1 Running the Demo
- 4.2 Demo 1: Writing (01write_simple.py)
- 4.3 Demo 2: Reading (01read_simple.py)
- 4.4 Demo 3: Alignment Tolerance (02tolerance_example.py)
- 4.5 Demo 4: Window Modes (03window_example.py)
- 4.6 Demo 5: Delta Timestamps (04delta_example.py)
- 4.7 Demo 6: Asynchronous Sensor Startup (05async_startup_example.py)
- 4.8 Demo 7: Audio (06audio_example.py)
- 4.9 Visualizer Controls
- 5. Core Concepts
- 6. API Reference
- 7. Architecture
- 8. Comparison with Native LeRobotDataset
- 9. Acknowledgements
1. Motivation: Why Multi-Frequency Support
Real-world data collection often requires multiple sensors to capture the same process: cameras record visual information, IMUs measure motion, and EEG and EMG capture physiological signals. Human behavior research, wearable devices, experimental recording, and environmental sensing all involve combining such data. Sensors differ in sampling rate, startup time, and data format, so their individual time series need to be preserved and aligned when used.
Common sensors operate at very different sampling rates:
| Sensor | Typical rate | Shape | Example |
|---|---|---|---|
| Camera RGB | 30 / 60 Hz | 480×640×3 | scene camera + wearable camera |
| IMU | 50-1000 Hz | 6 (accel×3 + gyro×3) | accelerometer + gyroscope |
| EEG | 256 / 512 / 1000 Hz | 8-256 channels | brain signals |
| EMG | 20-500 Hz | 4-16 channels | muscle signals |
Forcing streams with different sampling rates into a table at a single frame rate typically requires upsampling low-rate data and downsampling or aggregating high-rate data, which can discard original samples. This project builds on LeRobot’s dataset infrastructure to support independent sensor storage and time-based queries.
The humancoin solution: every sensor is stored independently as a time-indexed parquet file. At read time, a window parameter defines "how to aggregate data around the query timestamp" — no pre-resampling needed.
Writing — add_frame routes each key to its feature buffer; save_episode() flushes them to disk:
flowchart LR
AF["add_frame(key, value)"]
IX["MasterIndex record_frame(ts, task)"]
PF["ParquetFeature buffer.append(ts, data)"]
VF["VideoFeature save PNG + timestamp list"]
AuF["AudioFeature buffer chunks + timestamp list"]
MI[("master_index.parquet timestamp | task | frame_index")]
P1[("imu.parquet 1000Hz × 6d")]
P2[("eeg.parquet 256Hz × 8d")]
P3[("state.parquet 30Hz × 7d")]
C1[("head_rgb.parquet timestamps only")]
V1[("head_rgb.mp4 30fps video")]
A1[("mic.parquet timestamp | num_samples")]
W1[("mic.wav 16kHz audio")]
AF -->|"key = task"| IX
AF -->|"key = sensor<br/>imu / eeg / state"| PF
AF -->|"key = video"| VF
AF -->|"key = audio"| AuF
IX --> MI
PF --> P1
PF --> P2
PF --> P3
VF --> C1
VF --> V1
AuF --> A1
AuF --> W1
style AF fill:#fafafa,stroke:#9e9e9e,color:#000
style IX fill:#e1f5fe,stroke:#4fc3f7,color:#000
style PF fill:#fafafa,stroke:#9e9e9e,color:#000
style VF fill:#f3e5f5,stroke:#9c27b0,color:#000
style AuF fill:#e8f5e9,stroke:#66bb6a,color:#000
style MI fill:#e1f5fe,stroke:#4fc3f7,color:#000
style P1 fill:#fff3e0,stroke:#ff9800,color:#000
style P2 fill:#fff3e0,stroke:#ff9800,color:#000
style P3 fill:#fce4ec,stroke:#e91e63,color:#000
style C1 fill:#f3e5f5,stroke:#9c27b0,color:#000
style V1 fill:#f3e5f5,stroke:#9c27b0,color:#000
style A1 fill:#e8f5e9,stroke:#4caf50,color:#000
style W1 fill:#e8f5e9,stroke:#4caf50,color:#000
Loading
Reading — look up the master-clock timestamp in master_index, then query each feature by that timestamp:
flowchart TD
IDX["ds[10]"] -->|"① look up frame 10"| MI[("master_index.parquet timestamp | task | frame_index")]
MI -->|"② get timestamp"| TS["t = 0.333s"]
TS -->|"③ nearest<br/>closest row in time"| P3[("state.parquet 30Hz × 7d")]
TS -->|"③ window=(-0.033, 0] all rows in range"| P1[("imu.parquet 1000Hz × 6d")]
TS -->|"③ window=(-0.033, 0] all rows in range"| P2[("eeg.parquet 256Hz × 8d")]
TS -->|"③ nearest, closest row in time"| C1[("head_rgb.parquet timestamp")]
C1 -->|"④ get frame_index"| V1[("rgb → 480×640×3")]
P3 -->|"1 row"| RS["state → [7]"]
P1 -->|"33 rows"| RI["imu → [33, 6]"]
P2 -->|"8 rows"| RE["eeg → [8, 8]"]
V1 -->|"1 frame"| RV["head_rgb → [3, 480, 640]"]
style IDX fill:#fafafa,stroke:#9e9e9e,color:#000
style MI fill:#e1f5fe,stroke:#4fc3f7,color:#000
style TS fill:#fafafa,stroke:#9e9e9e,color:#000
style P3 fill:#fce4ec,stroke:#e91e63,color:#000
style P1 fill:#fff3e0,stroke:#ff9800,color:#000
style P2 fill:#e8f5e9,stroke:#4caf50,color:#000
style C1 fill:#f3e5f5,stroke:#9c27b0,color:#000
style V1 fill:#f3e5f5,stroke:#9c27b0,color:#000
style RS fill:#fce4ec,stroke:#e91e63,color:#000
style RI fill:#fff3e0,stroke:#ff9800,color:#000
style RE fill:#e8f5e9,stroke:#4caf50,color:#000
style RV fill:#f3e5f5,stroke:#9c27b0,color:#000
Loading
The result of ds[10] (t=0.333s):
state→ nearest match →[7]imu→ all 33 readings in the(-0.033, 0]window →[33, 6]eeg→ all 8 readings in the(-0.033, 0]window →[8, 8]head_rgb→ nearest row in timestamp parquet →frame_index→ decode that frame →[3, 480, 640]
2. Installation
Python 3.10–3.13 (verified with Python 3.12). LeRobot is currently pinned to 0.3.3.
Install from PyPI:
pip install humancoin
Or install from source:
git clone https://github.com/FlagOpen/HumanCOIN cd HumanCOIN pip install -e .
Dependencies:
| Dependency | Purpose |
|---|---|
lerobot |
base dataset framework (inherits LeRobotDataset) |
pyarrow |
parquet file I/O |
numpy / torch |
data processing |
av (PyAV) |
video encoding/decoding |
matplotlib |
visualization |
scipy |
web visualizer signal filtering |
Download HumanCOIN datasets
Browse dataset names on Hugging Face or ModelScope. These are organization pages; pass a dataset name from the selected hub to the command below.
pip install humancoin # Replace DATASET_NAME with an actual dataset repository name humancoin-download --hub huggingface --datasets DATASET_NAME --target-dir ./data humancoin-download --hub modelscope --datasets DATASET_NAME --target-dir ./data # Download only selected files (partial download; may not be loadable as a full dataset) humancoin-download --datasets DATASET_NAME --include "meta/*" README.md --target-dir ./data # Multiple datasets; preview without downloading humancoin-download --datasets DATASET_A DATASET_B --target-dir ./data --dry-run # Also available as: python -m humancoin.download ...
The default namespace is HumanCOIN; explicit owner/name IDs are also accepted. --ds_lists is an alias for --datasets. Files are stored under <target-dir>/HumanCOIN/<dataset-name>; the default target directory is ~/.cache/huggingface/lerobot. Repeating a download reuses files through the hub SDK. Use --revision to select a version and --max-workers to control parallel downloads. Private/gated datasets require access and HF_TOKEN or MODELSCOPE_TOKEN (or --token); ModelScope uses the lightweight modelscope-hub SDK; credentials are passed to the client or read from its login cache. MS_TOKEN is also accepted. A failed dataset does not stop the remaining downloads; the command exits nonzero if any fail.
from humancoin import HumanCOINDataset ds = HumanCOINDataset( repo_id="HumanCOIN/DATASET_NAME", root="data/HumanCOIN/DATASET_NAME", ) item = ds[0]
Downloads preserve the repository files; loading requires the multi-frequency dataset layout supported by this project. Downloading does not convert other formats.
3. Quick Start
import numpy as np from humancoin import HumanCOINDataset ds = HumanCOINDataset.create( repo_id="my_dataset", root="data/", fps=30, features={ # camera: dtype="video" auto-encodes to MP4 "observation.images.cam": { "dtype": "video", "shape": (480, 640, 3), "names": ["h", "w", "c"], "fps": 30, }, # high-rate IMU: window defines "return all readings in (t-33ms, t]" "observation.imu": { "dtype": "float32", "shape": (6,), "names": ["ax", "ay", "az", "gx", "gy", "gz"], "fps": 1000, "window": (-0.033, 0.0), "tolerance_s": 0.002, }, # mid-rate EEG "observation.eeg": { "dtype": "float32", "shape": (8,), "names": ["Fz", "Cz", "Pz", "Oz", "F3", "F4", "C3", "C4"], "fps": 256, "window": (-0.033, 0.0), }, # end-effector pose: no window → nearest neighbor by default "observation.state": { "dtype": "float32", "shape": (7,), "names": ["x", "y", "z", "roll", "pitch", "yaw", "gripper"], "fps": 30, }, "action": { "dtype": "float32", "shape": (7,), "names": ["dx", "dy", "dz", "droll", "dpitch", "dyaw", "dgrip"], "fps": 30, }, }, ) # recording: task = master clock, each frame is bounded by a task call for i in range(300): # high-rate sensors: multiple writes per frame for _ in range(33): # 1000Hz / 30fps ≈ 33 ds.add_frame("observation.imu", imu_sample) for _ in range(8): # 256Hz / 30fps ≈ 8 ds.add_frame("observation.eeg", eeg_sample) # a task call = frame boundary, timestamp auto-generated ds.add_frame("task", "reach_target") # camera-rate sensors: once per frame ds.add_frame("observation.state", joint_state) ds.add_frame("action", action_cmd) ds.add_frame("observation.images.cam", camera_image) ds.save_episode()
Reading — window features return all readings in the window, nearest features return a single vector:
ds = HumanCOINDataset(repo_id="my_dataset", root="data/") item = ds[10] item["observation.imu"] # tensor([33, 6]) — 33 readings in (-0.033, 0] item["observation.imu_timestamps"] # tensor([33]) item["observation.eeg"] # tensor([8, 8]) item["observation.state"] # tensor([7]) — nearest item["observation.images.cam"] # tensor([3, 480, 640]) — CHW item["task"] # "reach_target"
sequenceDiagram
participant User
participant DS as Dataset
participant PF as ParquetFeature<br/>(imu)
participant VF as VideoFeature<br/>(cam)
participant IX as MasterIndex
loop each frame
User->>DS: add_frame("imu", data)
DS->>DS: _auto_timestamp("imu")<br/>ts = counter × 1/1000
DS->>PF: add(data, ts)
Note over PF: buffer.append((ts, data))
User->>DS: add_frame("task", "reach")
DS->>DS: ts = len(index) / fps
DS->>IX: record_frame(ts, "reach")
User->>DS: add_frame("cam", image)
DS->>DS: _auto_timestamp("cam")<br/>ts = counter × 1/30
DS->>VF: add(image, ts)
Note over VF: save PNG + record timestamp
end
User->>DS: save_episode()
DS->>IX: write(ep_idx) → master_index.parquet
DS->>PF: save() → imu.parquet
DS->>VF: save() → timestamp parquet + MP4
DS->>DS: meta.save_episode()
DS->>DS: _check_episode_alignment()
Loading
4. Demo & Visualization
4.1 Running the Demo
# write 3 episodes of simulated data python -m examples.01write_simple # read and inspect the data python -m examples.01read_simple # interactive visualizer humancoin-visualize data/demo --episode 0 # web visualizer (browser UI, draggable time window) humancoin-visualize-web /path/to/dataset # audio: microphone stored as WAV + timestamp parquet python -m examples.06audio_example
The web visualizer supports optional zero-phase filtering of EEG/EMG strips: open the 滤波 panel on a strip and combine a high-pass / low-pass Butterworth with a comb notch at the line frequency and all its harmonics (EEG/EMG presets included). The API takes the same spec per feature: /api/frame?...&filt=observation.eeg:hp=0.5,lp=40,notch=50.
It also renders a 3D pose view in the SteamVR right-handed y-up world (*_pose 6-DoF features as coordinate triads + recent-history motion trails with time fading, fingertip end-effector paths from hand_skeleton FK; drag to rotate, wheel to zoom) and a color-coded electrode grid for static snapshot features such as eeg_impedance (quality-banded per channel). Pose rotation follows the recorder exactly (R = Rz(yaw)·Ry(pitch)·Rx(roll), the inverse of the recorder's ZYX euler decomposition of the OpenVR matrix); hand skeletons attach via a constant per-hand Manus→tracker mount map (+z = finger, +y = back of hand, plus a −5cm offset along −y since the tracker sits above the back of the hand).
4.2 Demo 1: Writing (01write_simple.py)
The main write demo: a set of simulated sensor streams — 3 cameras + IMU + EEG + EMG + temperature — recording three 2-second episodes (reach_target, pour_liquid, stack_blocks):
| Sensor | Shape | Rate | Window |
|---|---|---|---|
| head_rgb | 480×640×3 | 30 fps | — (video) |
| left_wrist_rgb | 480×640×3 | 30 fps | — (video) |
| right_wrist_rgb | 480×640×3 | 30 fps | — (video) |
| imu | 6 (accel+gyro) | 1000 Hz | (-0.033, 0.0) |
| eeg | 8 (EEG channels) | 256 Hz | (-0.033, 0.0) |
| emg_left | 4 (muscle channels) | 100 Hz | (-0.033, 0.0) |
| emg_right | 4 (muscle channels) | 100 Hz | (-0.033, 0.0) |
| temperature | 1 | 10 Hz | — (nearest) |
| state | 7 (x,y,z,r,p,y,grip) | 30 Hz | — (nearest) |
| action | 7 (velocity commands) | 30 Hz | — (nearest) |
What it shows:
dtype: "video"features auto-encode to MP4, exactly one frame per master frame- high-rate sensors are written multiple times between two task calls; timestamps are auto-generated per feature as
timestamp_start + counter × 1/fps - each
save_episode()writesmaster_index.parquet+ per-feature parquets + videos, then runs the alignment checker (every feature should report OK)
python -m examples.01write_simple
4.3 Demo 2: Reading (01read_simple.py)
Reads the dataset written by Demo 1 and exercises every read path end to end:
- window features (
observation.imu) return all readings in the window plus their timestamps - nearest features (
observation.temperature,observation.state,action) return a single vector - videos are decoded back to CHW tensors
window_overrideswiden the IMU window at read time without touching the stored data- prints the IMU segment sizes across frames — the first frames are empty because the IMU only comes online at
timestamp_start: 0.1
python -m examples.01read_simple # needs data/ from 01write_simple4.4 Demo 3: Alignment Tolerance (02tolerance_example.py)
Shows the alignment checker catching a sensor that violates the alignment contract: the IMU is configured with timestamp_start: 0.1 (it comes online 100 ms late) plus a tight tolerance_s: 0.002, so its samples are too far from the first master frames. After save_episode() the checker writes the offending frames to meta/alignment_check.jsonl and the demo prints the report.
Rule of thumb: tolerance_s must cover the distance from a master frame to the sensor's nearest sample — at most half its period (e.g. ≤ 50 ms for a 10 Hz sensor) — plus startup offset and jitter.
python -m examples.02tolerance_example
4.5 Demo 4: Window Modes (03window_example.py)
One episode, three aggregation modes side by side:
observation.imuat 1000 Hz withwindow: (-0.033, 0.0)— range query, ~33 readings per frameobservation.finger_poseat 120 Hz withwindow: "interpolate"— value interpolated at the frame timestampobservation.temperatureat 10 Hz with no window — nearest neighbor
Then the dataset is reopened with window_overrides — a wider IMU window, finger pose switched to a range query — proving that the window is a read-time decision, not a property of the stored data.
python -m examples.03window_example
4.6 Demo 5: Delta Timestamps (04delta_example.py)
Multi-frame temporal queries with delta_timestamps={"observation.state": [-0.1, 0.0, 0.1]}: reading ds[i] returns state stacked over three timepoints (t-0.1, t, t+0.1) instead of one vector — the building block for delta-pose training and temporal augmentation.
python -m examples.04delta_example
4.7 Demo 6: Asynchronous Sensor Startup (05async_startup_example.py)
Real sensors don't come online at the same instant: after the process starts, each one returns its first frame at its own wall-clock time (IMU at 0.04 s, EMG at 0.12 s, EEG amplifier at 0.31 s, …). This demo simulates such streams and aligns them to the master clock before adding data:
- t_all_started — the wall-clock moment every sensor has delivered its first frame (the max of all first-frame times).
- t = 0 — the master clock reference sensor's first frame at/after
t_all_started; its earlier frames are discarded. - timestamp_start — for every other sensor, the delta between its next frame (first frame at/after t = 0) and the t = 0 frame; set it in the feature spec so the stored timestamps stay phase-aligned with the master clock.
- Record everything from t = 0 onward. The episode ends when the master clock stops, but each sensor keeps streaming until its next frame after the master's last one — that trailing sample keeps the last frame's window full.
Simulated startup latencies:
| Sensor | Rate | First frame (wall clock) |
|---|---|---|
observation.imu |
1000 Hz | 0.04 s |
observation.emg |
100 Hz | 0.12 s |
observation.state (master clock reference) |
30 Hz | 0.18 s |
observation.temperature |
10 Hz | 0.22 s |
action |
30 Hz | 0.24 s |
observation.eeg |
256 Hz | 0.31 s |
The demo computes t_all_started = 0.31 s, takes the master's next frame as t = 0, derives each sensor's timestamp_start, writes the dataset, verifies on disk that every feature's first stored timestamp equals its timestamp_start and that its last sample lands past the master stop, then reads back frame 0, a mid-episode frame, and the last frame.
Two consequences worth knowing:
- Frame 0's backward window is empty. Every sensor's first stored frame sits at its own
timestamp_start> 0 and earlier data is discarded by design, so a window like(-0.033, 0.0)has nothing to return on the first frame(s). - Tolerance must cover half the sensor's period. The alignment checker compares each master frame against the sensor's nearest sample, whose distance is at most half a period (a 10 Hz sensor can be up to 50 ms away) plus jitter — e.g.
tolerance_s: 0.06for the 10 Hz temperature sensor here.
python -m examples.05async_startup_example
4.8 Demo 7: Audio (06audio_example.py)
One camera + state + a simulated microphone, showing the audio storage path end to end:
dtype: "audio"features defineshapeas one chunk of samples(n_samples,)(mono) or(n_samples, channels)and requiresample_rate- one
add_frameper master frame stores one chunk;save_episode()concatenates all chunks into one PCM16 WAV per episode underaudios/, plus a timestamp parquet index (timestamp | episode_index | num_samples) next to the other per-feature parquets - reading
ds[i]in nearest mode resolves the chunk nearest the master timestamp through the parquet index and returns it whole — the atomic unit, analogous to one video frame; with awindow(spec,window_overrides, or a directread(..., window=(start, end))argument) it returns every sample whose time lies in(t + start, t + end], sliced with sample precision across chunk boundaries and concatenated - the alignment checker reports WAV integrity: chunk count vs index rows, sample count vs summed index
python -m examples.06audio_example
4.9 Visualizer Controls
Visualizer (humancoin-visualize data/demo --episode 0):
space — play / pause
← → — step frame backward / forward
↑ ↓ — faster / slower playback
w s — widen / shrink the window
q / Esc — quit
5. Core Concepts
5.1 Per-Feature Independent Storage
Storage layout:
data/
└── chunk-000/
└── episode_000000/
├── master_index.parquet # timestamp | frame_index | task_index
├── observation.imu.parquet # 1000Hz · 1980 rows · 6 cols
├── observation.eeg.parquet # 256Hz · 512 rows · 8 cols
├── observation.state.parquet # 30Hz · 60 rows · 7 cols
└── observation.images.head_rgb.parquet # 30Hz · 60 rows · timestamps only
videos/
└── chunk-000/
└── observation.images.head_rgb/
└── episode_000000.mp4
audios/
└── chunk-000/
└── observation.audio.mic/
└── episode_000000.wav
Every non-video feature gets one parquet with a uniform column layout:
timestamp (float64) | episode_index (int64) | col_0 (float32) | ... | col_D (float32)
- camera-rate features (state, action): one row per frame, rows = frames
- high-rate features (imu, eeg): multiple samples between frames, rows = rate × duration
- camera features: timestamp parquet + MP4 video
- audio features: timestamp parquet (
timestamp | episode_index | num_samples, one row per added chunk) + one WAV per episode; chunks are concatenated in timestamp order
5.2 Task as the Master Clock
The whole dataset is organized around task calls as frame boundaries: add_frame("task", ...) declares "the current frame ends here, the next one begins".
- Frame = all feature data written between two task calls.
add_frame("task", ...)auto-generates the frame timestamp:t = frame_index / fps(override withtimestamp=if needed) - Master clock = the frame sequence. Reading
ds[i]first looks up the i-th frame timestamp t inmaster_index; every feature is then aligned around t (window / nearest neighbor, see 5.3) - Each feature's timestamps are generated independently of the master clock: one counter per feature,
ts = timestamp_start + counter × (1 / fps)on eachadd_frame; counters reset aftersave_episode() - No master_feature to configure — task is the master clock, and features of any sampling rate align to it
# standard call order per frame: high-rate first → task bounds the frame → low-rate last ds.add_frame("observation.imu", ...) # high-rate sensor, written multiple times between frames ds.add_frame("observation.eeg", ...) ds.add_frame("task", "pick_up") # ← frame boundary: frame_index++, master-clock timestamp ds.add_frame("observation.state", ...) # camera-rate sensor, once per frame ds.add_frame("observation.images.cam", image)
5.3 Window Parameter
The master-clock frame timestamp is the query anchor, but a feature's readings don't necessarily land on a frame — window defines "how to aggregate that feature's readings around the frame timestamp":
window value |
Behavior | Typical use |
|---|---|---|
None (default) |
nearest neighbor: return the single closest reading | camera-rate sensors (state, action); audio: one whole chunk |
"interpolate" |
linear interpolation at the query time | mid-rate sensors (finger pose) |
(start_s, end_s) |
range query: return all readings in (t+start, t+end] |
high-rate sensors (IMU, EEG, EMG); audio: sample-precise slice + concat |
# IMU at 1000Hz, 30fps master → ~33 readings per frame "window": (-0.033, 0.0) # (t-33ms, t] # wider window for context "window": (-0.066, 0.0) # (t-66ms, t], ~66 readings # override at read time ds = HumanCOINDataset( repo_id="my_dataset", root="data/", window_overrides={"observation.imu": (-0.100, 0.0)}, )
Readings returned per feature when querying t = 0.333s (frame 10):
Note that temperature (10Hz) has no sample at 0.333 — its sampling grid is 0.3, 0.4, …, so nearest neighbor falls back to the reading at t=0.300.
6. API Reference
Creating a Dataset
ds = HumanCOINDataset.create( repo_id="my_dataset", fps=30, features={...}, # see Feature Definition Format below )
| Parameter | Type | Default | Description |
|---|---|---|---|
repo_id |
str |
— | dataset name, directory name under root |
fps |
float |
— | master-clock frame rate; frame timestamp = frame_index / fps |
features |
dict[str, dict] |
— | feature definitions, format below |
root |
str | Path |
~/.cache/huggingface/lerobot/<repo_id> |
storage root directory |
robot_type |
str |
None |
robot type, metadata only |
use_videos |
bool |
True |
whether dtype="video" features are encoded to MP4 |
image_writer_processes |
int |
0 |
image writer subprocess count (0 = write in main process) |
image_writer_threads |
int |
0 |
image writer thread count |
video_backend |
str |
None |
video backend (e.g. "pyav"), None auto-selects |
batch_encoding_size |
int |
1 |
batch-encode videos every N episodes |
window_overrides |
dict |
None |
override each feature's window (same format as at read time) |
Feature Definition Format
features = { "feature_key": { "dtype": "video" | "audio" | "float32" | "int64", # required "shape": (H, W, C) | (N,) | (D,), # required "names": ["col_0", ...], # required # optional: "fps": 1000, # sampling rate of this feature "window": (-0.033, 0.0), # query window "timestamp_start": 0.1, # timestamp start offset "tolerance_s": 0.002, # alignment tolerance "sample_rate": 16000, # audio only: sample rate of the WAV } }
| Field | Type | Required | Description |
|---|---|---|---|
dtype |
str |
✓ | storage type: "float32" (continuous), "int64" (discrete/id), "video" (camera frames), "audio" (microphone chunks) |
shape |
tuple |
✓ | shape of a single reading: (D,) for numeric features, (H, W, C) for video, (n_samples,) or (n_samples, channels) for audio |
names |
list[str] |
✓ | column name per dimension (used as parquet column names) |
fps |
float |
optional | sampling rate of this feature; timestamps are auto-generated as timestamp_start + counter × 1/fps; falls back to master-clock fps if omitted |
window |
None | "interpolate" | (start, end) |
None |
how readings are aggregated at query time (see 5.3) |
timestamp_start |
float |
0.0 |
timestamp start offset, e.g. the sensor only starts recording at t=0.1s |
tolerance_s |
float |
None |
alignment tolerance: after save_episode(), per-frame nearest-reading gaps are checked and violations are logged to meta/alignment_check.jsonl |
sample_rate |
int |
audio only | sample rate of the WAV file (e.g. 16000); required for dtype="audio" |
Conventions and tips:
- feature keys follow LeRobot naming and must not contain
/(e.g.observation.imu,action) dtype="video"features only store a timestamp parquet; images are encoded as MP4 undervideos/dtype="audio"features store one WAV per episode underaudios/plus a timestamp parquet; eachadd_framepasses one chunk of float32 samples in [-1, 1] ((n_samples,)mono or(n_samples, channels)). Nearest read returns one whole chunk; awindow(start, end)read returns the samples in(t+start, t+end]concatenated (timing contract: a chunk's first sample sits at its label, samples advance at exactly1/sample_rate, real dropouts shorten the segment rather than stretch it)- always give high-rate features both
fpsandwindow— otherwise the nearest-neighbor query returns a single reading and the high-rate information is lost
Writing
ds.add_frame(key: str, value: np.ndarray | torch.Tensor | str, timestamp: float | None = None) ds.save_episode()
key:"task"or a defined feature keyvalue:np.ndarray/torch.Tensorfor numeric features (shape must matchshape);np.ndarray(H, W, C) for video features;strtask label for tasktimestamp:Noneauto-generates — task uses the master clocklen(records) / fps, other features use their own countertimestamp_start + counter × 1/fps; pass afloatto set it manually
A complete write loop:
FPS = 30 imu_per_frame = 1000 // FPS # 33 eeg_per_frame = 256 // FPS # 8 for frame in range(60): # 2 seconds / 30fps for _ in range(imu_per_frame): # high-rate: multiple writes per frame ds.add_frame("observation.imu", imu_sample) for _ in range(eeg_per_frame): ds.add_frame("observation.eeg", eeg_sample) ds.add_frame("task", "reach_target") # ← frame boundary ds.add_frame("observation.state", joint_state) # low-rate: once per frame ds.add_frame("action", action_cmd) ds.add_frame("observation.images.cam", camera_image) ds.save_episode() # flush all buffers, run alignment/video checks, reset counters
Reading
ds = HumanCOINDataset( repo_id="my_dataset", root="data/", episodes=[0, 2], # load only episodes 0, 2 image_transforms=my_transforms, # torchvision transforms delta_timestamps={"observation.state": [-0.1, 0, 0.1]}, # multi-frame query window_overrides={"observation.imu": (-0.066, 0)}, # override window video_backend="pyav", ) item = ds[10]
| Parameter | Type | Default | Description |
|---|---|---|---|
repo_id |
str |
— | dataset name |
root |
str | Path |
~/.cache/huggingface/lerobot/<repo_id> |
storage root directory |
episodes |
list[int] |
None |
load only the given episode indices |
image_transforms |
callable | None |
image transforms (torchvision style) |
delta_timestamps |
dict[str, list[float]] |
None |
multi-frame query: returns {key: master frame + each offset frame}, e.g. {"observation.state": [-0.1, 0, 0.1]} |
tolerance_s |
float |
1e-4 |
frame-alignment tolerance for delta_timestamps |
video_backend |
str |
None |
video decoding backend (e.g. "pyav") |
batch_encoding_size |
int |
1 |
(write side) video batch-encoding size |
window_overrides |
dict |
None |
override each feature's window at read time, e.g. {"observation.imu": (-0.066, 0)} |
Depth video reading: features with info["video.is_depth_map"] = true and HWC shape = [H, W, 1] are decoded directly with PyAV. Reads return native samples as a (1, H, W) tensor (e.g. uint16 for gray12le), without RGB conversion or division by 255. Units and invalid values remain those of the stored data. This applies to reading; depth-specific encoding is not implemented.
Return Structure
item = ds[10] # { # "timestamp": tensor(0.3333), # "frame_index": tensor(10), # "episode_index": tensor(0), # "index": tensor(10), # "task_index": tensor(0), # "task": "reach_target", # # # window feature: all readings + timestamps # "observation.imu": tensor([33, 6]), # "observation.imu_timestamps": tensor([33]), # # # nearest feature: a single vector # "observation.state": tensor([7]), # "observation.temperature": tensor([1]), # # # video feature: CHW format # "observation.images.head_rgb": tensor([3, 480, 640]), # # # audio feature: one chunk of samples, float32 in [-1, 1] # "observation.audio.mic": tensor([533]), # # # audio feature with window (-0.033, 0): samples in (t-33ms, t], concatenated # "observation.audio.mic": tensor([528]), # }
7. Architecture
flowchart TB
subgraph Entry["entry layer"]
DS["HumanCOINDataset<br/>dataset.py · orchestrator<br/>(inherits LeRobotDataset)"]
end
subgraph Core["core layer"]
IX["MasterIndex<br/>index.py"]
MD["HumanCOINDatasetMetadata<br/>metadata.py"]
end
subgraph Features["feature layer"]
PF["ParquetFeature<br/>parquet.py"]
VF["VideoFeature<br/>video.py"]
AuF["AudioFeature<br/>audio.py"]
end
subgraph Validate["validation layer"]
CK["DatasetChecker<br/>checker.py"]
end
subgraph Store["storage layer"]
PQ[("parquet files<br/>data/chunk-*/episode_*/")]
MP4[("MP4 videos<br/>videos/chunk-*/")]
WAV[("WAV audio<br/>audios/chunk-*/")]
META[("metadata<br/>meta/")]
end
DS -->|"frame index I/O"| IX
DS -->|"info.json / metadata"| MD
DS -->|"write readings / window query"| PF
DS -->|"write images / decode videos"| VF
DS -->|"write chunks / read WAV"| AuF
DS -->|"post-save validation"| CK
PF -->|"save / read"| PQ
VF -->|"encode MP4"| MP4
VF -->|"timestamp table"| PQ
AuF -->|"write WAV"| WAV
AuF -->|"timestamp table"| PQ
IX -->|"save"| PQ
MD -->|"save"| META
CK -->|"validation report"| META
style DS fill:#e8eaf6,stroke:#4fc3f7,stroke-width:2px,color:#000
style IX fill:#ffebee,stroke:#ef9a9a,color:#000
style MD fill:#fafafa,stroke:#9e9e9e,color:#000
style PF fill:#e8f5e9,stroke:#66bb6a,color:#000
style VF fill:#f3e5f5,stroke:#ce93d8,color:#000
style AuF fill:#e8f5e9,stroke:#66bb6a,color:#000
style CK fill:#fff3e0,stroke:#ffcc80,color:#000
style PQ fill:#fafafa,stroke:#9e9e9e,color:#000
style MP4 fill:#fafafa,stroke:#9e9e9e,color:#000
style WAV fill:#fafafa,stroke:#9e9e9e,color:#000
style META fill:#fafafa,stroke:#9e9e9e,color:#000
Loading
File layout:
src/humancoin/
├── dataset.py (341 lines) orchestrator — add_frame / save_episode / __getitem__
├── audio.py (185 lines) AudioFeature — buffer chunks, write WAV + timestamp parquet
├── parquet.py (168 lines) ParquetFeature — write buffer, save parquet, load, window query
├── video.py (168 lines) VideoFeature — save PNG, encode MP4, decode frames
├── index.py (127 lines) MasterIndex — frame index I/O
├── metadata.py (123 lines) HumanCOINDatasetMetadata — info.json management
├── checker.py (162 lines) DatasetChecker — video frame count + audio integrity + timestamp alignment checks
└── utils.py (236 lines) constants + info.json creation/validation
Design principles:
-
One object per feature:
ParquetFeature,VideoFeatureandAudioFeatureeach encapsulate their buffer, file paths, and read/write logic. The Dataset is only an orchestrator. -
Symmetric read/write: every Feature provides both
add/save(write) andload/query(read). -
Deterministic paths: file paths depend only on
episode_indexandfeature_key; no lookups inepisodes.jsonl. -
Inherit, don't rewrite: inherit
LeRobotDatasetforstart_image_writer,_save_imageand other basics; only 4 core methods are overridden.
8. Comparison with Native LeRobotDataset
| Aspect | Native LeRobotDataset | humancoin |
|---|---|---|
| Storage | 1 merged parquet | N per-feature parquets |
| Multi-rate sensors | pre-resampling required | native rates, independent storage |
| Time alignment | tolerance_s tolerance matching |
window mechanism (range / interpolate / nearest) |
| Master clock | camera frame | add_frame("task", ...) calls |
| Video encoding | unified batch encoding in save_episode |
independent per VideoFeature |
| Metadata | episodes.jsonl records everything |
only basic frame info; paths computed deterministically |
| per-feature fps | not supported | independent fps per feature |
| per-feature tolerance | global tolerance_s |
independent tolerance_s per feature |
| Audio storage | not supported | WAV per episode + timestamp parquet index, nearest-chunk read |
| Write API | add_frame(frame_dict, task) |
add_frame(key, value) per field |
| Alignment check | none | automatic alignment_check.jsonl |
| Video check | none | automatic frame count/duration check |
9. Acknowledgements
This project builds on the following open-source projects and work:
-
LeRobot — the base dataset framework. This project inherits the API design of
LeRobotDatasetandLeRobotDatasetMetadataand reuses its video encoding and image writing infrastructure. Thanks to the Hugging Face team for their contributions to the open-source robot learning ecosystem. -
PyArrow — high-performance columnar storage, the basis of per-feature parquet I/O.
-
PyAV — Python bindings for FFmpeg, powering video encoding (AV1) and decoding.
-
Matplotlib — visualization.
-
The Neuropixels data format and the BIDS standard from the Allen Institute for Neural Dynamics — inspiration for organizing multimodal neural data.
Thanks to all colleagues for feedback and discussions.
Building and checking a release
pip install -e ".[dev]" python -m unittest discover -s tests -v python -m build python -m twine check dist/*
The wheel includes dataset reading/writing, downloads, and both visualizers. Install humancoin to use all these features; no source checkout is required. The examples/ demonstration scripts require a source checkout.
来源:智源 FlagOpen:原创开源项目 · github.com
来源:RadarAI · aihot.news

