Chapter 4: Produce the Raw Material for Learning — Teleoperation and Dataset Engineering

Overview

The contact controller in Chapter 3 accepted bounded pose, velocity, and force references and stabilized the interaction between robot and environment. We now turn the operator’s production of those references, the scene time actually observed by each camera, and the command that survived safety projection into a learnable record. A good dataset is not a folder of successful videos. It is a versioned execution record that can explain an action, reproduce a failure, and audit the inputs and outputs of a new policy.

Merely increasing the episode count can mix calibrations, delayed cameras, operator habits, and states immediately preceding interventions. When a policy trained on that mixture fails, it becomes difficult to separate a model problem from an action-mapping error or a drifting clock. The first goal of this chapter is therefore not more demonstrations, but demonstrations with consistent, inspectable meaning.

Evidence-status note: The Parquet/video/metadata separation in LeRobotDataset v3 and the processor structure reflect official documentation inspected on 2026-07-15 ^{[18] [19]}. The combined contract for calibration epochs, raw/mapped/sent actions, clock uncertainty, protective stops, reset reasons, causal splits, and replay verdicts is an engineering synthesis for this book. This chapter recommends no port, device ID, command, control rate, or safety limit for a particular robot. Because main documentation can change, an implementation must pin the release or commit that it actually tests.

After reading this chapter... - You can separate the interfaces and authorities of the operator, teleoperator, mapper, safety supervisor, robot controller, and cameras. - You can distinguish raw, mapped, and sent actions and round-trip-test their units, signs, frames, and saturation. - You can record event time, arrival time, clock uncertainty, and calibration epochs. - You can preserve interventions, watchdog events, protective stops, resets, and failures as both learning labels and safety events. - You can release a dataset only after causal object/scene/operator/calibration splits and replay QA pass.

The experiment question is concrete: When one operator watches wrist and scene cameras while demonstrating a peg-like insertion with a 6/7-DoF arm and parallel gripper, what episode record and release gates prevent a later ACT policy from confusing what the person intended with what the robot actually received?

1. Treat teleoperation as a closed interface, not a command

Teleoperation is not a one-line copy from a leader arm or SpaceMouse to a joint command. The person observes camera and force cues, the mapper adapts the input to the robot embodiment, feasibility and safety layers constrain it, and the controller consumes the result before a deadline. Network transport, display delay, camera exposure, retargeting, and the controller all change the loop dynamics. This is why classical teleoperation work treats delay and stability as design problems in their own right ^{[4] [5]}.

In this chapter the operator is the source of intent, the teleoperator is a measurement device, the mapper is a representation transformer, the safety supervisor is the limiter of motion authority, and the robot controller is the executor. The recorder replaces none of them. Its job is to preserve the values and decisions crossing each boundary on a common episode timeline.

operator intent
  ↓
teleoperator raw sample ── device clock, clutch, mode
  ↓
mapping / retargeting ─── frame, scale, unit, sign, filter
  ↓
constraint projection ─── joint/workspace/rate/collision/force gate
  ↓
sent action ───────────── owner, deadline, sequence, validity
  ↓
vendor or real-time controller → robot → cameras / proprioception / force
  ↑                                        │
  └──────────── display and feedback ──────┘

independent safety supervisor:
  watchdog · protective stop · E-stop · authority transition

recorder:
  every boundary value + event time + arrival time + provenance

1.1 Build an owner–rate–deadline–failure matrix first

Writing fps: 30 on one stream does not complete its timing contract. A nominal 30 Hz camera still has distinct exposure-midpoint, driver-delivery, and decode-completion times. A teleoperator may sample at 100 Hz while a sender downsamples to 50 Hz. A vendor controller may run a faster inner loop on its own clock. Before designing storage, fill the following table with measured values from the cell.

Record target and measured rates together. A mean hides bursts and a long tail. At minimum, compute the median inter-arrival time, a high percentile, maximum gap, and counts of drops, duplicates, and reordered samples for every episode. These are dataset-health measures, not policy-performance claims.

1.2 A capture interface is an embodiment choice

For the single-arm, parallel-gripper beginner runbook, an already commissioned leader–follower device or bounded Cartesian input is the default candidate. UMI explores portable in-the-wild collection and embodiment transfer ^[7]. AnyTeleop addresses vision-based arm–hand retargeting ^[6]. DexCap and FastUMI occupy other points in portable capture design ^{[10] [11]}. Bunny-VisionPro, DOGlove, and DexUMI illustrate bimanual, dexterous, haptic, or exoskeleton branches ^{[12] [15] [16]}.

Those names are not a performance ranking. The table below is a scope decision for this book. The source packet does not yet contain exact primary-body locators supporting a general comparison across those devices. Read the non-interchangeability of portable, dexterous, bimanual, and haptic systems as an interface rule—different action dimensions and feedback channels require different schemas—not as a verified claim that one platform is generally better.

Require four artifacts instead of accepting that a device “feels natural.” First, an input-to-robot correspondence table. Second, a state machine for clutch and mode transitions. Third, a round-trip test covering values below, at, and beyond saturation. Fourth, a clock, thermal, and network drift log from a sustained collection interval. Without these artifacts, operator skill can conceal mapping defects.

2. Separate raw, mapped, and sent actions

Storing one field called action removes the most important causal variables. The raw value produced by the operator, the value transformed by the embodiment mapper, and the value sent after safety and controller processing can differ. Workspace projection, velocity clamps, collision rejection, clutching, and smoothing make “what the person attempted” and “what the robot executed” different targets.

2.1 The meanings of the three actions

• action.raw: the original device-coordinate reading, with axes, buttons, clutch, tracking confidence, and device sequence.
• action.mapped: the proposal transformed into robot joints or a Cartesian frame, with unit, sign, reference frame, scale, filter, and mapper version.
• action.sent: the command delivered to the controller after projection and rate limiting, with controller mode, validity, projection delta, and rejection reason.
• state.measured: robot feedback that must not be assumed simultaneous, including position, velocity, effort, gripper, status, and event time.

Current LeRobot processor documentation separates transformations among robot hardware values, model-compatible transitions, and robot actions ^[19]. The documented separation is verified, but unit, sign, saturation, and round-trip tests are validation requirements added by S12. The existence of a custom processor does not prove semantic correctness.

Test four properties independently for every processor.

1. Units: check degree/radian, meter/millimeter, and normalized/physical ranges with golden vectors.
2. Sign and ordering: apply one small positive input at a time and verify that only the expected joint or Cartesian direction changes.
3. Saturation: test inputs below, at, and beyond each limit and verify mapped-versus-sent differences and rejection codes.
4. Round trip: measure encode–decode or forward–inverse mapping error inside the admissible range and label irreversible regions.

No physical movement is required for the first stage. Run it on recorded samples and a simulator or controller stub. Hardware confirmation is a separate gate with reduced range and speed, a clear workspace, a spotter, and an independent stop path.

2.2 Fix meaning before tensor layout

A reusable episode schema must preserve observation and action semantics, timestamps, calibration identity, resets, failures, interventions, and replay status alongside frame tensors. This is a synthesis built on inspected elements from UMI, DROID, and official LeRobot documentation; no single source establishes the entire schema as a standard ^{[7] [8] [19] [20]}.

The following is a minimum logical schema. Storage formats and column names may change with the pinned tool version, but their semantics and provenance should not disappear.

Do not represent a task only with a natural-language sentence. Link object family and instance, fixture version, initial-state bin, target tolerance, allowed contact, and termination rule to a versioned task card. Natural language is a human-readable label; a separate evaluator computes physical success.

2.3 Do not claim that modalities were measured “at the same time”

Two sensors connected to one host do not necessarily observe the same instant. Camera exposure, USB buffering, compression, decoding, and middleware delivery add distinct delays. Force and joint-state sensors may use other clock domains. A shared row index must not imply simultaneity. Give every measurement an event time and an arrival time.

For a sample s_i selected from stream i at alignment time t, report age and uncertainty rather than only nearest-neighbor membership:

$$

\Delta t_i = t - t^{event}_i, \qquad

u_i = u^{clock}_i + u^{exposure}_i + u^{transport}_i

$$

The value u_i need not be a complete probabilistic model. It can be a measured quantity or conservative bound with provenance. The point is to avoid hiding uncertainty behind one aligned=true flag. When constructing a policy-input window, mark a stream exceeding maximum age or uncertainty with a padding/missing mask or quarantine the episode.

3. Manage calibration as epochs, not files

Camera extrinsics, leader–follower zeros, gripper range, joint offsets, and tool transforms are not constants to compute once and forget. Tightening a camera mount, changing a tool, re-homing the robot, or re-zeroing the leader changes the physical meaning of the same pixel and action value. An episode should reference a calibration epoch, not merely a file path.

An epoch record contains the calibrated entity, algorithm and version, input-sample hash, resulting parameters, covariance or residual summary, validation poses, operator, time, and the first and last affected episodes. Work on uncertainty-aware hand–eye calibration and multi-camera workcells motivates treating calibration uncertainty and joint geometry as separate engineering concerns ^{[13] [7]}. This chapter does not claim that either algorithm is generally more accurate. The method must be selected from its source conditions and local validation.

3.1 Drift gate

Run a short drift probe at the beginning and end of every collection block. For example, measure fiducial reprojection from wrist and scene cameras, tool-center offset, and leader–follower correspondence at several reachable validation poses. If the opening probe passes and the closing probe fails, do not automatically delete the entire block. Use episode time and the drift trend to quarantine an affected range for review.

Use four verdicts:

• pass: clocks, calibration, schema, and replay all satisfy release thresholds;
• conditional: a modality is missing, but the episode remains eligible for an experiment that does not use it;
• quarantine: exclude it from training until the cause or realignment is resolved;
• reject: semantic reconstruction or safety provenance is impossible.

Never overwrite the original dataset version. If v1.1 applies a calibration correction, record the parent, transform, affected episode list, and before/after validation. Re-encoding images or recomputing mapped actions must retain raw sources and lineage.

4. Preserve interventions, failures, and resets

A collection of successes exposes a policy only to states near successful trajectories. Yet admitting every failed action as a target can teach the wrong behavior. The answer is not a binary choice between deletion and mixture. Label authority and intent, then define separate admission rules.

4.1 An intervention may preserve the episode while changing authority

Current LeRobot HIL documentation describes a workflow in which autonomous, human-recovery, and correction segments can alternate inside one episode ^[20]. DAgger formalizes sequential imitation in which expert labels are collected at learner-visited states ^[2]. Intervention segments must therefore remain distinguishable from autonomous policy segments so evaluation and retraining do not silently mix authorities.

A minimum state machine is:

HUMAN_DEMO
POLICY_AUTONOMOUS
INTERVENTION_REQUESTED
HUMAN_RECOVERY
HUMAN_CORRECTION
RETURN_PENDING
SAFE_HOLD
PROTECTIVE_STOP
EPISODE_TERMINATED
RESET_IN_PROGRESS

For each transition record from, to, requester, approver, reason, event time, and sent-action sequence. Do not define an intervention merely because a person touched a joystick. Record when command authority actually changed and when the controller accepted the new owner’s command. Audit how many policy actions were still sent between INTERVENTION_REQUESTED and HUMAN_RECOVERY.

4.2 Watchdog, protective stop, and E-stop are different events

One failure=true flag erases the diagnosis order.

• A watchdog event means a heartbeat or fresh command missed its deadline. Network transport, a stalled process, a clock issue, or a queue can be responsible.
• A constraint rejection means a proposal violated a joint, workspace, rate, collision, or force gate. Separate policy failure from mapping failure.
• A protective stop means the robot safety system executed its defined protective behavior. Preserve vendor status and recovery requirements.
• An E-stop means a person or independent circuit exercised emergency authority. Follow the safety-incident process before assigning a learning label.
• A task failure means the task criterion was not met without necessarily causing a safety fault. Use a taxonomy such as grasp miss, insertion jam, or timeout.

Reset behavior is data too. The next initial-state distribution depends on whether a person replaced the object, aligned the fixture, homed the robot, or cleared a protective stop. Record reset_policy_version, reset_actor, initial_state_bin, reset_duration, and manual_adjustment. Comparing only episode counts in a cell without automated reset can distort both collection cost and the sampled distribution.

4.3 Worked episode trace

This tabletop-insertion trace reads an episode as events and interfaces rather than frame tensors. Times are illustrative relative values, not hardware defaults.

Do not count this as a policy-only success, but do not necessarily discard it. Evaluation separates unassisted from intervention-assisted success. The training pipeline assigns different masks and objectives to sent-action targets, human-correction targets, and rejected policy proposals. In Chapter 5, that authority mask becomes a starting point for analyzing covariate shift in BC and ACT.

5. Use LeRobotDataset as a learning contract, not just storage

The user-supplied “Robot Learning: A Tutorial” provides a useful teaching map for LeRobotDataset’s tabular/video/metadata separation, relative-time windows, padding, and batching ^[17]. It is nevertheless a 2025 v1 tutorial, not the sole authority for methods or APIs. Runnable calls and formats return to current official documentation, and the tested release or commit must be pinned.

Official LeRobotDataset v3 documentation inspected on 2026-07-15 describes low-dimensional signals and timestamps in Parquet, visual streams in per-camera MP4 shards, and schemas, FPS, statistics, tasks, and episode offsets in metadata ^[18]. Episode views can be reconstructed from metadata rather than file boundaries. A healthy “episode folder” therefore cannot serve as the integrity assumption.

5.1 Map the S12 logical schema into LeRobot

Fields not guaranteed by the current upstream schema can live in namespaced features or a sidecar manifest. Use names that do not collide with upstream fields and test preservation through the loader. A custom column should survive create → save → reload → sample window → batch → export → reload with value, dtype, shape, timestamp, and mask intact.

5.2 Windows and padding can change semantics

Relative timestamps are convenient when a policy requests past observations and a future action chunk around time t. Yet a request beyond an episode boundary requires padding. If the mask is lost, padding can look like the robot remained at the same pose before the episode began. Repeating a camera frame can confuse a dropped frame with a stationary scene.

Batch QA must check more than shape.

• Compare the actual event-time span with the requested window.
• Keep padding, missing, repeated, and interpolated samples as distinct masks.
• Separate observation interpolation from action-hold rules.
• Fit normalization statistics on the train split only.
• Display physical min/max beside normalized outliers to reveal unit defects.
• Test video-decode and tabular-frame drift on long episodes.

6. A split is a generalization question, not a frame ratio

A random frame split can place neighboring images from one episode, the fine-grained habits of one operator, one object texture, or one calibration epoch on both sides of train and test. The test metric then does not answer an independent generalization question. The Chapter 4 claim ledger, however, does not yet contain an exact primary section or table locator for this whole causal-leakage claim. The following is therefore an S12 evaluation-design rule, not a general performance conclusion drawn from RoboMimic, DROID, or UMI ^{[3] [8] [7]}. Do not attach a numerical performance penalty until a fact checker records the relevant primary locators.

Write the question first: a new object instance, a new pose of a known object, new lighting, a new operator, a new calibration epoch, or a new robot? Each requires a different split unit.

A release may contain multiple official split manifests, named by question: split_in_domain, split_new_object, or split_new_operator. Freeze episode UUID lists and the query hash so a training job cannot silently recreate a frame-level random split.

Scale does not automatically preserve local meaning. DROID foregrounds heterogeneous, large-scale real-world manipulation data ^[8], while RH20T addresses synchronized multimodal data across robots, viewpoints, and skills ^[9]. These corpora are useful design references, but they do not imply matching units, calibrations, resets, or force gates in our cell. When merging external data, distinguish native, converted, mapped, and unknown provenance; never silently zero-fill an unconvertible field.

7. Replay QA decides admission to training

A schema-valid episode is not automatically a valid demonstration. Replay QA consumes observations, actions, and events in order through three non-actuating modes: offline replay of the record, shadow replay in a kinematic model or simulator, and interface replay against a stub.

7.1 Three replay stages

1. Offline replay: plot image, state, action, authority, and safety events on one timeline. Detect drops, reorder, clock jumps, NaNs, shape changes, and impossible velocity.
2. Kinematic/shadow replay: apply sent actions to the robot model and collision scene. Test limits, discontinuities, and mode transitions, without claiming to reproduce contact outcomes.
3. Interface replay: send command packets only to a stub or read-only interface. Test serialization, units, sequence, deadline, and stale-message rejection.

Physical replay on the robot is not the default dataset-QA step. If uniquely necessary, it is a bounded validation with its own risk assessment and reduced-energy protocol. This chapter and its Codex prompt do not offer recorded-command retransmission as a default.

7.2 Failure symptoms and diagnosis order

Diagnose in the order schema → time → calibration → mapping → authority → safety → task labels → model. Starting with hyperparameters can disguise upstream corruption as an optimization issue.

7.3 Release manifest and metric checklist

Preserve these artifacts together for every release candidate:

• semantic version, immutable episode UUID list, and parent lineage;
• task card and observation/action dictionary with units, frames, shapes, and ranges;
• robot, teleoperator, camera, and hand–eye calibration-epoch manifest;
• code commit, dependency lock, tested LeRobot release/commit, and recorder hash;
• stream rate, age, jitter, drop, duplicate, reorder, and clock-uncertainty summaries;
• raw–mapped–sent projection and saturation statistics;
• authority duration, intervention count and latency, watchdog, protective-stop, and E-stop counts;
• task, unassisted, and intervention-assisted success with termination/failure taxonomy;
• reset actor, duration, reason, and initial-state distribution;
• split manifests, overlap audit, and train-only normalization statistics;
• offline/shadow/interface replay reports with pass/conditional/quarantine/reject verdicts.

Episode count and total hours are only two entries. Measure coverage over the cross product of objects, poses, scenes, operators, calibration epochs, failures, and recoveries. Empty cells, not an aggregate claim such as “ten hours collected,” should drive the next collection plan.

8. Minimal reproducible workflow and promotion gates

This workflow is an artifact sequence, not a command recipe.

Gate A — interface contract

Without moving the robot, complete the owner–rate–deadline–failure matrix, action dictionary, clock-domain map, and calibration registry. Pass golden raw samples through processors and compare expected mapped and sent values. Pass when unit, sign, ordering, frame, saturation, and round-trip tests are stored in a versioned log.

Gate B — dry capture

Run cameras and recorder without actuator authority. Inject clutch, mode, intervention, and simulated watchdog events, then replay the timeline. Pass when event/arrival times, sequences, drops/reorder, and authority transitions can be reconstructed from the episode.

Gate C — bounded teleoperation

In the validated #S11 cell, collect a small set under reduced workspace and speed, with a clear fixture, spotter, and independent stop path. Contact remains inside the force/impedance envelope from Chapter 3. Pass when sent commands, measured states, and safety events connect continuously and the closing drift probe stays within threshold.

Gate D — replay and split

Run offline, shadow, and interface replay, then create causal split manifests. Preserve failures and interventions under quarantine/admission rules. Pass when the overlap audit is empty, statistics are train-only, and every exclusion has a replay verdict and reason.

Gate E — dataset release candidate

Create an immutable version and lineage. Round-trip one episode window and one batch through save, reload, and export. Pass when values, dtypes, shapes, timestamps, masks, tasks, and boundaries survive, and an independent reviewer can trace one sample from the manifest.

9. A bounded Codex prompt

This prompt requests a schema and validator. Sending robot commands and moving hardware are out of scope.

Goal
Build a versioned episode-schema package and offline QA CLI for one tabletop
insertion dataset. Preserve raw, mapped, and sent actions; event and arrival
times; calibration epochs; authority transitions; safety events; resets; and
replay verdicts.

Context
- Input: five captured dry-run episodes plus the task card and calibration manifest.
- Target storage: the pinned LeRobotDataset release/commit recorded in the lockfile.
- Robot motion and command transmission are explicitly out of scope.

Constraints
- Do not invent missing values or silently zero-fill unknown modalities.
- Keep raw records immutable and write derived data to a new dataset version.
- Namespace S12-specific fields and preserve them through save/load/export.
- Treat clock uncertainty, padding, missing, repeated, and interpolated samples
  as distinct fields or masks.
- Generate split manifests only at episode/object/scene/operator/calibration units.
- Never connect to a robot, vendor controller, or teleoperator device.

Done when
- JSON/schema validation covers units, frames, shapes, ranges, enums, and lineage.
- Golden tests verify unit, sign, ordering, saturation, and processor round trips.
- The CLI reports drops, reorder, stale age, clock jumps, projection deltas,
  authority transitions, safety events, and reset/failure distributions.
- Save → reload → temporal window → batch → export → reload retains tested fields.
- An HTML/Markdown replay report and immutable split manifests are produced.
- Tests fail on one intentionally corrupted episode of each error class.

Safety
- Use fixtures, recorded data, and stubs only.
- Refuse any code path that calls send_action, enables actuators, changes safety
  limits, clears a protective stop, or performs hardware calibration.
- Print the pinned dependency and dataset versions in every report.

Do not send the result directly into training. A reviewer first examines the schema diff, test fixtures, corruption detection, and lineage. Examples in current official LeRobot documentation are API-reading aids, not hardware authorization or safe defaults for the local cell ^[21].

10. Evidence tiers and open questions

This chapter keeps different kinds of evidence separate.

• Original papers establish the structure of learning from demonstration, DAgger’s learner-visited-state correction, and specific designs such as UMI and DROID ^{[1] [2] [7] [8]}. They do not guarantee the same outcome on another robot.
• Current official documentation establishes current LeRobotDataset v3 storage and processor/HIL interfaces ^{[18] [19] [20]}. Mutable main pages require another check before release.
• The tutorial supplies a pedagogical data-to-policy map ^[17]. It does not substitute for validating summarized ACT, diffusion, RL, or API details.
• S12 engineering synthesis covers calibration epochs, raw/mapped/sent lineage, clock uncertainty, causal splits, and replay admission. It is a locally testable design, not an international standard or benchmark conclusion.

Open questions remain. How should clock uncertainty affect a training loss or attention mask? Which human-recovery segments are expert targets, and which should train an auxiliary “exit a risky state” objective? When is operator style valuable diversity rather than noise to normalize away? Can visual-only episodes remain a conditional subset when force or tactile streams are slower or missing? How should a public release exclude privacy-sensitive video while an internal audit trail remains complete?

RoboMimic brings a systematic benchmark view to demonstration data, observations, and algorithm choices ^[3]. Yet direct evidence for leakage and replay QA in an ordinary single-arm cell remains thinner than the literature on unusual capture systems. The strict provenance in this chapter is therefore not a solved universal recipe. It is a defensive layer that prevents the next model experiment from mistaking a data defect for model performance.

Manufacturing Cell Checkpoint

Before training, owners of the tabletop assembly cell answer these questions.

The collector is not the sole approver. The robot/controller owner signs action and safety fields; the perception/calibration owner signs cameras and epochs; the ML owner signs windows, masks, and splits; and the cell-safety owner signs authority and stop events. Concentrating those responsibilities allows collection speed to hide provenance loss.

What to Learn Next

Chapter 5 builds the simplest learned baseline—behavior cloning—and extends it to ACT with action chunking and temporal aggregation. Before choosing an observation horizon or chunk size, freeze this chapter’s event times, padding masks, authority labels, and sent-action targets. Otherwise, it will be unclear whether ACT learned a long temporal structure or merely averaged misalignment and mixed interventions.

The contact envelope from Chapter 3 remains in force. A learned policy emits a mapped proposal; the sent action reaches the robot only below constraints and the contact controller. The scale and cross-embodiment adaptation of VLA and generalist datasets belong to #S13. S12 completes an observation/action contract that a future upper-level policy can read and a safety boundary that can reject it.

References

1. Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009). A Framework for Learning from Demonstration. Robotics and Autonomous Systems. DOI: 10.1016/j.robot.2008.10.024.
2. Ross, S., Gordon, G., & Bagnell, J. A. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS.
3. Mandlekar, A., et al. (2021). RoboMimic: A Benchmark for Robot Learning from Demonstration. CoRL. arXiv:2108.03298.
4. Niemeyer, G., & Slotine, J.-J. E. (1991). Stable Adaptive Teleoperation. IEEE Journal of Oceanic Engineering. DOI: 10.1109/48.64895.
5. Lawrence, D. A. (1993). Stability and Transparency in Bilateral Teleoperation. IEEE Transactions on Robotics and Automation. DOI: 10.1109/70.258054.
6. Qin, Y., et al. (2023). AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System. Robotics: Science and Systems. DOI: 10.15607/RSS.2023.XIX.015.
7. Ha, H., et al. (2024). Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. RSS. arXiv:2402.10329.
8. Khazatsky, A., et al. (2024). DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset. RSS. arXiv:2403.12945.
9. Shen, L., et al. (2024). RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot. ICRA. DOI: 10.1109/ICRA57147.2024.10611615.
10. Wang, C., et al. (2024). DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation. RSS. arXiv:2403.07788.
11. Wu, Z., et al. (2024). FastUMI: A Scalable and Hardware-Independent Universal Manipulation Interface with Dataset. arXiv:2409.19499.
12. Ding, R., et al. (2024). Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning. arXiv:2407.03162.
13. Ulrich, M., & Hillemann, M. (2024). Uncertainty-Aware Hand–Eye Calibration. IEEE Transactions on Robotics. DOI: 10.1109/TRO.2023.3330609.
14. Allegro, D., Terreran, M., & Ghidoni, S. (2024). Multi-Camera Hand-Eye Calibration for Human-Robot Collaboration in Industrial Robotic Workcells. arXiv:2406.11392.
15. Fang, H.-C., et al. (2025). DOGlove: Dexterous Manipulation with a Low-Cost Open-Source Haptic Force Feedback Glove. RSS.
16. Xu, M., et al. (2025). DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation. CoRL, PMLR 305. DOI: 10.48550/arXiv.2505.21864.
17. Capuano, F., Pascal, C., Zouitine, A., Wolf, T., & Aractingi, M. (2025). Robot Learning: A Tutorial. arXiv:2510.12403v1. DOI: 10.48550/arXiv.2510.12403.
18. Hugging Face (2026a). LeRobotDataset v3.0. Official LeRobot documentation, accessed 2026-07-15.
19. Hugging Face (2026b). Processors for Robots and Teleoperators. Official LeRobot documentation, accessed 2026-07-15.
20. Hugging Face (2026c). Human-In-the-Loop Data Collection. Official LeRobot documentation, accessed 2026-07-15.
21. Hugging Face (2026d). Getting Started with Real-World Robots. Official LeRobot documentation, accessed 2026-07-15.
22. David J. Montana (1988). The Kinematics of Contact and Grasp. Annotated primary reading. canonical URL. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Singularity, joint limits, collision, scaling, and discrete-time behavior still require explicit handling.
23. Roger Y. Tsai & Reimar K. Lenz (1989). Position and Orientation Calibration of a Robot Camera as a Pan-Tilt Unit. Annotated primary reading. DOI: 10.1109/70.34770. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Decoupled estimation propagates rotation error into translation and is sensitive to degenerate motions and robot kinematic error.
24. Bruno Siciliano & Jean-Jacques Slotine (1991). A General Framework for Managing Multiple Tasks in Highly Redundant Robotic Systems. Annotated primary reading. DOI: 10.1109/icar.1991.240390. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Singularity, joint limits, collision, scaling, and discrete-time behavior still require explicit handling.
25. Frank C. Park & Bryan J. Martin (1994). Robot Sensor Calibration: Solving AX=XB on the Euclidean Group. Annotated primary reading. DOI: 10.1109/70.326576. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Still assumes consistent robot poses and a rigid camera mount; timing offsets and target-detection bias are separate errors.
26. Konstantinos Daniilidis (1999). Hand-Eye Calibration Using Dual Quaternions. Annotated primary reading. DOI: 10.1177/02783649922066213. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Algebraic least squares still needs nonlinear refinement and uncertainty checks under realistic noise.
27. Zhengyou Zhang (2000). A Flexible New Technique for Camera Calibration. Annotated primary reading. DOI: 10.1109/34.888718. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Low reprojection error can coexist with poor coverage, target nonplanarity, rolling shutter, depth bias, or extrapolation outside the calibrated volume.
28. Sudeep Dasari et al. (2019). RoboNet: Large-Scale Multi-Robot Learning. Annotated primary reading. canonical URL. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Dataset coverage, action semantics, covariate shift, horizon, and recovery policy limit transfer.
29. Ankur Handa et al. (2020). DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System. Annotated primary reading. arXiv:1910.03135. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Delay, sampling, saturation, filtering, environment stiffness, and hardware interfaces bound the reported behavior.
30. Catie Cuan et al. (2024). Leveraging Haptic Feedback to Improve Data Quality and Quantity for Deep Imitation Learning Models. Annotated primary reading. arXiv:2211.03020. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Platform, task, dataset, and evaluation assumptions must be checked in the primary paper before transferring a result.
31. Carolina Higuera et al. (2024). Sparsh: Self-Supervised Touch Representations for Vision-Based Tactile Sensing. Annotated primary reading. arXiv:2410.24090. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Platform, task, dataset, and evaluation assumptions must be checked in the primary paper before transferring a result.
32. Liu et al. (2024). Unified Calibration of Hand–Eye, Kinematic Parameters, and TCP. Annotated primary reading. DOI: 10.1108/ria-06-2023-0076. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Requires laser-tracker-grade measurements
33. Zilin Si et al. (2024). DiffTactile: A Physics-based Differentiable Tactile Simulator for Contact-Rich Robotic Manipulation. Annotated primary reading. arXiv:2403.08716. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Platform, task, dataset, and evaluation assumptions must be checked in the primary paper before transferring a result.
34. Various (2024). TacEx: GelSight Tactile Simulation in Isaac Sim. Annotated primary reading. arXiv:2411.04776. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Platform, task, dataset, and evaluation assumptions must be checked in the primary paper before transferring a result.
35. Yunlong Dong et al. (2025). GEX: Democratizing Dexterity with Fully-Actuated Dexterous Hand and Exoskeleton Glove. Annotated primary reading. arXiv:2506.04982. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Three fingers do not cover five-finger tasks
36. S. Satsevich et al. (2025). Prometheus: Universal, Open-Source Mocap-Based Teleoperation System with Force Feedback for Dataset Collection in Robot Learning. Annotated primary reading. arXiv:2510.01023. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Platform, task, dataset, and evaluation assumptions must be checked in the primary paper before transferring a result.
37. Ruoshi Wen et al. (2025). Dexterous Teleoperation of 20-DoF ByteDexter Hand via Human Motion Retargeting. Annotated primary reading. arXiv:2507.03227. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Report does not establish generalization outside its hand and task set
38. Xitong Chen et al. (2026). DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation. Annotated primary reading. arXiv:2603.17851. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Platform, task, dataset, and evaluation assumptions must be checked in the primary paper before transferring a result.
39. Dongmyoung Lee et al. (2026). DexTwist: Functional Twist Retargeting for Mixed-Reality Dexterous Teleoperation. Annotated primary reading. arXiv:2605.12182. — Role: primary reading for teleoperation, dataset, and calibration provenance. Limitation: Designed around twist tasks rather than general dexterity
40. Zhang et al. (2024). Multi-Camera Hand–Eye Calibration for Industrial Robotic Workcells. Annotated primary reading. arXiv:2406.11392. — Role: primary evidence for how multiple cameras expand observability and calibrated workcell coverage. Limitation: added cameras also increase synchronization, target-visibility, and extrinsic-drift validation burdens.