Chapter 7: Where Does RL Fit? — Reward, Residual Policies, and Fine-Tuning

Overview

Chapter 5 compared plain BC and ACT under one dataset, controller, and evaluation contract. Chapter 6 studies diffusion and flow policies when several action trajectories may be valid. The next question is not “How do we attach RL?” It is whether measurable value remains that is appropriately optimized through reward. If success, intervention, force events, latency, and cycle time already satisfy the task target, choosing not to run RL is a correct result.

Reinforcement learning (RL) optimizes long-horizon return through interaction. On a real robot, however, the optimizer is only one cost. The team must design rewards, reset the task, prepare demonstrations and replay, maintain controllers and logs, supervise trials, and expose robots and fixtures to repeated contact. This chapter does not make RL the final stage that removes classical control or imitation. It admits RL only in one bounded role: simulation skill learning, residual correction, gain or trajectory adaptation, offline refinement, or intervention-assisted online refinement.

Evidence boundary: The Capuano tutorial's list of unsafe exploration, sample inefficiency, manual resets, brittle rewards, and simulator fidelity is used as a teaching map for real-world constraints ^[21]. Task-specific results and numbers return to the original SERL and HIL-SERL sources ^{[18] [19]}. Residual RL, BCQ, CQL, and IQL have canonical identities in the packet, but their exact primary-body locators remain insufficient. This chapter therefore does not claim universal mechanisms or performance advantages for them ^{[11] [12] [13] [15]}.

After reading this chapter... - You can pre-register a BC-sufficient exit and an incremental-value gate for RL. - You can distinguish simulation, residual, offline, imitation-fine-tuning, and bounded online roles. - You can account for reward, resets, interaction, human supervision, and hardware exposure in one budget. - You can separate simulator throughput from contact fidelity and policy success from hardware safety. - You can keep reset and intervention authority outside the learned policy. - You can promote or reject a checkpoint through replay/offline, simulation, shadow, and bounded-hardware gates.

The experiment question is: If an approved ACT checkpoint makes slightly early contact under one fixture-tolerance slice, does reward-based refinement add enough value by reducing intervention or cycle time, or are dataset correction, contact-controller tuning, or system identification cheaper and safer?

1. Write the incremental-value contract before choosing an optimizer

A finite-horizon RL objective can be written as expected discounted reward over trajectories induced by policy \pi:

$$

J(\pi)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{T-1}\gamma^t r(s_t,a_t,s_{t+1})\right].

$$

A manufacturing cell is not promoted on J alone. Measure task completion, interventions, force and collision events, deadline misses, cycle time, reset labor, human minutes, and hardware cycles separately. Putting a penalty into the reward does not create an independent safety gate. A release evaluator should be owned outside the policy and learned critic.

Use a decision ledger such as

$$

\Delta V=

w_s\Delta S-w_i\Delta I-w_t\Delta T

-w_h C_{human}-w_r C_{reset}-w_x C_{exposure}-w_m C_{maint}.

$$

\Delta S is matched task improvement, \Delta I is change in interventions or unsafe events, and \Delta T is cycle-time change. The remaining terms account for skilled human time, reset labor, hardware exposure, and reward/simulator maintenance. Task, safety, and learning owners sign weights and the minimum acceptable \Delta V before results are visible. This is not a standardized economic measure. It is a way to expose hidden project costs.

1.1 A BC-sufficient exit is not failure

Issue a NO_RL verdict if the approved BC, ACT, or generative baseline satisfies pre-registered KPIs; if the remaining defect is explained by labels, calibration, action semantics, stale chunks, or controller gains; if a reward is likely to optimize a proxy; if reset and human coverage cannot support a comparison; or if expected benefit is smaller than exposure and maintenance cost.

S12 documents that exit and adds RL only after a pre-registered incremental-value gate. This is an engineering decision for this book, not a universal theorem derived from ACT, BAKU, or HIL-SERL ^{[16] [17] [19]}.

1.2 The real-world cost ledger

Robotics RL literature has long made physical exploration, samples, models, and safety part of the problem rather than external details ^[2]. Modern systems still assemble reward, demonstration replay, reset, controller, and data infrastructure around the optimizer ^{[18] [19]}. Real-world RL cost therefore includes reward design, resets, prior data, controller infrastructure, human supervision, interactions, and hardware exposure. SERL and HIL-SERL remain protocol-specific and do not establish universal transfer.

2. Five bounded roles for RL

PPO and SAC are canonical candidates in on-policy and off-policy deep RL, but this chapter packet does not carry enough exact result locators to transfer their performance ^{[3] [5]}. Do not begin with an optimizer name. First decide what learning may change and what remains fixed.

2.1 Simulation skill learning

Simulation can reduce real-robot exposure, but simulator success and real contact accuracy are separate claims. MuJoCo and Isaac Gym are important simulator lineages for robot learning ^{[1] [14]}. Their partial verification here does not justify importing throughput or fidelity numbers. Select a simulator by task role, contact and sensor models, controller integration, and deterministic replay.

Domain and dynamics randomization are candidates for placing visual and physical variation into the training distribution ^{[4] [6]}. Dactyl and SimOpt are major research lineages around randomization or simulation-parameter adaptation ^{[9] [10]}. A wide parameter range alone is not evidence that the real system lies inside it.

Continue the Chapter 3 and #S11 discipline: identify first, randomize second. Classify friction, actuator delay, gripper compliance, camera latency, mass, and contact stiffness by observability and task sensitivity. Do not hide an unidentifiable parameter behind a broad uniform distribution. A randomization manifest distinguishes nominal values, measured uncertainty, synthetic extension, distribution and correlations, and held-out stress ranges.

2.2 Residual and parameter adaptation

When a nominal planner, trajectory, and impedance controller solve most of a structured task, a policy need not regenerate the whole action. An S12 residual abstraction is

$$

a_t^{proposal}=a_t^{nominal}+\alpha(s_t)\,\delta a_\theta(s_t),

\qquad

a_t^{sent}=\Pi_{\mathcal{C}(s_t)}(a_t^{proposal}).

$$

a^{nominal} is the validated output, \delta a the learned correction, \alpha a phase-dependent bound, and \Pi_{\mathcal C} projection into joint, workspace, rate, collision, and force constraints. Setting the residual to zero must restore nominal behavior. Log projection delta and saturation in both training and evaluation.

Residual Reinforcement Learning for Robot Control is the canonical candidate for this branch ^[11]. The packet lacks an exact primary mechanism locator, so this chapter does not assert that retaining a conventional controller and learning a correction guarantees performance or safety. Projection, rollback, and an independent watchdog are S12 deployment rules; a software clamp is not a certified safety function.

Gain adaptation follows the same rule. A policy does not emit unbounded stiffness, damping, force targets, or trajectory offsets. Engineering owners define the admissible set, rate of change, phase transition, and passivity/stability test. Contact instability must immediately restore the nominal controller without waiting for model-weight rollback.

3. Offline RL is not automatically safe because data are fixed

Offline RL seeks a value or policy from a fixed dataset without new environment interaction. Demonstrations, failures, interventions, and resets may become useful inputs, but the value of an action outside dataset support is not directly observed. A wrong reward label or narrow action support can pull a learned policy toward this empty region.

BCQ, CQL, and IQL are distinct offline-RL candidate lineages ^{[12] [13] [15]}. Their exact primary-body locators remain incomplete in the packet, so no universal mechanism or advantage over BC is asserted here. The S12 operational boundary is to withhold promotion from proposals outside audited fixed-data support and reward labels. There is no assumed guarantee of outperforming a strong BC baseline.

3.1 Support audit

Slice observations and actions by task phase, initial-state bin, intervention state, object, scene, calibration epoch, and controller mode. Do not reduce support to a single Euclidean distance. Use separate physically meaningful distances for normalized joints, Cartesian deltas, gripper states, and contact phases.

Produce phase-specific action ranges and density summaries; nearest-neighbor replay; a reward-label confusion matrix; proposal projection and rejection histograms; matched rollouts for BC, behavior policy, and offline candidate; and coverage of rare failure precursors and interventions.

Demonstration-augmented policy gradients and large-scale off-policy grasping are historical candidates connecting imitation priors and RL interaction ^{[7] [8]}. Their unmatched apparatus results are not merged. They serve as a map reminding us that prior data are not a free input outside the optimizer budget.

3.2 Separate reward from the evaluator

An insertion reward might be decomposed as

$$

r_t=w_p\Delta p_t+w_d\Delta d_t+w_c\mathbf{1}[success]

-w_f\phi(F_t)-w_j\psi(jerk_t)-w_o\mathbf{1}[outside].

$$

This expression is not the true objective. A policy may increase pose progress by pressing a peg diagonally into the fixture, avoid a force penalty by stopping just before contact, or exploit an image cue in the success classifier. Log reward components, physical evaluator, and safety events separately. A changed reward weight creates a new experiment version; it must not overwrite an earlier run.

For a learned reward classifier, include false-positive challenge sets, camera shifts, occlusion, partial completion, and reset frames in addition to normal splits. Classifier confidence cannot be the sole owner of safety or task completion. Human labels also require a label policy, reviewer agreement, and drift tracking.

4. Bounded online and intervention RL are operating protocols

Online real-robot RL is not a required step in this book. It becomes a candidate only if imitation is insufficient, a simulation or offline branch leaves an incremental hypothesis, and reset and human authority are validated inside a narrow envelope.

SERL combines demonstration replay, off-policy learning, reward, reset strategy, and a robot controller in a task-specific system ^[18]. Its original protocol records policies trained in 25–50 minutes on three manipulation tasks. That duration belongs to the stated tasks, rewards, resets, controller, and prior data and is not a forecast for this insertion cell.

HIL-SERL combines demonstrations, human corrections, off-policy RL, and reset/control infrastructure ^[19]. Its 1–2.5 hours of training and improvement over its own imitation baselines are task- and baseline-dependent and include skilled human intervention. Do not reduce them to algorithm time while excluding human attention and hardware exposure.

4.1 Reset ownership

A reset is an environment operation, not an appendix to policy reward. Name who replaces objects, aligns fixtures, returns the robot to a safe pose, and clears a protective stop. A learned reset policy needs its own checkpoint, action envelope, evaluator, and rollback. Manual adjustments change the next initial-state distribution and must be logged.

Record reset frequency, duration, success, manual contact, replaced parts, and discarded episodes. A short training wall clock does not imply sample efficiency if a person continuously replaces objects and clears jams.

4.2 Human authority

A person may be dataset source, reward labeler, supervisor, intervention actor, and safety operator. Separate these roles in the log even when one person performs them. Reuse the Chapter 4–5 authority timeline for intervention request, controller acceptance, policy-queue purge, human recovery, and handback approval.

Using joystick contact as a reward signal differs from exercising safety authority. A reward learner may model takeover likelihood but cannot own the physical takeover or E-stop path. TRANSIC is a candidate using online correction for sim-to-real transfer ^[20], but this chapter does not claim that correction data cover unseen hazards.

4.3 External safety boundary

Keep hardware joint, speed, torque, and force limits; collision and workspace monitoring; watchdog, heartbeat, deadline, and stale rejection; E-stop and protective-stop authority; checkpoint approval and rollback; human takeover and incident logging; and controller/contact inner loops outside the learned policy.

Passing constraint projection is not application-safety certification. Safety owners separately maintain the applicable standard, vendor manual, and cell risk assessment. A policy that learns to avoid a reward penalty does not become the owner of the E-stop.

5. Worked incremental-value experiment card

The following is an illustrative card completed before collection, not an observed result. Local owners must replace thresholds after baseline and risk review.

Hypothesis

The approved ACT checkpoint meets free-space and grasp KPIs. A particular fixture-tolerance bin causes pre-contact pose residual and intervention. The hypothesis is that a bounded residual or offline refinement can reduce intervention without increasing force events, deadline misses, or reset labor.

Matched baseline

Freeze dataset, perception encoder, action semantics, controller, projection, and evaluator. Repeat the baseline across the same initial-state bins, block-randomize seed and trial order, and do not mix experiment blocks across calibration drift or maintenance.

Pre-registered admission threshold

• primary benefit: minimum effect and uncertainty bound for unassisted completion or intervention reduction;
• non-inferiority: no increase in collision, force, or protective-stop class;
• timing: deadline misses and stale rejection inside approved limits;
• operation: human supervision and reset minutes inside budget;
• exposure: maximum interactions, contact cycles, temperature, and wear stop;
• exit: retain BC/ACT and archive the RL branch if the threshold is missed.

Staged run

1. Find reward/evaluator disagreement in recorded replay.
2. Sweep residual bounds and offline support in simulation.
3. Apply held-out latency, observation, mass, friction, and contact stress.
4. In shadow mode, record RL proposals beside approved ACT sent actions.
5. Promote only checkpoints passing projection, deadline, and OOD gates to a reduced-envelope trial card.
6. After a bounded block, a blinded evaluator computes results and raw trials remain immutable.

Attribution ablation

Use the same evaluator for unchanged ACT; ACT retrained with more corrections; contact-controller or trajectory retuning; a residual candidate with \delta a=0; an offline refinement candidate; and a bounded online/HIL candidate only if approved. Otherwise, a benefit available from dataset or controller repair may be misattributed to RL.

6. Failure symptoms and diagnosis order

Diagnose in the order data/action semantics → evaluator/reward → clocks/controller → reset distribution → simulator/support → optimizer. Changing optimizer first can absorb an upstream defect into another hyperparameter.

7. Promotion from replay to bounded hardware

Stage A — replay and offline

Recompute reward on stored transitions and compare it with the physical evaluator. Mask episode boundaries, terminals, interventions, resets, and rejected actions correctly. Compare BC/ACT and the offline candidate on identical splits and attach the support audit.

Stage B — simulation regression

Separate nominal distribution from held-out stress. Manifest object mass, friction, latency, camera noise, contact parameters, and initial state as well as random seeds. Never describe simulation-only improvement as real-robot evidence.

Stage C — shadow

The approved policy moves the robot while the RL candidate only proposes. Put proposal, projection, sent baseline action, reward, evaluator, deadline, and authority on one timeline. Systematic limit violations or stale proposals stop promotion.

Stage D — bounded hardware

A human-reviewed card fixes workspace, speed, force, task phase, maximum episodes/interactions, reset owner, stop owner, abort threshold, and rollback checkpoint. Do not tune rewards during a block. Any change creates a new version and review.

Promotion checklist

• [ ] The baseline failure and minimum incremental value are pre-registered.
• [ ] A NO_RL/BC-sufficient exit and rollback checkpoint exist.
• [ ] Reward components and independent evaluator are separated.
• [ ] Reset owner, distribution, duration, and manual adjustment are recorded.
• [ ] Human supervision, intervention, and labeling time are in the budget.
• [ ] Offline support and proposal projection/rejection histograms are reviewed.
• [ ] Simulation nominal/stress and real evidence remain separate.
• [ ] Shadow passes deadline, stale-action, and authority-handoff tests.
• [ ] E-stop, watchdog, and safety owners remain outside policy for bounded trials.
• [ ] Success, intervention, force/collision, latency, reset, and exposure are reported together.

8. A bounded Codex implementation prompt

Goal
- Evaluate whether selective RL adds preregistered incremental value over the
  approved BC/ACT policy for one tabletop-insertion failure slice.

Context
- Read the versioned task card, dataset/split manifest, baseline checkpoint,
  controller contract, reward spec, independent evaluator, reset state machine,
  simulation manifest, and signed promotion thresholds.

Constraints
- Start with replay/offline analysis, then simulation and shadow only.
- Keep dataset, encoder, action semantics, controller, projection, evaluator,
  and test initial-state bins matched to the approved baseline.
- Implement a NO_RL result as a valid outcome.
- Log reward components, evaluator outcome, proposal/projected/sent actions,
  observation/action age, deadline, reset actor/time, human minutes,
  intervention, force/collision/stop events, and hardware-exposure counters.
- Treat residual, offline, and online branches as separate configs and reports.
- Do not let the learned policy own collision/force limits, watchdog, E-stop,
  protective-stop recovery, reset approval, or controller inner loops.

Done when
- The report reproduces the baseline and names the exact failure slice.
- Reward-versus-evaluator challenge tests and offline-support audit pass.
- Simulation includes nominal and held-out latency/contact/randomization stress.
- Shadow logs show deterministic stale/reorder rejection and rollback.
- The report accounts for optimizer, reset, human, interaction, maintenance,
  and exposure costs and recommends PROMOTE, REVISE, or NO_RL.
- No hardware command is executed.

Safety
- Stop at a human-reviewed bounded-hardware trial card. The card must specify
  reduced envelope, max interactions, independent stop authority, abort rules,
  reset ownership, incident logging, and the approved rollback checkpoint.

The output is not merely a log showing an RL repository ran. It is an evidence package comparing incremental value with incremental risk.

9. Open questions and evidence gaps

Matched BC-versus-RL studies rarely account completely for reward engineering, reset labor, human attention, and hardware wear. Training durations from different tasks and robots should not be placed in one ranking.

Measuring offline support under high-dimensional observations and contact phases remains open. One nearest-neighbor distance cannot represent semantic novelty.

Evidence that simulator parameters include the real system differs from evidence that a policy is robust across them. Separate identification uncertainty, synthetic extension, and held-out stress.

Interventions create recovery data but depend on skilled-human availability and reaction time. Human authority must remain distinguishable from a reward label.

Comparing RL with controller tuning should count baseline engineering effort too. It is also unfair to itemize every learning cost while assigning zero cost to classical maintenance.

Relation to Prior Surveys

#S11 provided the entry point for simulators, first motion, calibration, identification, and a small policy exercise. This chapter does not copy its prose, numbers, or claims. It carries forward identification-before-randomization, separation of simulation-only and real evidence, and bounded hardware authority, applying them to the new Chapter 4–6 dataset and policy artifacts. Public cross-links wait until #S11 is release-ready.

What to Learn Next

Even an approved RL branch does not output unrestricted motor commands. Chapter 8 builds a hybrid architecture in which a task/FSM selects skills, a learned policy proposes bounded actions, and planning, IK, constraint projection, contact control, watchdogs, and human fallback execute or reject them. This chapter hands forward the reward version, reset state machine, human authority, residual bound, offline-support audit, checkpoint rollback, timing, and safety verdict.

Detailed comparisons of VLA, world-model, or agentic planners that might propose rewards or select skills belong to #S13. A future model may propose a reward or long-horizon plan, but the evaluator, projection, watchdog, E-stop, and controller authority remain independent.

References

1. Todorov, E., Erez, T., & Tassa, Y. (2012). MuJoCo: A Physics Engine for Model-Based Control. IROS. DOI: 10.1109/IROS.2012.6386109.
2. Kober, J., Bagnell, J. A., & Peters, J. (2013). Reinforcement Learning in Robotics: A Survey. International Journal of Robotics Research. DOI: 10.1177/0278364913495721.
3. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
4. Tobin, J., et al. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS. DOI: 10.1109/IROS.2017.8202133.
5. Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML.
6. Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018). Dynamics Randomization for Sim-to-Real Transfer of Reinforcement Learning Policies. ICRA. DOI: 10.1109/ICRA.2018.8460528.
7. Rajeswaran, A., et al. (2018). Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. RSS. arXiv:1709.10087.
8. Kalashnikov, D., et al. (2018). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv:1806.10293.
9. OpenAI, et al. (2019). Learning Dexterous In-Hand Manipulation. International Journal of Robotics Research. DOI: 10.1177/0278364919887447.
10. Chebotar, Y., et al. (2019). SimOpt: Learning Robotic Skills in Simulation with Sim-to-Real Transfer. ICRA. DOI: 10.1109/ICRA.2019.8793789.
11. Johannink, T., et al. (2019). Residual Reinforcement Learning for Robot Control. IEEE ICRA. DOI: 10.1109/ICRA.2019.8794127. arXiv:1812.03201.
12. Fujimoto, S., Meger, D., & Precup, D. (2019). Off-Policy Deep Reinforcement Learning without Exploration. ICML. arXiv:1812.02900.
13. Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS. arXiv:2006.04779.
14. Makoviychuk, V., et al. (2021). Isaac Gym: High Performance GPU-Based Physics Simulation for Robot Learning. arXiv:2108.10470.
15. Kostrikov, I., Nair, A., & Levine, S. (2022). Offline Reinforcement Learning with Implicit Q-Learning. ICLR. arXiv:2110.06169.
16. Zhao, T. Z., Kumar, V., Levine, S., & Finn, C. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS. arXiv:2304.13705.
17. Haldar, S., Peng, Z., & Pinto, L. (2024). BAKU: An Efficient Transformer for Multi-Task Policy Learning. arXiv:2406.07539.
18. Luo, J., et al. (2024a). SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning. ICRA. arXiv:2401.16013.
19. Luo, J., Xu, C., Wu, J., & Levine, S. (2024b). Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. arXiv:2410.21845.
20. Jiang, Y., et al. (2024). TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction. CoRL.
21. Capuano, F., Pascal, C., Zouitine, A., Wolf, T., & Aractingi, M. (2025). Robot Learning: A Tutorial. arXiv:2510.12403v1. DOI: 10.48550/arXiv.2510.12403.
22. Pieter Abbeel & Andrew Y. Ng (2004). Apprenticeship Learning via Inverse Reinforcement Learning. Annotated primary reading. canonical URL. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Reward, resets, interactions, exploration bounds, dataset support, and hardware exposure are source-specific.
23. Nathan Koenig & Andrew Howard (2004). Gazebo: A Dynamic Multi-Robot Simulator. Annotated primary reading. DOI: 10.1109/IROS.2004.1389727. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Historical Gazebo architecture and physics backends differ from current Gazebo releases; fidelity is model dependent.
24. Chelsea Finn et al. (2016). Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization. Annotated primary reading. canonical URL. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Reward, resets, interactions, exploration bounds, dataset support, and hardware exposure are source-specific.
25. Jonathan Ho & Stefano Ermon (2016). Generative Adversarial Imitation Learning. Annotated primary reading. arXiv:1606.03476. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Dataset coverage, action semantics, covariate shift, horizon, and recovery policy limit transfer.
26. Sergey Levine et al. (2016). End-to-End Training of Deep Visuomotor Policies. Annotated primary reading. arXiv:1504.00702. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Dataset coverage, action semantics, covariate shift, horizon, and recovery policy limit transfer.
27. Scott Fujimoto et al. (2018). Addressing Function Approximation Error in Actor-Critic Methods. Annotated primary reading. arXiv:1802.09477. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Reward, resets, interactions, exploration bounds, dataset support, and hardware exposure are source-specific.
28. Xue Bin Peng et al. (2018). DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. Annotated primary reading. DOI: 10.1145/3197517.3201311. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Reward, resets, interactions, exploration bounds, dataset support, and hardware exposure are source-specific.
29. Ilge Akkaya et al. (2019). Solving Rubik’s Cube with a Robot Hand. Annotated primary reading. arXiv:1910.07113. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Large infrastructure, frequent hardware interventions, external vision, and task-specific reset/recovery make it unsuitable as a first exercise.
30. Fabio Ramos et al. (2019). BayesSim: Adaptive Domain Randomization Via Probabilistic Inference for Robotics Simulators. Annotated primary reading. DOI: 10.15607/RSS.2019.XV.029. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Posterior quality depends on summary statistics, simulator family, real trajectories, and prior support; structural mismatch remains.
31. Saran Tunyasuvunakool et al. (2020). dm_control: Software and Tasks for Continuous Control. Annotated primary reading. DOI: 10.1016/j.simpa.2020.100022. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Simulation throughput or benchmark success does not establish real contact fidelity or hardware safety.
32. Tianhe Yu et al. (2020). Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. Annotated primary reading. canonical URL. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Reward, resets, interactions, exploration bounds, dataset support, and hardware exposure are source-specific.
33. Denis Yarats et al. (2022). Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning. Annotated primary reading. arXiv:2107.09645. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Reward, resets, interactions, exploration bounds, dataset support, and hardware exposure are source-specific.
34. Mayank Mittal et al. (2023). Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments. Annotated primary reading. DOI: 10.1109/LRA.2023.3270034. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Platform, task, dataset, and evaluation assumptions must be checked in the primary paper before transferring a result.
35. Eric Heiden et al. (2025). Neural Robot Dynamics in Newton. Annotated primary reading. canonical URL. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Neural residuals can overfit the excitation distribution
36. RSS authors (2025). Physics-Aware Real2Sim2Real for Non-Prehensile Manipulation. Annotated primary reading. canonical URL. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Object-centric setup does not identify robot actuator dynamics
37. Kevin Zakka et al. (2025). MuJoCo Playground: GPU-Accelerated Robot Learning and Sim-to-Real. Annotated primary reading. DOI: 10.48550/arxiv.2502.08844. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Repository warns TF32 on Ampere can affect reproducibility
38. NVIDIA Isaac Sim Team (2026). NVIDIA Isaac Sim: Enabling Scalable, GPU-Accelerated Simulation for Robotics. Annotated primary reading. arXiv:2606.03551. — Role: primary reading for accounting of RL, simulation, resets, and sim-to-real cost. Limitation: Very recent preprint