Skip to content

RL environment

This downstream extension takes a validated asset package and produces an Isaac Lab RL task with complete evidence lineage. It runs only after stage 7 simready-verification and the target runtime's isaac-load check.

rl environment loop

Variant distributions

A policy learns from its observations and the effects of its actions. Geometry, texture variation, materials, contact behaviour and articulation shape that signal. An asset can look correct in a catalogue render while remaining unsuitable for grasping or pushing, or leaving joint-limit behaviour unspecified. The upstream stages make these properties explicit, measured and reviewable.

Existing systems generate simulation tasks, assets and rewards from language and image models (Y. Wang et al., 2024; Katara, Xian and Fragkiadaki, 2024)1 2. Others build procedural houses (Deitke et al., 2022)3 or reconstruct articulated objects from interaction and single images (Jiang, Hsu and Zhu, 2022; Chen et al., 2024)4 5, supported by part-level datasets and affordance models (Xiang et al., 2020; Mo et al., 2021)6 7. The cited systems do not provide a record of each asset's known properties, uncertainty and approvals. Asset Factory Blueprint carries that record into environment design.

The base asset supplies a prior over valid scenes rather than a single training target, following the same premise as ConTaRo. In the digital-cousins study, policies trained on affordance-preserving scene variants achieved 90% zero-shot success on the same manipulation task, compared with 25% for policies trained on the exact digital twin (Dai et al., 2024)8. This lane defines the variant model, admissible changes, evidence-backed bounds and the training evidence required for promotion.

Training contract

Training over asset variants is modelled as a contextual Markov decision process: one task with a family of transition functions indexed by mass, friction, drive stiffness, texture, placement and other context values (Hallak, Di Castro and Mannor, 2015)9. When the policy cannot observe that context directly, the problem becomes an epistemic POMDP. Most generalisation failures can then be understood as partial observability of the context (Ghosh et al., 2021; Kirk et al., 2023)10 11.

The asset factory records the prior over that context:

  • Every physical property proposal carries a value, unit, range, distribution, method and confidence.
  • Every authored mass carries a sealed uncertainty record.
  • Every grasp point carries a frame, an approach vector, a gripper width and a confidence.

The lane uses these fields as context priors. It records their source manifest identifiers and checksums in the environment manifest, then derives the training distribution from the recorded uncertainty.

Upstream records

Upstream record Fields consumed Use in the environment
simready-asset-manifest package identity, layer stack, units, axis policy, Profile, promotion status scene composition root and lineage
physics-articulation-manifest affordances.grasp_points, affordances.affordance_labels reset distributions, grasp-conditioned rewards, reachability gate
physics-articulation-manifest joints, drives, limits, articulation_roots action space for articulated tasks, limit terminations
physics-articulation-manifest mass_properties, physics_materials, sealed uncertainty record dynamics randomisation ranges with evidence provenance
physics-articulation-manifest tuning_scenarios, validation_scenarios smoke tests and scripted-oracle scenarios
material-inference-manifest physical_property_proposals (value, unit, range, distribution, confidence) context prior for vEp, friction &c.
nonvisual-material-manifest thermal, acoustic and electrical values with evidence nonvisual observation channels
isaac-runtime-evidence physics_dt, settle_steps, repeatability_tolerance_metres, runtime_identity.physics_backend timestep and backend binding, reset feasibility, rollout repeatability
asset-layout-manifest, mutation-plan, variants.usda placements, parametric patterns, gated operations with rollback notes layout randomisation and the editing curriculum
task-fitness-protocol scope: articulated_training tests, metrics and tolerances emitted by this lane, evaluated by the evaluation stage

How to use this lane

Agent skill: rl-environment-design-lead. The orchestrator invokes it only after the asset package has passed the stage 7 gates, including the isaac-load gate where the runtime is configured.

Declare the target behaviour: picking, placing, pushing, opening, closing, inspection or navigation. The skill checks that behaviour against the available asset evidence. Missing grasp points block a pick task; missing joint limits or drives block an opening task; and a missing friction range blocks a push task. The skill does not approximate missing records. Grasp tasks reuse the grasp affordances recorded in the physics-articulation manifest rather than re-deriving them from geometry.

The environment manifest records:

  • the runtime identity the environment is bound to
  • what the robot observes, and which of those observations are privileged
  • which actions it can take and from which recorded joints and bodies
  • what earns reward and which probes the reward passed
  • what ends an episode and how episodes reset
  • which variants are allowed during training and where each bound came from
  • how the environment is evaluated and which validation run proves it starts cleanly

Environment identity

Isaac Lab is the canonical RL environment framework for this lane. It succeeds Isaac Gym and ORBIT (Makoviychuk et al., 2021; Mittal et al., 2023)12 13 and supports Newton alongside PhysX, Warp and MuJoCo (Mittal et al., 2025; NVIDIA Corporation, 2026b)14 15. The same asset can produce different contact behaviour and failure modes under different solvers (Hu et al., 2026)16. The manifest therefore treats the physics backend as part of environment identity: a PhysX pass does not validate a Newton environment.

The timestep is part of the same identity. A manager-based Isaac Lab environment is defined by sim.dt, decimation and episode_length_s (NVIDIA Corporation, 2026a)17. Isaac runtime evidence records the physics_dt used to check the asset's contact behaviour. The training timestep must match it unless fresh runtime evidence is attached. Seeds, environment count and Isaac Lab version complete the identity record.

Observations and privilege

Observation terms are split into two groups. The policy group contains proprioception and the sensor channels available at deployment, each with its noise model. The critic group may also contain ground-truth state that the deployed policy never sees. Asymmetric actor-critic training (Pinto et al., 2018)18, learning by cheating (Chen et al., 2020)19 and teacher-student distillation (Lee et al., 2020; Kumar et al., 2021)20 21 use this privileged information, usually through simulator internals.

Every privileged input in this blueprint must come from a validated manifest field with evidence and a checksum. Each critic term cites the manifest path from which it is read.

The manifest also declares an adaptation_mode:

  • robust: the policy averages over the context prior and carries no explicit estimate of it.
  • adaptive: the policy infers the context online from its own observations, in the manner of rapid motor adaptation (Kumar et al., 2021)21.

Adaptive mode requires an identifiability audit. For each randomised parameter, the manifest records which policy observations can distinguish its values within one episode. Lifting can identify mass, pushing can identify friction, and joint motion can identify drive damping. Proprioception cannot identify texture variation. A randomised parameter that the policy cannot identify from its observations is handled robustly regardless of the declared mode, and the audit records that decision.

Sensor contract

For every channel, the sensor contract records the sensor model, noise model and parameters, update rate relative to decimation, and supporting upstream evidence. Isaac Lab's observation corruption and noise configuration is the rendering target for visual and proprioceptive channels (NVIDIA Corporation, 2026a)17.

Stage 6 grounds thermal, acoustic and electrical observations in the nonvisual materials manifest. A thermal channel, for example, cites the emissivity and temperature records from which it renders. Channels without supporting records are excluded from the policy group.

Actions, resets and terminations

Action terms are derived from the physics articulation manifest, never from geometry alone. Joint-space actions name the recorded joints, drive mode and limits. Task-space actions name the body they affect and the workspace bounds from the safety record. Manifest validation rejects commands beyond a recorded joint limit.

Reset distributions sample initial poses, joint positions and object placements within recorded bounds. Grasp tasks may seed resets from recorded grasp points. Every reset distribution carries feasibility evidence: sampled resets must settle without interpenetration within the runtime evidence's settle_steps, using its settled-speed threshold as the criterion.

Terminations include time-out, success, recorded joint-limit violation, illegal contact and workspace exit. The task-fitness definition is the sole success criterion.

Rewards

Every reward remains a proposal until it passes the probes, whether it came from an engineer or a language model. In the Eureka pattern, the model writes reward code, trains policies, receives training statistics and revises the reward (Yu et al., 2023; Ma, Liang, G. Wang, et al., 2024; Xie et al., 2024)22 23 24. DrEureka adds a safety instruction and reports that this makes the rewards deployable (Ma, Liang, H.-J. Wang, et al., 2024)25. Each iteration in this blueprint writes a proposal record, an evidence record and a report. Promotion requires both probe evidence and review.

Rigid Kriegsspiel before free Kriegsspiel

The Prussian war game began in 1824 as a rules-heavy exercise, played with dice and printed tables that fixed the outcome of every engagement (Reisswitz, 1824)26. Half a century later it split in two: rigid Kriegsspiel, governed by the tables, and free Kriegsspiel, governed by an umpire's judgement, which Verdy du Vernois argued was the only way to keep the game close to war (Verdy du Vernois, 1876)27. The Prussian General Staff kept both, in that order. The tables caught the errors an umpire would be too generous to notice.

Reward review follows the same order: deterministic probes precede reviewer judgement. The probes catch errors that an informal review may excuse.

Before review, every reward proposal passes the following deterministic probes, and the manifest records the result of each:

  • Random-policy baseline. Mean return and per-component magnitudes under uniformly random actions over the recorded seeds.
  • Scripted-oracle bound. A scripted policy built from the recorded affordances: move to the grasp frame along the approach vector, close to the recorded width, lift; or drive the joint through its recorded range. The oracle's return bounds what the task is worth, and its success proves the task is feasible for the embodiment.
  • Component audit. No component dominates the total by construction; every component has units or is dimensionless by construction; shaping terms are potential-based where the task admits it, so that they cannot change the optimal policy (Ng, Harada and Russell, 1999)28.
  • Specification-gaming probes. Known patterns of reward hacking (Amodei et al., 2016; Skalse et al., 2022; Pan, Bhatia and Steinhardt, 2022)29 30 31 checked against the proposal: proximity rewarded without contact, velocity rewarded without displacement, termination that pays, progress that can be earned by oscillation.
  • Evidence rule. No reward term depends on a quantity whose upstream record is proposal or review_required.

A proposal that fails a probe returns to the proposer with the probe output. A proposal that passes them all becomes review_required and goes to the reviewer with the probe evidence attached.

Randomisation with provenance

Domain randomisation began with visual variation (Tobin et al., 2017)32 and later covered dynamics (Peng et al., 2018)33. Adaptive methods let the range grow with the policy (OpenAI et al., 2019)34, while canonicalisation maps randomised observations back to a reference appearance (James et al., 2019)35. Surveys cover the resulting field (Zhao, Queralta and Westerlund, 2020; Muratore et al., 2022; Aljalbout et al., 2025)36 37 38. Ranges that are too broad can drive the optimiser into a conservative local optimum; ranges that are too narrow fail to generalise (Muratore et al., 2022; Chen et al., 2022)37 39.

Real-robot data provides another source of bounds. SimOpt and BayesSim infer a posterior over simulator parameters from real rollouts (Chebotar et al., 2019; Ramos, Possas and Fox, 2019)40 41, and DROPO performs offline inference from logged trajectories (Tiboni, Arndt and Kyrki, 2023)42. Active domain randomisation and entropy maximisation expand the distribution within policy tolerance (Mehta et al., 2020; Tiboni et al., 2024)43 44. DrEureka first perturbs the simulator around a trained policy to measure those tolerances, then uses a language model to set the ranges within them (Ma, Liang, H.-J. Wang, et al., 2024)25.

This blueprint also derives bounds from the asset's evidence. Every randomisation axis records its provenance.

Provenance Meaning Where it comes from
evidence interval taken from an upstream record material and physics manifests, the sealed uncertainty record
feasibility interval a trained policy tolerates a parameter sweep around the smoke-trained policy
posterior interval inferred from real rollouts BayesSim or DROPO style inference, when real data exists
policy_default declared fallback the randomisation policy in the run request

The manifest records how these sources combine into each training range. If the evidence and feasibility intervals do not overlap, the axis becomes review_required. The mismatch may indicate that the policy cannot tolerate the measured asset range or that the asset evidence needs review; the lane does not choose between them.

Requisite variety, and the good regulator

Ashby's law of requisite variety states that a regulator can only hold a system within bounds if the variety of its responses matches the variety of the disturbances it faces (Ashby, 1956)45. Domain randomisation is that law made operational: the disturbances are the contexts the policy will meet, and the recorded uncertainty in the manifests is a measured lower bound on their variety. A randomisation range narrower than the evidence interval is a regulator that has been told less than is known about its adversary.

Conant and Ashby went further and proved that every good regulator of a system must be a model of that system (Conant and Ashby, 1970)46. That is the cybernetic case for adaptive mode over robust mode: a policy that carries an estimate of the context is a better regulator than one that averages over it, provided the context is identifiable from what the policy can observe. The identifiability audit exists to check that proviso.

Variants must preserve recorded affordances. An affordance depends on the object and the agent together (Gibson, 1979)47. A texture change that makes a handle unrecognisable or a deformation that moves a grasp point beyond the gripper's reach changes the task and is rejected. Only variants that pass this check enter the training distribution as digital cousins (Dai et al., 2024)8.

Curriculum tiers

Domain randomisation is the simplest form of unsupervised environment design (Dennis et al., 2020)48. Prioritised level replay selects high-regret levels from random generation (Jiang, Grefenstette and Rocktäschel, 2021; Jiang et al., 2021)49 50. ACCEL edits high-regret levels so complexity can compound (Parker-Holder et al., 2022)51, while POET co-evolves environments and agents (Wang et al., 2019)52. Grounded curriculum learning keeps the curriculum tied to the real deployment task distribution (L. Wang et al., 2024)53.

ACCEL describes each edit as a mutation. The blueprint's mutation plan already declares the target layer, prim, operation, inputs, expected outputs, gates, rollback note and dry-run support. Layout plans add parametric placement patterns with unit policy and bounds. The lane supports three curriculum tiers:

  1. Static. A declared schedule over randomisation bounds and reset difficulty, in the Isaac Lab curriculum manager idiom, with the terrain-difficulty schedule as the canonical example (Portelas et al., 2020; Rudin et al., 2022)54 55.
  2. Replay. Prioritised replay over pre-validated variants, where the replay buffer is a set of variant identifiers with checksums and the priority is an estimated regret.
  3. Editing. ACCEL-style proposals emitted as mutation plans in validate_only mode and promoted only through the existing gates, with rollback notes. The curriculum generator is one more provider whose output is proposal material.

Editing curricula retain lineage, bounds, gates and rollback for every proposed change.

Affordance-weighted collision fidelity

Collision geometry limits manipulation transfer more directly than visual geometry. One recent study argues that replacing accurate collision meshes with crude convex hulls has a large, underexamined effect on robustness. It measures the error using a surface-sampling distance between the visual and collision shells (Xu et al., 2026)56. Region-specific decomposition tolerances, fine near contact surfaces and coarse elsewhere, cut simulation time by 69% on a pick-and-place task without losing fidelity at contact (Vu, Ganguly and Joshi, 2025)57. Bounded-stiffness contact reduction makes tight-clearance insertion learnable (Vuong and Pham, 2023)58.

The blueprint records grasp points with frames and approach vectors, together with the moving parts for articulation affordances. The lane measures visual-to-collision shell distance around each grasp point and articulated contact surface, then checks it against the task-fitness tolerance. A value outside tolerance emits a stage 5 request for finer decomposition in the failing region and blocks the lane until that request is resolved. The request uses the collision-aware decomposition already supported by the physics stage (Mamou and Ghorbel, 2009; Wei et al., 2022)59 60. Mandatory mesh verification makes this reliable because approximate convex decomposition behaves poorly on non-manifold input.

Safety constraints

Safety constraints are recorded as costs in a constrained MDP (Altman, 1999; Achiam et al., 2017; Ray, Achiam and Amodei, 2019)61 62 63 and as hard terminations. They are never represented by reward penalties alone. The safety record contains workspace bounds, joint-velocity and torque ceilings from the embodiment description, contact-force ceilings for the asset and robot, and deployment rate limits. DrEureka found that giving the reward proposer a safety instruction made the resulting rewards deployable, so every reward proposal request carries this record (Ma, Liang, H.-J. Wang, et al., 2024)25.

Evaluation protocol

Deep reinforcement learning results are sensitive to seeds, and small run counts can mislead (Henderson et al., 2018)64. The evaluation record declares the seeds, run count and aggregation method. Unless the protocol authority states otherwise, it uses the interquartile mean with stratified bootstrap confidence intervals (Agarwal et al., 2021; Patterson et al., 2024)65 66. Reports break success rates down by held-out variant and randomisation axis.

A policy can learn a proxy that matches the intended goal within the training distribution but diverges outside it (Langosco et al., 2022; Shah et al., 2022)67 68. The evaluation record therefore includes a proxy audit. Its held-out variants use the mutation machinery to separate candidate proxies from the objective, for example by matching a distractor to the goal colour or placing a handle on the wrong side. Reviewer approval depends on the proxy-audit evidence; the headline metric alone is insufficient.

Isaac Lab Arena is the evaluation harness of record for the Isaac runtime, and its zero-action runner is the reference noop rollout (NVIDIA Corporation and Lightwheel, 2026)69.

Updating asset evidence

The wind tunnel at Dayton

In 1901 the Wright glider produced a fraction of the lift its designers had calculated. The calculation rested on the Lilienthal tables and on Smeaton's coefficient of air pressure, a number that had been in circulation since 1759 and that nobody had thought to re-measure. The Wrights built a wind tunnel, measured, and found the coefficient wrong by about a third. The 1902 glider flew (Jakab, 1990)70.

The episode shows why simulator inputs should be treated as hypotheses. Flight tests provide evidence about those inputs, and the next calculation inherits any error that is not written back into the tables.

Evaluation runs and real-world rollouts provide new evidence about the asset. BayesSim or DROPO-style methods can infer a posterior over the context from those rollouts (Ramos, Possas and Fox, 2019; Tiboni, Arndt and Kyrki, 2023; Aljalbout et al., 2025)41 42 38. The lane writes that posterior as a new physical-property evidence record with lineage to the source rollouts, then proposes a revised asset manifest through the ordinary review gates. This return path lets recorded uncertainty shrink as evidence accumulates.

Environment card

The manifest renders an environment card containing intended tasks and embodiments, out-of-scope uses, provenance-backed randomisation coverage, known sim-to-real gaps, evaluation results and a reward report (Gilbert et al., 2023)71. It serves the same role as a model card or dataset datasheet (Mitchell et al., 2019; Gebru et al., 2021)72 73. A client's safety function can review it without opening a USD file.

Inputs

  • validated asset package (manifests/simready-asset-manifest.json)
  • physics-articulation, material-inference and, where present, nonvisual-material manifests
  • Isaac runtime evidence for the target backend
  • robot embodiment description with joint limits and actuator ceilings
  • task objective and success definition
  • sensor contract
  • randomisation policy and any real-rollout posterior evidence
  • safety record

Process

  1. Read the simready-asset-manifest, the upstream manifests and the runtime evidence; verify promotion state and checksums.
  2. Reconcile the requested behaviour against the evidence the asset carries; block on missing affordances, limits or ranges.
  3. Bind runtime identity: backend, timestep, episode length, seeds, embodiment.
  4. Assemble the context prior from the recorded ranges, distributions and uncertainty records.
  5. Define observation groups, the sensor contract, actions, resets and terminations from the recorded joints, limits and affordances.
  6. Propose rewards, run the deterministic probes and record their results.
  7. Derive randomisation ranges with provenance; check affordance invariance per variant; run the identifiability audit when the mode is adaptive.
  8. Attach the curriculum tier and, for editing curricula, the mutation-plan contract.
  9. Compute affordance-weighted collision fidelity; emit a stage 5 request on failure.
  10. Write the evaluation protocol, the articulated_training task-fitness protocol and the smoke, train and evaluate command contracts.
  11. Emit manifests/rl-environment-manifest.json, the environment card and the W&B plan.

Outputs

  • manifests/rl-environment-manifest.json
  • runtime identity and context prior records
  • observation, sensor, action, reset, reward and termination specs with probe evidence
  • randomisation spec with per-axis provenance and affordance-invariance results
  • curriculum spec and, where applicable, mutation-plan contracts
  • articulated_training task-fitness protocol
  • evaluation protocol with proxy-audit variants
  • environment card
  • W&B training plan
  • smoke, train and evaluate command contracts with digests

Promotion gates

RL environment generation is blocked until the simready-verification stage has promoted the package with load and physics evidence for the declared backend. Within the lane, promotion requires:

  • lineage: every consumed manifest is validated or released and checksums match
  • backend and timestep binding against the runtime evidence
  • affordance reachability under the scripted oracle
  • affordance-weighted collision fidelity within tolerance
  • reset feasibility within the recorded settle criteria
  • zero-action smoke rollout, random-policy baseline and oracle bound recorded
  • reward component audit and specification-gaming probes passed
  • evidence rule satisfied for every reward and observation term
  • randomisation provenance complete and affordance invariance checked per variant
  • identifiability audit passed when the mode is adaptive
  • evaluation protocol with seeds, held-out variants, aggregation and proxy audit
  • environment card rendered and checksummed
  • W&B plan status, review requirement and promotion decision

Commands

The manifest stores exact command lines and argument digests for smoke, train and evaluation contracts. The smoke contract steps the composed environment with zero actions and then random actions for a fixed number of frames. Isaac runtimes use the Isaac Lab Arena runner. The train contract names the Isaac Lab entry point, environment identifier, seeds and W&B project. The evaluation contract runs the approved protocol over held-out variants and writes the task-fitness evidence report. The evaluation stage recomputes every result from that protocol.

References

  1. Wang, Y. et al. (2024) "RoboGen: Towards unleashing infinite data for automated robot learning via generative simulation," International Conference on Machine Learning (ICML). Available at: https://arxiv.org/abs/2311.01455. ↩

  2. Katara, P., Xian, Z. and Fragkiadaki, K. (2024) "Gen2Sim: Scaling up robot learning in simulation with generative models," IEEE International Conference on Robotics and Automation (ICRA), pp. 6672–6679. Available at: https://doi.org/10.1109/ICRA57147.2024.10610968. ↩

  3. Deitke, M. et al. (2022) "ProcTHOR: Large-scale embodied AI using procedural generation," Advances in Neural Information Processing Systems (NeurIPS). ↩

  4. Jiang, Z., Hsu, C.-C. and Zhu, Y. (2022) "Ditto: Building digital twins of articulated objects from interaction," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5616–5626. Available at: https://doi.org/10.1109/CVPR52688.2022.00553. ↩

  5. Chen, Z. et al. (2024) "URDFormer: A pipeline for constructing articulated simulation environments from real-world images." Available at: https://arxiv.org/abs/2405.11656. ↩

  6. Xiang, F. et al. (2020) "SAPIEN: A simulated part-based interactive environment," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11097–11107. Available at: https://doi.org/10.1109/CVPR42600.2020.01111. ↩

  7. Mo, K. et al. (2021) "Where2Act: From pixels to actions for articulated 3D objects," IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6813–6823. Available at: https://doi.org/10.1109/ICCV48922.2021.00674. ↩

  8. Dai, T. et al. (2024) "Automated creation of digital cousins for robust policy learning," Conference on Robot Learning (CoRL). Available at: https://arxiv.org/abs/2410.07408. ↩↩

  9. Hallak, A., Di Castro, D. and Mannor, S. (2015) "Contextual Markov decision processes." Available at: https://arxiv.org/abs/1502.02259. ↩

  10. Ghosh, D. et al. (2021) "Why generalization in RL is difficult: Epistemic POMDPs and implicit partial observability," Advances in Neural Information Processing Systems (NeurIPS). Available at: https://arxiv.org/abs/2107.06277. ↩

  11. Kirk, R. et al. (2023) "A survey of zero-shot generalisation in deep reinforcement learning," Journal of Artificial Intelligence Research, 76, pp. 201–264. Available at: https://doi.org/10.1613/jair.1.14174. ↩

  12. Makoviychuk, V. et al. (2021) "Isaac Gym: High performance GPU-based physics simulation for robot learning," Advances in Neural Information Processing Systems: Datasets and Benchmarks Track. Available at: https://arxiv.org/abs/2108.10470. ↩

  13. Mittal, M. et al. (2023) "Orbit: A unified simulation framework for interactive robot learning environments," IEEE Robotics and Automation Letters, 8(6), pp. 3740–3747. Available at: https://doi.org/10.1109/LRA.2023.3270034. ↩

  14. Mittal, M. et al. (2025) "Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning." Available at: https://arxiv.org/abs/2511.04831. ↩

  15. NVIDIA Corporation (2026b) Newton physics integration. Available at: https://isaac-sim.github.io/IsaacLab/main/source/experimental-features/newton-physics-integration/index.html (Accessed: August 27, 2026). ↩

  16. Hu, W. et al. (2026) "SimWeaver: Zero-shot RGB sim-to-real for deformable manipulation." Available at: https://arxiv.org/abs/2606.15338. ↩

  17. NVIDIA Corporation (2026a) Creating a manager-based RL environment. Available at: https://isaac-sim.github.io/IsaacLab/main/source/tutorials/03_envs/create_manager_rl_env.html (Accessed: August 27, 2026). ↩↩

  18. Pinto, L. et al. (2018) "Asymmetric actor critic for image-based robot learning," Robotics: Science and Systems (RSS). Available at: https://doi.org/10.15607/RSS.2018.XIV.008. ↩

  19. Chen, D. et al. (2020) "Learning by cheating," Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research 100, pp. 66–75. ↩

  20. Lee, J. et al. (2020) "Learning quadrupedal locomotion over challenging terrain," Science Robotics, 5(47), p. eabc5986. Available at: https://doi.org/10.1126/scirobotics.abc5986. ↩

  21. Kumar, A. et al. (2021) "RMA: Rapid motor adaptation for legged robots," Robotics: Science and Systems (RSS). Available at: https://doi.org/10.15607/RSS.2021.XVII.011. ↩↩

  22. Ma, Y.J., Liang, W., Wang, G., et al. (2024) "Eureka: Human-level reward design via coding large language models," International Conference on Learning Representations (ICLR). Available at: https://arxiv.org/abs/2310.12931. ↩

  23. Yu, W. et al. (2023) "Language to rewards for robotic skill synthesis," Conference on Robot Learning (CoRL). Available at: https://arxiv.org/abs/2306.08647. ↩

  24. Xie, T. et al. (2024) "Text2Reward: Reward shaping with language models for reinforcement learning," International Conference on Learning Representations (ICLR). Available at: https://arxiv.org/abs/2309.11489. ↩

  25. Ma, Y.J., Liang, W., Wang, H.-J., et al. (2024) "DrEureka: Language model guided sim-to-real transfer," Robotics: Science and Systems (RSS). Available at: https://arxiv.org/abs/2406.01967. ↩↩↩

  26. Reisswitz, G.H.R.J. von (1824) Anleitung zur darstellung militairischer manöver mit dem apparat des kriegs-spieles. Berlin: Trowitzsch und Sohn. ↩

  27. Verdy du Vernois, J. von (1876) Beitrag zum kriegsspiel. Berlin: E. S. Mittler und Sohn. ↩

  28. Ng, A.Y., Harada, D. and Russell, S. (1999) "Policy invariance under reward transformations: Theory and application to reward shaping," International Conference on Machine Learning (ICML), pp. 278–287. ↩

  29. Amodei, D. et al. (2016) "Concrete problems in AI safety." Available at: https://arxiv.org/abs/1606.06565. ↩

  30. Skalse, J. et al. (2022) "Defining and characterizing reward hacking," Advances in Neural Information Processing Systems (NeurIPS). ↩

  31. Pan, A., Bhatia, K. and Steinhardt, J. (2022) "The effects of reward misspecification: Mapping and mitigating misaligned models," International Conference on Learning Representations (ICLR). Available at: https://arxiv.org/abs/2201.03544. ↩

  32. Tobin, J. et al. (2017) "Domain randomization for transferring deep neural networks from simulation to the real world," IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23–30. Available at: https://doi.org/10.1109/IROS.2017.8202133. ↩

  33. Peng, X.B. et al. (2018) "Sim-to-real transfer of robotic control with dynamics randomization," IEEE International Conference on Robotics and Automation (ICRA), pp. 3803–3810. Available at: https://doi.org/10.1109/ICRA.2018.8460528. ↩

  34. OpenAI et al. (2019) "Solving Rubik's cube with a robot hand." Available at: https://arxiv.org/abs/1910.07113. ↩

  35. James, S. et al. (2019) "Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12627–12637. Available at: https://doi.org/10.1109/CVPR.2019.01291. ↩

  36. Zhao, W., Queralta, J.P. textasciitilde na and Westerlund, T. (2020) "Sim-to-real transfer in deep reinforcement learning for robotics: A survey," IEEE Symposium Series on Computational Intelligence (SSCI), pp. 737–744. Available at: https://doi.org/10.1109/SSCI47803.2020.9308468. ↩

  37. Muratore, F. et al. (2022) "Robot learning from randomized simulations: A review," Frontiers in Robotics and AI, 9, p. 799893. Available at: https://doi.org/10.3389/frobt.2022.799893. ↩↩

  38. Aljalbout, E. et al. (2025) "The reality gap in robotics: Challenges, solutions, and best practices." Available at: https://arxiv.org/abs/2510.20808. ↩↩

  39. Chen, X. et al. (2022) "Understanding domain randomization for sim-to-real transfer," International Conference on Learning Representations (ICLR). Available at: https://arxiv.org/abs/2110.03239. ↩

  40. Chebotar, Y. et al. (2019) "Closing the sim-to-real loop: Adapting simulation randomization with real world experience," IEEE International Conference on Robotics and Automation (ICRA), pp. 8973–8979. Available at: https://doi.org/10.1109/ICRA.2019.8793789. ↩

  41. Ramos, F., Possas, R. and Fox, D. (2019) "BayesSim: Adaptive domain randomization via probabilistic inference for robotics simulators," Robotics: Science and Systems (RSS). Available at: https://doi.org/10.15607/RSS.2019.XV.029. ↩↩

  42. Tiboni, G., Arndt, K. and Kyrki, V. (2023) "DROPO: Sim-to-real transfer with offline domain randomization," Robotics and Autonomous Systems, 166, p. 104432. Available at: https://doi.org/10.1016/j.robot.2023.104432. ↩↩

  43. Mehta, B. et al. (2020) "Active domain randomization," Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research 100, pp. 1162–1176. ↩

  44. Tiboni, G. et al. (2024) "Domain randomization via entropy maximization," International Conference on Learning Representations (ICLR). Available at: https://arxiv.org/abs/2311.01885. ↩

  45. Ashby, W.R. (1956) An introduction to cybernetics. London: Chapman and Hall. ↩

  46. Conant, R.C. and Ashby, W.R. (1970) "Every good regulator of a system must be a model of that system," International Journal of Systems Science, 1(2), pp. 89–97. Available at: https://doi.org/10.1080/00207727008920220. ↩

  47. Gibson, J.J. (1979) The ecological approach to visual perception. Boston: Houghton Mifflin. ↩

  48. Dennis, M. et al. (2020) "Emergent complexity and zero-shot transfer via unsupervised environment design," Advances in Neural Information Processing Systems (NeurIPS), pp. 13049–13061. ↩

  49. Jiang, M., Grefenstette, E. and Rocktäschel, T. (2021) "Prioritized level replay," International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research 139, pp. 4940–4950. ↩

  50. Jiang, M. et al. (2021) "Replay-guided adversarial environment design," Advances in Neural Information Processing Systems (NeurIPS), pp. 1884–1897. ↩

  51. Parker-Holder, J. et al. (2022) "Evolving curricula with regret-based environment design," International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research 162, pp. 17473–17498. ↩

  52. Wang, R. et al. (2019) "Paired open-ended trailblazer (POET): Endlessly generating increasingly complex and diverse learning environments and their solutions," Genetic and Evolutionary Computation Conference (GECCO). Available at: https://arxiv.org/abs/1901.01753. ↩

  53. Wang, L. et al. (2024) "Grounded curriculum learning." Available at: https://arxiv.org/abs/2409.19816. ↩

  54. Rudin, N. et al. (2022) "Learning to walk in minutes using massively parallel deep reinforcement learning," Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research 164, pp. 91–100. ↩

  55. Portelas, R. et al. (2020) "Automatic curriculum learning for deep RL: A short survey," International Joint Conference on Artificial Intelligence (IJCAI), pp. 4819–4825. Available at: https://doi.org/10.24963/ijcai.2020/671. ↩

  56. Xu, K. et al. (2026) "Real-IKEA: Physical fidelity is the prerequisite for robust manipulation." Available at: https://arxiv.org/abs/2606.08564. ↩

  57. Vu, B., Ganguly, S. and Joshi, P. (2025) "Empart: Interactive convex decomposition for converting meshes to parts." Available at: https://arxiv.org/abs/2509.22847. ↩

  58. Vuong, N. and Pham, Q.-C. (2023) "Contact reduction with bounded stiffness for robust sim-to-real transfer of robot assembly." Available at: https://arxiv.org/abs/2306.06675. ↩

  59. Wei, X. et al. (2022) "Approximate convex decomposition for 3D meshes with collision-aware concavity and tree search," ACM Transactions on Graphics, 41(4), pp. 42:1–42:18. Available at: https://doi.org/10.1145/3528223.3530103. ↩

  60. Mamou, K. and Ghorbel, F. (2009) "A simple and efficient approach for 3D mesh approximate convex decomposition," IEEE International Conference on Image Processing (ICIP), pp. 3501–3504. Available at: https://doi.org/10.1109/ICIP.2009.5414068. ↩

  61. Altman, E. (1999) Constrained Markov decision processes. Chapman and Hall/CRC. ↩

  62. Achiam, J. et al. (2017) "Constrained policy optimization," International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research 70, pp. 22–31. ↩

  63. Ray, A., Achiam, J. and Amodei, D. (2019) "Benchmarking safe exploration in deep reinforcement learning." OpenAI technical report. Available at: https://cdn.openai.com/safexp-short.pdf. ↩

  64. Henderson, P. et al. (2018) "Deep reinforcement learning that matters," AAAI Conference on Artificial Intelligence. Available at: https://doi.org/10.1609/aaai.v32i1.11694. ↩

  65. Agarwal, R. et al. (2021) "Deep reinforcement learning at the edge of the statistical precipice," Advances in Neural Information Processing Systems (NeurIPS), pp. 29304–29320. ↩

  66. Patterson, A. et al. (2024) "Empirical design in reinforcement learning," Journal of Machine Learning Research, 25(318), pp. 1–63. ↩

  67. Langosco, L. et al. (2022) "Goal misgeneralization in deep reinforcement learning," International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research 162, pp. 12004–12019. ↩

  68. Shah, R. et al. (2022) "Goal misgeneralization: Why correct specifications aren't enough for correct goals." Available at: https://arxiv.org/abs/2210.01790. ↩

  69. NVIDIA Corporation and Lightwheel (2026) Isaac Lab-Arena: A composable framework for environment setup and policy evaluation. Available at: https://github.com/isaac-sim/IsaacLab-Arena (Accessed: August 27, 2026). ↩

  70. Jakab, P.L. (1990) Visions of a flying machine: The Wright brothers and the process of invention. Washington, DC: Smithsonian Institution Press. ↩

  71. Gilbert, T.K. et al. (2023) "Reward reports for reinforcement learning," AAAI/ACM Conference on AI, Ethics, and Society (AIES), pp. 84–130. Available at: https://doi.org/10.1145/3600211.3604698. ↩

  72. Mitchell, M. et al. (2019) "Model cards for model reporting," Conference on Fairness, Accountability, and Transparency (FAT*), pp. 220–229. Available at: https://doi.org/10.1145/3287560.3287596. ↩

  73. Gebru, T. et al. (2021) "Datasheets for datasets," Communications of the ACM, 64(12), pp. 86–92. Available at: https://doi.org/10.1145/3458723. ↩