Sim-to-Real Transfer

Training a real robot is slow, dangerous, and expensive. A biped needs millions of training episodes to learn to walk, and a real biped that falls over even once breaks hardware. Simulation fixes that: unlimited resets, deterministic reproducibility, parallel environments, no physical damage.

But simulators are wrong, in specific and predictable ways. The systematic gap between the sim distribution and the real distribution is the reality gap: the central problem of deployed RL for robotics. Bearings have more friction than MuJoCo models assume. Cameras carry lens distortion the simulator never modeled. Motors have delays, backlash (slack in the gear train), and saturation that 99% of sim models skip entirely. Wind, dust, and variable lighting sabotage a policy trained on sterile rendering.

You need a policy that's robust to sim-to-real distribution shift. Three approaches carry the field this far: randomize the simulator (domain randomization), adapt the policy with a little real data (domain adaptation, or fine-tuning), or measure the real system's parameters and match the sim to them (system identification). By 2026 the dominant recipe combines all three, running on massive parallel simulation such as Isaac Sim, Isaac Lab, or Mujoco MJX on GPU.

Domain Randomization (DR). Tobin et al. 2017, Peng et al. 2018. During training, randomize every sim parameter that might differ on the real robot: masses, friction coefficients, motor PD gains (the stiffness and damping settings controlling each joint), sensor noise, camera position, lighting, textures, contact models. The policy learns a conditional distribution over "which sim it is in today" and generalizes across the full span. If the real robot falls inside the training envelope, the policy works.

• Upside: no real data needed. One recipe, many robots.
• Downside: over-randomized training produces a "universal" but overly cautious policy. Too much noise acts like too much regularization.

System Identification (SI). Fit the simulator's parameters to real-world data before training. Measure arm-joint friction on the real robot, plug that number into the sim, then train a policy that expects it. SI needs access to the real system, but it reduces the reality gap directly.

• Upside: precise, low-noise training target.
• Downside: residual model error stays invisible to the policy. Small un-identified effects, like motor deadband, still break deployment.

Domain Adaptation. Train in sim, then fine-tune with a small amount of real data. Two flavors:

• Real2Sim2Real: learn a residual simulator f(s, a, z) - f_sim(s, a) from real rollouts, and train inside the corrected sim. Closes the gap without needing much real data.
• Observation adaptation: train a policy that maps real obs → sim-like obs through a learned feature extractor, a GAN doing pixel-to-pixel translation, for example. The controller itself never leaves sim.

Privileged learning, or teacher-student. Miki et al. 2022 trained this way on the ANYmal quadruped. A teacher trains in simulation with access to privileged information: ground truth friction, terrain height, IMU (inertial measurement unit) drift. A student is then distilled from it that only sees real-sensor observations, and learns to infer those privileged features from its own observation history, which makes it robust across physical parameters it was never told directly.

Massively parallel simulation. From 2024 to 2026, Isaac Lab, Mujoco MJX, and Brax all started running thousands of parallel robots on a single GPU. PPO with 4,096 parallel humanoids collects years of experience in hours. The reality gap shrinks as the training distribution widens, and DR gets almost free once each of those 4,096 environments carries its own randomized parameters.

The 2026 recipe for real-world deployment (quadruped walking, as an example):

1. Massively parallel sim with domain-randomized gravity, friction, motor gains, payload.
2. Teacher policy trained with privileged info (terrain map, body velocity ground truth).
3. Student policy distilled from the teacher using only proprioception (leg joint encoders).
4. Optional observation adaptation via an autoencoder on the real IMU.
5. Deploy. Zero-shot across 10+ environments. If it fails, run minutes of real-world fine-tuning with safety-constrained PPO.

This lesson's code is a small demonstration of domain randomization on a GridWorld with noisy transitions. We train a policy that experiences randomized slip probabilities in "sim," then evaluate it on "real" at a slip level it never saw during training. The shape maps directly onto MuJoCo-to-hardware transfer.

Step 1: parameterized sim

def step(state, action, slip):
    if rng.random() < slip:
        action = random_perpendicular(action)
    ...

slip is a parameter the simulator exposes. In real robotics that could be friction, mass, or motor gain: anything that shifts between sim and real.

Step 2: train with DR

At the start of each episode, sample slip ~ Uniform[0.0, 0.4]. Train PPO, Q-learning, whatever algorithm you like, for many episodes.

Step 3: evaluate zero-shot on "real" slips

Evaluate on slip ∈ {0.0, 0.1, 0.2, 0.3, 0.5, 0.7}. The first four sit within training support; 0.5 and 0.7 sit outside it. A DR-trained policy should stay near-optimal inside that support and degrade gracefully outside it. A fixed-slip-trained policy won't: it's brittle the moment it leaves its training slip.

Step 4: compare to narrow training

Train a second policy with slip = 0.0 only, and run it through the same sweep. Watch for a catastrophic drop the moment real slip climbs above zero.

• Too much randomization. Train on slip ∈ [0, 0.9] and the policy gets so risk-averse it never tries the optimal path. Match the expected real-world distribution, not "anything could happen."
• Too little randomization. Train on a thin slice and the policy can't generalize at all. Use an adaptive curriculum, Automatic Domain Randomization, that widens the distribution as the policy improves.
• Misidentified parameter space. Randomize the wrong thing, camera hue when the real gap is actually motor delay, and DR doesn't help at all. Profile the real robot first.
• Privileged info leakage. A teacher that acts on global state instead of just observations can produce a student that never catches up. Make sure the teacher's policy is realizable by the student given its observation history.
• Sim-to-sim transfer failure. A policy that isn't robust to a harder sim variant won't be robust to the real world either. Always test on a held-out sim variant before deploying.
• No real-world safety envelope. A policy that works in sim, and seems to work in real, can still break hardware without a low-level safety shield. Add rate limits, torque limits, and joint limits in a non-learned controller.

Here's the 2026 sim-to-real stack, by domain:

Across all of these, the workflow stays the same: fit the sim as well as you can, randomize whatever you can't fit, train enormous policies, distill them down, and deploy behind a safety shield.

Save as outputs/skill-sim2real-planner.md:

---
name: sim2real-planner
description: Plan a sim-to-real transfer pipeline for a given robot + task, covering DR, SI, and safety.
version: 1.0.0
phase: 9
lesson: 11
tags: [rl, sim2real, robotics, domain-randomization]
---
 
Given a robot platform, a task, and access to real hardware time, output:
 
1. Reality gap inventory. Suspected sources ranked by expected impact (contact, sensing, actuation delay, vision).
2. DR parameters. Exact list, ranges, distribution. Justify each range against real measurements.
3. SI steps. Which parameters to measure; measurement method.
4. Teacher/student split. What privileged info the teacher uses; what obs the student uses.
5. Safety envelope. Low-level limits, emergency stops, backup controller.
 
Refuse to deploy without (a) a zero-shot sim-variant test, (b) a safety shield, (c) a rollback plan. Flag any DR range wider than 3× measured real variability as likely over-randomized.

1. Easy. Train a Q-learning agent on the fixed-slip GridWorld (slip=0.0). Evaluate on slip ∈ {0.0, 0.1, 0.3, 0.5}. Plot return vs slip.
2. Medium. Train a DR Q-learning agent sampling slip ~ Uniform[0, 0.3]. Evaluate the same sweep. How much does DR buy at slip=0.5 (out-of-distribution)?
3. Hard. Implement a curriculum: start with slip=0.0, widen the DR range every time the policy hits 90% of optimal. Measure total environment steps to reach slip=0.3 zero-shot vs. a fixed DR baseline.

• Tobin et al. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World: the original DR paper, for vision in robotics.
• Peng et al. (2018). Sim-to-Real Transfer of Robotic Control with Dynamics Randomization: DR for dynamics, quadruped locomotion.
• OpenAI et al. (2019). Solving Rubik's Cube with a Robot Hand: Dactyl, ADR at scale.
• Miki et al. (2022). Learning robust perceptive locomotion for quadrupedal robots in the wild: teacher-student for ANYmal.
• Makoviychuk et al. (2021). Isaac Gym: High Performance GPU Based Physics Simulation for Robot Learning: the massively parallel sim driving 2025–2026 deployments.
• Akkaya et al. (2019). Automatic Domain Randomization: the ADR curriculum method.
• Sutton & Barto (2018). Ch. 8 — Planning and Learning with Tabular Methods: the Dyna framing (using a model for planning plus rollouts) that underpins modern sim-to-real pipelines.
• Zhao, Queralta & Westerlund (2020). Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey: a taxonomy of sim-to-real methods with benchmark results.