TAI BUI
Back to curriculum
LearnPython

Embodied VLAs: RT-2, OpenVLA, π0, GR00T

RT-2 (Google DeepMind, July 2023) was the first model to carry web knowledge into a robot arm. It discretized actions into text tokens, co-fine-tuned a VLM on web data plus robot-action data, and showed that web-scale vision-language knowledge transfers to control. OpenVLA (June 2024) shipped the open 7B reference. Physical Intelligence's π0 series (2024-2025) added flow-matching action experts. NVIDIA's GR00T N1 (March 2025) brought dual-system control to humanoids. The VLA primitive, a single model that sees, reads, and acts, is the bridge between this phase's understanding models and the autonomous systems in Phase 15.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 05 (LLaVA), Phase 15 (Autonomous Systems, referenced)

Learning Objectives

  • Describe action tokenization: discrete bin encoding (RT-2), FAST efficient action tokens, continuous flow-matching actions (π0).
  • Explain why co-fine-tuning on web + robot data preserves general-knowledge transfer to novel tasks.
  • Compare OpenVLA (open 7B Llama+VLM), π0 (flow-matching), and GR00T N1 (dual-system) on the same robot task.
  • Name the Open X-Embodiment dataset and its role as the RT-X training corpus.

The Problem

Robots that do chores from natural-language instructions have been a research target since the 1970s. The 2020s answer is the vision-language-action (VLA) model: the same VLM architecture you use for VQA, except the output is actions (joint torques, end-effector poses, discrete commands) instead of text.

VLAs run into four problems that text models don't:

  1. Action spaces are continuous and high-dimensional. A 7-DOF arm plus a 3-DOF gripper is 10 dims, emitted at 30 Hz.
  2. Robot training data is scarce. Open X-Embodiment has ~1M trajectories, while web image-text data runs past 5B.
  3. Control frequency sets a hard budget. A 30 Hz loop leaves 33 ms per action.
  4. Safety. A wrong action can damage hardware, people, or property.
Warm-up0/3
1. In a LLaVA-style VLM, how does image information reach the language model?Warm-up
An image of a table with a red cube, a blue block and a green cylinder on the left, joined by a dashed arrow with a question mark to a box labelled language model. A text prompt box reading what is on the table points into the language model.
Typed schematic, not captured.
2. A VLM decodes text autoregressively, one token per forward pass at about 25 ms each. Roughly how long does it take to emit a 10-token answer?Warm-up
A row of ten word chips forming the sentence The mug is on the left side of the table, a box saying one forward pass takes about 25 ms and gives one token, and a dashed box asking for the time to emit all 10 tokens.
Example sentence is typed. The 25 ms and 10 tokens are the stem's. Typed schematic, not captured.
3. A control loop must send a command to a motor 50 times per second. Roughly how much time does each cycle allow for computing that command?Warm-up
A box for a control loop that sends 50 commands per second with an arrow to a motor box that gets one command each cycle, and a dashed box asking for the time to compute each command.
Figures are the stem's. Typed schematic, not captured.
Check: The Problem0/1
1. What is a vision-language-action (VLA) model?Define
A tabletop camera image and the instruction pick up the red cube both feeding a box labelled the model, with an arrow to a dashed box asking what it outputs.
Scene and sentence are typed. Typed schematic, not captured.

The Concept

Action tokenization (RT-2)

RT-2 represents each joint target as a quantized text token. Discretize the normalized [-1, 1] range into 256 bins and map each bin to a vocabulary ID, so a 10-DOF action becomes 10 tokens per control step.

RT-2 co-fine-tunes a PaLM-X VLM on a mixture of two sources:

  • Web image-text pairs (captioning, VQA).
  • Robot demonstrations, with actions written as tokens.

The model sees "pick up the red cube" (language), then an image (vision), and emits a 10-token action sequence (discretized joint targets). Web pretraining is what keeps general knowledge available: RT-2 can follow "move towards the fast-moving object" even though "fast-moving" isn't in training data.

Inference runs at 3-5 Hz in the RT-2 paper, limited by autoregressive VLM decode.

OpenVLA, the open 7B reference

OpenVLA (Kim et al., June 2024) is the open-weights counterpart to RT-2. It has a 7B Llama backbone, a DINOv2 + SigLIP dual vision encoder, and action tokenization over 256 bins.

It trains on Open X-Embodiment (970k trajectories across 22 robots) and ships with LoRA fine-tuning support for adapting to a new robot.

Inference reaches 4-5 Hz on an A100 with quantization. That is fast enough for slow manipulation and too slow for high-frequency control.

FAST tokenizer: faster action decode

Pertsch et al. (2024) argued that discrete-bin tokenization is inefficient because most actions cluster in a small region of bin-space. FAST (Frequency-domain Action Sequence Tokenizer) compresses action sequences with a DCT and quantizes the coefficients.

A 30-step action trajectory becomes ~10 FAST tokens instead of 300 discrete-bin tokens, and inference speeds up 3-5x without quality loss.

π0 and flow-matching actions

Physical Intelligence's π0 (Black et al., October 2024) replaces discrete action tokens with a flow-matching action expert:

  • A small action transformer reads the VLM's hidden states and outputs a continuous 50-step action sequence via rectified flow.
  • The action head trains with a flow-matching loss while the VLM pretraining stays unchanged.
  • The full action sequence comes out in ~5 denoising steps, which works out to roughly 50 Hz control.

π0 reports beating OpenVLA and Octo on a wide suite of manipulation tasks. A continuous formulation keeps the smoothness that discretization throws away.

π0.5 and π0-FAST are incremental upgrades. π0-FAST combines FAST tokenization with flow matching.

GR00T N1: dual-system control for humanoids

NVIDIA's GR00T N1 (March 2025) targets humanoid robots (>30 DOF, full body) and splits the work in two:

  • System 2 is a large VLM that reads the scene and instruction and produces high-level subgoals at ~1 Hz.
  • System 1 is a small action-head transformer that produces low-level 50-100 Hz joint commands conditioned on those subgoals.

The split maps to Kahneman's fast-and-slow thinking: System 2 plans, System 1 acts. Slow VLM-sized planning does not block fast control, and System 1 stays small to keep latency down.

GR00T N1.7 (late 2025) improves data scaling. GR00T fine-tunes with sim-to-real data from Omniverse.

Open X-Embodiment

This is the training data. RT-X (October 2023) assembled 22 datasets covering 1M trajectories across 22 robots, and Open X-Embodiment is the corpus most of these models train on:

  • ALOHA, Bridge V2, Droid, RT-2 Kitchen, Language Table.
  • Each sample is (robot state, camera views, instruction, action sequence).
  • Training hygiene means unifying the action space, normalizing joint ranges, and resizing cameras.

OpenVLA and π0 train on Open X-Embodiment. The domain gap to a specific robot is closed by LoRA fine-tuning on 100-1000 task-specific demos.

Co-fine-tuning vs robot-only

Co-fine-tuning mixes web VQA data with robot trajectories, and the ratio matters. Too much VQA and the model forgets actions. Too much robot data and it loses general knowledge.

RT-2 uses roughly 1:1. OpenVLA uses ~0.5:1 web-to-robot, and π0 is similar. Treat the exact ratio as a hyperparameter to tune per dataset size.

Robot-only training gives task-specific models that fail on out-of-distribution instructions. Co-fine-tuning is the difference between "pick up the red cube" (as in the demo) and "pick up the third largest object from the left" (novel phrasing).

Safety and action limits

Every production VLA ships with:

  • Hard joint limits, so the arm can't torque past spec.
  • Velocity limits (soft clipping).
  • Workspace bounds, so the end-effector stays over the table.
  • Human-in-the-loop approval for novel tasks.

These sit outside the VLA as control-layer checks. The VLA's output is a suggestion, not a command.

Check: The Concept0/21
1. What is RT-2's core trick for representing robot actions?Define
A simple robot arm drawing and ten labelled joint values, seven arm joints and three gripper joints, each between minus one and one, next to a dashed box asking how the VLM emits this action.
Example values are typed. Typed schematic, not captured.
2. What does the FAST tokenizer change, and what speedup is claimed for it?Define
A plot of three smooth joint curves over 30 steps with an arrow to a box labelled FAST tokenizer with the text what does it change.
Curves are the sin trajectory from main.py's compare_formats. Plot is drawn from its values.
3. What does it mean to co-fine-tune a VLA?Define
Two data cards, web VQA data with a sample question about a mug and robot demonstrations with an instruction and actions, beside a pretrained VLA box and a dashed box asking to co-fine-tune it.
Sample content is typed. Typed schematic, not captured.
4. What is the flow-matching action head used in pi0?Define
A box labelled pi0 containing a VLM box and an orange-outlined box reading flow-matching action head with a question mark. A camera image and an instruction enter the pi0 box and an actions box leaves it.
Block labels follow the lesson. Typed schematic, not captured.
5. What is the System 1 / System 2 split in GR00T N1?Define
A GR00T N1 box with a dashed question box about a System 1 and System 2 split inside it, receiving a scene and instruction and producing joint commands, above a card naming the target robot as a humanoid with more than 30 degrees of freedom.
Typed schematic, not captured.
6. What is Open X-Embodiment?Define
Five name cards, ALOHA, Bridge V2, Droid, RT-2 Kitchen and Language Table, under the heading names the lesson lists under Open X-Embodiment.
Names are from the lesson's list. Typed schematic, not captured.
7. Which description of OpenVLA matches its architecture and training data?Define
A tabletop image and the instruction pick up the red cube feeding a box labelled OpenVLA that outputs a 10-DOF action, above a dashed box asking about architecture and training data.
Scene and sentence are typed. Typed schematic, not captured.
8. RT-2 can follow "move towards the fast-moving object" even though that phrase is absent from training data. What explains this?Explain
A tabletop scene, the asked instruction move towards the fast-moving object, a card stating the phrase is absent from the training data, and a dashed box asking why RT-2 still follows it.
Scene is typed. Sentences are from the lesson. Typed schematic, not captured.
9. Why does FAST need far fewer tokens than per-step 256-bin tokenization for the same trajectory?Explain
A row of 30 tick marks with dots, one per control step, labelled per-step 256-bin format, above a line about the FAST format using far fewer tokens and a dashed box asking why.
Tick per control step is typed. Typed schematic, not captured.
10. How does pi0's action head differ from the discrete-bin approach of RT-2 and OpenVLA?Explain
Two cards side by side: RT-2 and OpenVLA with each joint target quantized to a bin, and pi0 with the text action head: how does it differ, above a dashed box asking for the difference.
Descriptions follow the stem. Typed schematic, not captured.
11. Why does GR00T N1 pair a large VLM with a small action head rather than using one model for everything?Explain
A GR00T N1 humanoid diagram: scene and instruction go in, a large VLM box and a smaller action head box sit side by side, joint commands come out, and a dashed box asks for the reason for the large-plus-small pairing.
Sizes are schematic. Typed schematic, not captured.
12. Open X-Embodiment mixes data from many different robots. What does the lesson's training hygiene do about that?Explain
Five dataset name cards, ALOHA, Bridge V2, Droid, RT-2 Kitchen and Language Table, with an arrow down to a dashed box asking about training hygiene for one model.
Names are from the lesson. Typed schematic, not captured.
13. Which statement about the web-to-robot co-fine-tuning ratio is correct?Explain
Two rows labelled RT-2 and OpenVLA, each with an empty dashed bar holding a question mark, and a legend with blue for web VQA data and green for robot trajectories.
Typed schematic, not captured.
14. Why is a production VLA's output treated as a suggestion rather than a command?Explain
A VLA box with output joint targets, a dashed arrow with a question mark toward a robot arm drawing, and a card saying the output is treated as a suggestion.
Typed schematic, not captured.
15. A robot-only policy completes "pick up the red cube" from its training demos but fails on "pick up the third largest object from the left". Which missing ingredient best explains this?Analyze
A box labelled robot-only policy with arrows to two instruction cards: pick up the red cube, which completes, and pick up the third largest object from the left, which fails. A dashed box asks for the missing ingredient.
Phrases are from the stem. Typed schematic, not captured.
16. After fine-tuning on a mix that is almost all web VQA, a VLA answers questions well but its arm actions are poor. What does the lesson's guidance say went wrong?Analyze
A wide bar that is almost entirely blue web VQA with a thin green robot sliver at the right end, two cards saying the model answers questions well and its arm actions are poor, and a dashed box asking what went wrong.
Bar proportions are schematic. Typed schematic, not captured.
17. A trajectory contains fast, high-frequency motion such as drumming. Why would a FAST-style tokenizer that keeps only a few DCT coefficients struggle with it?Analyze
A plot of one joint over 30 steps swinging rapidly up and down, a card saying the FAST-style tokenizer keeps only the first few DCT coefficients per dimension, and a dashed box asking what happens to this trajectory.
Signal is typed (0.9 sin(1.6 t)). Typed schematic, not captured.
18. A VLA emits a joint target that would torque the arm past its rated spec, and the arm stalls hard. Which safeguard should have caught it?Analyze
A horizontal joint track with a green rated range and a red marker labelled VLA target sitting beyond its right end, a line saying the arm is asked to go past its rating and stalls, and a dashed box asking which safeguard should have caught it.
Positions are schematic, no real spec. Typed schematic, not captured.
19. Your 10-DOF arm needs 30 Hz control, but an RT-2-style discrete decoder only reaches about 4 Hz on your hardware. Which change most directly addresses the bottleneck?Design
Two horizontal bars compare the needed control rate of 30 Hz with a much shorter bar for the measured rate of about 4 Hz, above a dashed box asking which change addresses the bottleneck.
Rates are the stem's. Typed schematic, not captured.
20. You are building a humanoid with more than 30 DOF. Planning from a scene can run at about 1 Hz, but joint commands must update at 50-100 Hz. Which design fits?Design
A one-second timeline with a single blue tick for planning at about 1 Hz and 50 green ticks for joint commands at 50 to 100 Hz, above a dashed box asking which design fits.
Tick counts show 1 and 50 updates in one second (the low end of each range). Typed schematic, not captured.
21. You must adapt an Open X-Embodiment-pretrained OpenVLA to a new robot arm, and you have 300 demos of your task. Which plan fits?Design
A box for OpenVLA with 7B parameters pretrained on Open X-Embodiment and a box for a new robot arm with 300 demos of the task, joined by a dashed arrow, above a dashed box asking for a plan to adapt it.
Figures are the stem's. Typed schematic, not captured.

Use It

code/main.py is stdlib Python and does four things:

  • Implements 256-bin action tokenization and de-tokenization, with a round-trip check on a 10-DOF action.
  • Sketches a FAST-style tokenizer: a per-dimension DCT, the first few coefficients kept, then quantized.
  • Compares token counts for a 30-step, 10-DOF trajectory across discrete-bin, FAST, and continuous flow.
  • Prints a lineage table of RT-2, OpenVLA, Octo, π0, π0-FAST, and GR00T.
Check: Use It0/4
1. In the lesson's discretize function with 256 bins, which token does an action value of 0.0 map to?Apply
A terminal listing the discretize function from main.py, followed by the call discretize of a list holding 0.0 with 256 bins and a question mark for the result.
Function lines copied from main.py (docstring omitted). The call is typed from the stem; its result is left blank.
2. fast_compress runs on a 30-step, 10-DOF trajectory with keep_coeff=4. How many tokens does it return, and how does that compare with the discrete 256-bin count?Apply
A terminal listing the fast_compress function from main.py and a typed call on a 30-step, 10-DOF trajectory with keep_coeff 4, with a question mark in place of the result.
Function lines copied from main.py (docstring omitted). The call is typed from the stem; its result is left blank.
3. A 10-DOF arm runs at 30 Hz with 256-bin tokenization, emitting one token per joint per step. How many action tokens must the VLM decode each second?Apply
A small robot arm drawing, ten chips labelled joint 1 to joint 10, a line stating one token per joint at every control step, a ruler of 30 ticks for one second, and a dashed box asking how many tokens the VLM must decode per second.
Chips and ticks are typed from the stem. Typed schematic, not captured.
4. In the round-trip demo, the action value 0.1 goes through discretize and then undiscretize with 256 bins. What comes back?Apply
A terminal listing the discretize and undiscretize functions from main.py and a typed round trip of 0.1 with 256 bins, with a question mark in place of the result.
Function lines copied from main.py (docstrings omitted). The call is typed from the stem; its result is left blank.

Ship It

This lesson produces outputs/skill-vla-action-format-picker.md. Give it a robot task (manipulation, navigation, or humanoid whole-body control) and it picks between discrete-bin + RT-2, FAST + OpenVLA, flow-matching + π0, or dual-system + GR00T.

Author's Note

I'd run main.py first and look at the token counts before reading anything else. For a 30-step, 10-DOF trajectory, the discrete 256-bin format prints 300 tokens, and the FAST-style compressor prints 40. That is about 7.5x, because the toy keeps 4 DCT coefficients per dimension. Treat the ratio as a feel for the idea, not a benchmark. The ~10 token figure in the body is a general claim about FAST made in the body, not something this script measures.

The flow-matching row prints a count of 1. That is a placeholder for a single head output, not a measurement. The code has no action expert and no denoising loop, so it says nothing about π0's speed or quality. The same goes for the lineage table and the co-fine-tuning ratios: they are printed constants, not results from training anything.

The round-trip demo is the part I'd poke at. The max error comes out at 0.0069 against a bin width of about 0.0078, and it never exceeds one bin because discretize truncates with int(). Try swapping that for rounding and see how much the error shrinks. Then ask whether an error of that size per joint matters on a real arm.

Exercises

  1. A 10-DOF arm at 30 Hz control rate. Discrete-bin tokenization at 256 bins emits how many tokens per second? Can a 7B VLM keep up?

  2. FAST tokenization compresses 30-step trajectories to ~10 tokens. What does the user lose if the trajectory has high-frequency motion (e.g., drumming)?

  3. π0's flow-matching head denoises in ~5 steps. Compare throughput to OpenVLA's autoregressive decode at 4-5 Hz.

  4. GR00T's System 1 / System 2 split maps to Kahneman. Propose a different split (System 3?) that might help bipedal walking.

  5. Read Open X-Embodiment Section 4 on dataset curation. Name the three curation rules that prevent domain leakage.

Key Terms

TermWhat people sayWhat it actually means
VLA"Vision-language-action"Model that takes image + instruction and outputs action commands
Action tokenization"Discrete bins"Quantize continuous joint targets into 256 bins per dim, each a vocab ID
FAST tokenizer"Frequency action tokens"DCT + quantize to compress 30-step trajectories to ~10 tokens
Co-fine-tune"Mix web + robot"Train on web VQA data alongside robot demos to preserve general knowledge
Flow-matching action head"π0 continuous output"Small transformer that outputs a 50-step action sequence via rectified flow
System 1 / System 2"Dual-system control"Large VLM plans slowly, small action head acts quickly; GR00T pattern
Open X-Embodiment"RT-X dataset"1M-trajectory cross-robot dataset; the training corpus

Further Reading

Code

Ships With