Phoenix studies how robots can learn from large collections of manipulation recordings and adapt that knowledge to physical tasks. The work, developed by Anyverse Dynamics and the reinforcement learning team at CASIA, combines a pretrained world-action model with two post-training methods: learning contact skills from demonstrations and improving policies through interaction with a learned simulator.
Phoenix-MWA predicts future visual states and actions from current observations and language instructions. Phoenix Encoder learns geometric and temporal features from two teachers; a causal-semantic VAE compresses these features for joint video-action generation. The data pool contains approximately 54,400 raw hours, 39,300 effective hours, and 3.38 billion raw frames from robot demonstrations, egocentric videos, and UMI recordings.
After task-specific post-training, Phoenix-MWA reaches 94.63% on LIBERO-Plus, 66.48% on LIBERO-Pro, 82.67% on RoboCasa-GR1, and 45.6% on Bench2Dex, ranking first in the reported comparisons. It also records 72.42% on RoboTwin 2.0 C2R and 26.11% on RoboDojo-RealWorld. Each result uses the corresponding benchmark’s post-training and evaluation setting.
Phoenix-ILA adapts the pretrained model to contact-rich tasks by learning latents that encode both motion and tactile changes. Phoenix-PhysCanvas provides a visual-tactile simulator for reinforcement learning, allowing a policy to compare the outcomes of different actions. We evaluate these two post-training methods on peeling, wiping, test-tube insertion, and block manipulation. Code and model releases are linked above.
The eight videos below show multi-step manipulation, sustained tool contact, and precise alignment. All run at 1× speed with policy inference on an NVIDIA RTX 4090. Real-time chunking (RTC) and asynchronous execution let the model prepare the next action segment while the robot executes the current one.
The pencil-pouch and sandwich-preparation demonstrations show how the robot carries out several actions in sequence. After executing a short segment, it observes the new object state and plans the next segment.
This video could not be loaded. Try opening the original video.
In peeling and wiping, the tool must stay in contact with the surface as it moves. Phoenix-ILA learns motion and tactile changes together so that its action targets include the contact patterns present in demonstrations. The videos show sustained contact and transitions between contact states; success rates are reported in the analysis section.
This video could not be loaded. Try opening the original video.
Demonstrations may not cover every approach angle or object-position error encountered during insertion. Phoenix-PhysCanvas lets the policy try different actions in a learned simulator and update from task rewards. We test the resulting policy on test-tube insertion and block manipulation under changes in lighting and object position. Bottle-cap twisting and cube repositioning provide additional qualitative demonstrations.
This video could not be loaded. Try opening the original video.
Phoenix-MWA learns a visual representation, compresses it into latents, and jointly predicts future observations and actions. The following sections describe each stage. Phoenix-ILA and PhysCanvas then adapt this pretrained policy through demonstration-based and interactive post-training, respectively.
Given current multi-view RGB images, robot state, language, and embodiment ID, Phoenix-MWA predicts future visual latents and actions. Training samples pair 32 actions with nine frames over the same interval. States and actions share a unified Action Space, with dataset–embodiment normalization and masks for missing views, invalid action dimensions, and padded timesteps.
Phoenix Encoder augments DINOv3 ViT-L/16 with a spatiotemporal adapter and learns from two frozen teachers. VGGT-Ω supplies geometric supervision through multi-layer features, depth predictions, and adjacent-frame relations, without additional ground-truth depth. V-JEPA supplies temporal supervision: the teacher sees the full video, while the student predicts teacher features in consistently masked regions across frames. We alternate the two distillation tasks and maintain an exponential moving average (EMA) of the student weights. CS-VAE and the policy use the resulting EMA DINO backbone to extract dense features frame by frame.
Dense visual features retain semantics and geometry, but their high dimensionality and non-Gaussian distributions can make generation costly and denoising difficult. CS-VAE compresses channels and time: a spatial Transformer and two causal temporal stages reduce nine frames to three temporal positions while preserving the spatial grid. The first latent encodes only the current frame; later latents use preceding information.
The decoder reconstructs standardized features from samples of the Gaussian posterior. Reconstruction uses MSE and cosine losses, while KL regularization shapes the latent distribution toward a standard Gaussian prior for generative modeling. We train with the following objective:
We concatenate two visual latents along the channel dimension: Wan VAE latents, which capture appearance and spatial structure, and Phoenix Encoder–CS-VAE latents containing geometric and semantic features. VideoDiT and ActionDiT process video and actions, connected by a Mixture of Transformers (MoT) with masked joint attention at each layer and modality-specific processing. T5 language embeddings, projected robot states, and learned embodiment-specific soft prompts condition the action branch through cross-attention.
During flow-matching training, current visual latents stay clean, while future video latents and actions receive Gaussian noise. The model learns to jointly denoise both modalities, conditioned on current observations. Wan VAE, Phoenix Encoder, and CS-VAE remain frozen; training updates the video and action experts, state projection, and embodiment soft prompts. The joint pretraining loss combines video and action flow terms:
At inference, actions are unnormalized with the training statistics and mapped to the robot’s control interface. The robot executes a short action prefix, observes the scene again, and replans. Predicted visual latents can be decoded with Wan VAE for inspection. RTC and asynchronous execution overlap planning with movement and maintain continuity between action segments.
Phoenix-MWA uses a joint video-action backbone with approximately 6B parameters, comprising a 5B video expert and a 1B action expert. Joint pretraining updates both experts, the state projection, and embodiment soft prompts, totaling approximately 6B trainable parameters. Wan VAE (approximately 705M), Phoenix Encoder (approximately 621M), and CS-VAE (approximately 72M) remain frozen during joint video-action pretraining and are excluded from the trainable parameter count.
Large-scale pretraining provides a starting policy, but contact-rich tasks still need task-specific training. A visually plausible path does not ensure that a peeler presses against a surface or that a cloth maintains contact. Phoenix-ILA introduces tactile information into the pretrained model by replacing direct action targets with latents learned from synchronized motion and tactile demonstrations.
First, the Interaction Encoder embeds motion and tactile sequences separately. Joint self-attention exchanges information between them, and cross-attention incorporates multi-view visual features before temporal compression. A time-conditioned decoder reconstructs motion, three-dimensional tactile deformation, and a force proxy at queried times. Force supervision is applied both to the force head and to spatially aggregated tactile reconstructions. The decoder receives no separate visual input, so the latent must retain the information needed for reconstruction.
Second, we freeze the interaction codec and encode demonstration chunks offline as normalized latent targets. Conditional flow matching trains the policy to generate these targets from Gaussian noise. We update the action branch, its input/output projections, and temporal positional encoding; the visual encoder and video branch remain frozen. Generation is conditioned on visual and dynamic context, language, robot state, and tactile history.
At inference, the policy uses only currently available observations, including tactile history; it does not require future tactile measurements. Generated latents are denormalized and decoded into joint targets on the control-time grid. The robot executes a short prefix, observes again, and replans.
Phoenix-ILA learns from fixed demonstrations. For tasks where those demonstrations do not cover enough approach angles or position errors, Phoenix-PhysCanvas provides an interactive post-training environment. It learns from real robot trajectories, arranging time-aligned multi-view RGB, tactile images, and tactile displacements on a shared spatial canvas. Robot motion supplies camera-aligned geometric conditions. A pretrained video model with a frozen visual codec learns to predict future observations through conditional flow matching.
Policy training starts from real initial observations. Phoenix-MWA proposes actions, and PhysCanvas predicts the next visual and tactile observations. We sample several rollouts from each initial condition, score task completion with a reward model, and compute relative advantages within each group for GRPO updates.
We then evaluate the updated policy on the physical robot. This separates improvement inside the learned simulator from improvement on real tasks; the test-tube and block results are reported below.
The data pool contains robot demonstrations, egocentric human videos, and UMI recordings: 54,413 hours of raw video, approximately 3.38 billion frames, and 39,306 hours retained after processing. Robot recordings account for 31.09% of the effective duration, egocentric videos for 63.58%, and UMI recordings for 5.33%. The figure lists the sources and their durations.
Raw recordings can contain long idle periods, broken tracks, or instructions that do not match the action. Robot, Ego, and UMI data first pass through format conversion and quality filtering. Source-specific processing then aligns coordinates, annotates actions, and constructs training windows. Final review checks that video, trajectories, and language describe the same operation.
Format conversion. We convert sources to LeRobot V3 and record position units and orientation conventions explicitly. We align video timestamps with state acquisition times and check frame counts, state sequences, and task boundaries.
Quality filtering. We screen recordings for decoding failures, invalid timestamps, poor visual quality, abnormal motion, and duplicates. Visual and temporal similarity are jointly considered for duplicate detection, while camera-motion compensation helps distinguish local activity from viewpoint changes in Ego videos. For robot demonstrations, stationary detection uses measured states rather than control commands, combining adjacent and cumulative changes to preserve slow movement. Only redundant stationary segments at episode boundaries are trimmed; pauses within a task are retained, and video, states, and annotation timelines remain synchronized.
Data annotation. For Ego data, we screen available wrist tracks for abrupt changes that may indicate tracking errors. Language annotations capture the overall task, scene context, and time-localized hand actions. These annotations guide window descriptions: only actions with sufficient temporal overlap are included, and their combined coverage must adequately describe the selected window. We split data by original episode before constructing samples to prevent overlapping observations from leaking between training and validation sets.
For Robot data, we align end-effector poses through workspace alignment, end-effector alignment and standardizing position units while maintaining synchronization with video. We augment task instructions through semantic paraphrasing, conversational rewriting, and context framing, preserving objects, arm assignments, spatial relationships, action order, and constraints. Each sample pairs an instruction with synchronized observations and state–action sequences drawn entirely from one continuous task interval.
Review platforms. Reviewers inspect synchronized video, wrist trajectories, and speed curves to distinguish real movement from tracking errors. Annotation tools align window descriptions with hand-action timelines. Final review checks crop boundaries, speed peaks, windows near filtering thresholds, and simultaneous hand actions, alongside ordinary retained samples.
The 80-dimensional state and action space assigns fixed slots to arm joints, end-effector poses, grippers, and dexterous hands, with additional channels for the torso, lift, head, and mobile base. Components with the same meaning map to the same slots; single-arm data uses the right-arm slots. Missing, invalid, and unused dimensions are filled with zeros and masked out of the loss. The shared interface preserves each robot’s available controls without assuming identical kinematics.
We report results on LIBERO-Plus, LIBERO-Pro, RoboCasa-GR1, RoboTwin 2.0 Clean2Random, Bench2Dex, and RoboDojo-RealWorld. Each benchmark uses its corresponding post-training and evaluation configuration.
LIBERO-Plus tests seven perturbation categories: camera, robot, language, lighting, background, noise, and layout. Phoenix-MWA reaches 94.63% overall success, with the highest scores among the compared methods in the robot, language, lighting, noise, and layout categories.
Without pretraining, Phoenix-MWA reaches 88.16% overall success, including 79.36% under camera perturbations and 94.82% under noise perturbations. Pretraining raises overall success by 6.47 percentage points to 94.63%, with gains in all seven categories.
Bench2Dex includes 12 arm–hand combinations and 26 task–embodiment settings. Tests vary object pose, table height, backgrounds, lighting, distractors, and camera pose. Phoenix-MWA scores 38.5% on equivariance (Equi.), 44.1% on invariance (Inv.), and 45.6% on the combined (Full) evaluation, the highest values in the comparison. Its mean across the four evaluation channels is 45.6%, 17.3 percentage points above the strongest compared baseline.
| Metric | ACT | DP | π₀.₅ | GR00T N1.5 | Phoenix-MWA (w. Pretrain) |
|---|---|---|---|---|---|
| None | 29.5% | 12.9% | 35.5% | 48.5% | 54.2% |
| Equi. | 15.4% | 6.3% | 21.2% | 18.8% | 38.5% |
| Inv. | 14.0% | 5.2% | 26.7% | 26.0% | 44.1% |
| Full | 13.0% | 3.8% | 25.5% | 19.8% | 45.6% |
| SR | 18.0% | 7.1% | 27.2% | 28.3% | 45.6% |
| LSCR | 36.5% | 23.0% | 44.2% | 47.3% | 65.6% |
| Task | Embodiment | ACT | DP | π₀.₅ | GR00T N1.5 | Phoenix-MWA (w. Pretrain) | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| SR | LSCR | SR | LSCR | SR | LSCR | SR | LSCR | SR | LSCR | ||
| Baking Tray Prep | IIWA7 + Sharpa | 6.0% | 11.6% | 2.0% | 7.9% | 27.5% | 38.4% | 10.0% | 29.3% | 30.0% | 61.8% |
| Canned Food Tray Arrangement | 9.5% | 34.6% | 0.0% | 14.8% | 12.0% | 38.8% | 12.5% | 37.3% | 13.5% | 52.6% | |
| Jigsaw Puzzle Assembly | 0.0% | 12.5% | 0.0% | 6.3% | 0.0% | 8.4% | 5.0% | 30.3% | 0.0% | 27.6% | |
| Gaming Desk Setup | JAKA ZU7 + DexHand021 | 31.0% | 46.0% | 9.0% | 30.7% | 33.5% | 49.3% | 59.5% | 72.2% | 52.0% | 73.0% |
| Medicine Shoebox Pack | Panda + Orca | 7.5% | 21.5% | 0.0% | 9.3% | 21.5% | 39.8% | 13.0% | 26.2% | 49.5% | 65.5% |
| Tool Box Loading | 20.0% | 37.5% | 16.5% | 39.8% | 72.0% | 86.0% | 40.0% | 51.0% | 68.5% | 84.2% | |
| Sports Ball Cup Sort | Panda + Allegro | 2.0% | 2.5% | 0.5% | 4.4% | 21.5% | 33.6% | 9.0% | 21.5% | 23.0% | 43.1% |
| Faucet Cup Water Fill | RM65 + Revo2 | 7.5% | 35.4% | 4.0% | 14.0% | 15.5% | 40.5% | 33.0% | 55.0% | 35.5% | 66.5% |
| Stationery Category Sorting | 1.5% | 23.1% | 2.0% | 18.5% | 0.0% | 2.3% | 3.0% | 16.9% | 24.5% | 55.5% | |
| Bimanual Piano Melody | xArm7 + Ability | 4.0% | 4.5% | 0.5% | 0.5% | 7.5% | 16.0% | 23.0% | 23.5% | 4.0% | 6.2% |
| Wine Glass Plate Balance | 26.0% | 72.0% | 3.5% | 39.8% | 35.0% | 76.2% | 28.5% | 72.5% | 40.0% | 81.8% | |
| Cleaner Box Loading | xArm7 + LEAP | 33.0% | 76.2% | 9.0% | 32.8% | 55.0% | 88.5% | 47.0% | 82.3% | 72.5% | 95.8% |
| Shoebox Accessory Pack | 30.0% | 56.3% | 9.5% | 35.2% | 51.0% | 68.3% | 40.5% | 60.3% | 64.5% | 79.8% | |
| Toilet Lid Cleaner Pour | 34.0% | 50.3% | 8.5% | 23.5% | 45.5% | 53.3% | 45.0% | 63.3% | 74.5% | 84.2% | |
| Breadbasket Fast-Food Loading | UR5 + RH5DG2 | 10.5% | 15.8% | 4.5% | 7.3% | 12.5% | 15.6% | 9.0% | 13.5% | 65.5% | 71.9% |
| Citrus Plate Loading | 51.5% | 69.9% | 22.5% | 44.4% | 56.5% | 67.3% | 59.0% | 73.1% | 92.5% | 96.1% | |
| Fridge Wine Interhand Pour | 11.0% | 48.5% | 2.5% | 41.8% | 8.5% | 36.1% | 21.0% | 58.6% | 64.5% | 82.1% | |
| Fruit Bowl Loading | UR5 + RH56DFX | 37.0% | 62.0% | 18.0% | 47.8% | 38.5% | 71.6% | 32.5% | 68.1% | 60.0% | 82.4% |
| Screwdriver Box & Hammer | 24.5% | 56.2% | 20.5% | 49.0% | 33.0% | 57.5% | 48.0% | 73.7% | 64.0% | 85.9% | |
| Trash Disposal | 18.5% | 30.3% | 14.0% | 21.5% | 41.0% | 53.7% | 34.5% | 47.8% | 44.5% | 64.8% | |
| Fridge Fruit Shelf Sorting | UR5 + Shadow | 0.5% | 2.6% | 0.0% | 15.1% | 6.0% | 29.6% | 1.0% | 19.6% | 30.0% | 49.8% |
| Soup Serving | 1.0% | 11.5% | 0.0% | 3.1% | 0.0% | 14.0% | 3.0% | 13.9% | 3.0% | 18.8% | |
| Frypan Stand & Pour | UR5 + Schunk | 19.0% | 51.2% | 14.5% | 30.0% | 15.0% | 33.3% | 54.5% | 72.3% | 60.0% | 85.8% |
| Microwave Bowl Loading | 62.5% | 74.9% | 21.0% | 48.6% | 70.0% | 78.4% | 70.5% | 82.0% | 90.5% | 94.1% | |
| Ball Box Loading | UR5 + Wuji | 8.5% | 10.4% | 0.0% | 7.1% | 14.0% | 19.4% | 23.0% | 37.1% | 9.5% | 19.9% |
| Condiment Box Loading | 10.5% | 31.8% | 1.0% | 4.2% | 15.0% | 33.2% | 10.5% | 27.7% | 49.5% | 76.7% | |
| Task mean (26 tasks) | 18.0% | 36.5% | 7.1% | 23.0% | 27.2% | 44.2% | 28.3% | 47.3% | 45.6% | 65.6% | |
We measure changes in visual features and future-video prediction, then evaluate contact post-training and simulator-based reinforcement learning on real tasks.
Phoenix Encoder starts from the DINO backbone in VGGT-Ω, denoted DINOv3 (VGGT-Ω). This initialization has a lower effective rank than the original DINOv3. We test whether dual-teacher distillation spreads feature variation across more directions while retaining the geometric initialization.
We sample 10,000 frames each from the pretraining validation set and LIBERO-Plus, using identical frames for all three encoders. CLS and Patch Mean refer to the CLS token and the average of the remaining image patch tokens, respectively. We compute effective rank from the singular values of the centered feature matrices to measure how evenly feature variation is distributed across principal directions.
Compared with DINOv3 (VGGT-Ω), the CLS / Patch Mean effective ranks increase by 5.81% / 7.86% on the pretraining validation set and 2.44% / 6.55% on LIBERO-Plus. This measures a more even distribution of feature variation. It is evidence about the representation, not a direct measure of manipulation performance.
We compare FastWAM with Phoenix-MWA trained with and without pretraining on LIBERO-Plus and RoboTwin 2.0 C2R. The comparison measures whether the visual representation and pretraining improve future-video prediction.
We evaluate three models on 200 complete windows per benchmark, matched by task and window identifiers. Only the eight future frames are scored against each model’s own rollout observations. Scores are averaged equally across views (main and wrist for LIBERO-Plus; main, left wrist, and right wrist for RoboTwin) and then across windows.
| Model | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DINO-GT ↑ | CLIP-GT ↑ |
|---|---|---|---|---|---|
| LIBERO-Plus | |||||
| FastWAM | 20.75 | 0.77 | 0.19 | 85.33 | 91.50 |
| Phoenix-MWA (w/o. Pretrain) | 20.96 | 0.78 | 0.17 | 86.06 | 91.46 |
| Phoenix-MWA (w. Pretrain) | 23.31 | 0.81 | 0.12 | 90.28 | 93.09 |
| RoboTwin 2.0 C2R | |||||
| FastWAM | 16.51 | 0.61 | 0.39 | 60.33 | 83.27 |
| Phoenix-MWA (w/o. Pretrain) | 21.19 | 0.75 | 0.30 | 76.02 | 88.96 |
| Phoenix-MWA (w. Pretrain) | 26.30 | 0.82 | 0.16 | 86.12 | 93.95 |
Phoenix-MWA without pretraining exceeds FastWAM on PSNR, SSIM, LPIPS, and DINO-GT on both benchmarks. Pretraining improves all five reported metrics. Relative to FastWAM, the pretrained model gains 2.56 dB PSNR and reduces LPIPS by 36.84% on LIBERO-Plus; the corresponding changes on RoboTwin 2.0 C2R are 9.79 dB and 58.97%. The videos below show examples from LIBERO-Plus and RoboTwin 2.0 C2R.
Video unavailable.
Video unavailable.
We compare direct action generation with interaction-latent generation on peeling and plate wiping. Both require the tool to maintain contact while moving along a surface. These tests measure the effect of learning motion and tactile changes as a shared prediction target.
We report task success and Fext L1, the error between the expert reference and a resultant-force proxy computed from tactile deformation. The latter measures agreement with demonstrated contact patterns.
| Method | Peeling SR ↑ | Peeling Fext L1 ↓ | Wiping SR ↑ | Wiping Fext L1 ↓ |
|---|---|---|---|---|
| FastWAM | 10% | 2.01 ± 0.53 | 50% | 1.54 ± 0.35 |
| Phoenix-MWA | 20% | 1.85 ± 0.63 | 75% | 1.62 ± 0.41 |
| Phoenix-MWA-ILA | 40% | 1.22 ± 0.51 | 90% | 1.38 ± 0.45 |
With interaction latents, success rises from 20% to 40% on peeling and from 75% to 90% on wiping. Force-proxy error falls from 1.85 to 1.22 and from 1.62 to 1.38, reductions of 34.05% and 14.81%. In these two tasks, more trials succeed and the executed contact patterns are closer to the expert reference.
Fixed demonstrations may not cover the corrective actions needed when an object is displaced or approached from a different angle. We test whether GRPO updates in PhysCanvas improve the pretrained policy when it returns to the real robot.
We start from a policy trained for 30,000 SFT steps, then apply GRPO in the learned simulator. We evaluate test-tube insertion and block manipulation under three lighting conditions with varying object positions.
| Task | Method | Success rates across three conditions | Mean success rate ± population SD |
|---|---|---|---|
| Test Tube | SFT 30k steps | 55%, 70%, 50% | 58.33% ± 8.50 pp |
| Test Tube | SFT 30k + PhysCanvas GRPO | 95%, 75%, 55% | 75.00% ± 16.33 pp |
| Blocks | SFT 30k steps | 80%, 80%, 80% | 80.00% ± 0.00 pp |
| Blocks | SFT 30k + PhysCanvas GRPO | 90%, 85%, 85% | 86.67% ± 2.36 pp |
After PhysCanvas GRPO, mean real-world success increases from 58.33% to 75.00% on test-tube insertion and from 80.00% to 86.67% on block manipulation: gains of 16.67 and 6.67 percentage points. Both tasks improve in all three tested conditions. These results cover two tasks; transfer and stability across more tasks and environments remain to be evaluated.
@misc{openphoenix2026phoenix,
title = {Phoenix: Scaling Robotic Manipulation through Visual and Interaction Representations},
author = {{Anyverse Dynamics} and {OpenPhoenix Team}},
year = {2026},
month = {October},
url = {https://openphoenix-team.github.io/Phoenix-pages},
note = {Blog post}
}Qichao Zhang, Haoran Li, Dongbin Zhao
Yupeng Zheng, Zhongpu Xia
Yupeng Zheng, Chaoyue Li, Linbo Wang, Xiang Li, Secant, Zebin Xing
Correspondence to: zhengyupeng18@mails.ucas.ac.cn
Linbo Wang, Xiaotian Liu, Chaoyue Li, Yupeng Zheng, Songen Gu, Secant, Shuai Tian, Ning Ma, Xiaotong Su, Jing Li, Youhao Hu, Bo Li, Qiupeng Gu, Kun Zhang, Kangyang Chen, Jun Wan, Yujie Zang, Zichen Li, Yuchen Liu
Xiaotong Su, Linbo Wang, Yichen Zhang
Xiang Li, Yupeng Zheng, Chaoyue Li, Secant, Yichen Zhang
Join the WeChat group to discuss model training, benchmark evaluation, and robot deployment, and share reproduction results and questions.
The QR card is marked valid before October 29, 2026.
