GIFT
Robot learning · Research project

GIFT: Guided Intermediate Feature Trainingvia Action-Oriented Structural Supervision
for Robotic Manipulation

Yupeng Zheng1,2*Xiang Li3*Songen Gu4*Yuhang Zheng5*Shuai Tian1,2Weize Li5Linbo Wang1,2Chaoyue Li1,2Qichao Zhang1,2†Haoran Li1,2†Ya-Qin Zhang3Shuicheng Yan5Dongbin Zhao1,2†
1 Institute of Automation, Chinese Academy of Sciences2 University of Chinese Academy of Sciences3 Tsinghua University4 Fudan University5 National University of Singapore
* Equal contribution   ·   † Corresponding authors
Explore results BibTeX
GeometryAffordanceGoalsVLA + WAM
One principle. Three policy configurations.Click any figure to view full size ↗
GIFT framework: geometry, affordance, and goal objectives supervise intermediate visual features in a VLA and two WAM action formulations.
Structure the features, retain the action formulation. GIFT attaches geometry alignment, affordance prediction, and goal-region reconstruction to intermediate visual tokens. In the default design, auxiliary predictions provide training supervision without being injected into the action pathway.
87.8%
LIBERO-Plus success
GIFT-WAM-IDM · +5.2 pp over Fast-WAM-IDM
82.3%
RoboCasa overall success
GIFT-WAM-IDM · +8.4 pp over Fast-WAM-IDM
87.5%
Real-world success
GIFT-WAM-IDM · Four original task settings
The idea

Visual richness.
Control relevance.

Robot policies inherit rich visual and semantic knowledge. But what looks informative is not always what matters for action. We call this mismatch the action-sufficiency gap.

GIFT guides intermediate features to preserve three recurring structures: geometry for motion feasibility, affordance for object-centric interaction, and goals for instruction-conditioned spatial grounding. The same supervision principle transfers across a VLA and two world-action models.

Read the abstract

Vision-language pre-training and predictive world modeling provide robot policies with rich semantic and dynamic visual features, but their native action and visual-prediction objectives may omit critical physical and task structure while retaining control-irrelevant visual redundancy, leading to suboptimal performance. We call this mismatch between visual richness and control utility the action-sufficiency gap. We investigate whether this gap can be bridged by guiding intermediate features to preserve three recurring types of control-relevant structure in robotic manipulation: geometry governing motion feasibility, affordance encoding instruction-relevant entities and object-centric end-effector configurations, and goals grounding instructions in task-relevant action regions. To this end, we present GIFT (Guided Intermediate Feature Training), an architecture-flexible framework for learning intermediate features that translates these structures into training-time constraints through geometry alignment, affordance prediction, and goal-region reconstruction. We instantiate GIFT in a semantics-centered Vision-Language-Action (VLA) policy, a direct-action World-Action Model (WAM), and an inverse-dynamics WAM, and compare injection and no-injection designs while retaining each model's action formulation. Under zero-shot transfer to LIBERO-Plus, GIFT-VLA, GIFT-WAM-Fast, and GIFT-WAM-IDM outperform StarVLA-OFT, Fast-WAM, and Fast-WAM-IDM by 4.6, 12.6, and 5.2 percentage points, reaching 79.6%, 72.6%, and 87.8%, respectively, across seven distribution shifts. On RoboCasa, the three GIFT variants reach 61.4%, 83.6%, and 82.3%, outperforming their counterparts by 12.6, 9.0, and 8.4 points; on articulated-object tasks, GIFT-WAM-Fast and GIFT-WAM-IDM outperform Fast-WAM and Fast-WAM-IDM by 21.3 and 24.6 points, respectively. Together, these results establish learning functionally structured intermediate features as a reusable principle across model-specific action formulations, with especially large gains on articulated-object tasks and high-precision real-world manipulation under unseen visual and spatial perturbations. Ablation studies further show that the no-injection variants perform better, indicating that GIFT internalizes task-relevant structure into intermediate features without relying on auxiliary conditioning.

Zero-shot robustness at a glance

Radial bar chart comparing GIFT-VLA and GIFT-WAM-Fast with their baselines across LIBERO-Plus perturbations.
The paper's overview chart compares GIFT-VLA and GIFT-WAM-Fast with their respective baselines.
01 / Method

Three signals.
More useful features.

Complementary training objectives teach intermediate visual tokens to retain the physical and task structure needed for manipulation.

01 — SPATIAL STRUCTURE

Geometry

Where can the robot move?

Align shallow visual features with a frozen VGGT teacher, preserving geometric structure through feature direction and magnitude supervision.

02 — INTERACTION STRUCTURE

Affordance

What to interact with, and how?

Predict instruction-relevant entity roles, object-centric end-effector poses, and closure states with a structured interaction representation.

03 — TASK STRUCTURE

Goals

Which regions matter for the task?

Ground language instructions in task-relevant image regions through goal-region prediction, complementing object-centric interaction guidance.

ℒGIFT = ℒnative + λgeo ℒgeo + λaff ℒaff + λgoal ℒgoal

GIFT-VLA

Semantics-centered features with direct action prediction.

GIFT-WAM-Fast

Direct action diffusion from cached current-frame world-model features.

GIFT-WAM-IDM

Future-conditioned inverse dynamics: imagine a trajectory, then derive actions.

Supervision at training time.
Native action conditions at inference.

The default no-injection design shapes shared features through auxiliary losses. The geometry teacher and auxiliary prediction heads do not need to run at deployment.

Observations and predicted geometry, object-centric affordances, and goal regions for GIFT-VLA and GIFT-WAM-IDM.
What the features learn. Structured intermediate predictions for GIFT-VLA (top) and GIFT-WAM-IDM (bottom), shown alongside the current observations.
02 / Benchmark results

Generalization across
policy families.

Compare each GIFT variant with its corresponding baseline. Gains are reported in percentage points (pp); scores are success rates (%).

Zero-shot transfer to LIBERO-Plus

Trained on LIBERO. Evaluated across seven distribution shifts.

Baseline GIFTSuccess rate (%) ↑
GIFT-VLAvs. StarVLA-OFT
75.0
79.6
+4.6pp gain
GIFT-WAM-Fastvs. Fast-WAM
60.0
72.6
+12.6pp gain
GIFT-WAM-IDMvs. Fast-WAM-IDM
82.6
87.8
+5.2pp gain

Source: Table 2. All methods are evaluated without fine-tuning on LIBERO-Plus. Overall improvements do not imply improvements on every individual perturbation.

Full LIBERO-Plus comparison · seven distribution shifts
Original Table 2: all baseline and GIFT success rates under camera, robot, language, light, background, noise, and layout shifts.

Original Table 2 from the paper. Scroll horizontally on smaller screens.

Full RoboCasa comparison · articulated objects and pick-and-place
Original Table 3: articulated-object, pick-and-place, and overall success rates on RoboCasa.

Original Table 3. Art. denotes articulated-object tasks; P&P denotes pick-and-place.

03 / Real-world manipulation

From visual understanding
to precise interaction.

Four tasks on two platforms probe instruction following, articulated-object interaction, and precise bimanual insertion. Each task uses 10 evaluation trials per method.

Single-arm UFACTORY xArm7 with RealSense D435 cameras and dual-arm ARX X5 with RealSense D405 cameras.
Two platforms, shared guidance. Single-arm UFACTORY xArm7 (left) and dual-arm ARX X5 (right).

Stronger under
unseen perturbations.

GIFT-WAM-IDM reaches 87.5% overall success in the original settings and 67.5% across the four tested perturbation conditions, compared with 52.5% and 15.0% for Fast-WAM-IDM.

Real-world aggregate success rates for original and perturbed settings
MethodOriginalPerturbed
StarVLA-OFT35.0%5.0%
GIFT-VLA57.5%27.5%
Fast-WAM-IDM52.5%15.0%
GIFT-WAM-IDM87.5%67.5%

Original: 40 trials over Tasks 1–4. Perturbed: 40 trials over two levels each for Tasks 2 and 4. Tables 8–9.

Single-arm task sequences for color-conditioned and layer-conditioned placement, including rotating colored light and a changed tabletop background.
Color- and layer-conditioned placement. Level 1 introduces a rotating colored lamp; Level 2 additionally changes the tabletop background. These conditions are absent from the training data.
Original per-task results and perturbation protocols
Original Table 8: per-task successes out of 10 trials and overall averages for four robot policies.
Task 1: color-conditioned placement. Task 2: layer-conditioned placement. Task 3: articulated-object manipulation. Task 4: bimanual test-tube insertion.
Original Table 9: successes out of 10 trials at each perturbation level for Tasks 2 and 4.
Two perturbation levels for Tasks 2 and 4. All lighting, background, and task-relevant pose changes are unseen in training.
04 / Inside the representation

Learning the structure
is what matters.

Ablations examine each supervision signal, auxiliary feature injection, and the task-relevant information encoded by the policy.

Auxiliary supervision.
Without auxiliary conditioning.

All three no-injection variants achieve higher reported overall success than their injection counterparts on both LIBERO-Plus and RoboCasa. The result supports internalizing task structure in shared features.

Each individual guidance signal improves overall LIBERO-Plus success over its named baseline. Combining all three gives the highest overall success in each ablation group.

With-injection and without-injection architectures for VLA and WAM policies.
In the no-injection design, auxiliary predictions supervise the visual features without becoming action inputs.
LIBERO-Plus action-to-vision attention maps showing the task-relevant focus of GIFT compared with the corresponding VLA and WAM-IDM baselines.
More consistent focus on interaction regions. Qualitative action-to-vision attention maps on LIBERO-Plus compare GIFT with its corresponding baselines.
Guidance and injection ablations · original tables
Original Table 4: geometry-only, affordance-only, goal-only, and joint supervision results for all three models.

Table 4. Joint supervision yields the best overall success in each model's ablation group.

Original Table 5: with-injection and without-injection results on LIBERO-Plus and RoboCasa.

Table 5. No-injection variants are the default GIFT design.

Attention maps on RoboCasa
Action-to-vision attention maps comparing GIFT-VLA and GIFT-WAM-IDM with their baselines on RoboCasa.

Qualitative attention comparisons on RoboCasa.

Limitations and failure cases

Structured guidance remains imperfect. The paper documents failure cases in geometry, affordance, and goal prediction. Overall gains vary with the policy and perturbation; intermediate supervision does not guarantee correct action in every scene.

Examples of geometry, affordance, and goal prediction failures reported in the paper.
Reference

Cite GIFT

Download .bib
BibTeX forthcoming.
Manuscript citation.