Sifei Liu

ASENA: Self-Evolving Agents for Embodied Navigation

ASENA brings coding agents into the physical world through a real robot harness for perception, navigation, whole-body control, and execution feedback. Agents write, test, and refine robot programs, turning experience into reusable skills.

Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation

Recova uses an agent to build digital twins, discover recovery strategies in simulation, and recover from manipulation failures on real robots. Task and recovery policies improve from successful rollouts and human-assisted corrections, supporting longer autonomous operation with less human intervention.

Vesta: A Generalist Embodied Reasoning Model

Vesta is a powerful, state-of-the-art Robot System 2 embodied-reasoning VLM spanning localization, navigation, memory, reasoning, tool use, and long-horizon planning.

Reinforcing Dual-Path Reasoning in Spatial Vision Language Models

SR-REAL jointly reinforces language-only deduction and detect-then-reason 3D grounding for stronger spatial reasoning.

SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning

SpatialClaw gives multimodal agents a stateful code interface for adaptive tool use across 3D and 4D spatial reasoning tasks.

4DP-QA: Scalable QA for 4D Perception in Vision Language Models

4DP-QA provides scalable training data and evaluation for disentangling object and camera motion in 4D scene understanding.

Scaling Parallel Sequence Models to Vision Foundation Models

Compact GSPN scales linear-time 2D spatial propagation to foundation-scale vision encoders with fused kernels and cross-operator distillation.

Grounded 3D-Aware Spatial Vision-Language Modeling

GR3D unifies explicit 2D, implicit 2D, and monocular 3D grounding for grounded spatial chain-of-thought reasoning.

Long-Horizon Manipulation via Trace-Conditioned VLA Planning

LoHo-Manip uses progress-aware subtask plans and visual traces to extend VLA policies to long-horizon manipulation.

3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing

3D-Layout-R1 performs language-conditioned layout editing through explicit scene-graph reasoning.

4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation

4D-RGPT distills 4D perception into a region-prompted multimodal model for depth-aware dynamic-scene reasoning.

OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

OmniVinci advances joint image, video, audio, and text understanding through cross-modal alignment and large-scale omni-modal data.

QeRL: Beyond Efficiency—Quantization-Enhanced Reinforcement Learning for LLMs

QeRL uses low-precision quantization both to accelerate RL and to improve policy exploration.

3D Aware Region Prompted Vision Language Model

SR-3D connects 2D and 3D representation spaces for flexible region prompting and grounded spatial reasoning.

Real Deep Research for AI, Robotics and Beyond

Real Deep Research provides a generalizable pipeline for mapping research areas, identifying emerging trends, and uncovering cross-domain opportunities, with extensive analyses of AI and robotics.

EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos

EgoVLA scales robot-policy learning with egocentric human video and action retargeting.

Scaling RL to Long Videos

Long-RL combines large-scale reasoning data, reinforcement learning, and sequence-parallel infrastructure for hour-long video understanding.

SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation

SSE selects semantically diverse data and enriches it from large unlabeled pools while preserving a fixed training budget.

M3: 3D-Spatial MultiModal Memory

M3 stores foundation-model features in a compact 3D Gaussian memory for grounded multimodal retrieval.

Describe Anything: Detailed Localized Image and Video Captioning

DAM generates detailed localized image and video captions for user-specified regions while preserving local detail and global context.

Token-Efficient VLM: High-Resolution Image Understanding via Dynamic Region Proposal

TEVA uses dynamic region proposal and adaptive patch sampling to handle high-resolution image understanding efficiently.

Parallel Sequence Modeling via Generalized Spatial Propagation Network

GSPN is a spatially coherent attention mechanism for vision models that improves high-resolution efficiency and fidelity.

Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks

Omni-RGPT unifies region-level understanding for images and videos with token marks and region-aware video instruction tuning.

BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations

BlobGEN-Vid introduces blob video representations for controllable compositional text-to-video generation.

Scaling Vision Pre-Training to 4K Resolution

PS3 scales CLIP-style vision pre-training to 4K resolution with near-constant cost for stronger high-resolution perception.

NaVILA: Legged Robot Vision-Language-Action Model for Navigation

NaVILA combines vision-language-action modeling with locomotion policies for embodied navigation.

NVILA: Efficient Frontier Visual Language Models

NVILA is an efficient frontier family of visual language models with strong training and inference efficiency.

No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images

NoPoSplat reconstructs 3D Gaussian splats from sparse unposed images with a simple pose-free pipeline.

Compositional Text-to-Image Generation with Feedforward Layout Generation

BlobGen-Next trains a feedforward BlobLLM to generate detailed object layouts for compositional text-to-image generation.

GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

GroPrompt uses grounded prompt generation and lightweight adaptation for weakly supervised referring video object segmentation.

RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos

WildRGB-D provides 20,000 in-the-wild RGB-D videos of 8,500 objects across 44 categories for scalable real-world 3D object learning.

AGG: Amortized Generative 3D Gaussians for Single Image to 3D

AGG directly predicts high-quality 3D Gaussians from a single image without per-instance optimization.

CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation

CamCo introduces camera-pose conditioning and epipolar attention for controllable, 3D-consistent video generation.

BlobGEN-3D: Compositional 3D-Consistent Freeview Image Generation with 3D Blobs

BlobGEN-3D uses compositional 3D blob controls for coherent free-view image generation.

Compositional Text-to-Image Generation with Dense Blob Representations

BlobGEN turns dense, modular blob representations into controllable compositional image generation.

COLMAP-Free 3D Gaussian Splatting

COLMAP-Free 3D Gaussian Splatting enables view synthesis without SfM preprocessing by progressively growing 3D Gaussians from video.

CosAE: Learnable Fourier Series for Image Restoration

CosAE is a cosine autoencoder for image restoration that uses Fourier coefficients for compact yet detail-preserving representations.

SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models

SpatialRGPT improves grounded spatial reasoning in VLMs through regional representation learning and depth-aware modeling.

RegionGPT: Towards Region Understanding Vision Language Model

RegionGPT enhances region-level captioning and understanding by improving spatial awareness in vision-language models.

Dream-in-4D: A Unified Approach for Text- and Image-guided 4D Scene Generation

Dream-in-4D provides a unified framework for text-guided, image-guided, and personalized 4D scene generation.

3D Reconstruction with Generalizable Neural Fields using Scene Priors

NFP learns scene priors for fast, flexible 3D reconstruction and single-image novel-view synthesis.

HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data

HOIDiffusion generates realistic 3D hand-object interaction data for perception tasks such as 6D pose estimation.

TUVF: Learning Generalizable Texture UV Radiance Fields

The paper introduces TUVF, a method for learning generalizable texture UV radiance fields.

Affordance diffusion: Synthesizing hand-object interactions

The paper proposes a method for interaction synthesis that addresses issues using diffusion models. They build upon the classic idea of disentangling where to interact (layout) from how to interact (content).

Open-vocabulary panoptic segmentation with text-to-image diffusion models

We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation.

Scraping Textures from Natural Images for Synthesis and Editing

CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs

This work introduces Coordinate GAN (CoordGAN), a structure-texture disentangled GAN that learns a dense correspondence map for each generated image.

GroupViT: Semantic Segmentation Emerges from Text Supervision

This paper proposes a hierarchical Grouping Vision Transformer (GroupViT), which learns to group image regions into progressively larger arbitrary-shaped segments.

Learning continuous environment fields via implicit functions

Autoregressive 3D Shape Generation via Canonical Mapping

The paper demonstrates a solution for 3D point cloud generation using transformers. The key idea is to decompose a point cloud into a sequence of semantically meaningful shape compositions, which are further encoded by an autoregressive model for point cloud generation.

Learning Continuous Image Representation with Local Implicit Image Function

The paper presents a method for learning continuous image representation with local implicit image function.

Coupled Segmentation and Edge Learning via Dynamic Graph Propagation

Video Autoencoder: self-supervised disentanglement of static 3D structure and motion

This paper presents a video autoencoder for learning disentangled representations of 3D structure and camera pose from videos in a self-supervised manner.

Learning 3D Dense Correspondence via Canonical Point Autoencoder

The paper presents a method for learning 3D dense correspondence using a canonical point autoencoder.

Learning to track instances without video annotations

Contrastive syn-to-real generalization

Self-supervised object detection via generative image synthesis

Video matting via consistency-regularized graph neural networks

Synthesizing long-term 3d human motion and interaction in 3d scenes

Self-supervised single-view 3D reconstruction via semantic consistency

Hierarchical contrastive motion learning for video action recognition

Regularizing meta-learning via gradient dropout

Self-supervised viewpoint learning from image collections

Online Adaptation for Consistent Mesh Reconstruction in the Wild

Joint-task self-supervised learning for temporal correspondence

This paper proposes to learn reliable dense correspondence from videos in a self-supervised manner.

Learning Propagation for Arbitrarily-structured Data

Learning Linear Transformations for Fast Image and Video Style Transfer

Few-Shot Viewpoint Estimation

Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments

SCOPS: Self-Supervised Co-Part Segmentation

Switchable temporal propagation network

Rendering Portraitures from Monocular Camera and Beyond

Context-Aware Synthesis and Placement of Object Instances

Learning Video-Story Composition via Recurrent Neural Network

Learning Dual Convolutional Neural Networks for Low-Level Vision

Learning Affinity via Spatial Propagation Networks

Face Parsing via Recurrent Propagation

Unsupervised Domain Adaption for Face Recognition in Unlabeled Videos

Learning to Segment Instances in Videos with Spatial Propagation Network

Generative Face Completion

Learning Recursive Filters for Low-Level Vision via a Hybrid Neural Network

Deep Cascaded Bi-Network for Face Hallucination

Multi-Objective Convolutional Learning for Face Labeling

Compressed Face Hallucination

Structured Face Hallucination