Sifei Liu

Principal Research Scientist / Tech Lead

A portrait of Sifei Liu

sifeiliu.net

Liusifei

Sifei30488L

zwrq0star@gmail.com

Education

University of California, Merced
Ph.D. in EECS, advised by Ming-Hsuan Yang.
Fall 2012 - Fall 2017
University of Science and Technology of China
M.S. in Electronic Engineering and Information Sciences, advised by Stan Z. Li.
Fall 2008 - Spring 2011

Employment

NVIDIA
Researcher
Nov 2017 - Current
Santa Clara, CA
NVIDIA
Research Intern with NVIDIA Research.
March 2017 - Aug 2017
Santa Clara, CA
Chinese University of HongKong
Visiting Scholar at the MMLAB.
Jul 2016 - Dec 2016
Hong Kong
Baidu Inc.
Applied Scientist Intern on IDL. Worked on face parsing and beautification apps.
May 2013 - Jan 2016
Beijing, China

Workshop and tutorial organization

4D Hand Object Interaction: Geometric Understanding and Applications in Dexterous Manipulation
CVPR 2023
Human-centric Trustworthy Computer Vision From Research to Applications
ICCV 2021
Sensing, Understanding and Synthesizing Humans
ECCV 2020
New Frontiers for Learning with Limited Labels or Data
ECCV 2020. (Co-orgainizor and Speaker)
Learning Representations via Graph-structured Networks Tutorial
CVPR 2019 and 2020. (Co-orgainizor and Speaker)

(Co)-Mentees at NVIDIA Research

Recent Publications

3D Aware Region Prompted Vision Language Model
A. Cheng, Y. Fu, Y. Chen, Z. Liu, X. Li, S. Radhakrishnan, S. Han, Y. Lu, J. Kautz, P. Molchanov, H. Yin, X. Wang, S. Liu
SR-3D connects 2D and 3D representation spaces for flexible region prompting and grounded spatial reasoning.
ICLR, 2026
Describe Anything: Detailed Localized Image and Video Captioning
L. Lian, Y. Ding, Y. Ge, S. Liu, H. Mao, B. Li, M. Pavone, M. Liu, T. Darrell, A. Yala, Y. Cui
DAM generates detailed localized image and video captions for user-specified regions while preserving local detail and global context.
ICCV, 2025
Token-Efficient VLM: High-Resolution Image Understanding via Dynamic Region Proposal
Y. Jiang, J. Gu, T. Xue, K. Cheung, P. Molchanov, H. Yin, S. Liu
TEVA uses dynamic region proposal and adaptive patch sampling to handle high-resolution image understanding efficiently.
ICCV, 2025
Parallel Sequence Modeling via Generalized Spatial Propagation Network
H. Wang, W. Byeon, J. Xu, J. Gu, K. Cheung, X. Wang, K. Han, J. Kautz, S. Liu
GSPN is a spatially coherent attention mechanism for vision models that improves high-resolution efficiency and fidelity.
CVPR, 2025
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks
M. Heo, M. Chen, D. Huang, S. Liu, S. Radhakrishnan, S. Kim, Y. Wang, R. Hachiuma
Omni-RGPT unifies region-level understanding for images and videos with token marks and region-aware video instruction tuning.
CVPR, 2025
BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations
W. Feng, C. Liu, S. Liu, W. Wang, A. Vahdat, W. Nie
BlobGEN-Vid introduces blob video representations for controllable compositional text-to-video generation.
CVPR, 2025
Scaling Vision Pre-Training to 4K Resolution
B. Shi, B. Li, H. Cai, Y. Lu, S. Liu, M. Pavone, J. Kautz, S. Han, T. Darrell, P. Molchanov, H. Yin
PS3 scales CLIP-style vision pre-training to 4K resolution with near-constant cost for stronger high-resolution perception.
CVPR, 2025
NaVILA: Legged Robot Vision-Language-Action Model for Navigation
A. Cheng, Y. Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, X. Wang
NaVILA combines vision-language-action modeling with locomotion policies for embodied navigation.
arXiv, 2025
NVILA: Efficient Frontier Visual Language Models
Z. Liu, L. Zhu, B. Shi, Z. Zhang, Y. Lou, S. Yang, H. Xi, S. Cao, Y. Gu, D. Li, X. Li, Y. Fang, Y. Chen, C. Hsieh, D. Huang, A. Cheng, V. Nath, J. Hu, S. Liu, R. Krishna, D. Xu, X. Wang, P. Molchanov, J. Kautz, H. Yin, S. Han, Y. Lu
NVILA is an efficient frontier family of visual language models with strong training and inference efficiency.
CVPR, 2025
No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images
B. Ye, S. Liu, H. Xu, X. Li, M. Pollefeys, M. Yang, S. Peng
NoPoSplat reconstructs 3D Gaussian splats from sparse unposed images with a simple pose-free pipeline.
ICLR, 2025
COLMAP-Free 3D Gaussian Splatting
Y. Fu, S. Liu, A. Kulkarni, J. Kautz, A. Efros, X. Wang
COLMAP-Free 3D Gaussian Splatting enables view synthesis without SfM preprocessing by progressively growing 3D Gaussians from video.
CVPR, 2024
CosAE: Learnable Fourier Series for Image Restoration
S. Liu, S. De Mello, J. Kautz
CosAE is a cosine autoencoder for image restoration that uses Fourier coefficients for compact yet detail-preserving representations.
NeurIPS, 2024
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
A. Cheng, H. Yin, Y. Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, S. Liu
SpatialRGPT improves grounded spatial reasoning in VLMs through regional representation learning and depth-aware modeling.
NeurIPS, 2024
RegionGPT: Towards Region Understanding Vision Language Model
Q. Guo, S. De Mello, H. Yin, W. Byeon, K. Cheung, Y. Yu, P. Luo, S. Liu
RegionGPT enhances region-level captioning and understanding by improving spatial awareness in vision-language models.
CVPR, 2024
Dream-in-4D: A Unified Approach for Text- and Image-guided 4D Scene Generation
Y. Zheng, X. Li, K. Nagano, S. Liu, K. Kreis, O. Hilliges, S. De Mello
Dream-in-4D provides a unified framework for text-guided, image-guided, and personalized 4D scene generation.
CVPR, 2024
3D Reconstruction with Generalizable Neural Fields using Scene Priors
Y. Fu, S. De Mello, X. Li, A. Kulkarni, J. Kautz, X. Wang, S. Liu
NFP learns scene priors for fast, flexible 3D reconstruction and single-image novel-view synthesis.
ICLR, 2024
HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data
M. Zhang, Y. Fu, Z. Ding, S. Liu, Z. Tu, X. Wang
HOIDiffusion generates realistic 3D hand-object interaction data for perception tasks such as 6D pose estimation.
CVPR, 2024
TUVF: Learning Generalizable Texture UV Radiance Fields
A. Cheng, X. Li, S. Liu, X. Wang
The paper introduces TUVF, a method for learning generalizable texture UV radiance fields.
arXiv, 2023
Affordance diffusion: Synthesizing hand-object interactions
Y. Ye, X. Li, A. Gupta, S. De Mello, S. Birchfield, J. Song, S. Tulsiani, S. Liu
The paper proposes a method for interaction synthesis that addresses issues using diffusion models. They build upon the classic idea of disentangling where to interact (layout) from how to interact (content).
CVPR, 2023
Scraping Textures from Natural Images for Synthesis and Editing
X. Li, S. Liu, X. Wang, M. Yang, A. Efros
ECCV, 2022
Open-vocabulary panoptic segmentation with text-to-image diffusion models
J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, S. De Mello
We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation.
CVPR, 2023
CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs
J. Mu, S. De Mello, Z. Yu, N. Vasconcelos, X. Wang, J. Kautz, S. Liu
This work introduces Coordinate GAN (CoordGAN), a structure-texture disentangled GAN that learns a dense correspondence map for each generated image.
CVPR, 2022
GroupViT: Semantic Segmentation Emerges from Text Supervision
J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, X. Wang
This paper proposes a hierarchical Grouping Vision Transformer (GroupViT), which learns to group image regions into progressively larger arbitrary-shaped segments.
CVPR, 2022
Learning continuous environment fields via implicit functions
X. Li, S. De Mello, X. Wang, M. Yang, J. Kautz, S. Liu
ICLR, 2022
Autoregressive 3D Shape Generation via Canonical Mapping
A. Cheng, X. Li, S. Liu, M. Sun, M. Yang
The paper demonstrates a solution for 3D point cloud generation using transformers. The key idea is to decompose a point cloud into a sequence of semantically meaningful shape compositions, which are further encoded by an autoregressive model for point cloud generation.
ECCV, 2022
Learning Continuous Image Representation with Local Implicit Image Function
Y. Chen, S. Liu, X. Wang
The paper presents a method for learning continuous image representation with local implicit image function.
CVPR, 2021
Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
Z. Yu, R. Huang, W. Byeon, S. Liu, G. Liu, T. Breuel, A. Anandkumar, J. Kautz
NeurIPS, 2021
Video Autoencoder: self-supervised disentanglement of static 3D structure and motion
Z. Lai, S. Liu, A. Efros, X. Wang
This paper presents a video autoencoder for learning disentangled representations of 3D structure and camera pose from videos in a self-supervised manner.
ICCV, 2021
Learning 3D Dense Correspondence via Canonical Point Autoencoder
A. Cheng, X. Li, M. Sun, M. Yang, S. Liu
The paper presents a method for learning 3D dense correspondence using a canonical point autoencoder.
NeurIPS, 2021
Learning to track instances without video annotations
Y. Fu, S. Liu, U. Iqbal, S. De Mello, H. Shi, J. Kautz
CVPR, 2021
Contrastive syn-to-real generalization
W. Chen, Z. Yu, S. De Mello, S. Liu, J. Alvarez, Z. Wang, A. Anandkumar
ICLR, 2021
Self-supervised object detection via generative image synthesis
S. Mustikovela, S. De Mello, A. Prakash, U. Iqbal, S. Liu, T. Nguyen-Phuoc, C. Rother, J. Kautz
CVPR, 2021
Video matting via consistency-regularized graph neural networks
T. Wang, S. Liu, Y. Tian, K. Li, M. Yang
CVPR, 2021
Synthesizing long-term 3d human motion and interaction in 3d scenes
J. Wang, H. Xu, J. Xu, S. Liu, X. Wang
CVPR, 2021
Joint-task self-supervised learning for temporal correspondence
X. Li, S. Liu, S. De Mello, X. Wang, J. Kautz, M. Yang
This paper proposes to learn reliable dense correspondence from videos in a self-supervised manner.
NeurIPS, 2019
Self-supervised single-view 3D reconstruction via semantic consistency
X. Li, S. Liu, K. Kim, S. De Mello, V. Jampani, M. Yang, J. Kautz
ECCV, 2021
Hierarchical contrastive motion learning for video action recognition
X. Yang, X. Yang, S. Liu, D. Sun, L. Davis, J. Kautz
BMVC, 2021
Regularizing meta-learning via gradient dropout
H. Tseng, Y. Chen, Y. Tsai, S. Liu, Y. Lin, M. Yang
ACCV, 2020
Self-supervised viewpoint learning from image collections
S. Mustikovela, V. Jampani, S. De Mello, S. Liu, U. Iqbal, C. Rother, J. Kautz
CVPR, 2020
Online Adaptation for Consistent Mesh Reconstruction in the Wild
X. Li, S. Liu, S. De Mello, K. Kim, X. Wang, M. Yang, J. Kautz
NeurIPS, 2020
Learning Propagation for Arbitrarily-structured Data
S. Liu, X. Li, V. Jampani, S. De Mello, J. Kautz
ICCV, 2019
Learning Linear Transformations for Fast Image and Video Style Transfer
X. Li, S. Liu, J. Kautz, M. Yang
CVPR, 2019
Few-Shot Viewpoint Estimation
H. Tseng, S. De Mello, J. Tremblay, S. Liu, S. Birchfield, M. Yang, J. Kautz
BMVC, 2019
Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments
X. Li, S. Liu, K. Kim, X. Wang, M. Yang, J. Kautz
CVPR, 2019
SCOPS: Self-Supervised Co-Part Segmentation
W. Hung, V. Jampani, S. Liu, P. Molchanov, M. Yang, J. Kautz
CVPR, 2019
Switchable temporal propagation network
S. Liu, G. Zhong, S. De Mello, J. Gu, V. Jampani, M. Yang, J. Kautz
ECCV, 2018
Rendering Portraitures from Monocular Camera and Beyond
X. Xu, D. Sun, S. Liu, W. Ren, Y. Zhang, M. Yang, J. Sun
ECCV, 2018
Context-Aware Synthesis and Placement of Object Instances
D. Lee, S. Liu, J. Gu, M. Liu, M. Yang, J. Kautz
NeurIPS, 2018
Learning Video-Story Composition via Recurrent Neural Network
G. Zhong, Y. Tsai, S. Liu, Z. Su, M. Yang
WACV, 2018
Learning Dual Convolutional Neural Networks for Low-Level Vision
J. Pan, S. Liu, D. Sun, J. Zhang, Y. Liu, J. Ren, Z. Li, J. Tang, H. Lu, Y. Tai, M. Yang
CVPR, 2018
Learning Affinity via Spatial Propagation Networks
S. Liu, S. De Mello, J. Gu, G. Zhong, M. Yang, J. Kautz
NeurIPS, 2017
Face Parsing via Recurrent Propagation
S. Liu, D. Yi, Z. Lei, S. Li
BMVC, 2017
Unsupervised Domain Adaption for Face Recognition in Unlabeled Videos
K. Sohn, S. Liu, G. Zhong, X. Yu, M. Yang, M. Chandraker
ICCV, 2017
Learning to Segment Instances in Videos with Spatial Propagation Network
J. Cheng, S. Liu, S. De Mello, J. Gu, J. Kautz, S. Wang, M. Yang
CVPR workshop, 2017
Generative Face Completion
Y. Li, S. Liu, J. Yang, M. Yang
CVPR, 2017
Learning Recursive Filters for Low-Level Vision via a Hybrid Neural Network
S. Liu, J. Pan, M. Yang
ECCV, 2016
Deep Cascaded Bi-Network for Face Hallucination
S. Zhu, S. Liu, C. Loy
ECCV, 2016
Multi-Objective Convolutional Learning for Face Labeling
S. Liu, J. Yang, C. Huang, M. Yang
CVPR, 2015
Compressed Face Hallucination
S. Liu, M. Yang
ICIP, 2014
Structured Face Hallucination
C. Yang, S. Liu, M. Yang
CVPR, 2013