Hello, I'm Sifei Liu

I currently hold the position of a staff-level Senior Research Scientist at NVIDIA, where I am part of the LPR team led by Jan Kautz. My work primarily revolves around the development of generalizable visual representation learning for images, videos, and 3D content. Prior to this, I pursued my Ph.D. at the VLLAB, under the guidance of Ming-Hsuan Yang.

Over the years, I’ve been fortunate to receive several prestigious awards and recognitions. In 2013, I was honored with the Baidu Graduate Fellowship. This was followed by the NVIDIA Pioneering Research Award in 2017, and the Rising Star EECS accolade in 2019. Additionally, I was nominated for the VentureBeat Women in AI Award in 2020.

CV Scholar Github Nvidia Twitter

News

May 2023: We’ve released the code for Affordance Diffusion.
April 2023: We will host the Learning with Noisy and Unlabeled Data for Large Models beyond Categorization Tutorial at ICCV 2023. See you in Paris.
Mar 2023: Four papers accepted at CVPR on representation learning for (i) Affordance generation, (ii) Open-vocabulary discriminative network (Highlight top 2.5% paper), (iii) zero-shot pose transfer, and (iv) self-supervised 3D reconstruction.

Recent Research

Full publications can be found at Google Scholar and CV

TUVF: Learning Generalizable Texture UV Radiance Fields

An-Chieh Cheng, Xueting Li, Sifei Liu, Xiaolong Wang

arXiv 2023

The paper introduces TUVF, a method for learning generalizable texture UV radiance fields.

Project Page PDF arXiv Code

Affordance diffusion: Synthesizing hand-object interactions

Yufei (Judy) Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, Sifei Liu

CVPR 2023

The paper proposes a method for interaction synthesis that addresses issues using diffusion models. They build upon the classic idea of disentangling where to interact (layout) from how to interact (content).

Project Page PDF arXiv Code

Open-vocabulary panoptic segmentation with text-to-image diffusion models

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello

CVPR 2023

We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation.

Project Page PDF arXiv Code

CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs

Jiteng Mu, Shalini De Mello, Zhiding Yu, Nuno Vasconcelos, Xiaolong Wang, Jan Kautz, Sifei Liu

CVPR 2022

This work introduces Coordinate GAN (CoordGAN), a structure-texture disentangled GAN that learns a dense correspondence map for each generated image.

Project Page PDF arXiv Code

GroupViT: Semantic Segmentation Emerges from Text Supervision

Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas B Breuel, Jan Kautz, Xiaolong Wang

CVPR, 2022

This paper proposes a hierarchical Grouping Vision Transformer (GroupViT), which learns to group image regions into progressively larger arbitrary-shaped segments.

Project Page PDF arXiv Code

Autoregressive 3D Shape Generation via Canonical Mapping

An-Chieh Cheng, Xueting Li, Sifei Liu, Min Sun, Ming-Hsuan Yang

ECCV 2022

The paper demonstrates a solution for 3D point cloud generation using transformers. The key idea is to decompose a point cloud into a sequence of semantically meaningful shape compositions, which are further encoded by an autoregressive model for point cloud generation.

Project Page PDF arXiv Code

Learning Continuous Image Representation with Local Implicit Image Function

Yinbo Chen, Sifei Liu, Xiaolong Wang

CVPR, 2021

The paper presents a method for learning continuous image representation with local implicit image function.

Project Page PDF arXiv Code

Video Autoencoder: self-supervised disentanglement of static 3D structure and motion

Zihang Lai, Sifei Liu, Alexei A. Efros, Xiaolong Wang

ICCV, 2021

This paper presents a video autoencoder for learning disentangled representations of 3D structure and camera pose from videos in a self-supervised manner.