About Me

Hi! I am a second-year Ph.D. student of MMLab, The Chinese University of Hong Kong, supervised by Prof. Dahua Lin. Prior to this, I received my Bachelor’s degree in Computer Science and Technology from Chu Kochen Honors College (Pursuit Science Class) at Zhejiang University in 2022. In ZJU, I was fortunate to be supervsied by Prof. Guofeng Zhang, at the State Key Lab of CAD&CG.

My reserach interest lies in 3D understanding for robotics (spatial AI). My long-term goal is to develop an intellectual model capable of universally perceiving and reasoning about our 3D physical world—primarily through visual information and various multi-modal data—that can be deployed on robots or wearable devices (like AR glasses). This pursuit aims to augment human intelligence and to benefit society at large. Achieving this goal involves overcoming three core challenges:

Output: defining what perceptions are essential for the intelligent agent.
Representation: establishing a viable world (scene) representation.
Data: addressing the difficulty in acquiring necessary training data.

I am generally interested in research that seeks solutions to these problems. Currently, my exploration focuses on leveraging the successes of foundation models in both vision and language, as well as the advancements in generative modeling to address these issues. If you share my interests, have articles to recommend that are helpful, or simply have any queries, please do not hesitate to contact me. :)

News

[Sep. 2023] Our HC-Net is accepted by NeurIPS 2023! 🎉
[Aug. 2023] We release PointLLM, a multi-modal large language model capable of understanding point clouds! Try our demo here. 🤗
[Aug. 2023] We release HC-Net, a SOTA fine-grained cross-view geo-localization model! Demo here. 📊
[Mar. 2023] Our paper on MV-JAR, a LiDAR-based self-supervised pre-training method along with a new data-efficient benchmark, has been accepted by CVPR 2023. 🎉
[Jun. 2022] Graduated from Zhejiang University. Forever cherishing the memories from ZJU. 🎓
[Apr. 2022] Honored to receive the Hong Kong PhD Fellowship (HKPFS) and the CUHK Vice-Chancellor’s PhD Scholarship. Deeply grateful for the recognition! 🏆

Education


The Chinese University of Hong Kong (CUHK): August 2022 - June 2026 (Expected); Ph.D. in Information Engineering

Zhejiang University (ZJU): September 2018 - June 2022; B.Eng. in Computer Science and Technology

Publications

3D Perception and Understanding

PointLLM: Empowering Large Language Models to Understand Point Clouds: Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang^#, Dahua Lin; arXiv Preprint, 2023; [Paper] [Code] [Project] [Demo] [Cite]
Self-Supervised 3D Representation Learning

MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Training: Runsen Xu, Tai Wang, Wenwei Zhang, Runjian Chen, Jinkun Cao, Jiangmiao Pang^#, Dahua Lin; Computer Vision and Pattern Recognition, CVPR 2023; [Paper] [Code] [Video] [Slides] [Cite]

COˆ3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving: Runjian Chen, Yao Mu, Runsen Xu, Wenqi Shao, Chenhan Jiang, Hang Xu, Zhenguo Li, Ping Luo; International Conference on Learning Representations, ICLR 2023; [Paper] [Code] [Cite]
Robot Localization and Navigation

Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator: Xiaolong Wang, Runsen Xu, Zuofan Cui, Zeyu Wan, Yu Zhang; Neural Information Processing Systems, NeurIPS 2023; [Paper] [Code] [Demo] [Cite]

RNIN-VIO: Robust Neural Inertial Navigation Aided Visual-Inertial Odometry in Challenging Scenes: Danpeng Chen, Nan Wang, Runsen Xu, Weijian Xie, Hujun Bao, Guofeng Zhang; International Symposium on Mixed and Augmented Reality, ISMAR 2021, Oral Presentation; [Paper] [Code] [Project] [Cite]

Selected Awards

Hong Kong PhD Fellowship (most prestigious scholarship for Ph.D. studies in Hong Kong), 2022
CUHK Vice-Chancellor’s PhD Scholarship, 2022
Outstanding Graduates of Zhejiang University, 2022
Outstanding Undergraduate Thesis of College of Computer Science and Technology, Zhejiang University, 2022
National Scholarship (highest honor nationwide for Chinese undergraduates), 2019

Academic Services

Reviewer: ICRA 2024, CVPR 2024, ECCV 2024, ACM MM 2024

Teaching

IERG4998: Final Year Project, Spring 2023
IERG4998: Final Year Project, Fall 2022