Hello there!
I am a research scientist at NVIDIA AI. My primary focus is to develop generally capable autonomous agents. To tackle this grand challenge, my research efforts span foundation models, policy learning, robotics, multimodal learning, and large-scale systems. I obtained my Ph.D. degree at Stanford Vision Lab, advised by Prof. Fei-Fei Li. Previously, I did research internships at NVIDIA, Google Cloud AI, OpenAI, Baidu Silicon Valley AI Lab, and Mila-Quebec AI Institute. I was the Valedictorian of Class 2016 and a recipient of the Illig Medal at Columbia University. Feel free to follow me on for latest research announcements and team updates!
✨ News ✨
Nov. 2022:
has won 🎉 Outstanding Paper Award 🎉 at NeurIPS [announcement]! I am also invited as a Speaker at the 1st NeurIPS Foundation Model for Decision Making (FMDM) workshop — please join us at New Orleans!Oct. 2022: We trained a transformer called VIMA that ingests multimodal prompt and outputs controls for a robot arm. A single agent is able to solve visual goal, one-shot imitation from video, novel concept grounding, visual constraint, etc. Strong scaling with model capacity and data! We open-source everything: code, pretrained models, training dataset, and simulation benchmark. Check out our paper and website!
Jun. 2022:
has launched! MineDojo is a new framework for building generally capable agents with internet-scale knowledge in Minecraft. Paper, code, and databases are all open access. Check it out today!
- Foundation Models
- General-purpose Agents
- Reinforcement Learning
- Robotics
- Multimodal Learning
- Large-scale AI Systems
Ph.D. in Computer Science, 2016 - 2021
Stanford University
B.S. in Computer Science, 2012 - 2016
Columbia University, Summa Cum Laude
Valedictorian of Class 2016
Columbia University
Featured
Research Highlights
Publications
Visit my Google Scholar page for a comprehensive listing!
Experience
- Conducting bleeding edge research on foundation models for general-purpose autonomous agents.
- Leading the MineDojo effort for open-ended agent learning in Minecraft.
- Mentoring interns on diverse research topics.
- Collaborating with universities: Stanford, Berkeley, Caltech, MIT, UW, etc.
- Doctoral advisor: Prof. Fei-Fei Li.
- Ph.D. Thesis “Training and Deploying Visual Agents at Scale”.
- Co-designed World of Bits, an open-domain platform for teaching AI to use the web browser. World of Bits was part of the OpenAI Universe initiative.
- Paper published at ICML 2017.
- Systematically analyzed and proposed novel variants of the Ladder Network, a strong semi-supervised deep learning technique.
- Mentored by Turing Award Laureate Yoshua Bengio.
- Paper published at ICML 2016.
- Co-developed DeepSpeech 2, a large-scale end-to-end system that achieved world-class performance on English and Chinese speech recognition.
- Mentored by Dario Amodei, Adam Coates, and Andrew Ng.
- Paper published at ICML 2016.
- DeepSpeech and derivative works have been featured in various media: MIT Technology Review, TechCrunch, Forbes, NPR, VentureBeats, etc.
- Columbia NLP Group, advised by Prof. Michael Collins. Studied kernel methods for speech recognition. Paper published in Journal of Machine Learning Research.
- Columbia Vision Lab, advised by Prof. Shree Nayar. Implemented a computer vision system in Matlab to infer astrophysics parameters from galactic images.
- Columbia CRIS Lab, advised by Prof. Venkat Venkatasubramanian. Developed ML and NLP techniques to automate ontology curation for pharmaceutical engineering. Paper published in Computers & Chemical Engineering.