Foundation Model

Eureka

GPT-4 writes reward functions to teach a 5-finger robot hand how to do extremely dexterous tasks like pen spinning.

Voyager

LLM-powered agent that masters Minecraft by in-context lifelong learning.

VIMA

Multimodal LLM for robot manipulation; unifies diverse robotics tasks in a single prompting framework.

VIMA: General Robot Manipulation with Multimodal Prompts

✨ Oral Presentation ✨. 1st NeurIPS Workshop on Foundation Models for Decision Making (FMDM), 2022

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, Linxi "Jim" Fan

VIMA: General Robot Manipulation with Multimodal Prompts

MineDojo

✨NeurIPS Outstanding Paper Award✨. Large-scale open-ended agent learning framework in Minecraft.

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

💫✨ Outstanding Paper Award ✨💫. Neural Information Processing Systems (NeurIPS) Dataset & Benchmark, 2022

Linxi "Jim" Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, Anima Anandkumar

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Pre-Trained Language Models for Interactive Decision-Making

✨ Oral Presentation ✨. Neural Information Processing Systems (NeurIPS), 2022

Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi "Jim" Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, Yuke Zhu

Pre-Trained Language Models for Interactive Decision-Making

MetaMorph

Learning universal controller over diverse morphologies with transformers

Mar 22, 2022

MetaMorph: Learning Universal Controllers with Transformers

International Conference on Learning Representations (ICLR), 2022

Agrim Gupta, Linxi "Jim" Fan, Surya Ganguli, Li Fei-Fei

MetaMorph: Learning Universal Controllers with Transformers

Training and Deploying Visual Agents at Scale

My Stanford Ph.D. thesis advised by Fei-Fei Li. Learn the rules like a pro, so you can break them like an artist — Picasso.

Sep 27, 2021

Training and Deploying Visual Agents at Scale