Hi! I'm Yujie Alice Lu.

social-16px_logo-github ui-16px-1_email-84 Created with Sketch. social-16px_logo-linkedin Created with Sketch. social-16px_logo-linkedin Created with Sketch. social-16px_logo-instagram Created with Sketch.

I am a third year CS PhD at UC Santa Barbara, advised by William Wang at Natural Language Processing Group. I obtained my bachelor’s degree from Chu Kochen Honors College, Zhejiang University.

My research is focused on developing advanced multimodal models capable of enhancing their intelligence through interactions with humans and the real world.

News! Check out our Vision Arena demo on HuggingFace! You can directly chat with or compare the large multimodal models (GPT4-V, Gemini-Pro Vision, LLaVA-NEXT 34b, QwenVL Chat, etc.) side by side easily!

New Preprints

VIM: Probing Multimodal Large Language Models for Visual Embedded Instruction Following

Yujie Lu*, Xiujun Li*, William Yang Wang, Yejin Choi

Gpt-4v (ision) as a generalist evaluator for vision-language tasks

Xinlu Zhang*, Yujie Lu*, Weizhi Wang*, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, Linda Ruth Petzold

Multimodal Procedural Planning via Dual Text-Image Prompting

Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, William Yang Wang

Selected Publications

LAD: Language Augmented Diffusion for Reinforcement Learning

Edwin Zhang, Yujie Lu, William Yang Wang, Amy Zhang

International Conference on Learning Representations (ICLR), 2024 | NeurIPS Workshop LaReL, 2022

Imagenhub: Standardizing the evaluation of conditional image generation models

Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, Wenhu Chen

International Conference on Learning Representations (ICLR), 2024

Empowering Psychotherapy with Large Language Model: Cognitive Distortion Detection through Diagnosis of Thought Prompting

Zhiyu Chen, Yujie Lu, William Yang Wang

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation

Wanrong Zhu, Xinyi Wang, Yujie Lu, Tsu-Jui Fu, Xin Eric Wang, Miguel Eckstein, William Yang Wang

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

Let's Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought

Vaishnavi Himakunthala, Andy Ouyang, Daniel Philip Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, William Yang Wang

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation

Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, William Yang Wang

Conference on Neural Information Processing Systems (NeurIPS), 2023

Few-Shot Document-Level Event Argument Extraction

Xianjun Yang, Yujie Lu, Linda Petzold

Annual Meeting of the Association for Computational Linguistics (ACL), 2023

Neuro-Symbolic Causal Procedural Planning with Commonsense Prompting

Yujie Lu, Weixi Feng, Wanrong Zhu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, William Yang Wang

International Conference on Learning Representations (ICLR), Spotlight, 2023

WikiWhy: Answering and Explaining Cause-and-Effect Questions

Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, William Yang Wang

International Conference on Learning Representations (ICLR), Oral, 2023

Visualize Before You Write: Imagination-Guided Open-Ended Text Generation

Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, William Yang Wang

The European Chapter of the Association for Computational Linguistics (EACL), 2023

Breaking Out of the Ivory Tower: A Large-scale Analysis of Patent Citations to HCI Research

Hancheng Cao, Yujie Lu, Yuting Deng, Daniel McFarland, Michael S. Bernstein

The ACM CHI Conference on Human Factors in Computing Systems (CHI), Best Paper, 2023

ULN: Towards Underspecified Vision-and-Language Navigation

Weixi Feng, Tsu-Jui Fu, Yujie Lu, William Yang Wang

The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022

Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis

Wenda Xu, Yi-Lin Tuan, Yujie Lu, Michael S. Saxon, Lei Li, William Yang Wang

The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022

MIC: Model-agnostic Integrated Cross-channel Recommenders

Yujie Lu*, Ping Nie*, Shengyu Zhang, Ming Zhao, Ruobing Xie, William Yang Wang, Yi Ren

The Conference on Information and Knowledge Management (CIKM), Oral Presentation, 2022 (* indicates equal contribution)

Imagination-Augmented Natural Language Understanding

Yujie Lu, Wanrong Zhu, Xin Eric Wang, Miguel Eckstein, William Yang Wang,

North American Chapter of the Association for Computational Linguistics (NAACL), Oral Presentation, 2022

Re4: Learning to Re-contrast, Re-attend, Re-construct for Multi-interest Recommendation

Shengyu Zhang, Lingxiao Yang, Dong Yao, Yujie Lu, Fuli Feng, Zhou Zhao, Tat-Seng Chua, Fei Wu

The Web Conference (WWW), 2022

Future-Aware Diverse Trends Framework for Recommendation

Yujie Lu, Shengyu Zhang, Yingxuan Huang, Luyao Wang, Xinyao Yu, Zhou Zhao, Fei Wu

The Web Conference (WWW), 2021

CLOUD: Contrastive Learning of Unsupervised Dynamics

Yujie Lu*, Jianren Wang*, Hang Zhao

The Conference on Robot Learning (CoRL), 2020 (* indicates equal contribution)

Experience

  • 2023, Research Intern, Amazon AWS AI
    Advisor: Zhaowei Cai, Yonatan Dukler, Yusheng Xie, Hao Yang, Zhuowen Tu, Stefano Soatto
  • 2022, Research Intern, Microsoft Research
    Advisor: Oriana Riva
  • 2019-2021, Applied Researcher, Tencent
  • 2019, Research Assistant, Digital Media Computing & Design Lab, Zhejiang University
    Advisor: Zhou Zhao
  • 2019, Research Intern, Robotics Institute, Carnegie Mellon University
    Advisor: David Held
  • 2018, Research Assistant, Massachusetts General Hospital
  • 2018, Research Intern, FABU Technology Co., Ltd.
    Advisor: Deng Cai
  • 2018, Software Engineer Intern, Tencent
  • 2017, Game Developer Intern, NetEase
  • 2016, Research Assistant, Visual Intelligence and Pattern Analysis, Zhejiang University
    Advisor: Mingli Song

    • Talks

  • Invited Talk at MLNLP. 02/2023 [Slides and video to be released.]
  • Paper Presentation at CIKM. 10/2022 [Slides to be released.]
  • Paper Presentation at NAACL. 07/2022 [Slides]
  • Invited Talk at When CV Meets NLP. 05/2022 [Slides and video to be released.]
  • Paper Presentation at WWW 2021. 2021/04 [Video]

    • Services

  • Organizer: SoCalNLP 2022 [Website].
  • Program Committee: NeurIPS, ICLR, ICML, ACL, EMNLP, NAACL, EACL, ECCV, ICCV, AAAI, ICASSP.
  • Volunteer and NSF Travel Award, CIKM.
  • Robert Noyce Fellow.
  • UCSB Faculty Recruitment CS Grad Representative.

    • Dancer

    Soccer Player

    Drummer