Dong Won Lee

Hi, My name is Dong Won. My advisors and friends usually call me “Don" :)

I’m a PhD Student at MIT working on the foundations of multimodal social agents, where I focus on developing machine learning methods and evaluation for social intelligence. I am extremely grateful to be co-advised by Professor Cynthia Breazeal and Dr. Hae Won Park in the Personal Robots Group at the Media Lab and Professor Louis-Philippe Morency at the Language Technologies Institute at CMU. Prior to MIT, I graduated with a M.S in Machine Learning and a B.S in Statistics and Machine Learning from Carnegie Mellon University.

Email  /  CV  /  Google Scholar  /  Linkedin  /  Twitter  /  Github  /  Medium

We introduce a framework which uses a frozen large language model (LLM) to decompose global session-level feedback into fine-grained turn-level rewards for dialogue agents. Our method works in both text-only and multimodal settings (using cues like pitch and gaze), enabling reinforcement learning without dense supervision. The resulting reward models improve dialogue quality in human evaluations, showing that LLMs can effectively serve as general-purpose reward decomposers.

We introduce a large-scale collection of datasets of real-world human-robot interaction videos with 10K+ annotations to benchmark AI models' ability to identify and reason social interactions, providing a foundation for advancing socially intelligent AI.

We show that augmenting older models like GPT-3.5 with reasoning strategies (e.g. DoT, CoT, Self-Consistency) outperforms state-of-the-art pre-trained models (e.g., DeepSeek-R1, o1) in recognizing and reframing unhelpful thoughts.

We introduce an approach named GELI, which automatically decomposes a single Global Explicit post-interaction score while incorporating Local Implicit feedback from multimodal signals to adapt a language model to become more conversational.

We introduce the Multimodal Lecture Presentations dataset and PolyViLT a multimodal transformer trained with a multi-instance learning loss. We propose a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational content.

We model the Historical, Intra-and Inter-personal (HIINT) Dynamics in conversation by incorporating memory modules in the Cross-person Memory Transformer to address temporal coherence and better represent the context of conversational behaviors.

We introduce a new transformer architecture to model contingent behaviors in multiparty group conversations.

We propose a new approach in crossmodal generative modeling in low-resource settings in the hopes to to create a personalized gesture generation model (e.g. as part of a personalized avatar) with limited data from a new speaker.

We propose a new crossmodal contrastive learning loss to encourage a stronger grounding between gestures and spoken language.

We study relationships between spoken language and co-speech gestures to account for the long tail of text-gesture distribution.

We propose a new style transfer model to learn individual styles of speaker's gestures.

PATS was collected to study correlation of co-speech gestures with audio and text signals. The dataset consists of a diverse and large amount of aligned pose, audio and transcripts.

I have been blessed to meet amazing mentors who have guided me to become a better researcher (and more importantly, a good person). I believe that I can only repay what they've done for me by assisting others in their journey in any way I can. Please don't hesitate to reach out!

• Ben Eysenbach - CMU
• Chaitanya Ahuja - CMU
• Cynthia Breazeal - MIT
• David Kosbie - CMU
• Hae Won Park - MIT
• Louis-Phillipe Morency - CMU
• Mark Stehlik - CMU
• Paul Pu Liang - CMU
• Roz Picard - MIT
• Yoon Kim - MIT
• Sid Sen - Microsoft Research

Website Credits Here: source code