Dong Won Lee

Hi, My name is Dong Won. My advisors and friends usually call me “Don" :)

I’m a PhD Student at MIT working on the foundations of multimodal social agents, where I focus on developing machine learning methods and evaluation for social intelligence. I am extremely grateful to be co-advised by Professor Cynthia Breazeal in the Personal Robots Group at the Media Lab and Professor Louis-Philippe Morency at the Language Technologies Institute at CMU. Prior to MIT, I graduated with a Master's in Machine Learning and Bachelor's in Statistics and Machine Learning from Carnegie Mellon University.

Email  /  CV  /  Google Scholar  /  Linkedin  /  Twitter  /  Github  /  Medium

We introduce an approach named GELI, which automatically decomposes a single Global Explicit post-interaction score while incorporating Local Implicit feedback from multimodal signals to adapt a language model to become more conversational.

We introduce the Multimodal Lecture Presentations dataset and PolyViLT a multimodal transformer trained with a multi-instance learning loss. We propose a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational content.

We model the Historical, Intra-and Inter-personal (HIINT) Dynamics in conversation by incorporating memory modules in the Cross-person Memory Transformer to address temporal coherence and better represent the context of conversational behaviors.

We introduce a new transformer architecture to model contingent behaviors in multiparty group conversations.

We propose a new approach in crossmodal generative modeling in low-resource settings in the hopes to to create a personalized gesture generation model (e.g. as part of a personalized avatar) with limited data from a new speaker.

We propose a new crossmodal contrastive learning loss to encourage a stronger grounding between gestures and spoken language.

We study relationships between spoken language and co-speech gestures to account for the long tail of text-gesture distribution.

We propose a new style transfer model to learn individual styles of speaker's gestures.

PATS was collected to study correlation of co-speech gestures with audio and text signals. The dataset consists of a diverse and large amount of aligned pose, audio and transcripts.

I have been blessed to meet amazing mentors who have guided me to become a better researcher (and more importantly, a good person). I believe that I can only repay what they've done for me by assisting others in their journey in any way I can. Please don't hesitate to reach out!

• Ben Eysenbach - CMU
• Chaitanya Ahuja - CMU
• Cynthia Breazeal - MIT
• David Kosbie - CMU
• Hae Won Park - MIT
• Louis-Phillipe Morency - CMU
• Mark Stehlik - CMU
• Paul Pu Liang - CMU
• Roz Picard - MIT
• Yoon Kim - MIT
• Sid Sen - Microsoft Research

Website Credits Here: source code