UMD Researchers Develop SteerDuplex for Enhanced Conversational AI Control
Alum Utkarsh Tyagi (M.S. ’25, computer science) from the University of Maryland, advised by Distinguished University Professor Dinesh Manocha, led the development of SteerDuplex, a speech-to-speech AI model designed to adjust its speaking behavior in response to user instructions while maintaining real-time conversations. The project, a collaboration with Scale AI and UMD's GAMMA Lab, also introduced SteerBench, a benchmark for evaluating how well voice-based AI systems respond to instructions regarding tone, speaking speed, persona, and vocal style. SteerDuplex was developed using supervised training to introduce new capabilities, followed by a two-stage reinforcement learning process to improve conversational timing and handle interruptions. This approach addressed the challenge of models forgetting existing capabilities when new ones are introduced. The research paper indicates that supervised training improved the model's average pass rate on audio-steering tasks by 44.5 percentage points compared to t...
