Nuance Labs raises $50M Series A to fix AI avatar awkwardness

The company said existing investors Accel and South Park Commons returned for the round, while Nvidia Corp. and Define Ventures joined as new investors. Nuance is developing the model to give AI avatars more emotional intelligence, co-founder and CEO Fangchang Ma told Business Insider in an interview.

Ma, a former AI researcher at Apple Inc., said existing voicebots and AI avatars are built by combining separate models: a voice-to-text model, a large language model that responds to the text, and a text-to-voice model that turns responses into audio. AI avatars add a fourth step to animate a face to match the words. Those handoffs create delayed responses and strip away the feeling of talking with a human, he said, leaving avatars that appear frozen while a person is speaking. “If you use existing AI avatars, when they’re listening, they’re not reacting, or they’re just doing random things,” Ma said.

Nuance Labs says it has developed a model that perceives and generates responses through one full-duplex system. The system takes in audiovisual perception of the user’s stream and streams back an audiovisual response, allowing the avatar to react to verbal and non-verbal cues as the person speaks. Ma said the model can perceive words, gaze, gestures, tone and timing, and respond in real time with facial and vocal expressions. It improves by studying how people behave during its conversations, according to Ma, and can show understanding while the user is still speaking. “We decided to build this on one system, on one model,” he said. “There’s audio video in and audio video out.”

The company has posted a demo of the model and described it as a work in progress. Nuance is targeting use cases such as sales and customer service, coaching, professional training and education. Those scenarios could include a video call to learn a new language or practice an interview, Ma said.

Nuance has not launched any product yet. It is still refining the model to make conversations more fluid and hopes to have its first research preview ready to roll out to the public later this year, Ma said. The new funding will accelerate development and allow the company to hire more researchers.

Lightspeed Venture Partners’ Nnamdi Iregbulem said there is a “massive opportunity” to fix the way people interact with AI systems. He said the founding team has a rare mix of technical talent and operating excellence, and has turned that into a single model that follows users in real time. “We can see this becoming a foundational layer for AI products everywhere, and this is the team to build it,” he said.

Editor's Summary

Nuance Labs has raised $50 million in Series A funding led by Lightspeed Venture Partners to build a human foundation model for more natural AI avatars. The company says its full-duplex audiovisual model can respond to verbal and non-verbal cues in real time, with target uses in sales, customer service, coaching, training and education. Nuance has not launched a product and plans a research preview later this year.