A month ago, Mistral conquered speech-to-text with its Voxtral Transcribe 2 model family. Now, the AI lab is flipping the script with its first text-to-speech model.

On Thursday, Mistral launched its Voxtral TTS, an open-weight text-to-speech model that is three times smaller than the industry standard, more natural-sounding, and multilingual. The company claims that it's the best open-source text-to-speech model released to-date.

The model was built to be as natural-sounding as possible by understanding the context of what people say and modeling how people naturally speaks, including pauses, rhythm, intonation, emotion and more. Some other highlights include:

• Support for nine languages: English, French, German, Spanish, Dutch, Portuguese, Italian, Hindi, and Arabic. It can accept input in one language and output in another.
• Speed: Adapts to voices with references of just five seconds and offers low-latency, taking a voice sample of 10 seconds and 500 characters, and producing audio within 90 milliseconds
• On-device: 3 billion parameter model that fits in a smartwatch, a phone, or a laptop
• Human evaluations: In comparative human evaluation of Voxtral TTS and ElevenLabs v2.5 Flash, Mistral’s model achieved superior naturalness

While there are many other text-to-speech models on the market, Pierre Stock, VP of science and the first employee hired at Mistral AI, said the model’s overall efficiency sets it apart.

“The model is optimal for both efficiency — including the cost, the speed, the ease of deployment – and state-of-the-art, conversational capability in which the model feels both super natural and conversational,” Stock told The Deep View.

Some use cases, Mistral notes, include customer support, sales and marketing, real-time translation and personal voice agents. The model is available to test out in Mistral Studio playground, and a model with several reference voices is available as open weights on Hugging Face.

Our Deeper View

Since the dawn of computing, users have been dependent on a keyboard to get tasks done. As a result, one of the most exciting applications of generative AI is the ability to interact with applications more intuitively and hands-free via multimodal understanding, including voice. However, this poses many challenges, as seen by the failure of AI voice assistants such as Alexa and Siri to deliver really conversational and helpful experiences. To bridge the gap, more developments are needed in text-to-speech and speech-to-text, and Mistral is doing just that with this new model.