India– ElevenLabs, the AI research and product company, today announced new model and agent capabilities designed to help companies shape AI systems for their specific context.
“Within a few years, conversational agents will become critical infrastructure for every enterprise, as essential as cloud or payments,” said Mati Staniszewski, co-founder and CEO of ElevenLabs. “But if AI is going to work for us, and alongside us, it will need to interact naturally and intuitively with people in many different settings. Our goal is to build the interaction layer for AI, and today’s announcements make it possible for businesses to use their data to craft experiences that resonate in their unique circumstances.”
Fine-tuning and orchestrating models for highly customized interactions
Over the last month, ElevenLabs researchers have shipped three new voice models, all of which are leading independent benchmarks. Scribe v3, a new speech-to-text transcription model that captures what people say; Eleven v4, a text-to-speech model that delivers expressive speech that reads the tone, pacing, and context of a script; and Eleven Realtime, a new class of speech-to-speech models for fast and accurate voice agents.
To help customers deliver accurate, useful interactions in every part of the economy, ElevenLabs is building more domain-optimized models. It launched Scribe v3 Medical, its healthcare-optimized transcription model, and set out plans to build more models for industries such as financial services.
Building on their expertise in audio, ElevenLabs researchers will expand their work to cover all kinds of models that power interactions. Select enterprises will be able to work with the research team to fine-tune audio models and LLMs, tailoring model and agent output for their own context. Using data from live deployments and reinforcement learning, businesses can continually refine the models to improve metrics such as customer satisfaction score (CSAT) or reliability.
Speaking on stage, Staniszewski said: “We believe in a future where each of you will have your own RL stack. It’s one of the best ways to make your data work for you, and extend your own unique advantage.”
With companies increasingly concerned about balancing model performance and token costs, ElevenLabs also unveiled Interaction Engine, a new system to orchestrate voice models and LLMs across a conversation.
Not every agentic task requires a frontier model at every step to succeed. Interaction Engine can help bring in the best voice models and LLMs for a specific use case, balancing performance and reliability, with cost. “Interaction Engine orchestrates all this under the hood, so you can be confident you’re delivering the right outcome for the customer, without wasting money on tokens,” Staniszewski said.
Building high EQ agents that work for and alongside people
Getting AI to interact effectively with a company’s customers and employees takes three things: communication that feels natural, context about who it’s serving, and clear collaboration between people and agents on tasks, escalation, and accountability.
ElevenLabs builds the interaction layer that delivers all three, from the models that power conversation to the agents that complete work. ElevenAgents is the company’s platform for voice and chat agents that can be used in any part of a business — from sales and support, to recruitment and operations.
Alongside major new model capabilities, the company announced ElevenAgents upgrades focused on how people and AI interact.
- Memory now gives ElevenAgents the full context of every returning customer, recalling what they’ve said before, from a new shirt size to a pending move, so no one has to repeat themselves. This means enterprises can deliver more personalized customer service that gets customers the results they need faster and more accurately.
- Routines, the new automation layer in ElevenAgents, runs multi-step work before, during, and after every conversation, helping companies process leads and follow-up actions quickly. The result is agents that respond quickly and act proactively.
- Architect, a built-in expert that helps teams build and improve agents just by talking or typing, with every proposed change drafted, tested in simulation, and held for your approval.
Alongside upgrades to ElevenAgents, the company also opened up early access for ElevenAI, a new app that acts as an AI teammate for enterprise teams. It can turn spoken conversations, including in-person meetings, into notes and next steps, with notice and consent mechanisms
built in. It can also give live assistance during a call, sharing context and enhancing collaboration between people and agents.
Accelerating deployment across the economy
To accelerate deployment in large complex enterprises, ElevenLabs launched its new Partner Program with more than 550 partners in 44 countries. These are trusted integrators, consultancies, and software vendors that handle integrations into companies’ existing systems or work alongside ElevenLabs’ forward-deployed engineers on complex deployments.
As part of the program, ElevenLabs announced an alliance with McKinsey to help organizations build and scale agents with ElevenAgents. ElevenLabs also expanded its partnership with
NVIDIA, announcing that Nemotron 3.5 Super VL is now available in ElevenAgents, and ElevenLabs will add 15 MW of new Vera Rubin compute capacity. The two companies will also work on an on-premises voice agent for Poland’s national e-Health platform.
Many companies will want agents already optimized for their specific industry niche. To make it easier for them to find industry-specific agents, ElevenLabs is launching a new Marketplace. Instead of building from scratch, companies can find and adopt agents built by more than 20 ElevenLabs partners that are already customized to fit their workflows, and improve outcomes in their industry.
At the Summit, enterprises showcased how they are making voice agents part of how they operate, including Yahoo’s voice-powered AI assistant, and Westgate’s new front-desk agent. They join other companies like Stripe, TELUS, FOX Sports, Klarna, and SevenRooms by DoorDash, already putting ElevenAgents to work in their enterprise.
Concluding the keynote, Staniszewski said: “People are understandably concerned that AI could centralise and homogenize everything. That businesses won’t be able to protect their moats. That each new iteration of AI regresses to the mean. We’re building the opposite: models, products, and a community to amplify what makes your business unique, and help you extend your advantage. We will not solve human-AI interaction without reflecting what makes people and businesses distinctive.
