SEPTEMBER 2023 RELEASES
September 2023 Releases
How We Built a Voicemail Detection Model that Outperforms the Industry
Last week at our customer conference, Regal Rise we announced Regal's new Voicemail Detection model and the headline results: ~45% fewer voicemails routed to agents on outbound calls, 96% screener recall, and no tradeoff on human recall. This post is the technical follow-up: a look at the methodology behind the model and the decisions that shaped it.
Most AMD solutions on the market have two problems that result in their lower accuracy and usefulness:
1. They treat the answer space as binary: human or machine. But the real world isn't binary. Google and iOS call screeners are automated, but there can be a live person behind them. IVRs can be navigated to eventually get to a live person. Collapsing these into a single "machine" label limits your routing and handling options.
2. They rely on acoustic heuristics: detecting silence patterns, listening for a beep, measuring pause length after the greeting. Those signals can work on simple carrier voicemail, but they break down on personal greetings that sound like live speech, on call screeners that introduce a delay before a human picks up, and on any audio that doesn't fit the templates the heuristic was built around.
We started from scratch: a speech-encoder-based model trained on real call audio from 163k outbound calls, with a richer taxonomy, a principled labeling pipeline, and a decision framework built around the asymmetric costs of each type of error, for example, it's more costly to hang up on a human than it is to let a voicemail through to an agent.
This post walks through our technical approach.
Why "human or machine" isn't enough
Most AMD systems frame the problem as a binary classification: human or machine. That framing is too coarse for the range of call behaviors encountered in production.
A Google or iOS call screener is automated speech, but some percentage of time, there’s a real person behind it. Treating that as an ordinary voicemail discards a potentially valuable connection. A personal voicemail greeting can sound nearly identical to live speech (remember your high school voicemail message: Hello?…hello? Just kidding, please leave a voicemail). A carrier-generated greeting has very different acoustic properties. Collapsing all of these into a single "machine" label makes the problem look simpler than it is and limits what you can optimize.
We trained the system on a richer internal taxonomy:
The classifier produces a probability estimate for every class. A separate policy layer then maps those estimates to a routing action, and that mapping is fully configurable per brand and campaign. For a live human pickup, the AI agent is connected and the conversation begins. For voicemail, the system can leave a message or end the call, depending on campaign configuration. For call screeners, customers can also choose their routing policy based on data.
A high-intent campaign, like outbound calls for a healthcare follow up, may find that a large share of screener interactions result in a live conversation; routing every screener to an agent makes sense. A lower-intent re-engagement campaign for lapsed insurance customers may find that very few screener interactions convert; in that case, treating screeners as voicemail protects agent minutes. Because the model and the policy are separate, customers can tune this without retraining.
.png)
How we built it: from audio to routing decision
1. Start with the audio the model will actually hear
We collected contact-side audio from real outbound calls and focused on the first two seconds after the answer. That budget is a product constraint: more time provides better model signal, if you wait too long to route, then the person who picked up will hang up.
Two properties of that clip have to be right. The first is alignment, which matters more than it might seem. At a two-second budget, a few hundred milliseconds of offset is not a rounding error: it can push the greeting's first syllable out of the clip, leaving the model to score mostly silence on a call a human would classify instantly. Training and inference audio must be aligned to the same starting point and normalized the same way otherwise offline accuracy measures a slightly different problem than the one running in production. We verified that predictions agreed across both paths before trusting any offline result.
The second property is format. Telephony audio arrives at 8 kHz as mulaw-encoded audio, while the speech encoder we selected expects 16 kHz PCM, so every clip is decoded and converted to mono 16 kHz before it reaches the model, in training and in production alike.
Here is an example standard carrier voicemail input:
“Your call has been forwarded to voicemail. The person you're trying to reach is not available. At the tone, please record your message. When you have finished recording, you may hang up. [BEEP]”
2. Let the representation inform the taxonomy
Before training a classifier, we passed sample clips through pretrained speech encoders and inspected the resulting embeddings. We projected those high-dimensional representations into two dimensions using UMAP and inspected density-based clusters to answer two questions: Which candidate labels form coherent acoustic regions? And which labels overlap so heavily that adding more examples won't solve the problem?
The analysis was instructive. Carrier-generated voicemail and some screeners formed recognizable regions. Live speech and personal voicemail greetings overlapped significantly. The broad "other screener or IVR" bucket was less coherent than the named screener classes. These findings shaped the taxonomy, manual-review priorities, and the decision to preserve uncertainty rather than forcing every ambiguous clip into a confident label.
3. Scale labels with an LLM judge, then audit the hard cases with people
Manually reviewing nearly 200,000 recordings would have been slow and expensive. Many of their transcripts, however, contained clear linguistic signals, carrier greetings, named call-screening prompts, and live conversational exchanges, that could be labeled automatically
We first created a human-labeled evaluation set, stratified across our taxonomy, and used it to measure the LLM judge’s agreement with human reviewers. The labeling rubric was deliberately conservative: the judge assigned a class only when the transcript contained sufficient evidence; otherwise, it returned unknown.
We iterated by examining disagreements between the judge and the human labels, refining the rubric, and rerunning the same evaluation set. Once the results were stable, we compared several LLMs on both label agreement and cost at scale.
We then applied the selected judge to approximately 196,000 transcript-backed calls, our results are below:
Random human audits confirmed that language-distinct classes, such as named screeners and common voicemail patterns were suitable for automated labeling. Incomplete transcripts, silence, dead air, and short ambiguous fragments were not. Those unknowns, disagreements, and boundary cases were sent to our team for manual audio review.
This created a practical division of labor:
- Automation for scale: assign high-confidence candidate labels to common patterns
- Humans for ambiguity: review unknowns, short fragments, disagreements, and rare classes
- Human labels for final claims: measure production performance on audio labeled independently of either system's verdict
This approach concentrated human effort where product context and careful listening mattered most, while allowing us to prepare the broader training corpus efficiently and consistently.
4. Seal the test set before model selection
Before the model evaluation, we set aside a stratified holdout representing ~15% of the usable labeled corpus, with class proportions preserved so minority classes remained visible. We used stratified cross-validation inside the remaining training data for iteration, then returned to the sealed holdout for model comparison. Threshold sweeps create many opportunities to overfit a test set indirectly: training metrics, threshold-selection data, and live-audio validation answer different questions and shouldn't be collapsed into one score.
5. Choose the encoder before optimizing the classifier head
We tested multiple open speech encoders, including W2V-BERT, WavLM-Base-Plus*, and XLS-R. The largest performance improvement came from the encoder, not the classifier head. WavLM was the best fit for our accuracy-and-latency tradeoff: it was designed to preserve spoken content while remaining robust to speaker, channel, and background-noise variation.
We use WavLM as a frozen feature extractor rather than an end-to-end fine-tuned network. The pretrained encoder converts a short waveform into frame-level representations; Regal trains only the compact downstream aggregation and classification layers. Partial or full encoder fine-tuning remains a future experiment.
6. Pool frame sequences without discarding their variation
The audio encoder returns a sequence of vectors (one every 20 ms) rather than a single fixed-size array. A simple baseline averages the frames, but two clips with the same average and very different variation over time can look identical to such a model.
To address this, we compress the time axis within each selected WavLM layer using masked mean-and-standard-deviation pooling, then train a supervised scalar-mixing layer to learn how much each encoder layer should contribute. The mixer is optimized jointly with a temporary linear probe; the learned layer weights are retained for inference.
7. Keep the decision head deliberately lightweight
On top of the pooled WavLM representation, we compared a regularized linear head with random forests, gradient-boosted trees (LightGBM, XGBoost), and feedforward neural networks. More flexible models can fit nonlinear boundaries, but that flexibility wasn't overwhelmingly useful in practice. Several alternatives improved individual offline slices while adding overfitting risk and operational complexity.
The final system retains the lightweight linear head. The encoder and pooling stages already make much of the problem separable; the head adds negligible latency and is straightforward to package and serve. Human recall is controlled at the policy layer through the decision threshold, not through class weighting in the loss.
8. Find right threshold for your needs
The classifier emits a probability estimate for each class. For each candidate model, we swept the human threshold and measured human recall, precision, false-connection rate, and accuracy. The decision rule is intentionally asymmetric:
- Preserve the call when the human score clears the chosen risk threshold
- Preserve supported screener outcomes when a screener is the strongest class
- Otherwise treat the call as an answering machine
That threshold can be adjusted by business context. A campaign selling a high-value product or service, where each live conversation is worth more, may accept more voicemail leakage to protect every possible human connection. A campaign with a lower-value offer may choose a stricter operating point, prioritizing agent minutes efficiency over capturing every marginal call.
9. Validate on live audio before granting routing authority
Offline test data can't reproduce every carrier, codec, microphone, background, and timing pattern seen in production. We ran the new Regal model in parallel with our incumbent AMD solution (which leverages Twilio AMD + keyword identification from Deepgram realtime transcription to disambiguate “unknown” responses from Twilio AMD) on live calls before using it as the routing authority.
The validation used the same waveform seen by production inference. Both systems' proposed outcomes were retained, and uniform samples were drawn from live traffic for manual review. Reviewers labeled audio without seeing the model-comparison stratum. We reported confidence intervals and inspected results by class and campaign rather than trusting a single aggregate.
Production Pipeline: An End to End Example
Training a model is one thing, but the production pipeline you build is just as critical in order to go from audio input to output decision in 2 seconds. Lets again consider our carrier voicemail example from the beginning of this post and how it runs through our production system.
1. Collect a short contact-audio window. Live μ-law telephony frames are accumulated in sequence. When configured, a silence-aware path can wait for enough voiced audio before making a final decision.
2. Normalize the waveform. Decode the telephony signal and resample it from 8 kHz to the 16 kHz input expected by WavLM.
3. Extract frame representations. Frozen WavLM layers turn the waveform into a sequence of high-dimensional vectors.
4. Pool the sequence. Learned layer weights pooling combine those summaries into one vector.
5. Estimate all classes. After inputting the embedding in the lightweight multiclass head, return probabilities for live speech, voicemail variants, and screeners.
6. Apply business policy. Human and screener rules map probabilities to the final agent-versus-system decision.
What the human-labeled production comparison showed
At the documented operating point, we compared our Regal Voicemail Detection Model against the incumbent on a human-labeled sample of live calls:
At the same 96% human recall, Regal's voicemail leakage rate was 6.1%, compared with 11.0% for the incumbent stack, a 44.7% relative reduction, or roughly 45% fewer machine connections without sacrificing a single human connection in this sample.
The screener result is the other notable figure. Regal correctly identified 96.4% of call screeners; the incumbent caught 43.1%. Those missed screeners represent calls where there may have been a real person reachable behind the automated prompt, and the incumbent stack was dropping them.
The important result is the shape of the tradeoff: the selected Regal operating point moves voicemail leakage down while holding human recall at the incumbent level, with a model that can be tuned further toward human recall if business needs require it.
What's next
Every model has a lifecycle. Voicemail greetings, call-screening products, lead sources, and carrier behavior change over time. We've built the feedback loop as a first-class part of the system:
- Monitor runtime latency, errors, and class-probability distributions by model version
- Draw recurring human-labeled samples, including new brands and campaigns, to detect silent quality drift
- Track class-level confusion or break out new classes, especially the heterogeneous other IVR/screneer bucket
- Retrain the lightweight head when labeled production data reveals stable new patterns
- Continue controlled experiments with temporal pooling, CNNs, recurrent models, transformer heads, and partial or full encoder fine-tuning
The central lesson from this project: better Voicemail Detection didn't come from a single clever classifier. It came from aligning the taxonomy with the actual audio, scaling labels without trusting automation blindly, choosing representations before adding model complexity, and treating the decision threshold as an explicit business choice.
_______________________________________________________________________________
*WavLM-Base-Plus by Microsoft (Chen et al., arXiv:2110.13900, 2021), licensed under CC BY-SA 3.0. License: https://github.com/microsoft/UniSpeech/blob/main/LICENSE. Model: https://huggingface.co/microsoft/wavlm-base-plus.
Frequently Asked Questions
Treat your customers like royalty
Ready to see Regal in action?
Book a personalized demo.
Thank you! Click here if you are not redirected.
Oops! Something went wrong while submitting the form.
Founded in 2020, Regal is an enterprise voice AI agent platform for contact centers. Regal helps businesses build, deploy, and manage autonomous AI agents across sales, support, and operations teams.
Latest Blog Posts
Dialing Compliance: How We Scaled Outbound Calling Reliably
Behind a fragmented and ever-changing regulatory landscape is a simple goal: protecting people from unwanted or poorly timed outreach and giving them control over how they’re contacted. In practice, that responsibility translates into a set of checks that must be validated.
No Need to "Press 1": How AI Phone Agents Go Beyond Automation
AI phone agents aren't just better phone trees. They are autonomous systems that can understand caller intent, retrieve relevant information from your CRM in real time, execute actions, handle objections and clarifications, and complete the conversation without human involvement.
.webp)
.webp)
The PII Question Every Regulated Industry Asks Before Deploying AI Agents
When prospects in healthcare and financial services hear "PII," they often imagine bulk data transfer: the idea that deploying an AI agent means copying a database somewhere. In most modern deployments, that's not what's happening.
.webp)
Why Your Appointment Scheduling Is Still Broken (And What AI Agents Actually Fix)
With AI agents, every appointment gets a confirmation call. Each contact gets a chance to reschedule, and the confirmation rate goes to 100%, not because the AI is better at conversations than your agents, but because the AI never runs out of capacity.
.webp)
The Knowledge Gap: Why Your AI Agent Can’t Answer Real Customer Questions
AI agent performance depends on how well it aligns with real customer language, not just fluency. Regal Improve helps teams identify gaps and improve outcomes by analyzing real conversations and surfacing high-impact opportunities
March 2026 Releases
In March, we made it faster than ever to build, launch, and continuously improve AI agents with Copilot, Regal's AI agent for building AI agents. Together, these updates lessen the time needed to launch and maintain agents, so you can focus on understanding your customers, unlocking new use cases, and driving faster business outcomes.
.webp)
.webp)
February 2026 Releases
In February, we made AI agents sound and feel more natural, even when handling complex, multi-step conversations with conditional logic. These updates give you the control to build AI agents that don't just follow a script: they adapt, respond, and engage naturally with each customer.
.webp)
.webp)
.webp)
Pinpoint Critical Knowledge Base Gaps with Regal Improve
Customer conversations are inherently dynamic, and Regal Improve’s Knowledge Base Coverage tracks changes in retrieval performance over time and at scale. This makes it easy to verify that KB updates increase coverage and if your documentation is keeping up with evolving customer requests.
November 2025 Releases
November's releases make it easier to scale your transfer-heavy call flows, with new tools that align dialer pacing to downstream agent capacity and streamline complex handoffs. You’ll also see improved post-call workflow accuracy, with reliable data extraction and disposition outcomes driven by LLM configurability and earlier validation.
.webp)
Teaching Voice AI to Behave: Prompting Principles Every Parent Already Knows
Building a great AI voice agent isn’t so different from parenting a five-year-old: clarity, examples, and structure make all the difference. This post breaks down proven prompting principles—from positive framing and persona design to conversational guardrails—that help AI agents sound natural, stay on task, and follow directions every time.
.webp)
How Kin Insurance & a360inc Achieved Faster Outreach and Better Customer Experiences with AI Agents
In this fireside chat, Regal Co-Founder & CEO Alex Levin sits down with Kin Insurance’s Austin Ewell and a360inc’s Henry Davidson to share how their organizations are using AI agents to transform outreach, customer experience, and operational efficiency. They discuss real-world deployments—from qualification and human agent handoffs to complex negotiation workflows—break down the results, and offer practical guidance for leaders adopting AI agents at scale.
.webp)
Single vs Multi-State: How to Pick the Right AI Agent for the Job
Learn about the difference between single-state and multi-state AI agents, and how each impacts speed, scale, and reliability. Discover when simplicity is enough and when enterprise workflows demand structured orchestration, so you can choose the right design for your use case.
.webp)
The Rise of Voice AI Agents: Real-World Deployments Across the Enterprise
In this AI4 fireside chat, Regal.ai COO Sahil Mehta and TaskUs CCO Jarrod Johnson explore how enterprise organizations are deploying voice AI agents across customer experience, support, and sales. They share real-world deployments, discuss quality and ROI trade-offs, and outline best practices for aligning operations and product teams when adopting AI agents at scale.
Automatically Evaluate Test Suites with Simulations
Discover how you can use Simulations to evaluate scenario-specific conversational flows that pinpoint AI Agent failures before launch. Speed up regression testing, validate prompts, knowledge bases, and custom actions at scale, and deploy reliable AI Agents with confidence.
%20(1).webp)
Apple iOS 26 Caller Screening: An Enterprise Guide
Apple’s iOS 26 introduces new call-screening controls that will reshape outbound performance. This guide explains what’s changing, how adoption may impact enterprise contact centers, and the proactive steps leaders can take now to protect answer rates and customer trust.
.webp)
The RAG Playbook: Structuring Scalable Knowledge Bases for Reliable AI Agents
Learn how to structure a knowledge base that keeps AI Agents accurate and on-script. Discover why human-style KBs fail, and apply best practices like single-topic chunking, concrete instructions, and explicit outcomes to reduce hallucinations and scale reliable RAG performance.
RAG Hygiene: How to Scale and Maintain AI Agent Knowledge
Learn how to maintain a clean, reliable RAG system for AI Agents. Discover best practices for structuring source docs, chunking content, titling for retrieval, avoiding redundancy, and keeping knowledge bases fresh to ensure accurate, scalable performance.
Announcing Regal + Rime: Human-Like Voices at Enterprise Scale
Regal is partnering with Rime to deliver ultra-realistic, emotionally nuanced AI voices for enterprise contact centers. Discover new voice options that capture human qualities like pacing, resonance, and subtle expression, and learn how Rime’s technology powers conversations that build trust at scale.
.webp)
.webp)
.webp)
%20(1).webp)
Announcing Regal + Cartesia: Expanding Voice Options for Enterprise AI
Regal is partnering with Cartesia to deliver ultra-low latency, high-fidelity AI voices for enterprise contact centers. Explore new voice options, hear them in action, and learn about Cartesia's industry-leading voice AI offerings.
.webp)
Anatomy of an AI Voice: What Makes It Sound Human
This article outlines the core characteristics that influence how voice AI is perceived on live calls. From mechanical traits like speed and volume, to more emotional and conversational behaviors, we’re going to look at what those characteristics mean, why they matter, and how they impact your bottom line.
.webp)
Voice AI Customization 101: Settings That Work Best
Learn how to configure your AI Voice Agent for real performance. This guide covers the most important voice settings in Regal, what ranges top brands use in production, and how adjusting speed, tone, and responsiveness impact cost, containment, and overall customer experience.
%20(1).webp)
%20(1).webp)
.webp)
%20(1).webp)
.webp)
.webp)
.webp)
.webp)
.webp)
%20(1).webp)
%20(1).webp)
%20(1).webp)
.webp)
.webp)
.webp)
.webp)
%20(1).webp)
%20(1).webp)
.webp)
%202.webp)
AI Agents for Education: 8 Use Cases for More Meaningful Student Outcomes
AI Agents for Education are transforming student engagement—boosting enrollment, improving retention, and making support more human. Discover 8 game-changing use cases that free up your staff while delivering better student outcomes.
Regal Named One of Forbes America’s Best Startup Employers 2025!
Regal is officially one of Forbes America’s Best Startup Employers 2025, ranking #164 out of 500. This recognition is a testament to our incredible team, our innovative work culture, and our unwavering commitment to advancing AI technology.
AI in Education: The Future of Student Engagement & Enrollment
AI in education is helping to streamline admissions, automate student engagement, and enhance higher ed outreach. Discover key education technology trends to boost enrollment and learn why automated student engagement tools are the future.
.webp)
.webp)
How are Generative AI Voice Agents Different from AI Voice Assistants?
When it comes to comparing AI Agents vs. AI Assistants, Siri & Alexa handle simple tasks, but Gen AI Voice Agents—like Regal’s AI Phone Agent—drive real business impact with human-like conversations, automation, and seamless integration.
.webp)
.webp)
.webp)
7 Best Use Cases for AI Voice Agents in Your Contact Center
As AI technology continues to evolve, the use cases for AI Voice Agents in contact centers will only increase. By answering these six key questions, you can identify where AI agents fit best today in your contact center and plan for future integrations.
.avif)
The Next Step Forward in AI will come from Consumer Businesses
Consumer businesses are implementing AI-enabled customer experiences without any change in behavior on the part of consumers, which is leading AI to become common with more speed than past transitions like the internet and the smartphone. And it means that the next step in AI will not come from the LLM providers. The next big step forward in AI rests in the hands of consumer businesses.
Innovating on Patient Care | B2C Sales Podcast Episode 7
Explore Eric Hauser's remarkable career journey from GovTech to healthcare disruption at Cadence, highlighting the transformative power of innovation, collaboration, with a special focus on the pivotal role of Regal in driving patient engagement and outcomes.
