GitHubJoin the waitlistWaitlist

Voice agent benchmark · independent, dated

How well do voice agents hold up on a real phone line?

Independent, dated measurements of the caller-side voice stack callerlab runs (LLM, TTS and STT). Whole agents on real calls will appear after the first real run. Timing figures publish after we re-run the timing measurements.

Real: measured by callerlabWithheld: measured, published after the re-run
Published now
LLM, TTS and STT (agents not run yet; end-to-end timings withheld)
Standard
proposed, not frozen yet (rows below predate it)
Run dates (ET, 2026)
LLM Oct 13, 01:00TTS Oct 13, 01:01STT Oct 13, 01:01E2E Oct 11–12

Caller LLM

Real

How callerlab picks the model behind its own test caller, not a ranking of models for your agent. Candidates run against each provider’s API on 12 real call contexts. Persona adherence and plan-following are judged 1 to 5 by an LLM judge (Model 1 on Host A). Rankings need at least 10 of 12 replies.

voicebench v0.1Oct 13, 2026, 01:00 ET6 models from 3 providers12 contextstext-only replaylist-price cost unless gateway-reported

How it’s measured

What this run can tell you. Click a tile for its chart.

Showing 6 of 6 models

Persona and plan scores

Judged 1 to 5 by an LLM judge: persona, plan. Gaps of 0.1 or less are ties. Values read persona / plan, on a 4.4 to 5 axis because bars from zero would hide every difference. One reply rated a point lower moves a score by a tenth of a point or less.

Cost per turn vs quality

2 of 6 models plotted. $ per turn (log scale, estimated or gateway-reported) against quality. Up and left is better.

n/m · price re-check

No cost yet: Baseten and Groq list prices are being re-checked, so these models sit in the n/m column at their quality score.

LLM caller models, Oct 13, 2026, 01:00 ET
1 to 5Judged1 to 5JudgedUSDest. or gatewayof 12Measured
Also the judgeHost A5.005.00$0.000168 est.12/12
Host A5.005.00$0.000044 est.12/12
how measuredGroq5.005.00n/m12/12
how measuredBaseten4.925.00n/m12/12
how measuredBaseten4.924.83n/m12/12
how measuredGroq4.754.92n/m12/12

$/turn n/m: the provider’s list price is being re-checked. Judge: Model 1 on Host A (lenient, and one of the candidates). Click a row, bar or point for the drill-down. Run conditions

Named vendors
  • Baseten: As measured by callerlab on Oct 13, 2026; not endorsed by Baseten. Terms checked: Oct 14, 2026.
  • Groq: As measured by callerlab on Oct 13, 2026; not endorsed by Groq. Terms checked: Oct 14, 2026.

How these numbers were measured: run settings, prompt, sample size and cost formula.

Vendors can request a correction or re-test: see Conflict and corrections.

Text to speech (TTS)

Real

Candidate voices for callerlab’s caller: six emotional caller lines per voice. Word error is measured after 8 kHz telephone downsampling.

voicebench v0.1Oct 13, 2026, 01:01 ETstreaming synthesis, not on a call11 to 12 clips per voice

How it’s measured
TTS voices
VoiceWER at 8 kHz% · DerivedClipsMeasured
TTS 2Realvoice, anonymized5.3%12/12
TTS 1Realvoice, anonymized6.8%11/12
  • Up to 12 clips per voice (6 caller lines, 2 repetitions each); the Clips column shows how many came back.
  • WER number formatting (“twelve hundred” against “$1,200”) inflates the money-move line.
  • Naturalness and emotion were not rated: no human panel.
  • With this few clips per voice the word-error gap is indicative only, so no better voice is named.

Speech to text (STT)

Real, non-streaming

Transcribers for callerlab’s own call checks: word error on 8 kHz phone audio.

voicebench v0.1Oct 13, 2026, 01:01 ETSTT 134 clips

How it’s measured

Streaming STT is not measured yet. The one real row is a non-streaming upload of clean synthetic clips, so it says nothing about noisy callers. The CLI’s callerlab transcribe checks spoken digits; it is not what this table measures.

STT rows
TranscriberWER at 8 kHz% · DerivedClipsMeasuredMethod
STT 1Realmodel5.3%34Non-streaming file upload
  • Word error compares the transcript with the text the clips were made from, so number formatting counts against it.
  • One transcriber on 34 clips is a reference point, not a ranking.

Agents on real calls: not run yet

No whole-agent benchmark results are published yet. Agent results will appear here after the first real run.

Nothing is shown for agents until a real run exists.

End-to-end reply gap

Withheld

End of caller audio to first agent audio, read off two-channel PSTN recordings of calls to our own reference agents over the older Gather path. Includes the carrier both ways and the agent platform’s end-of-turn wait.

16 live calls, Oct 11–12, 2026Twilio Gather (half-duplex)quiet calls

How it’s measured

We recorded the reply gap on these 16 live calls to our own reference agents, over the older Gather path. The figures publish after the re-run.

Methodology

Every number names its conditions, and the numbers themselves are in benchmark-data.json (schema callerlab.benchmark/1.0). These rows predate the Standard; its rubric applies to agent results once it is frozen.

Dataset generated Oct 13, 01:09 ET; revised Oct 15 (rows removed under vendor terms, timings withheld). No layer carries a Standard version yet: the real layers ran before the Standard.
LayerStatusConditionsnWhen (ET)
End-to-endWithheld
Setup
Live PSTN calls over Twilio Gather (half-duplex) to our own test agents on a hosted voice-agent platform; quiet calls; one measured reply gap per call.
Measured
Reply gap = end of caller audio to first agent audio, read off the two-channel recording.
Region
US
Audio
8 kHz mu-law (PSTN)
16callsOct 11–12
LLMReal
Setup
12 real phone-agent contexts; streaming chat completions; max_tokens 400; caller persona prompt; sequential.
Client
callerlab box, 8 vCPU Linux, single-threaded sequential requests.
Region
US (us-east, Cloudflare IAD)
Audio
text only (no audio)
Left out
4 providers, under their terms of service (see Fairness).
12contexts per modelOct 13, 01:00 ET
TTSReal
Setup
6 caller lines (money-move consent, a dispute, anger with mild profanity, sarcasm, joking, frustration), 2 repetitions each, streaming synthesis, sequential requests.
Region
US (us-east, Cloudflare IAD)
Audio
8 kHz mu-law (native, or transcoded by ffmpeg where a voice has no 8 kHz output; transcode time excluded from time to first audio)
Left out
1 provider, under its terms of service (see Fairness).
11–12clips per voiceOct 13, 01:01 ET
STTReal
Setup
Non-streaming file upload of every clip from the TTS run, including clips from the provider left out of the TTS table. Not streaming 'final' latency and not noisy real callers.
Measured
Latency = request to full transcript.
Region
US (us-east, Cloudflare IAD)
Audio
8 kHz mu-law clips from the TTS run (clean synthetic speech)
34clipsOct 13, 01:01 ET

How the LLM numbers were measured

Every LLM number on this page, named vendors’ rows included, comes from one run on Oct 13, 2026, 01:00 ET. These are its settings; the scripts and contexts are not published yet, so the run can’t be repeated from this page alone.

Harness
voicebench v0.1, task caller_next_line (bench_llm.py). It predates the callerlab Standard, so these rows carry no Standard version. The scripts and the 12 frozen contexts are not published yet.
Region and network
One Linux box (8 vCPU) with egress at Cloudflare colo IAD (Ashburn, VA, us-east). Each provider’s public API is called directly.
Model IDs
As each provider’s API names them, under the model name in the drill-down of each named model. IDs are withheld for the 3 rows shown as “Model N” (Groq Model 3, Host A Model 1, Host A Model 2): the ID, or another host serving the model, would name an anonymized vendor.
Prompt and contexts
A fixed system prompt (the caller’s role and goal, persona and affect guidance, and “one short spoken line ≤ 30 words”) plus the real conversation so far, taken from live callerlab transcripts: agent turns are sent as user, caller turns as assistant. 12 contexts drawn with seed 7 from scenarios S1–S5 and R1–R5; every model gets the same ones. Text only, no audio.
Sample size
12 requests per model, one per context, sent one at a time. A model is ranked only with at least 10 of 12 non-empty replies; each row shows its count.
Streaming and retries
POST /chat/completions with stream: true and stream_options.include_usage, max_tokens 400 (the equivalent field where an API names it differently). Reasoning is kept to the minimum each API allows; a row that sets reasoning_effort shows it in its drill-down. If a provider rejects a param, the param is dropped and the request retried once; params are recorded as requested, not as applied.
Cost formula
Per reply: input tokens × input list price + output tokens × output list price, where output tokens include reasoning tokens if the provider counts them. Cost per turn is the mean over a model’s successful replies.
Price sheet
List prices as of Oct 12, 2026. Not shown for Baseten and Groq: the list price is being re-checked. Anonymized vendors’ price lists are not linked, since a link would name them. Prices change, so these may be stale.

Known limits

  • US PSTN only
  • End-to-end calls used the Gather path (half-duplex); streaming calls are not in these rows yet
  • No SIP/WebRTC direct, no tool-call correctness, no WER of the agent
  • LLM, TTS and STT: one box, one region, one request at a time, in one time window; provider load varies by hour
  • Quality scores come from an LLM judge, not human ratings; prices are list prices except gateway rows

Fairness

  • Rule. Vendor terms are checked before a name is published. Providers whose terms forbid benchmarking, or need their consent before results are published, are left out; vendors whose terms only limit use of their name are anonymized.
  • Named. 2 of 2 named vendors have a recorded check.
  • Anonymized. 4 more (Host A, TTS 1, TTS 2, STT 1) until their terms clearly allow naming them.
  • Left out. 4 LLM providers and 1 TTS provider, because their terms forbid benchmarking or need consent first (terms checked Oct 14, 2026).

Conflict and corrections

callerlab sells testing. We don’t sell or resell any voice platform, have no affiliate deals with them, and never audit a platform we rank. If that ever changes, we disclose it on the affected vendor’s benchmark row. Verified factual errors anywhere on the benchmark are corrected and logged below.

Vendor? Request a correction or re-test: support@callerlab.ai. We keep the raw run logs behind these numbers for 12 months (voicebench: LLM, TTS, STT), so a correction can be checked against them; they are not published.

Corrections and revisions
  • Oct 15, 2026 Revision: rows removed under vendor terms; timing figures withheld until the re-run.

No corrections yet.

What each number is

Cost per turn
Gateway-reported usage cost when the gateway gave one, otherwise list price times measured tokens. List prices are as of Oct 12, 2026 and may be stale. est.
Persona and plan
An LLM judge (temperature 0) rates each reply 1 to 5: persona is in character and sounds like a phone caller; plan advances the goal and answers what was asked. A lenient judge, not human ratings: use it for gross failures. The judge is also one of the candidate models, so its own scores may be inflated. Judged
WER at 8 kHz
The 8 kHz clip is transcribed by the model in the STT table and compared with the input text (word-level Levenshtein, lowercase, punctuation stripped). Number formatting inflates it. Derived

Test your own agent

The callerlab harness, pointed at your agent. Dry run first. (The LLM, TTS and STT rows above come from voicebench scripts that aren’t published yet.)

1. Dry run (no call, no cost)
# first: export CALLERLAB_DIAL_TO=<a number you own, E.164>
# dry run: plans and prices the calls, dials nothing
callerlab explore --pack examples/packs/home-services.json \
  --to "$CALLERLAB_DIAL_TO" --repeat 3 --transport stream --record
2. Real call (paid)
callerlab explore --pack examples/packs/home-services.json \
  --to "$CALLERLAB_DIAL_TO" --repeat 3 --transport stream --record --yes
--yes enables paid calls. Confirm permission and consent first.

Configure a speech-to-text endpoint for the stream before placing this call. --record saves the two-channel audio locally for the gaps command below.

3. Read the results
callerlab scorecard --report .cache/reports/<stamp>
callerlab gaps --recording .cache/runs/<stamp>   # reply gaps off the recording

From a dry run to a CI gate

The quickstart covers install, Twilio setup, the first real call and gating a build on the scorecard.

Read the quickstartRead the Standard