Draft · not frozen yet

The callerlab Standard

A draft way to score a voice agent on real phone calls. Nothing in it is frozen or published, no one has measured whether two runs agree, and this page promises no reproducibility, no published agreement figure and no frozen version. Five lanes, never blended. Every pass rate carries its interval, and every rate its n.

If a version is ever frozen, its reports would carry this stampthe callerlab Standard, version set at freezeharness 0.1.0 · judge pinned at freezemanifest sha256 set at freeze

scenarios planned

× 3

repeats each, 72 calls per agent (planned)

caller temperaments; hash pinning planned

lanes, never one blended score

The five lanes

From docs/SCORECARD.md. An agent that is fast and wrong is not average, so the lanes are never added up. Task is the headline. Lane thresholds are set in the manifest at freeze.

Task

Judged

Did the caller get what they called for?

Comes from: Judge verdicts; pass = passed / (passed + failed + measured failures + unclear). Unclear counts against.
Reported as: Pooled pass rate with a clustered Wilson 95% interval (the headline), and pass^3 (all 3 repeats pass) for each scenario that ran 3 times
“Thin” below: 5 judged calls

Speed

Measured

How long does the caller wait?

Comes from: End of caller audio as sent → first agent audio, read off the recording of a streamed call. Quiet calls only.
Reported as: p50/p95/p99, tail ratio p95/p50, share over 1/2/3/5/10 s, timing bound ±ms
“Thin” below: 20 answered turns

Conversation

Measured

How does the exchange behave?

Comes from: Recording and turn transcript.
Reported as: Talk-over, unanswered and repeat rates (n of N, no interval)
“Thin” below: 20 caller turns

Robustness

Judged

How much survives realistic conditions?

Comes from: Task verdicts grouped by voice, difficulty, noise, language, impairment.
Reported as: Largest drop vs clean, Newcombe interval when same scenarios
“Thin” below: a clean group + 1 other

Compliance

Judged

Did the agent say only what it may?

Comes from: Fact checks and must-not rules from the pack, each quote checked by code.
Reported as: Fact accuracy, must-not breaches, verified fabrications
“Thin” below: 5 verified claims

Planned: 24 scored scenarios, plus 8 held back

The suite is not built or run yet. Filter by family.

Scenario IDs and seeds are set in standard/v0/manifest.json at freeze. Tile labels such as TK1 are positions in the wall, not those IDs. Unverified Escalation and DTMF still need live verification.

Family	Scenarios	Calls (× 3)	What it covers
Task	8	24	Booking, change or cancel, and information calls with must-not rules
Temperament	6	18	One per level, Easygoing through Heated
Barge-in	3	9	The caller cuts in on the agent’s reply
Noise	3	9	Cafe babble at 10 dB, car cabin, speakerphone
Keypad / DTMF	2	6	Menu navigation and a PIN entry
Escalation	2	6	Correct handoff or an honest “can’t help”, never a false “done”
Total	24	72	Per agent under test

Six caller temperaments

From docs/CALLERS.md. The six levels exist today. Pinning their stance, challenges and duplex timings by content hash is planned for the frozen version. On the Gather path only the wording applies; cut-ins need streaming over Media Streams.

Level L1EasygoingNothing added. The default.Challenges: noneCuts in (stream): no
Level L2ImpatientShort lines, no small talk; asks once for the short version.Challenges: noneCuts in (stream): no
Level L3PushyAsks for the best option and for specifics.Challenges: up to 1Cuts in (stream): 5 s into reply 2
Level L4ArgumentativeDoubts answers and says why, citing only the call.Challenges: up to 2Cuts in (stream): 3 s into replies 1–3
Level L5RelentlessBlunt, persistent; restates the request each time.Challenges: up to 3Cuts in (stream): 1.5 s into replies 1–6
Level L6HeatedOpenly angry; mild profanity (needs --allow-profanity).Challenges: up to 3Cuts in (stream): as Relentless

Why calls fail

Filed by code from the verdicts and checks: no model picks the reason. A call can sit under more than one reason, so counts never become a share.

Connection faultno answer, voicemail, silence, timeout
Mission criterion not metwith the judge’s quote
Verified fabrication
Verified wrong claim
Verified must-not rule
The caller technique the call usedonly when nothing more specific applies

Intervals that respect repeats

Calls of one scenario share its script, so the planned 72 calls over 24 scenarios would carry less information than 72 independent calls. Pass rates use Wilson at the effective sample size n / DEFF (Kish design effect, clustered by scenario), capped at both ends.

Baseline comparisons use Newcombe’s difference on the same scenarios
A lane below its sample floor reads “thin”; with no evidence it reads “not measured”
Simulation check: clustered covers 94.6% vs 68.6% for plain Wilson (12 × 5, nominal 95%)

scorecard line, example from tests/bench/render.test.ts

task  75% (45/60; 95% interval 49% to 90%;
      clustered by scenario (Wilson, design effect 4.10,
      effective n 14.6), 12 scenarios x 5 calls each)

What the first frozen version would pin

Planned: written to standard/v0/manifest.json with SHA-256s. Today only the caller side is hashed, for the CLI preset.

Scenarios and seeds

Fixed seed list per scenario: caller planning, noise offset, barge-in jitter.

Caller model, TTS, STT

Dated model IDs and voice IDs. The agent under test is not pinned: a platform runs its agent on its own default model, and every report names that model.

Judge

Pinned snapshot(s), temperature 0, prompt hash. 3-model majority panel if ready for the first frozen version; otherwise single judge, panel in a later version.

Thresholds

Lane thresholds fixed in the manifest.

Golden set

≥50 blind-graded calls for the first frozen version, 150 after it (planned).

Harness

Container pinned by digest plus a git tag. Neither exists yet.

Judge drift control

The judge must quote evidence. A verdict is a named judge model’s opinion, with the lines it quoted. No representative agreement figure has been published for the default judge yet. No agreement figure with blind human labels has been published. The bars below are proposed.

Check	Bar (proposed)
Cohen’s kappa vs blind human labels, golden set	≥ 0.70
False-pass rate	≤ 5%
Self-consistency, same transcript × 5 at temp 0	≥ 98%
Judge swap moves per-lane pass rates	< 2 ppa swap is still a major version
Measured kappa for the first frozen version	Set at freeze

Stability targets

Targets for the first frozen version, to be measured on 3 reruns before it freezes. Nothing has been measured against them yet.

Measure	Target (proposed)
Per-scenario verdict agreement, 3 reruns	≥ 90%
Suite pass-rate SD, 3 reruns	≤ 3 pp
Reply-gap p50 rerun to rerun	±10%
Barge-in yield latency spread	≤ 250 ms IQR

Semver for scores

Anything that can change a score on the same calls is a major. Old versions stay runnable; both stay on the benchmark for at least 6 months after a major (proposed). Pick a change:

From an illustrative 1.0.0

MAJOR, version becomes 2.0.0: Can change a score on the same calls. Re-score the golden set and the last benchmark under both versions and publish a crosswalk.

Run it on your agent

Dry run first: it prints the plan and its price and dials nothing. The CLI supports a pinned caller configuration. That preset does not implement this proposed 24-scenario scoring rubric. See CLI preset documentation.

Dry run

# first: export CALLERLAB_DIAL_TO=<a number you own, E.164>
# dry run: dials nothing
callerlab explore --pack examples/packs/home-services.json \
  --to "$CALLERLAB_DIAL_TO"

Real, paid calls, then score offline

# real, paid calls: run the pack three times, then score offline
callerlab explore --pack examples/packs/home-services.json \
  --to "$CALLERLAB_DIAL_TO" --repeat 3 --yes
callerlab scorecard --report .cache/reports/<stamp>

--yes enables paid calls. Confirm permission and consent first.

Start testing today

The quickstart goes from install to a dry run, a first real call and its scorecard.

Read the quickstart See the benchmark