Days 27-25Nov 4-6Live

pulse-eot-hi: end-of-turn on Hindi calls for Smallest.ai

A silence timer is the default end-of-turn detector, and on real Hindi speech it forces a choice between cutting callers off and leaving dead air.

Data: REAL benchmark (LiveKit eot-bench hi, real Hindi turns) and REAL live Pulse API runs (2026-10-24). Training data is SYNTHETIC (Chirp3 TTS Hindi).

The problem

A voice agent has to decide when the caller has finished speaking. Hindi and Hinglish callers pause mid-sentence on fillers like "haan, matlab..." and while reading out digits, so a plain silence timer either cuts in on those pauses or waits too long on every turn.

This build is a causal, CPU-only streaming replay of LiveKit's eot-bench hi set that reproduces the published Hindi rows, a real-VAD timer sweep, a live sweep of Pulse's own eou_timeout_ms, and an attempt at a Hindi end-of-turn model.

The honest result so far: Pulse behaves like the silence timer plus about 0.33 s, and the model does not beat Smart Turn v3.2 or the timer.

Key result

Pulse's own end-of-turn is a silence timer plus about 0.33 s: at its default 600 ms it cuts in on 2.0% of real Hindi holds and waits 931 ms; treating every is_final as end-of-turn ends about half of turns early.

Line chart: share of pauses where the agent cuts in vs mean wait after the caller finished, for silence timers from 300 to 1200 ms. 400 ms gives 27.5% cut-offs, 600 ms 6.9%, 800 ms 2.9%, 1000 ms 0.9%.
Silence timers on all 400 real Hindi turns (real Silero VAD timing). Going from 600 ms to 1000 ms cuts false cut-offs from 6.9% to 0.9% and adds 400 ms of dead air on every turn. Figure from the repo: results/timer_cost_hi.png.

Comparison with baselines

eot-bench hi, all 400 turns, eot-bench knob sweep (historical, best-on-test operating points)

SystemFCR@300FCR@600Lat@5%Lat@10%
LiveKit Turn Detector v1 (published, cloud)10.5%2.0%454 ms306 ms
Soniox stt-rt-preview (published, cloud)-3.4%553 ms553 ms
ultraVAD (published, open weights, 0.7B)39.1%6.5%616 ms509 ms
Gradium (published, cloud)49.5%5.8%618 ms530 ms
LiveKit v1-mini (published)27.4%5.4%624 ms505 ms
Pulse-EoT-hi v0.1 (ours, CPU)46.4%6.7%634 ms514 ms
SmartTurn v3.2 fp32, run by us on CPU42.6%6.5%639 ms532 ms
SmartTurn v3.2 int8 CPU build, run by us45.1%6.9%646 ms507 ms
deveshu/hinglish-turn-detector, run by us49.5%7.4%667 ms584 ms
Pulse-EoT-hi v0 (Hindi TTS-only training)48.9%7.2%696 ms568 ms
Silero VAD silence timer, run by us (real VAD)46.6%7.4%700 ms554 ms
VAD baseline, oracle silence (eot-bench)49.5%7.4%700 ms600 ms

FCR = false cut-off rate on holds. These rows use eot-bench's definition (best knob triple on the test sweep), so they are comparable to the published rows but not deployable. The only deployable row for v0.1 uses knobs frozen on dev: FCR 0.103 at mean latency 1.107 s on the 400 turns (results/report.md), worse than a 600 ms timer (0.069 at 0.579 s).

What a silence timer costs (real Silero VAD timing, 400 turns)

TimerFalse cut-offs on holdsMean latency after true end
400 ms27.5%379 ms
600 ms6.9%579 ms
800 ms2.9%779 ms
1000 ms0.9%979 ms

A timer has no tuned knob, so these numbers are not affected by the test-set use described below.

Pulse eou_timeout_ms sweep (REAL, live API 2026-10-24) vs the real-VAD timer, 200 dev-test turns

eou_timeout_msPulse FCR [95% CI]Pulse premature turnsPulse mean latencylatency minus timeoutEvery is_final as end-of-turn: premature turnsTimer FCR / premature / latency
3009.3% [6.3%, 13.5%]36.5%627 ms+327 ms54.5%50.4% / 40.5% / 281 ms
4005.2% [3.1%, 8.8%]31.5%717 ms+317 ms52.5%27.0% / 36.0% / 381 ms
600 (Pulse default)2.0% [0.9%, 4.6%]11.5%931 ms+331 ms49.0%6.0% / 18.0% / 581 ms
8000.8% [0.2%, 2.9%]4.0%1197 ms+397 ms47.5%3.2% / 6.5% / 781 ms
10000.8% [0.2%, 2.9%]1.5%1368 ms+368 ms48.0%1.2% / 4.0% / 981 ms
12000.4% [0.1%, 2.2%]1.0%1577 ms+377 ms49.0%1.2% / 1.5% / 1181 ms

model=pulse, language=hi, endpointing=true, streamed in real time (128 ms chunks). Pulse's end-of-turn event is the first is_final after speech_ended. Premature turns = share of turns with any end-of-turn event before the true end (a late event can land after the caller resumed, which FCR does not count). n = 200 turns, 248 holds. Sweep 254.8 audio-minutes, $1.53; with the partials replay and probe, total Smallest spend about $2.08.

Other findings

What this does not show

How to reproduce

From the repository root (commands from its README; Windows PowerShell unless noted).

python -m venv .venv && .venv/Scripts/pip install -e ".[dev]"
bash eval/run_all.sh --quick        # Windows: .\run.ps1 quick
python -m pytest -q tests           # metric parity with eot-bench, causality, split guard
python -m eval.rescore_devtest      # dev-test half rescored from saved predictions
python -m demo.timer_cost           # writes results/timer_cost_hi.png
python -m adapters.pulse_endpointing --sweep 300 400 600 800 1000 1200   # live Pulse, needs SMALLEST_API_KEY
python -m eval.pulse_vs_timer       # Pulse vs timer table

Source of every number on this page: results/report.md, results/eot_bench_hi_devtest/pulse_vs_timer.md, README.md and STATUS.md (commits 2741c40, 1fedba7) in the build's repository.