pulse-eot-hi: end-of-turn on Hindi calls for Smallest.ai
A silence timer is the default end-of-turn detector, and on real Hindi speech it forces a choice between cutting callers off and leaving dead air.
Data: REAL benchmark (LiveKit eot-bench hi, real Hindi turns) and REAL live Pulse API runs (2026-10-24). Training data is SYNTHETIC (Chirp3 TTS Hindi).
The problem
A voice agent has to decide when the caller has finished speaking. Hindi and Hinglish callers pause mid-sentence on fillers like "haan, matlab..." and while reading out digits, so a plain silence timer either cuts in on those pauses or waits too long on every turn.
This build is a causal, CPU-only streaming replay of LiveKit's eot-bench hi set that reproduces the published Hindi rows, a real-VAD timer sweep, a live sweep of Pulse's own eou_timeout_ms, and an attempt at a Hindi end-of-turn model.
The honest result so far: Pulse behaves like the silence timer plus about 0.33 s, and the model does not beat Smart Turn v3.2 or the timer.
Key result
Pulse's own end-of-turn is a silence timer plus about 0.33 s: at its default 600 ms it cuts in on 2.0% of real Hindi holds and waits 931 ms; treating every is_final as end-of-turn ends about half of turns early.

results/timer_cost_hi.png.Comparison with baselines
eot-bench hi, all 400 turns, eot-bench knob sweep (historical, best-on-test operating points)
| System | FCR@300 | FCR@600 | Lat@5% | Lat@10% |
|---|---|---|---|---|
| LiveKit Turn Detector v1 (published, cloud) | 10.5% | 2.0% | 454 ms | 306 ms |
| Soniox stt-rt-preview (published, cloud) | - | 3.4% | 553 ms | 553 ms |
| ultraVAD (published, open weights, 0.7B) | 39.1% | 6.5% | 616 ms | 509 ms |
| Gradium (published, cloud) | 49.5% | 5.8% | 618 ms | 530 ms |
| LiveKit v1-mini (published) | 27.4% | 5.4% | 624 ms | 505 ms |
| Pulse-EoT-hi v0.1 (ours, CPU) | 46.4% | 6.7% | 634 ms | 514 ms |
| SmartTurn v3.2 fp32, run by us on CPU | 42.6% | 6.5% | 639 ms | 532 ms |
| SmartTurn v3.2 int8 CPU build, run by us | 45.1% | 6.9% | 646 ms | 507 ms |
| deveshu/hinglish-turn-detector, run by us | 49.5% | 7.4% | 667 ms | 584 ms |
| Pulse-EoT-hi v0 (Hindi TTS-only training) | 48.9% | 7.2% | 696 ms | 568 ms |
| Silero VAD silence timer, run by us (real VAD) | 46.6% | 7.4% | 700 ms | 554 ms |
| VAD baseline, oracle silence (eot-bench) | 49.5% | 7.4% | 700 ms | 600 ms |
FCR = false cut-off rate on holds. These rows use eot-bench's definition (best knob triple on the test sweep), so they are comparable to the published rows but not deployable. The only deployable row for v0.1 uses knobs frozen on dev: FCR 0.103 at mean latency 1.107 s on the 400 turns (results/report.md), worse than a 600 ms timer (0.069 at 0.579 s).
What a silence timer costs (real Silero VAD timing, 400 turns)
| Timer | False cut-offs on holds | Mean latency after true end |
|---|---|---|
| 400 ms | 27.5% | 379 ms |
| 600 ms | 6.9% | 579 ms |
| 800 ms | 2.9% | 779 ms |
| 1000 ms | 0.9% | 979 ms |
A timer has no tuned knob, so these numbers are not affected by the test-set use described below.
Pulse eou_timeout_ms sweep (REAL, live API 2026-10-24) vs the real-VAD timer, 200 dev-test turns
| eou_timeout_ms | Pulse FCR [95% CI] | Pulse premature turns | Pulse mean latency | latency minus timeout | Every is_final as end-of-turn: premature turns | Timer FCR / premature / latency |
|---|---|---|---|---|---|---|
| 300 | 9.3% [6.3%, 13.5%] | 36.5% | 627 ms | +327 ms | 54.5% | 50.4% / 40.5% / 281 ms |
| 400 | 5.2% [3.1%, 8.8%] | 31.5% | 717 ms | +317 ms | 52.5% | 27.0% / 36.0% / 381 ms |
| 600 (Pulse default) | 2.0% [0.9%, 4.6%] | 11.5% | 931 ms | +331 ms | 49.0% | 6.0% / 18.0% / 581 ms |
| 800 | 0.8% [0.2%, 2.9%] | 4.0% | 1197 ms | +397 ms | 47.5% | 3.2% / 6.5% / 781 ms |
| 1000 | 0.8% [0.2%, 2.9%] | 1.5% | 1368 ms | +368 ms | 48.0% | 1.2% / 4.0% / 981 ms |
| 1200 | 0.4% [0.1%, 2.2%] | 1.0% | 1577 ms | +377 ms | 49.0% | 1.2% / 1.5% / 1181 ms |
model=pulse, language=hi, endpointing=true, streamed in real time (128 ms chunks). Pulse's end-of-turn event is the first is_final after speech_ended. Premature turns = share of turns with any end-of-turn event before the true end (a late event can land after the caller resumed, which FCR does not count). n = 200 turns, 248 holds. Sweep 254.8 audio-minutes, $1.53; with the partials replay and probe, total Smallest spend about $2.08.
Other findings
- Pulse is the silence timer shifted by about 0.33 s: its mean latency is the timeout plus 317 to 397 ms, and at matched latency it is no better than the timer (FCR intervals overlap). At its default 600 ms it cuts into 2.0% of Hindi holds and makes every turn wait 0.93 s on average.
- Trap: with default settings Pulse also sends word-count finals mid-speech, so an agent that treats every is_final as end-of-turn ends 47.5% to 54.5% of turns early.
- Real Pulse partials in place of Vosk text change nothing detectable for v0.1 (dev-test, paired bootstrap latency -16.0 ms, 95% CI -51.5 to +20.5).
- A Smart Turn v3.2 fine-tune on real Hindi (IndicVoices) ran on a Modal L4 (0.31 h, $0.45) and is negative: no epoch beat v3.2 on Lat@5%, so nothing was exported. Only 13 end-of-turn windows in its dev set, so that metric is very noisy.
- The yardstick reproduces LiveKit's Hindi rows: our CPU run of Smart Turn v3.2 fp32 gives the published row exactly (42.6% / 6.5% / 639 ms).
- On the 200-turn dev-test half, paired bootstrap of v0.1 minus Smart Turn v3.2 fp32: latency -12.5 ms (95% CI -30.0 to +6.5), FCR +0.40 pp (95% CI -1.38 to +2.12). No detectable difference.
- The community Hinglish fine-tune (deveshu) is no better than a VAD timer on real Hindi.
What this does not show
- Test-set use: the 400 turns informed one design iteration (v0 to v0.1). Since 2026-10-20 the set is split into a 200-turn dev-test half and a sealed half; the sealed half is sealed from now on, not never seen.
- No real Hindi speech in training: every Hindi training and dev clip is Google Chirp3 TTS.
- Test-time text comes from Vosk small-hi, whose Hindi transcripts are poor; swapping in real Pulse partials did not move v0.1, so the lexical branch carries little weight and is effectively unproven.
- Efficiency was measured with x86 Python under emulation on an ARM laptop: p99 401 ms per hop, 622 MB RSS against a 300 MB budget. Not yet re-measured on native x86.
- The barge-in arbiter is not a result: it was trained and tested on TTS clips, with backchannel recall 2/15.
- The Pulse sweep covers the 200-turn dev-test half only, one network path and default Pulse settings. The sidecar has not been run against the live Pulse socket, and a v1 refit on Pulse text and a two-party Hindi set are still PENDING.
How to reproduce
From the repository root (commands from its README; Windows PowerShell unless noted).
python -m venv .venv && .venv/Scripts/pip install -e ".[dev]"
bash eval/run_all.sh --quick # Windows: .\run.ps1 quick
python -m pytest -q tests # metric parity with eot-bench, causality, split guard
python -m eval.rescore_devtest # dev-test half rescored from saved predictions
python -m demo.timer_cost # writes results/timer_cost_hi.png
python -m adapters.pulse_endpointing --sweep 300 400 600 800 1000 1200 # live Pulse, needs SMALLEST_API_KEY
python -m eval.pulse_vs_timer # Pulse vs timer table
Source of every number on this page: results/report.md, results/eot_bench_hi_devtest/pulse_vs_timer.md, README.md and STATUS.md (commits 2741c40, 1fedba7) in the build's repository.