Kikitori-bench: A Personal ASR Benchmark for Voice Journaling

The capture screen in Ayumi, my voice-journaling app, sends a recording to a transcription model and shows the text back. I had to decide which model. My working answer had been the Gemini family, and I could explain why. Its transcripts read well to me, it came back fast, and on top of that it takes a prompt alongside the audio, so I can pass names, vocabulary and context, and I can ask for the output to be cleaned up or restructured in the same request. Accuracy, speed and that flexibility together had left me with the impression that Gemini was simply the best option for journaling.

The problem was that this was a feeling. When I asked myself how much better Gemini was than Whisper on my own recordings, or whether a cheaper model would do, I had qualitative impressions and nothing else. That is a thin basis for a decision that shapes the app. So I built kikitori-bench, a benchmark built on my private recordings that turns those impressions into numbers I can argue with.

Public benchmarks did not answer my question

Two things bothered me about relying on published leaderboards.

The first is the audio. The public benchmarks I found tended to use read speech, broadcast audio, or clean interviews. What Ayumi receives is unscripted Japanese dictation, recorded on a phone, full of names, numbers and half-finished sentences, mostly one speaker with the occasional passage of conversation. A model’s rank on a public set told me little about how it would handle a tired monologue recorded on a walk.

The second is the metric. Japanese is my native language, and when I read transcripts by hand I noticed distinctions that word error rate and character error rate flatten. Written Japanese does not mark word boundaries with spaces, so WER depends heavily on the chosen segmentation. CER avoids that choice, but it is sensitive to orthographic variation: kanji versus kana, okurigana, and punctuation the model may or may not emit. In my recordings, a harmless spelling variant and a meaning-breaking substitution could receive similar edit distances even though one preserves the sentence and the other does not. That made CER a poor proxy for what I cared about when reading an entry back.

Both problems pointed the same way. If I wanted a number I could trust, it had to be computed on my own audio, and it had to be built from what I actually notice when I read an entry back.

Scoring what I notice when I read an entry back

When I read a transcribed journal entry, I ask two things. Did it keep what I said, without asserting things I never said? And did it get the names and numbers right? Those became the two axes of the benchmark. Neither WER nor CER is computed at all.

Propositions. Every sample carries a verified ground-truth transcript and a list of key points, the claims the transcript must convey. An LLM judge (GPT-5.6 Luna, through the Codex App Server) rules on each key point separately: present, missing, or contradicted, with a critical or minor severity. It also lists insertions, content the hypothesis asserts that the ground truth does not support. From those verdicts I get proposition recall and proposition precision, combined with F-beta. Judging per claim rather than assigning one pass-or-fail grade per clip preserves resolution on long recordings. On a 90-second sample, one slip does not make every model look equally wrong.

Proper nouns and numbers. These never go through the LLM. Names, numbers and coined words are extracted from the reference and the hypothesis with the same local morphological analyser (kuromoji) and compared as sets, giving recall, precision and F1. A name transcribed as a different name is wrong even when the sentence reads perfectly, and I did not want a judge to be talked out of that by a fluent transcript.

When the name-and-number axis applies, the overall score is proper-noun F1 weighted 0.3 plus proposition F-beta weighted 0.7. If neither the reference nor the hypothesis contains an extracted name, number, or coined word, proposition F-beta receives the full weight. An empty transcript scores zero and is counted separately in the report. A request the provider rejects is recorded as an error rather than silently scored as a bad transcript. I set beta to 1, weighting omissions and fabrications equally.

Most of the work was the dataset

The tooling ends up being a pipeline for building ground truth rather than a scorer. I drop recordings into a dataset directory, Gemini drafts a transcript and a first set of key points, and a refine step re-derives the key points from the verified transcript at a granularity computed from the audio length. Then I sit in a local review UI, listen to each recording, fix the transcript, adjust the key points, and approve the sample with Cmd + Enter. The current corpus is 122 samples and 3 hours 5 minutes of audio, every sample under 10 minutes so that every provider sees the same set.

The corpus comes from my journal, so I do not publish it. That does not mean the data stays on my machine: running the benchmark sends each recording to every ASR provider under test, then sends the ground truth, key points and hypotheses to the judge model through the Codex App Server.

The public repository contains the tooling, aggregate results and a synthetic demo dataset. Audio, transcripts, run artefacts and full reports are git-ignored. The publish command does not read per-sample evaluations, so its generated summary contains only aggregate fields and no sample ids or transcripts. I also run a separate leak check over staged files before committing; it looks for audio, transcripts, personal paths, API keys, and phrases copied from a recording into a comment. These checks reduce the risk of publishing private material, but they do not replace reviewing the staged files.

What the numbers said

Six configurations, judged by GPT-5.6 Luna at temperature 0. The summary below keeps the two components used in the overall score. Recall, precision, confidence intervals, costs and all pairwise tests are in the repository’s results page.

Model	Overall	Names / numbers F1	Proposition F-beta
Gemini 3.8 Flash	75.1%	65.9%	81.1%
Gemini 3.8 Flash (thinking off)	71.7%	56.0%	80.8%
Gemini 3.5 Transcribe	66.3%	52.8%	74.7%
Meta Muse Voice Transcribe	66.2%	39.9%	79.2%
Groq Whisper large-v3	63.9%	47.4%	74.7%
xAI Grok STT	48.0%	23.9%	60.4%

Gemini 3.8 Flash leads, and its gap to everything except its own thinking-off variant survives a Bonferroni correction over all 15 pairs. That is the difference I had been feeling and could not point at.

The part I could not have guessed by feel is the thinking budget. Turning thinking off lowered the point estimate by 3.4 percentage points, with a confidence interval of [-0.4, +7.1], so the two configurations are not distinguishable on this corpus. It reduced ASR spend by 64% and median latency by about 47%. Thinking improved names and numbers (65.9% against 56.0%), while turning it off slightly improved proposition precision (80.8% against 79.1%). The precision gain only partly offset the loss on names and numbers. I had a single vague impression that thinking was “a bit better”. The benchmark split that impression into two opposing effects and priced them.

Two practical findings came out for free. Meta Muse rejects audio longer than 10 minutes, which is why the corpus is capped there, and that remains a hard constraint in production. And the whole run cost $9.18, of which the judge cost more than the transcription itself ($5.96 against $3.22). A benchmark that scores with an LLM has an LLM bill.

One off-topic discovery: xAI’s Grok STT is fast. Its real-time factor was 0.01x, with a median latency of 934 ms and a p95 of 2,087 ms on this corpus. Groq, whose whole pitch is fast inference, served Whisper large-v3 at a similar median (1,004 ms) but with a p95 of 8,329 ms. Grok STT lost badly on accuracy for my Japanese audio, but I had not expected it to match Groq on speed with a tighter tail.

The score agreed with me, which is both the point and the caveat

I built the metric from my qualitative impressions and then the metric agreed with them. That is the outcome I wanted, and it is also a reason for suspicion: I designed the scoring to see what I already saw. The confidence intervals and pairwise tests quantify sampling uncertainty, while the judge audit measures repeatability and agreement with my review. They do not validate the metric I designed. The judge’s self-consistency on re-judged samples was Cohen’s kappa 0.81, and its agreement with my own 30-item audit was 0.84. Good, but not an oracle, so I treat differences of a few points as noise. The thinking-budget result was still useful because it separated effects I could not separate by reading.

If you are considering these numbers for your own decision, please read them as one person’s evidence for one decision. The corpus is one speaker, one language, one microphone, mostly dictation. Nothing here measures meetings or speaker attribution. The dataset cannot be published, so the results are not independently reproducible. Prices are list prices from September 2026 and will go stale, and a model version is a snapshot of a given day. The tooling works on any directory of recordings, so if the decision matters to you, point it at your own audio.

I intend to keep journaling by voice and to keep building Ayumi, so the benchmark will stay in use. As new models appear, I plan to add adapters and re-run it, as far as time allows. The next time my feeling about a model changes, I want a number to check it against.