[xuanruli]

What

Four accuracy fixes to hyperframes audio --read (what Desktop's read_audio wraps). They come from a DSH e2e on a 30 s fixture: speech, silence, clicks at 8.0/8.5/9.0/9.5, a 1 kHz tone burst, a 120 BPM bed from 10.0 with a kick every 0.5 s, and speech over music.

1. Music onset (item 3). Music was read from 7.9 s instead of 10.0. The 8.0 click and the tone burst were missing, and "riser ×2 at 10.0" was the music's own entrance.
2. Effect levels (item 4). The same clicks read −10/−7/−2 dB at mono 22 kHz and +12/+15/+14 at mono 44.1 kHz.
3. Pauses (item 5). pause 3.8–21.0 said nothing about the sound that kept playing.
4. Voice against music (item 6). There was no voice-vs-music level, so an agent answered "does the music fight the voiceover?" wrongly.

All four are in this PR.

Why: causes found

I reproduced these on a rebuilt fixture: macOS say speech, synthesized clicks, burst, kick and pad, plus synthetic stems with cross-stem leakage. The base read reproduces the reported failures:

• the 8.0 click is merged into the 7.9 sound;
• a kick run reads as one 4 s boom/riser;
• clicks where the music is out read +34 dB over music.

The causes:

• 7.9 music onset. The tone burst lands in the pitched bed, not in the drums. Click leakage in the bed fills the gaps, so runs() sees one run from 7.9 to the end, and the run's tonality makes it all music. Effects were searched only in the drum stem, so the burst could never become an effect.
• Missing 8.0 click. It sits inside the burst. Summed into one track, the click's 20 ms energy barely clears the tone, so it never makes a 6 dB step.
• "riser at 10.0". extent() ran for up to 4 s while the level stayed above the silent floor from before the music. Every following kick extended the same sound, which gave a long attack and a riser/boom kind.
• Level spread. All three encodings took the "over music" branch. The music level was read at the click's own peak, in a stem that holds the click's leakage after separation and re-encoding.

How

• music.ts. Each run of the bed is split into sustained parts. A frame counts as sustained when sound fills at least half of the 0.1 s frames within ±1 s. Part edges move in to where the sound holds for 0.5 s without a gap. Sparse stretches are neither music nor ambience.
• effects.ts, onsets. A sound ends where the next one steps 6 dB up out of it, and the onset walk-back stops at the trough after the previous sound.
• effects.ts, per-track search. Effects are found in each source track on its own: the drums, and the bed with its music and ambience stretches silenced ±0.1 s. Duplicates within 30 ms keep the louder finding. The speech skip gets 0.1 s of slack for word timing.
• effects.ts, levels.
  • peakDb: the loudest 20 ms RMS of the decoded mix within ±50 ms of the effect's peak.
  • overMusicDb: measured against the music's (bed + drums) mean level in the second before the onset, or after the end when the music starts with the effect.
  • riseDb: measured against the mix's median level in the half second before; null when that level is under −55 dBFS.
• voice.ts / read.ts.
  • Pauses carry recordingDb.
  • Sentences over music carry againstMusic: { voiceLufs, musicLufs }. This is BS.1770 integrated loudness (core loudness()) of the vocals stem and of everything else, measured over the overlap only.
  • Both are measured on the mono downmix, so absolute values run about 3 dB under a stereo meter; the difference between them is unaffected.
• Version and docs. READ_VERSION goes 3 → 4, so cached reads recompute. Docs are updated in docs/packages/cli.mdx.

Related work

Builds on #6536 (noise beds as ambience), now merged into main. This PR is rebased onto it. The bed is silenced for effect search outside both music segments and ambience, and pause lines name ambience too.

Stem separation billing

/v3/audio/stems is a billed call. It is charged per second of audio to the caller's HeyGen credits, or to the HyperFrames usage pool for Desktop, and refunded on failure. Only --quality best triggers it, and cached stems are reused.

Model-facing text (for Xuanru's review)

Line	Old	New	中文
effect, music playing	…, tail 0.0 s, +34 dB over music, …	…, tail 0.0 s, peak -23 dBFS, +4 dB over music, …	音效峰值 −23 dBFS（整段混音），比效果前一秒的音乐高 4 dB
effect, after silence	…, while the music is out, … (stems) / +104 dB over what played before (mix)	…, peak -17 dBFS, out of silence, …	峰值 −17 dBFS，从静音中出现
effect, after sound	+12 dB over what played before	peak -20 dBFS, +12 dB over what played before	峰值 −20 dBFS，比前半秒的整体电平高 12 dB
pause, with sound	pause 19.4 s	pause 19.4 s; during it: 5 effects (whoosh, click, tick ×3), music 10.0–21.0	停顿 19.4 秒，期间有 5 个音效，10.0–21.0 有音乐
pause, no stems	pause 19.4 s	pause 19.4 s; during it: 7 effects (…), sound continues at -18 dBFS	停顿期间声音仍在继续，整体 −18 dBFS
pause, silent	pause 1.0 s	pause 1.0 s, silent	停顿 1 秒，静音
voice over music, music louder	voice "…" 3.2 words/s (mean 4.2)	voice "…" 3.2 words/s (mean 4.2) voice -24 LUFS, music -20 LUFS: music 4 dB over voice	人声 −24 LUFS，音乐 −20 LUFS：音乐比人声响 4 dB
voice over music, voice louder	voice "…" …	voice "…" … voice -14 LUFS, music -18 LUFS: voice 4 dB over music	人声 −14 LUFS，音乐 −18 LUFS：人声比音乐响 4 dB
pause, ambience under it	pause 2.0 s	pause 2.0 s; during it: ambience 4.0–6.0	停顿期间有环境声（4.0–6.0）
voice over music, within 0.5 dB	voice "…" …	voice "…" … voice -18 LUFS, music -19 LUFS: level with each other	人声与音乐响度相当（相差不到 0.5 dB）

The pause token stays first on pause lines, because Desktop's show filter takes the first word as the kind.

Test plan

Fixture. 30 s, built locally: say speech plus synthesized clicks, burst, 120 BPM kick and pad, and synthetic stems with leakage re-encoded as 192 kb/s MP3. It was encoded as mono 22.05 kHz, mono 44.1 kHz and stereo 48 kHz.

• Before (main):
  • music from 9.9;
  • 4 effects, with the 8.0 click merged into the 7.9 burst;
  • clicks where the music is out read +34 dB over music.
• After:
  • all three encodings give the same events: the 7.9 burst and the 8.0, 8.5, 9.0 and 9.5 clicks;
  • music 10.0–30.0, with the kicks left out as 40 drum hits;
  • peaks agree within 1 dB, and the voice line reads music 4 dB over voice (5 dB on the 22 kHz encoding, where the voice measures −25 LUFS).
• Mix-only mode: the base read turned the music entrance into a 4 s boom at +114 dB; it now reads thump ×3 … peak -7 dBFS.

New tests. Each one is confirmed to fail on the base code where that applies:

• kicks over bleed are not read as one long sound;
• a click inside a tone held in another track is found;
• levels come from the mix and from the music before the effect;
• music starts where it is sustained (the base reads 2.8 instead of 5);
• an end-to-end read of the three encodings gives the same events and peaks within 1 dB, and checks the pause line and the voice-vs-music line;
• format.test.ts covers the three voice-vs-music wordings (music louder, voice louder, level within 0.5 dB).

Checks.

• vitest run src/audio/read/: 46 passed.
• CLI tsc is clean for these files.
• oxlint, oxfmt, the comment ratchet/citations and fallow complexity all pass.

Not run for real.

• HeyGen's stem separation: there is no HeyGen credential on this machine, and it is a billed call. Every stem path used synthetic stems placed in the stem cache.
• The reported "stereo 48 kHz drops one click" depended on real separation output and was not reproduced. On the synthetic stems, with levels read from the mix and detection per stem, the three encodings agree.
• No live Desktop/DSH re-run.

Known limits.

• The 1 kHz burst is named whoosh: the kind table has no plain tone kind.
• The 8.0 click, which sits inside the burst, reads click: mid while the other clicks read tick: high.
• Commits are grouped by item. Only the full set is meant to be built.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
hyperframes	🟢 Ready	View Preview	Oct 27, 2026, 10:31 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[heygengenesis]

Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits · head 587d9974

One blocker: zeroing the bed over music and ambience spans adds a fake onset where each span ends, so an ordinary music fade-out reads as an effect. Details inline.

Scope

Read all of effects.ts, music.ts, read.ts, format.ts, voice.ts, signal.ts and types.ts, plus the tests. vitest run src/audio/read/ passes locally (46/46). I probed the bed path with synthetic beds through readMusic → findEffects, which is what fromStems does. No prior reviews. CI and Comments were green on this head when I posted; one CI and one Windows render run were still in progress.

— Genesis

[heygengenesis]

Verdict: APPROVE — 1 resolved, 0 open, 0 new · 587d997 → a45cee5

— Genesis