Skip to content

Fix frozen RNG in Whisper temperature sampling - #1461

Open
PCcoding666 wants to merge 1 commit into
ml-explore:mainfrom
PCcoding666:fix/whisper-sampling-rng
Open

PCcoding666 wants to merge 1 commit into
ml-explore:mainfrom
PCcoding666:fix/whisper-sampling-rng

Conversation

@PCcoding666

Copy link
Copy Markdown

Fixes #1460.

Whisper's compiled categorical sampler captured a random key when MLX traced the function and reused that key on later calls. This change passes mx.random.state as both an input and an output of mx.compile, the pattern already used by MusicGen in this repository.

The new model-free regression test seeds MLX and compares two consecutive compiled draws and their final RNG state with eager mx.random.categorical. It fails on main because the second draw repeats the first and passes with this change.

Tested on Apple Silicon

Apple M2 Pro, macOS 27.2, Python 3.12.9, MLX 0.32.4, Metal backend.

  • Ran Whisper Tiny through mlx_whisper.transcribe on the same 1.75-second synthetic clip at temperature 1 with five seeds. On main, only 1 of 40 sampler calls advanced the RNG, and all five runs returned the same tokens. With this change, all 45 calls advanced the RNG and the five runs produced four distinct transcripts. The output lengths varied, accounting for the different call counts.
  • With the model already loaded, running seeds 101, 102, then 101 through the public entry point reproduced the first transcript and token sequence exactly on the third run. Greedy decoding returned the same text on both revisions. Forcing temperature fallback on the clip also exercised 0.5 and 1.0, with RNG advancement on all 23 calls after the change.
  • All 12 Whisper tests passed, including the optional 12-minute Alice audiobook test with its actual fixture, model parity, quantization, and word timestamps. Black 25.1.0 and isort 6.0.0 checks passed on both changed files.

Prepared and tested with Codex assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[whisper] Temperature sampling uses a frozen RNG: categorical is compiled without mx.random.state

1 participant