Fish Audio Drama 3 requested !

[AndyLeAI]

Hi! Would you consider integrating Fish Audio Drama 3 (.GGUF) into the prebuilt release of audio.cpp?

Model: https://huggingface.co/fishaudio/drama-3

Drama 3 is an expressive TTS model designed for natural voice acting, emotion control, multiple speakers, and long-form dialogue. It would be a great addition for users interested in audiobooks, storytelling, dialogue generation, and voice-over production.

It would be especially helpful if users could download a ready-to-run prebuilt package without having to manually set up the entire inference environment and its dependencies.

If native integration isn't feasible, perhaps an optional backend or a simple integration with the official inference engine could be considered.

Thanks for your work on audio.cpp!

[0xShug0]

@AndyLeAI Thanks! This is definitely an important model. It would be the first MoE model in audio.cpp, and I'm sure it will introduce useful new framework patterns and primitives. Supporting it will require careful framework extensions.

I'm currently tied up with wrapping up the next release, so implementation and optimization will be deferred until after the release.

I also found an existing GGUF conversion here: https://huggingface.co/Jmica/Drama-3-GGUF. The inference implementation appears to be based on a fork of audio.cpp.

@JarodMica, thanks for your work on this! If you're interested, I'd love to collaborate and bring your work upstream into audio.cpp. It would be great to combine our efforts and build on what you've already accomplished. Of course, I completely understand if you'd prefer to continue developing it independently. Either way, I really appreciate your contribution to the ecosystem!

[0xShug0]

@JarodMica Oh, I just saw your GitHub profile! When I first started exploring TTS models, I actually followed your YouTube channel after seeing it recommended in a Reddit post :)

[JarodMica]

@JarodMica Oh, I just saw your GitHub profile! When I first started exploring TTS models, I actually followed your YouTube channel after seeing it recommended in a Reddit post :)

Hi! I'm glad people have found me through various means, I've been vibing around for awhile xd. I'd have to say, awesome work on audio.cpp, I love having a one stop shop for audio models and I have high praise for the ease of use here.

I'd love to contribute the work I've done so far, but to be frank, it's all coded by codex. But if that's acceptable, I'll take a look at contributing standards and get a PR going.

I'm still testing things out and from what I've tried so far, it works very well.

[0xShug0]

@JarodMica Great! If you plan to contribute, just make sure the changes are well-scoped and any framework or GGML changes are additive and opt-in. I'd be happy to pick this up after the next release.

[AndyLeAI]

Thanks a lot, @0xShug0 and @JarodMica! No rush on the timeline, I really appreciate you both working on this. Happy to test whenever it's ready.

[0xShug0]

@AndyLeAI @GalenMarek14 Drama3 is now in main! Kudos to @JarodMica for the contribution! Please help test it and spread the word. Once the model is stable enough, I'll create a minor release to include it.

[0xShug0]

Demo here. 8GB VRAM and 2x faster than real time on an RTX 5090 https://www.reddit.com/r/LocalLLaMA/s/SNCa6GS3Mx

[5uck1ess]

Opened #935 from a teacher-forced parity check of fish_drama3 against the ComfyUI-Drama3 port: fish_drama3.codec_backend=cpu mis-encodes the voice reference, so cloning keeps the words but loses the speaker (WavLM similarity about 0.75 with the CUDA codec vs about −0.08 with the CPU codec). Generation itself tracks the port at Q4_K_S, and the residual gap looks like quantization. This only shows audio.cpp matches Fish's code, not that the model or Fish's code is bug-free. Details and repro are in the issue.

[0xShug0]

CPU tensor layout issue. Fixed in #936.

[JarodMica]

In case anyone else is running into this, it seems there is an issue with Drama 3 where audio continues a generation in silence, usually after a pause or end of first sentence. Related issues: fishaudio/fish-speech#1357 and fishaudio/fish-speech#1356

tldr: So far, Codex says the evidence points mainly to learned model behavior.

One thing was found that a shared sampling behavior in the original Python implementation and C++ port can delay EOS, using the same top-p filtering rule. It can remove EOS even when EOS is the second-most-likely token, because including it crosses the probability cutoff. However, some tested sampler changes did not reliably fix the silence or missing speech and so ATM, we did not identify a C++ specific runtime defect that explains the problem.

For an initial baseline, I ran with 5000 samples with the drama-3-backbone-8bit-experts-Q4_K_S-F32-routers-BF16-embed-head.gguf across English, Japanese, Chinese, French, and Spanish, resulting in 42 clips that were flagged for trailing silence of at least two seconds, and 4 of these reached the length limit without EOS - meaning about a 0.82% trailing-silence rate in my samples.

Some evidence suggests that this happens more seriously with tagged prompts ([tags like this], [pause], etc), but this also occurs in samples without tagged prompts and the groups also contained different sentences, so these counts alone do not establish causation.

Several possible explanations were tested:

• Runtime and precision checks covered prompt construction, attention and recurrent caches, expert routing, codebook feedback, conversion and numerical precision. Native reference calculations also favored quiet codes and assigned very low EOS probabilities when given the same failing histories. These comparisons used saved histories, including CPU FP32 calculations with dequantized released FP8 weights. They did not constitute a complete native-versus-C++ free-generation benchmark.
• Both implementations discard the token that crosses the cumulative top-p threshold, except for the highest-ranked token. In selected traces, EOS ranked second with roughly 35 to 49 percent raw probability but received zero probability after filtering. Sampling changes reduced some limit stops, but did not provide a reliable fix for silence or missing speech.
• Context-retention tests held the current quiet suffix and position constant. Predictions still differed depending on whether speech had occurred earlier. This argues against complete context loss after every pause. It does not establish exact tracking of remaining words or identify an MoE-specific defect.

I did try to have Codex work on a LoRA experiment to try and figure out a way to recover these lost sections, however, it resulted in a regression to the base model where it did not follow whisper tags as well so it might not be appropriate.

As it stands, the working diagnosis is a model-side continuation weakness once a silent trajectory begins. The initial trigger and exact training cause remain unresolved. The investigation ended without an accepted fix or adoption of the experimental weights. These findings describe a fixed benchmark and selected diagnostic cases, rather than a general failure-rate estimate.

[JarodMica]

From the PR #978, top_k=4097 and chunk=128 resolve 41/42 of the failures.

Currently re-running the 5000 samples to see if these silences appear in new samples, but it's promising to mitigate the issue.

[JarodMica]

Probably my last investigation into the model for now, but reporting that the new changes do not fully resolve the issue, however it does decrease it by a substantial amount, from 42 -> 29 for the 5000 samples (31% fewer instances). These were 28 new cases which had previously generated fine under top_k=30, but can't win them all I guess xd.

For all intents and purposes, it does well enough to where I won't be spending much more time into this thought. These are just my tests so results may vary.

[0xShug0]

@JarodMica Thanks for spending so much time testing this! Really appreciate all the effort and detailed feedback!

[ewb-git]

Hmm...well the transcribe not having an auto feature is worse than the comfyui version for sure. I have to open comfy to run this basically.

Otherwise, Drama 3 seems interesting. The failure rate is astonishingly high. I've played around with it for about 2 hours but get a lot of partials. Shorter prompt is better. Seems higher context and higher characters per chunk help get some more full outputs. Well, it will be interesting to see how this evolves. When it gets it right, the output can be awesome. But the failure rate, speed, seed reliance, etc definitely keep it at preview level. Hopefully improves!

Thanks

[0xShug0]

I also tested one complete passage from #920. The official SGLang-Omni impl used the original checkpoint with FP8 experts and BF16 elsewhere; C++ used Q4 with a chunk size of 64. audio.cpp produced less silence than Python in this test. Diarization detected one speaker in both outputs, with no detected speaker switching.

File	Duration	ASR WER	Speakers	Detected speech
python_stress.wav	174.61 s	57.67%	1	43.93 s
cpp_stress.wav	174.24 s	3.17%	1	101.78 s

python_stress.wav
cpp_stress.wav

[0xShug0]

On why the official HF Space demo looks better:

The official HF Space’s source code reports that top_k=30 left 11 of 40 H200 takes stuck in silence until the generation limit, while removing the cap eliminated that failure in its tests. The Space therefore uses top_k=4097, keeping top-p filtering enabled.

The offiical HF Space also adds application-level handling:

• Trims leading/trailing silence and limits retained pauses.
• Stops generation after sustained silence.
• Retries once if the opening is silent or incomplete and nothing has played yet.

If you invoke the API directly, you’ll receive events like:
{"retry_count":1,"audio_s":2.23,"finish_reason":"stop"}