mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-31 02:29:43 -04:00
* feat(tts): stream Qwen text through audio pipelines Feed cleaned LLM deltas into one persistent Qwen synthesis session. Return audio before input ends and preserve the live RPC through model cache, staging and in-flight wrappers. Stop generation on sink failures or cancellation without replaying speech through a fallback. Pin the incremental native fork and verify early audio, late text, finalization and cancellation with lifecycle and real-model checks. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(tts): use the organization Qwen fork Fetch the reviewed incremental synthesis commit from localai-org so backend builds use the organization-maintained fork. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add a Qwen streaming assistant preset Expose an English pipeline using existing VAD, ASR, LLM and named-voice TTS models. Enable incremental text and audio delivery, and document separate component installs because the gallery has no dependency installer. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): use Parakeet TDT v3 for the assistant Select the existing Parakeet TDT 0.6B v3 transcription entry and update its prerequisite install command. Keep the preset's English voice and incremental Qwen synthesis configuration. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(kokoros): reject unsupported backend RPCs Implement the MotionStream and SlotCache methods required by the shared protobuf service. Return Unimplemented so Kokoros compiles without advertising support for either operation. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tts): mark cleanup and logging results ignored Explicitly ignore connection shutdown, test server shutdown and Ginkgo diagnostic writer errors. These operations do not replace the stream's inference or sink errors, and errcheck requires their return values to be acknowledged. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(tts): stream buffered clauses in realtime pipelines Start clause synthesis while LLM text is still arriving when both streaming flags are enabled. Keep explicit clause opt-out and native Qwen sessions, bound automatic queued work, and choose streaming support on resolved failover targets. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(tts): bound synthetic voice clone generation The sine-wave reference checks speaker conditioning and WAV output, but does not provide a meaningful speech EOS target. Limit that smoke request to 128 frames and verify the request limit reaches native synthesis. Leave production sampling defaults unchanged. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tts): require explicit clause buffering opt-in Keep whole-reply synthesis for nonnative engines when clause_chunking is unset or false. Use the bounded cancellable worker when explicitly enabled, with unary clauses when TTS streaming is disabled. Preserve native Qwen precedence for every clause setting. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add VITS and Piper streaming assistants Provide comparable full pipeline configurations with shared VAD, Parakeet transcription, and Qwen chat components. Enable clause buffering explicitly and document the separate prerequisite model installs and Talk selection. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
56 lines
2.7 KiB
C
56 lines
2.7 KiB
C
#pragma once
|
|
#include <stddef.h>
|
|
|
|
extern "C" {
|
|
|
|
// Streaming PCM chunk callback. samples is mono float PCM at 24 kHz, valid
|
|
// only for the duration of the call. Return non-zero to continue, 0 to abort.
|
|
typedef int (*qt3_chunk_cb)(const float *samples, int n_samples,
|
|
void *user_data);
|
|
|
|
// Load the talker + codec/tokenizer GGUFs. use_fa / clamp_fp16 map to
|
|
// qt_init_params (the qt ABI exposes no thread count; ggml uses its own
|
|
// default). Returns 0 on success, non-zero on failure.
|
|
int qt3_load(const char *talker_path, const char *codec_path, int use_fa,
|
|
int clamp_fp16);
|
|
|
|
// Synthesize to a malloc'd float PCM buffer (caller frees via qt3_pcm_free).
|
|
// The synthesis mode (base / custom_voice / voice_design) is auto-detected by
|
|
// qt from the talker GGUF; speaker is honoured only for custom_voice, instruct
|
|
// for voice_design / custom_voice, and ref_samples (+ optional ref_text) drive
|
|
// base-mode cloning. qt enforces the rules and we surface qt_last_error() on
|
|
// QT_STATUS_MODE_INVALID. Writes the sample count to *out_n. Returns NULL on
|
|
// failure (out_n set to 0).
|
|
float *qt3_tts(const char *text, const char *lang, const char *instruct,
|
|
const char *speaker, const float *ref_samples, int ref_n,
|
|
const char *ref_text, long long seed, float temperature,
|
|
int top_k, float top_p, float repetition_penalty,
|
|
int max_new_tokens, int *out_n);
|
|
|
|
// Streaming synthesis: cb is invoked per PCM chunk as audio is produced. Same
|
|
// param semantics as qt3_tts. Returns 0 on success.
|
|
int qt3_tts_stream(const char *text, const char *lang, const char *instruct,
|
|
const char *speaker, const float *ref_samples, int ref_n,
|
|
const char *ref_text, long long seed, float temperature,
|
|
int top_k, float top_p, float repetition_penalty,
|
|
int max_new_tokens, qt3_chunk_cb cb, void *user_data);
|
|
|
|
typedef int (*qt3_text_cb)(void *user_data, const char **text, size_t *n_bytes);
|
|
int qt3_tts_live(const char *lang, const char *instruct, const char *speaker,
|
|
const float *ref_samples, int ref_n, const char *ref_text,
|
|
long long seed, float temperature, int top_k, float top_p,
|
|
float repetition_penalty, int max_new_tokens,
|
|
qt3_text_cb next_text, qt3_chunk_cb cb, void *user_data);
|
|
|
|
// Free a buffer returned by qt3_tts.
|
|
void qt3_pcm_free(float *p);
|
|
|
|
// Release the qt context.
|
|
void qt3_unload(void);
|
|
|
|
// Named-speaker introspection (custom_voice models). Returns 0 / NULL when no
|
|
// model is loaded or the index is out of range.
|
|
int qt3_n_speakers(void);
|
|
const char *qt3_speaker_name(int i);
|
|
}
|