mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-03 19:14:56 -05:00
mudler-agentandEttore Di Giacinto
ae8092e1c6
* feat(sound): expand simple prompts with chat models Allow callers to select an installed chat model for sound prompt expansion. Validate its JSON before audio generation and preserve explicit instrumental and language controls. Keep backend defaults when expansion is omitted, including native Python ACE-Step sampling. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(sound): map expansion by loaded audio family Map neutral expanded fields where the resolved audio family is known. Keep unmarked requests unchanged and reject unsupported expansion before inference. Clear YuE2 and HeartMuLa transcripts to prevent lyric fallback. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(sound): choose a prompt expansion model Require an explicit Simple choice and remember it separately from chat. Clear unavailable choices instead of silently switching language models. Preserve the choice in history and restore it when editing a past result. Keep Backend default and Advanced requests compatible with existing APIs. Document family support and align the published optional request schema. Verify selection, request fields, persistence and history in the browser. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(sound): authorize both generation models Check the resolved audio model and explicit expansion model before loading either. Permission lookup failures stop generation. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(sound): enforce JSON during prompt expansion Constrain expansion output with the existing JSON grammar before decoding. Require a llama-cpp chat model so unsupported backends cannot ignore it. Keep strict field validation and fail before generating audio. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
61 lines
3.0 KiB
Go
61 lines
3.0 KiB
Go
package schema
|
|
|
|
type ElevenLabsTTSRequest struct {
|
|
Text string `json:"text" yaml:"text"`
|
|
ModelID string `json:"model_id" yaml:"model_id"`
|
|
LanguageCode string `json:"language_code" yaml:"language_code"`
|
|
}
|
|
|
|
// DO NOT ADD A SOURCE-AUDIO FIELD TO THIS STRUCT.
|
|
//
|
|
// SoundGenerationRequest.src on the wire is the input clip for the editing
|
|
// routes, and setting it on a stable_audio model CORRUPTS THE HEAP AND ABORTS
|
|
// THE PROCESS in the audio-cpp backend's pinned audio.cpp: "free(): invalid
|
|
// size" / "munmap_chunk(): invalid pointer", SIGABRT, backend gone. It is
|
|
// upstream's bug, not LocalAI's, and it was attributed rather than assumed:
|
|
// upstream's own audiocpp_cli, built from the same checkout, aborts identically
|
|
// with --audio at exit 134 and completes cleanly without it. The long version
|
|
// is on the SoundGeneration handler in backend/cpp/audio-cpp/grpc-server.cpp.
|
|
//
|
|
// What keeps that off the network today is exactly this omission. The
|
|
// ElevenLabs endpoint (core/http/endpoints/elevenlabs/soundgeneration.go)
|
|
// passes nil for the source file because this struct has nowhere to put one,
|
|
// so the only caller that can set src is core/cli/soundgeneration.go, a local
|
|
// CLI. Adding the field here turns a local CLI abort into remotely reachable
|
|
// heap corruption with fully attacker-influenced input. Wait for the pin to
|
|
// move past the fix.
|
|
type ElevenLabsSoundGenerationRequest struct {
|
|
ExpansionModel *string `json:"expansion_model,omitempty" yaml:"expansion_model,omitempty"`
|
|
Text string `json:"text" yaml:"text"`
|
|
ModelID string `json:"model_id" yaml:"model_id"`
|
|
Duration *float32 `json:"duration_seconds,omitempty" yaml:"duration_seconds,omitempty"`
|
|
Temperature *float32 `json:"prompt_influence,omitempty" yaml:"prompt_influence,omitempty"`
|
|
DoSample *bool `json:"do_sample,omitempty" yaml:"do_sample,omitempty"`
|
|
// Advanced mode
|
|
Think *bool `json:"think,omitempty" yaml:"think,omitempty"`
|
|
Caption string `json:"caption,omitempty" yaml:"caption,omitempty"`
|
|
Lyrics string `json:"lyrics,omitempty" yaml:"lyrics,omitempty"`
|
|
BPM *int `json:"bpm,omitempty" yaml:"bpm,omitempty"`
|
|
Keyscale string `json:"keyscale,omitempty" yaml:"keyscale,omitempty"`
|
|
Language string `json:"language,omitempty" yaml:"language,omitempty"`
|
|
VocalLanguage string `json:"vocal_language,omitempty" yaml:"vocal_language,omitempty"`
|
|
Timesignature string `json:"timesignature,omitempty" yaml:"timesignature,omitempty"`
|
|
// Simple mode: use text as description; optional instrumental / vocal_language
|
|
Instrumental *bool `json:"instrumental,omitempty" yaml:"instrumental,omitempty"`
|
|
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"`
|
|
}
|
|
|
|
func (elttsr *ElevenLabsTTSRequest) ModelName(s *string) string {
|
|
if s != nil {
|
|
elttsr.ModelID = *s
|
|
}
|
|
return elttsr.ModelID
|
|
}
|
|
|
|
func (elsgr *ElevenLabsSoundGenerationRequest) ModelName(s *string) string {
|
|
if s != nil {
|
|
elsgr.ModelID = *s
|
|
}
|
|
return elsgr.ModelID
|
|
}
|