mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 10:28:43 -04:00
e9d03c7c48
Four defects the whole-branch review found on the Go side, plus two comment corrections. sample_rate is a disk-exhaustion hazard. The branch added the `form:` tag that makes the field bind for the first time, so the resample path went from dead to live, and utils.AudioResample interpolates the int straight into ffmpeg's -ar with no bound. Measured with ffmpeg 7: -ar 999999999 on a 0.01 s clip writes 20 MB and exits 0, which scales linearly to the reported 3.9 GB for one second, into a GeneratedContentDir nothing sweeps, and convertStems repeats it once per separation stem. Clamped to 8000..192000 in the handler, before the temp dir and before the model is touched, and rejected with a 400 outside it. The low end was reported as "a 0-byte file". It is not: -ar 1 writes a 78-byte header with no audio behind it, whose declared data size still claims 70 bytes, so go-audio parses it as a 35 SECOND file and a size check does not see it. The guard therefore compares the declared data chunk against the bytes actually on disk, and AudioResample now fails rather than returning a WAV carrying nothing. Both parts of a transform request land in one temp dir, and the raw copy was named only after the client's basename, so `-F audio=@mic/clip.wav -F reference=@loopback/clip.wav` wrote "raw-clip.wav" twice. Since AudioToWavPreservingShape hardlinks an already-PCM16 WAV rather than copying it, the reference part's os.Create truncated the inode audio.wav pointed at: mic and reference came out identical, which makes an echo canceller null everything and return near-silence with a 200. The raw copy now carries the form field name. audio-cpp had no BackendCapabilities entry, so VoiceCloningForModel returned nil before it ever consulted the model's tts.voice_cloning override and every `voice: "profile:<id>"` request was refused with a 400, on a backend that ships audio-cpp-chatterbox whose family serves cloning and not plain TTS. Registered with its RPCs, usecases and the reference-audio contract, and deliberately without the 16 kHz mono fold, which its separation families cannot survive. GetBackendCapability was exact-match only, so every pinned gallery variant read as an unknown backend: vulkan-localvqe lost the 16 kHz mono fold that used to be unconditional and started failing inside LocalVQE, and the usecase gate does not stand in for it because BuildFilteredFirstAvailableDefaultModel returns early once the client names a model. Lookup now falls back to the meta name by stripping the gallery's hardware prefix and release-channel suffix, exact match first so nothing can be shadowed. Same class as #10945. Also corrected: the AudioTransformRequest comment claimed echo's binder falls back to the field name, which it does not in either direction (bindData binds ONLY tagged fields and `continue`s otherwise; `model` arrives from setModelNameFromRequest's c.FormValue). And the stable_audio `src` heap corruption caveat now lives on ElevenLabsSoundGenerationRequest, where the Go developer who would add the field can see it, instead of only in C++. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
71 lines
3.5 KiB
Go
71 lines
3.5 KiB
Go
package schema
|
|
|
|
// @Description Audio transform request body — multipart form-data only.
|
|
// `audio` (the primary input file) is required; `reference` (auxiliary
|
|
// signal: loopback for echo cancellation, target speaker for voice
|
|
// conversion, etc.) is optional. Backend-specific tuning lives in the
|
|
// `params[<key>]=<value>` form fields, collected into a generic map so
|
|
// the schema doesn't bake in any one transform's vocabulary.
|
|
//
|
|
// The `form` tags on the two snake_case fields are load bearing, and NOT for
|
|
// the reason first written here. echo's binder has no field-name fallback at
|
|
// all: bindData is documented as binding "ONLY fields in destination struct
|
|
// that have EXPLICIT tag", and a field whose tag is empty and whose kind is not
|
|
// a struct is skipped outright with a `continue`
|
|
// (labstack/echo/v4@v4.15.1 bind.go). So an untagged Format or SampleRate is
|
|
// not matched loosely, it is not looked at, which is why both documented form
|
|
// fields were silently ignored until these tags were added.
|
|
//
|
|
// `model` is not the counter-example it looks like. It is not bound by the
|
|
// binder either: it arrives because setModelNameFromRequest asks for it by
|
|
// name, c.FormValue("model") (core/http/middleware/request.go).
|
|
//
|
|
// SampleRate is validated in the handler, not here: it is interpolated into
|
|
// ffmpeg's -ar and an unbounded value is a disk-exhaustion hazard. See
|
|
// minAudioTransformSampleRate in
|
|
// core/http/endpoints/localai/audio_transform.go.
|
|
type AudioTransformRequest struct {
|
|
BasicModelRequest
|
|
Format string `json:"response_format,omitempty" yaml:"response_format,omitempty" form:"response_format"` // wav | mp3 | ogg | flac
|
|
SampleRate int `json:"sample_rate,omitempty" yaml:"sample_rate,omitempty" form:"sample_rate"` // desired output sample rate; 0 = backend default, otherwise 8000..192000
|
|
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"` // backend-specific tuning
|
|
}
|
|
|
|
// AudioTransformStreamControl is the JSON envelope used on the
|
|
// /audio/transformations/stream WebSocket. The first frame on a new
|
|
// connection MUST be a session.update; subsequent frames are binary PCM.
|
|
// Server may emit error / session.closed text frames.
|
|
type AudioTransformStreamControl struct {
|
|
Type string `json:"type"`
|
|
Model string `json:"model,omitempty"`
|
|
SampleFormat string `json:"sample_format,omitempty"`
|
|
SampleRate int `json:"sample_rate,omitempty"`
|
|
FrameSamples int `json:"frame_samples,omitempty"`
|
|
Params map[string]string `json:"params,omitempty"`
|
|
Reset bool `json:"reset,omitempty"`
|
|
Error string `json:"error,omitempty"`
|
|
}
|
|
|
|
// AudioTransformStreamControl Type values.
|
|
const (
|
|
AudioTransformCtrlSessionUpdate = "session.update"
|
|
AudioTransformCtrlSessionClose = "session.close"
|
|
AudioTransformCtrlSessionClosed = "session.closed"
|
|
AudioTransformCtrlError = "error"
|
|
)
|
|
|
|
// AudioTransformStreamControl SampleFormat values (mirror the proto enum
|
|
// names so the wire format stays self-describing).
|
|
const (
|
|
AudioTransformSampleFormatS16LE = "S16_LE"
|
|
AudioTransformSampleFormatF32LE = "F32_LE"
|
|
)
|
|
|
|
// LocalVQE param keys — backend-specific but referenced by both the
|
|
// HTTP layer (form-field shortcuts, defaults) and the localvqe backend
|
|
// itself. Hoisted so renames stay in lockstep.
|
|
const (
|
|
AudioTransformParamNoiseGate = "noise_gate"
|
|
AudioTransformParamNoiseGateThreshold = "noise_gate_threshold_dbfs"
|
|
)
|