mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 10:28:43 -04:00
b93e7bdfb3
Four fixes from the second review, plus one bug they made visible. isPCM16Wav tested only the bit depth, and go-audio's IsValidFile never looks at the format tag, so a 16-bit WAVE_FORMAT_EXTENSIBLE (0xFFFE) upload was passed through untouched where the old fold would have transcoded it. audio.cpp's WAV reader accepts 16-bit only when the tag is 1, so such a file died with "unsupported WAV encoding". Extensible is what many DAWs and Windows tools write and music files are this endpoint's new headline input, so it is a first-contact failure rather than a corner. The check now requires tag 1, with a spec that fails against the old implementation. Stem URLs are percent-escaped. A stem name is the model's own string and legally contains a space, a '#', a '?' or a '%'; an unescaped '#' truncates the URL before the request is even sent. The name field keeps the raw name. sample_rate and response_format are applied to the stems as well as to dst. Applying beat documenting: dst IS one of those stems, so leaving them alone broke the "dst duplicates the selected stem" invariant the whole design rests on, and both conversions are no-ops when unset. A stem whose conversion fails is dropped from the header rather than advertised in the wrong shape. Verifying that turned up why it had never been noticed: the two fields were never bound at all. The request arrives as multipart/form-data and echo's binder falls back to the FIELD NAME without a form tag, matching only case-insensitively, so "SampleRate" never matched "sample_rate" and "Format" never matched "response_format". Both were documented in the endpoint table and silently ignored. Two form tags fix it, and with them the conversion is observable end to end. Docs: audio-transform.md now documents what LocalAI does to an upload before the backend sees it, which backend gets the 16 kHz mono fold and why, params[stem], and the X-Audio-Stems header with a worked example. Also records the known limitation that the fold lookup is on the bare backend name, so pinned variants (vulkan-localvqe) do not match, and points at IsLlamaCppBackend as the suffix-tolerant precedent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
61 lines
3.0 KiB
Go
61 lines
3.0 KiB
Go
package schema
|
|
|
|
// @Description Audio transform request body — multipart form-data only.
|
|
// `audio` (the primary input file) is required; `reference` (auxiliary
|
|
// signal: loopback for echo cancellation, target speaker for voice
|
|
// conversion, etc.) is optional. Backend-specific tuning lives in the
|
|
// `params[<key>]=<value>` form fields, collected into a generic map so
|
|
// the schema doesn't bake in any one transform's vocabulary.
|
|
//
|
|
// The `form` tags on the two snake_case fields are load bearing. This request
|
|
// arrives as multipart/form-data, and echo's binder falls back to the FIELD
|
|
// NAME when a form tag is missing, matching it only case-insensitively:
|
|
// "Format" never matches "response_format" and "SampleRate" never matches
|
|
// "sample_rate", so both documented form fields were silently ignored until
|
|
// these tags were added. `model` worked all along because its field name and
|
|
// its form key differ only in case.
|
|
type AudioTransformRequest struct {
|
|
BasicModelRequest
|
|
Format string `json:"response_format,omitempty" yaml:"response_format,omitempty" form:"response_format"` // wav | mp3 | ogg | flac
|
|
SampleRate int `json:"sample_rate,omitempty" yaml:"sample_rate,omitempty" form:"sample_rate"` // desired output sample rate; 0 = backend default
|
|
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"` // backend-specific tuning
|
|
}
|
|
|
|
// AudioTransformStreamControl is the JSON envelope used on the
|
|
// /audio/transformations/stream WebSocket. The first frame on a new
|
|
// connection MUST be a session.update; subsequent frames are binary PCM.
|
|
// Server may emit error / session.closed text frames.
|
|
type AudioTransformStreamControl struct {
|
|
Type string `json:"type"`
|
|
Model string `json:"model,omitempty"`
|
|
SampleFormat string `json:"sample_format,omitempty"`
|
|
SampleRate int `json:"sample_rate,omitempty"`
|
|
FrameSamples int `json:"frame_samples,omitempty"`
|
|
Params map[string]string `json:"params,omitempty"`
|
|
Reset bool `json:"reset,omitempty"`
|
|
Error string `json:"error,omitempty"`
|
|
}
|
|
|
|
// AudioTransformStreamControl Type values.
|
|
const (
|
|
AudioTransformCtrlSessionUpdate = "session.update"
|
|
AudioTransformCtrlSessionClose = "session.close"
|
|
AudioTransformCtrlSessionClosed = "session.closed"
|
|
AudioTransformCtrlError = "error"
|
|
)
|
|
|
|
// AudioTransformStreamControl SampleFormat values (mirror the proto enum
|
|
// names so the wire format stays self-describing).
|
|
const (
|
|
AudioTransformSampleFormatS16LE = "S16_LE"
|
|
AudioTransformSampleFormatF32LE = "F32_LE"
|
|
)
|
|
|
|
// LocalVQE param keys — backend-specific but referenced by both the
|
|
// HTTP layer (form-field shortcuts, defaults) and the localvqe backend
|
|
// itself. Hoisted so renames stay in lockstep.
|
|
const (
|
|
AudioTransformParamNoiseGate = "noise_gate"
|
|
AudioTransformParamNoiseGateThreshold = "noise_gate_threshold_dbfs"
|
|
)
|