mirror of
https://github.com/mudler/LocalAI.git
synced 2026-05-17 13:10:23 -04:00
bb033b16a9
feat(audio-transform): add LocalVQE backend, bidi gRPC RPC, Studio UI
Introduce a generic "audio transform" capability for any audio-in / audio-out
operation (echo cancellation, noise suppression, dereverberation, voice
conversion, etc.) and ship LocalVQE as the first backend implementation.
Backend protocol:
- Two new gRPC RPCs in backend.proto: unary AudioTransform for batch and
bidirectional AudioTransformStream for low-latency frame-by-frame use.
This is the first bidi stream in the proto; per-frame unary at LocalVQE's
16 ms hop would be RTT-bound. Wire it through pkg/grpc/{client,server,
embed,interface,base} with paired-channel ergonomics.
LocalVQE backend (backend/go/localvqe/):
- Go-Purego wrapper around upstream liblocalvqe.so. CMake builds the upstream
shared lib + its libggml-cpu-*.so runtime variants directly — no MODULE
wrapper needed because LocalVQE handles CPU feature selection internally
via GGML_BACKEND_DL.
- Sets GGML_NTHREADS from opts.Threads (or runtime.NumCPU()-1) — without it
LocalVQE runs single-threaded at ~1× realtime instead of the documented
~9.6×.
- Reference-length policy: zero-pad short refs, truncate long ones (the
trailing portion can't have leaked into a mic that wasn't recording).
- Ginkgo test suite (9 always-on specs + 2 model-gated).
HTTP layer:
- POST /audio/transformations (alias /audio/transform): multipart batch
endpoint, accepts audio + optional reference + params[*]=v form fields.
Persists inputs alongside the output in GeneratedContentDir/audio so the
React UI history can replay past (audio, reference, output) triples.
- GET /audio/transformations/stream: WebSocket bidi, 16 ms PCM frames
(interleaved stereo mic+ref in, mono out). JSON session.update envelope
for config; constants hoisted in core/schema/audio_transform.go.
- ffmpeg-based input normalisation to 16 kHz mono s16 WAV via the existing
utils.AudioToWav (with passthrough fast-path), so the user can upload any
format / rate without seeing the model's strict 16 kHz constraint.
- BackendTraceAudioTransform integration so /api/backend-traces and the
Traces UI light up with audio_snippet base64 and timing.
- Routes registered under routes/localai.go (LocalAI extension; OpenAI has
no /audio/transformations endpoint), traced via TraceMiddleware.
Auth + capability + importer:
- FLAG_AUDIO_TRANSFORM (model_config.go), FeatureAudioTransform (default-on,
in APIFeatures), three RouteFeatureRegistry rows.
- localvqe added to knownPrefOnlyBackends with modality "audio-transform".
- Gallery entry localvqe-v1-1.3m (sha256-pinned, hosted on
huggingface.co/LocalAI-io/LocalVQE).
React UI:
- New /app/transform page surfaced via a dedicated "Enhance" sidebar
section (sibling of Tools / Biometrics) — the page is enhancement, not
generation, so it lives outside Studio. Two AudioInput components
(Upload + Record tabs, drag-drop, mic capture).
- Echo-test button: records mic while playing the loaded reference through
the speakers — the mic naturally picks up speaker bleed, giving a real
(mic, ref) pair for AEC testing without leaving the UI.
- Reusable WaveformPlayer (canvas peaks + click-to-seek + audio controls)
and useAudioPeaks hook (shared module-scoped AudioContext to avoid
hitting browser context limits with three players on one page); migrated
TTS, Sound, Traces audio blocks to use it.
- Past runs saved in localStorage via useMediaHistory('audio-transform') —
the history entry stores all three URLs so clicking re-renders the full
triple, not just the output.
Build + e2e:
- 11 matrix entries removed from .github/workflows/backend.yml (CUDA, ROCm,
SYCL, Metal, L4T): upstream supports only CPU + Vulkan, so we ship those
two and let GPU-class hardware route through Vulkan in the gallery
capabilities map.
- tests-localvqe-grpc-transform job in test-extra.yml (gated on
detect-changes.outputs.localvqe).
- New audio_transform capability + 4 specs in tests/e2e-backends.
- Playwright spec suite in core/http/react-ui/e2e/audio-transform.spec.js
(8 specs covering tabs, file upload, multipart shape, history, errors).
Docs:
- New docs/content/features/audio-transform.md covering the (audio,
reference) mental model, batch + WebSocket wire formats, LocalVQE param
keys, and a YAML config example. Cross-links from text-to-audio and
audio-to-text feature pages.
Assisted-by: Claude:claude-opus-4-7 [Bash Read Edit Write Agent TaskCreate]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
99 lines
3.1 KiB
Makefile
99 lines
3.1 KiB
Makefile
CMAKE_ARGS?=
|
|
BUILD_TYPE?=
|
|
NATIVE?=false
|
|
|
|
GOCMD?=go
|
|
GO_TAGS?=
|
|
JOBS?=$(shell nproc --ignore=1)
|
|
|
|
# LocalVQE upstream version pin. Bump to a specific commit when picking up
|
|
# a new release; `main` works for development but is not reproducible.
|
|
LOCALVQE_REPO?=https://github.com/localai-org/LocalVQE
|
|
LOCALVQE_VERSION?=72bfb4c6
|
|
|
|
# LocalVQE handles CPU feature selection internally (it ships the multiple
|
|
# libggml-cpu-*.so variants and its loader picks the best one at runtime
|
|
# via GGML_BACKEND_DL), so we build a single liblocalvqe.so + the per-CPU
|
|
# ggml shared libs and let it sort itself out. No need for a wrapper
|
|
# MODULE library or per-AVX backend variants here.
|
|
|
|
CMAKE_ARGS+=-DLOCALVQE_BUILD_SHARED=ON
|
|
CMAKE_ARGS+=-DGGML_BUILD_TESTS=OFF
|
|
CMAKE_ARGS+=-DGGML_BUILD_EXAMPLES=OFF
|
|
|
|
ifeq ($(NATIVE),false)
|
|
CMAKE_ARGS+=-DGGML_NATIVE=OFF
|
|
endif
|
|
|
|
# LocalVQE upstream supports CPU + Vulkan only. Other BUILD_TYPE values
|
|
# fall through to the default CPU build — Vulkan is already as fast as the
|
|
# specialised GPU paths would be on this 1.3 M-parameter model.
|
|
ifeq ($(BUILD_TYPE),vulkan)
|
|
CMAKE_ARGS+=-DGGML_VULKAN=ON -DLOCALVQE_VULKAN=ON
|
|
else ifeq ($(OS),Darwin)
|
|
CMAKE_ARGS+=-DGGML_METAL=OFF
|
|
endif
|
|
|
|
# --- Sources ---
|
|
|
|
sources/LocalVQE:
|
|
mkdir -p sources/LocalVQE
|
|
cd sources/LocalVQE && \
|
|
git init && \
|
|
git remote add origin $(LOCALVQE_REPO) && \
|
|
git fetch origin && \
|
|
git checkout $(LOCALVQE_VERSION) && \
|
|
git submodule update --init --recursive --depth 1 --single-branch
|
|
|
|
# --- Native build ---
|
|
#
|
|
# Drives cmake directly against the upstream LocalVQE/ggml CMakeLists.
|
|
# Produces liblocalvqe.so plus the per-CPU libggml-cpu-*.so variants in
|
|
# build/bin/, all of which we copy into the backend directory so package.sh
|
|
# can pick them up. The `liblocalvqe.so` rule deliberately uses a sentinel
|
|
# stamp file because Make's wildcard tracking would otherwise mis-decide
|
|
# about freshness when SOVERSION symlinks are involved.
|
|
|
|
LIB_SENTINEL=.localvqe-build.stamp
|
|
|
|
$(LIB_SENTINEL): sources/LocalVQE
|
|
mkdir -p build && \
|
|
cd build && \
|
|
cmake ../sources/LocalVQE/ggml $(CMAKE_ARGS) -DCMAKE_BUILD_TYPE=Release && \
|
|
cmake --build . --config Release -j$(JOBS)
|
|
# Upstream's CPU build sets GGML_BACKEND_DL=ON + GGML_CPU_ALL_VARIANTS=ON,
|
|
# which produces multiple libggml-cpu-*.so files (SSE4.2 / AVX2 / AVX-512)
|
|
# that the loader picks at runtime. We must build every target — the
|
|
# default `--target localvqe_shared` drops these. CMAKE_LIBRARY_OUTPUT_DIRECTORY
|
|
# routes all of them into build/bin; copy them out next to the binary.
|
|
cp -P build/bin/liblocalvqe.so* . 2>/dev/null || cp -P build/liblocalvqe.so* .
|
|
cp -P build/bin/libggml*.so* . 2>/dev/null || true
|
|
touch $(LIB_SENTINEL)
|
|
|
|
liblocalvqe.so: $(LIB_SENTINEL)
|
|
|
|
# --- Go binary + packaging ---
|
|
|
|
localvqe: main.go golocalvqe.go $(LIB_SENTINEL)
|
|
CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o localvqe ./
|
|
|
|
package: localvqe
|
|
bash package.sh
|
|
|
|
build: package
|
|
|
|
clean: purge
|
|
rm -rf liblocalvqe.so* libggml*.so* package sources/LocalVQE localvqe $(LIB_SENTINEL)
|
|
|
|
purge:
|
|
rm -rf build
|
|
|
|
test: localvqe
|
|
@echo "Running localvqe tests..."
|
|
bash test.sh
|
|
@echo "localvqe tests completed."
|
|
|
|
all: localvqe package
|
|
|
|
.PHONY: build package clean purge test all
|