mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-03 12:57:02 -04:00
bb033b16a9
feat(audio-transform): add LocalVQE backend, bidi gRPC RPC, Studio UI
Introduce a generic "audio transform" capability for any audio-in / audio-out
operation (echo cancellation, noise suppression, dereverberation, voice
conversion, etc.) and ship LocalVQE as the first backend implementation.
Backend protocol:
- Two new gRPC RPCs in backend.proto: unary AudioTransform for batch and
bidirectional AudioTransformStream for low-latency frame-by-frame use.
This is the first bidi stream in the proto; per-frame unary at LocalVQE's
16 ms hop would be RTT-bound. Wire it through pkg/grpc/{client,server,
embed,interface,base} with paired-channel ergonomics.
LocalVQE backend (backend/go/localvqe/):
- Go-Purego wrapper around upstream liblocalvqe.so. CMake builds the upstream
shared lib + its libggml-cpu-*.so runtime variants directly — no MODULE
wrapper needed because LocalVQE handles CPU feature selection internally
via GGML_BACKEND_DL.
- Sets GGML_NTHREADS from opts.Threads (or runtime.NumCPU()-1) — without it
LocalVQE runs single-threaded at ~1× realtime instead of the documented
~9.6×.
- Reference-length policy: zero-pad short refs, truncate long ones (the
trailing portion can't have leaked into a mic that wasn't recording).
- Ginkgo test suite (9 always-on specs + 2 model-gated).
HTTP layer:
- POST /audio/transformations (alias /audio/transform): multipart batch
endpoint, accepts audio + optional reference + params[*]=v form fields.
Persists inputs alongside the output in GeneratedContentDir/audio so the
React UI history can replay past (audio, reference, output) triples.
- GET /audio/transformations/stream: WebSocket bidi, 16 ms PCM frames
(interleaved stereo mic+ref in, mono out). JSON session.update envelope
for config; constants hoisted in core/schema/audio_transform.go.
- ffmpeg-based input normalisation to 16 kHz mono s16 WAV via the existing
utils.AudioToWav (with passthrough fast-path), so the user can upload any
format / rate without seeing the model's strict 16 kHz constraint.
- BackendTraceAudioTransform integration so /api/backend-traces and the
Traces UI light up with audio_snippet base64 and timing.
- Routes registered under routes/localai.go (LocalAI extension; OpenAI has
no /audio/transformations endpoint), traced via TraceMiddleware.
Auth + capability + importer:
- FLAG_AUDIO_TRANSFORM (model_config.go), FeatureAudioTransform (default-on,
in APIFeatures), three RouteFeatureRegistry rows.
- localvqe added to knownPrefOnlyBackends with modality "audio-transform".
- Gallery entry localvqe-v1-1.3m (sha256-pinned, hosted on
huggingface.co/LocalAI-io/LocalVQE).
React UI:
- New /app/transform page surfaced via a dedicated "Enhance" sidebar
section (sibling of Tools / Biometrics) — the page is enhancement, not
generation, so it lives outside Studio. Two AudioInput components
(Upload + Record tabs, drag-drop, mic capture).
- Echo-test button: records mic while playing the loaded reference through
the speakers — the mic naturally picks up speaker bleed, giving a real
(mic, ref) pair for AEC testing without leaving the UI.
- Reusable WaveformPlayer (canvas peaks + click-to-seek + audio controls)
and useAudioPeaks hook (shared module-scoped AudioContext to avoid
hitting browser context limits with three players on one page); migrated
TTS, Sound, Traces audio blocks to use it.
- Past runs saved in localStorage via useMediaHistory('audio-transform') —
the history entry stores all three URLs so clicking re-renders the full
triple, not just the output.
Build + e2e:
- 11 matrix entries removed from .github/workflows/backend.yml (CUDA, ROCm,
SYCL, Metal, L4T): upstream supports only CPU + Vulkan, so we ship those
two and let GPU-class hardware route through Vulkan in the gallery
capabilities map.
- tests-localvqe-grpc-transform job in test-extra.yml (gated on
detect-changes.outputs.localvqe).
- New audio_transform capability + 4 specs in tests/e2e-backends.
- Playwright spec suite in core/http/react-ui/e2e/audio-transform.spec.js
(8 specs covering tabs, file upload, multipart shape, history, errors).
Docs:
- New docs/content/features/audio-transform.md covering the (audio,
reference) mental model, batch + WebSocket wire formats, LocalVQE param
keys, and a YAML config example. Cross-links from text-to-audio and
audio-to-text feature pages.
Assisted-by: Claude:claude-opus-4-7 [Bash Read Edit Write Agent TaskCreate]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
147 lines
6.6 KiB
JavaScript
147 lines
6.6 KiB
JavaScript
import { createBrowserRouter, Navigate, useParams } from 'react-router-dom'
|
|
import { routerBasename } from './utils/basePath'
|
|
import App from './App'
|
|
import Home from './pages/Home'
|
|
import Chat from './pages/Chat'
|
|
import Models from './pages/Models'
|
|
import Manage from './pages/Manage'
|
|
import ImageGen from './pages/ImageGen'
|
|
import VideoGen from './pages/VideoGen'
|
|
import TTS from './pages/TTS'
|
|
import Sound from './pages/Sound'
|
|
import AudioTransform from './pages/AudioTransform'
|
|
import Talk from './pages/Talk'
|
|
import Backends from './pages/Backends'
|
|
import Settings from './pages/Settings'
|
|
import Traces from './pages/Traces'
|
|
import P2P from './pages/P2P'
|
|
import Agents from './pages/Agents'
|
|
import AgentCreate from './pages/AgentCreate'
|
|
import AgentChat from './pages/AgentChat'
|
|
import AgentStatus from './pages/AgentStatus'
|
|
import Collections from './pages/Collections'
|
|
import CollectionDetails from './pages/CollectionDetails'
|
|
import Skills from './pages/Skills'
|
|
import SkillEdit from './pages/SkillEdit'
|
|
import AgentJobs from './pages/AgentJobs'
|
|
import AgentTaskDetails from './pages/AgentTaskDetails'
|
|
import AgentJobDetails from './pages/AgentJobDetails'
|
|
import ModelEditor from './pages/ModelEditor'
|
|
// PipelineEditor removed — the Model Editor with templates handles all model types
|
|
import ImportModel from './pages/ImportModel'
|
|
import BackendLogs from './pages/BackendLogs'
|
|
import Explorer from './pages/Explorer'
|
|
import Login from './pages/Login'
|
|
import FineTune from './pages/FineTune'
|
|
import Quantize from './pages/Quantize'
|
|
import Studio from './pages/Studio'
|
|
import FaceRecognition from './pages/FaceRecognition'
|
|
import VoiceRecognition from './pages/VoiceRecognition'
|
|
import Nodes from './pages/Nodes'
|
|
import NodeBackendLogs from './pages/NodeBackendLogs'
|
|
import NotFound from './pages/NotFound'
|
|
import Usage from './pages/Usage'
|
|
import Users from './pages/Users'
|
|
import Account from './pages/Account'
|
|
import RequireAdmin from './components/RequireAdmin'
|
|
import RequireAuth from './components/RequireAuth'
|
|
import RequireFeature from './components/RequireFeature'
|
|
|
|
function BrowseRedirect() {
|
|
const { '*': splat } = useParams()
|
|
return <Navigate to={`/app/${splat || ''}`} replace />
|
|
}
|
|
|
|
|
|
function Admin({ children }) {
|
|
return <RequireAdmin>{children}</RequireAdmin>
|
|
}
|
|
|
|
function Feature({ feature, children }) {
|
|
return <RequireFeature feature={feature}>{children}</RequireFeature>
|
|
}
|
|
|
|
const appChildren = [
|
|
{ index: true, element: <Home /> },
|
|
{ path: 'models', element: <Admin><Models /></Admin> },
|
|
{ path: 'chat', element: <Chat /> },
|
|
{ path: 'chat/:model', element: <Chat /> },
|
|
{ path: 'image', element: <ImageGen /> },
|
|
{ path: 'image/:model', element: <ImageGen /> },
|
|
{ path: 'video', element: <VideoGen /> },
|
|
{ path: 'video/:model', element: <VideoGen /> },
|
|
{ path: 'tts', element: <TTS /> },
|
|
{ path: 'tts/:model', element: <TTS /> },
|
|
{ path: 'sound', element: <Sound /> },
|
|
{ path: 'sound/:model', element: <Sound /> },
|
|
{ path: 'transform', element: <Feature feature="audio_transform"><AudioTransform /></Feature> },
|
|
{ path: 'transform/:model', element: <Feature feature="audio_transform"><AudioTransform /></Feature> },
|
|
{ path: 'studio', element: <Studio /> },
|
|
{ path: 'talk', element: <Talk /> },
|
|
{ path: 'face', element: <Feature feature="face_recognition"><FaceRecognition /></Feature> },
|
|
{ path: 'face/:model', element: <Feature feature="face_recognition"><FaceRecognition /></Feature> },
|
|
{ path: 'voice', element: <Feature feature="voice_recognition"><VoiceRecognition /></Feature> },
|
|
{ path: 'voice/:model', element: <Feature feature="voice_recognition"><VoiceRecognition /></Feature> },
|
|
{ path: 'usage', element: <Usage /> },
|
|
{ path: 'account', element: <Account /> },
|
|
{ path: 'users', element: <Admin><Users /></Admin> },
|
|
{ path: 'manage', element: <Admin><Manage /></Admin> },
|
|
{ path: 'backends', element: <Admin><Backends /></Admin> },
|
|
{ path: 'settings', element: <Admin><Settings /></Admin> },
|
|
{ path: 'traces', element: <Admin><Traces /></Admin> },
|
|
{ path: 'backend-logs/:modelId', element: <Admin><BackendLogs /></Admin> },
|
|
{ path: 'p2p', element: <Admin><P2P /></Admin> },
|
|
{ path: 'nodes', element: <Admin><Nodes /></Admin> },
|
|
{ path: 'node-backend-logs/:nodeId/:modelId', element: <Admin><NodeBackendLogs /></Admin> },
|
|
{ path: 'agents', element: <Feature feature="agents"><Agents /></Feature> },
|
|
{ path: 'agents/new', element: <Feature feature="agents"><AgentCreate /></Feature> },
|
|
{ path: 'agents/:name/edit', element: <Feature feature="agents"><AgentCreate /></Feature> },
|
|
{ path: 'agents/:name/chat', element: <Feature feature="agents"><AgentChat /></Feature> },
|
|
{ path: 'agents/:name/status', element: <Feature feature="agents"><AgentStatus /></Feature> },
|
|
{ path: 'collections', element: <Feature feature="collections"><Collections /></Feature> },
|
|
{ path: 'collections/:name', element: <Feature feature="collections"><CollectionDetails /></Feature> },
|
|
{ path: 'skills', element: <Feature feature="skills"><Skills /></Feature> },
|
|
{ path: 'skills/new', element: <Feature feature="skills"><SkillEdit /></Feature> },
|
|
{ path: 'skills/edit/:name', element: <Feature feature="skills"><SkillEdit /></Feature> },
|
|
{ path: 'agent-jobs', element: <Feature feature="mcp_jobs"><AgentJobs /></Feature> },
|
|
{ path: 'agent-jobs/tasks/new', element: <Feature feature="mcp_jobs"><AgentTaskDetails /></Feature> },
|
|
{ path: 'agent-jobs/tasks/:id', element: <Feature feature="mcp_jobs"><AgentTaskDetails /></Feature> },
|
|
{ path: 'agent-jobs/tasks/:id/edit', element: <Feature feature="mcp_jobs"><AgentTaskDetails /></Feature> },
|
|
{ path: 'agent-jobs/jobs/:id', element: <Feature feature="mcp_jobs"><AgentJobDetails /></Feature> },
|
|
{ path: 'fine-tune', element: <Feature feature="fine_tuning"><FineTune /></Feature> },
|
|
{ path: 'quantize', element: <Feature feature="quantization"><Quantize /></Feature> },
|
|
{ path: 'model-editor', element: <Admin><ModelEditor /></Admin> },
|
|
{ path: 'model-editor/:name', element: <Admin><ModelEditor /></Admin> },
|
|
{ path: 'import-model', element: <Admin><ImportModel /></Admin> },
|
|
{ path: '*', element: <NotFound /> },
|
|
]
|
|
|
|
export const router = createBrowserRouter([
|
|
{
|
|
path: '/login',
|
|
element: <Login />,
|
|
},
|
|
{
|
|
path: '/invite/:code',
|
|
element: <Login />,
|
|
},
|
|
{
|
|
path: '/explorer',
|
|
element: <Explorer />,
|
|
},
|
|
{
|
|
path: '/app',
|
|
element: <RequireAuth><App /></RequireAuth>,
|
|
children: appChildren,
|
|
},
|
|
// Backward compatibility: redirect /browse/* to /app/*
|
|
{
|
|
path: '/browse/*',
|
|
element: <BrowseRedirect />,
|
|
},
|
|
{
|
|
path: '/',
|
|
element: <Navigate to="/app" replace />,
|
|
},
|
|
], { basename: routerBasename })
|