mirror of
https://github.com/mudler/LocalAI.git
synced 2026-05-17 04:56:52 -04:00
feat(audio-transform): add LocalVQE backend, bidi gRPC RPC, Studio UI
Introduce a generic "audio transform" capability for any audio-in / audio-out
operation (echo cancellation, noise suppression, dereverberation, voice
conversion, etc.) and ship LocalVQE as the first backend implementation.
Backend protocol:
- Two new gRPC RPCs in backend.proto: unary AudioTransform for batch and
bidirectional AudioTransformStream for low-latency frame-by-frame use.
This is the first bidi stream in the proto; per-frame unary at LocalVQE's
16 ms hop would be RTT-bound. Wire it through pkg/grpc/{client,server,
embed,interface,base} with paired-channel ergonomics.
LocalVQE backend (backend/go/localvqe/):
- Go-Purego wrapper around upstream liblocalvqe.so. CMake builds the upstream
shared lib + its libggml-cpu-*.so runtime variants directly — no MODULE
wrapper needed because LocalVQE handles CPU feature selection internally
via GGML_BACKEND_DL.
- Sets GGML_NTHREADS from opts.Threads (or runtime.NumCPU()-1) — without it
LocalVQE runs single-threaded at ~1× realtime instead of the documented
~9.6×.
- Reference-length policy: zero-pad short refs, truncate long ones (the
trailing portion can't have leaked into a mic that wasn't recording).
- Ginkgo test suite (9 always-on specs + 2 model-gated).
HTTP layer:
- POST /audio/transformations (alias /audio/transform): multipart batch
endpoint, accepts audio + optional reference + params[*]=v form fields.
Persists inputs alongside the output in GeneratedContentDir/audio so the
React UI history can replay past (audio, reference, output) triples.
- GET /audio/transformations/stream: WebSocket bidi, 16 ms PCM frames
(interleaved stereo mic+ref in, mono out). JSON session.update envelope
for config; constants hoisted in core/schema/audio_transform.go.
- ffmpeg-based input normalisation to 16 kHz mono s16 WAV via the existing
utils.AudioToWav (with passthrough fast-path), so the user can upload any
format / rate without seeing the model's strict 16 kHz constraint.
- BackendTraceAudioTransform integration so /api/backend-traces and the
Traces UI light up with audio_snippet base64 and timing.
- Routes registered under routes/localai.go (LocalAI extension; OpenAI has
no /audio/transformations endpoint), traced via TraceMiddleware.
Auth + capability + importer:
- FLAG_AUDIO_TRANSFORM (model_config.go), FeatureAudioTransform (default-on,
in APIFeatures), three RouteFeatureRegistry rows.
- localvqe added to knownPrefOnlyBackends with modality "audio-transform".
- Gallery entry localvqe-v1-1.3m (sha256-pinned, hosted on
huggingface.co/LocalAI-io/LocalVQE).
React UI:
- New /app/transform page surfaced via a dedicated "Enhance" sidebar
section (sibling of Tools / Biometrics) — the page is enhancement, not
generation, so it lives outside Studio. Two AudioInput components
(Upload + Record tabs, drag-drop, mic capture).
- Echo-test button: records mic while playing the loaded reference through
the speakers — the mic naturally picks up speaker bleed, giving a real
(mic, ref) pair for AEC testing without leaving the UI.
- Reusable WaveformPlayer (canvas peaks + click-to-seek + audio controls)
and useAudioPeaks hook (shared module-scoped AudioContext to avoid
hitting browser context limits with three players on one page); migrated
TTS, Sound, Traces audio blocks to use it.
- Past runs saved in localStorage via useMediaHistory('audio-transform') —
the history entry stores all three URLs so clicking re-renders the full
triple, not just the output.
Build + e2e:
- 11 matrix entries removed from .github/workflows/backend.yml (CUDA, ROCm,
SYCL, Metal, L4T): upstream supports only CPU + Vulkan, so we ship those
two and let GPU-class hardware route through Vulkan in the gallery
capabilities map.
- tests-localvqe-grpc-transform job in test-extra.yml (gated on
detect-changes.outputs.localvqe).
- New audio_transform capability + 4 specs in tests/e2e-backends.
- Playwright spec suite in core/http/react-ui/e2e/audio-transform.spec.js
(8 specs covering tabs, file upload, multipart shape, history, errors).
Docs:
- New docs/content/features/audio-transform.md covering the (audio,
reference) mental model, batch + WebSocket wire formats, LocalVQE param
keys, and a YAML config example. Cross-links from text-to-audio and
audio-to-text feature pages.
Assisted-by: Claude:claude-opus-4-7 [Bash Read Edit Write Agent TaskCreate]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
224 lines
6.7 KiB
Go
224 lines
6.7 KiB
Go
package auth
|
|
|
|
import (
|
|
"github.com/google/uuid"
|
|
"github.com/labstack/echo/v4"
|
|
"gorm.io/gorm"
|
|
)
|
|
|
|
const contextKeyPermissions = "auth_permissions"
|
|
|
|
// GetCachedUserPermissions returns the user's permission record, using a
|
|
// request-scoped cache stored in the echo context. This avoids duplicate
|
|
// DB lookups when multiple middlewares (RequireRouteFeature, RequireModelAccess)
|
|
// both need permissions in the same request.
|
|
func GetCachedUserPermissions(c echo.Context, db *gorm.DB, userID string) (*UserPermission, error) {
|
|
if perm, ok := c.Get(contextKeyPermissions).(*UserPermission); ok && perm != nil {
|
|
return perm, nil
|
|
}
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
c.Set(contextKeyPermissions, perm)
|
|
return perm, nil
|
|
}
|
|
|
|
// Feature name constants — all code must use these, never bare strings.
|
|
const (
|
|
// Agent features (default OFF for new users)
|
|
FeatureAgents = "agents"
|
|
FeatureSkills = "skills"
|
|
FeatureCollections = "collections"
|
|
FeatureMCPJobs = "mcp_jobs"
|
|
FeatureLocalAIAssistant = "localai_assistant"
|
|
|
|
// General features (default OFF for new users)
|
|
FeatureFineTuning = "fine_tuning"
|
|
FeatureQuantization = "quantization"
|
|
|
|
// API features (default ON for new users)
|
|
FeatureChat = "chat"
|
|
FeatureImages = "images"
|
|
FeatureAudioSpeech = "audio_speech"
|
|
FeatureAudioTranscription = "audio_transcription"
|
|
FeatureVAD = "vad"
|
|
FeatureDetection = "detection"
|
|
FeatureVideo = "video"
|
|
FeatureEmbeddings = "embeddings"
|
|
FeatureSound = "sound"
|
|
FeatureRealtime = "realtime"
|
|
FeatureRerank = "rerank"
|
|
FeatureTokenize = "tokenize"
|
|
FeatureMCP = "mcp"
|
|
FeatureStores = "stores"
|
|
FeatureFaceRecognition = "face_recognition"
|
|
FeatureVoiceRecognition = "voice_recognition"
|
|
FeatureAudioTransform = "audio_transform"
|
|
)
|
|
|
|
// AgentFeatures lists agent-related features (default OFF).
|
|
var AgentFeatures = []string{FeatureAgents, FeatureSkills, FeatureCollections, FeatureMCPJobs, FeatureLocalAIAssistant}
|
|
|
|
// GeneralFeatures lists general features (default OFF).
|
|
var GeneralFeatures = []string{FeatureFineTuning, FeatureQuantization}
|
|
|
|
// APIFeatures lists API endpoint features (default ON).
|
|
var APIFeatures = []string{
|
|
FeatureChat, FeatureImages, FeatureAudioSpeech, FeatureAudioTranscription,
|
|
FeatureVAD, FeatureDetection, FeatureVideo, FeatureEmbeddings, FeatureSound,
|
|
FeatureRealtime, FeatureRerank, FeatureTokenize, FeatureMCP, FeatureStores,
|
|
FeatureFaceRecognition, FeatureVoiceRecognition, FeatureAudioTransform,
|
|
}
|
|
|
|
// AllFeatures lists all known features (used by UI and validation).
|
|
var AllFeatures = append(append(append([]string{}, AgentFeatures...), GeneralFeatures...), APIFeatures...)
|
|
|
|
// defaultOnFeatures is the set of features that default to ON when absent from a user's permission map.
|
|
var defaultOnFeatures = func() map[string]bool {
|
|
m := map[string]bool{}
|
|
for _, f := range APIFeatures {
|
|
m[f] = true
|
|
}
|
|
return m
|
|
}()
|
|
|
|
// isDefaultOnFeature returns true if the feature defaults to ON when not explicitly set.
|
|
func isDefaultOnFeature(feature string) bool {
|
|
return defaultOnFeatures[feature]
|
|
}
|
|
|
|
// GetUserPermissions returns the permission record for a user, creating a default
|
|
// (empty map = all disabled) if none exists.
|
|
func GetUserPermissions(db *gorm.DB, userID string) (*UserPermission, error) {
|
|
var perm UserPermission
|
|
err := db.Where("user_id = ?", userID).First(&perm).Error
|
|
if err == gorm.ErrRecordNotFound {
|
|
perm = UserPermission{
|
|
ID: uuid.New().String(),
|
|
UserID: userID,
|
|
Permissions: PermissionMap{},
|
|
}
|
|
if err := db.Create(&perm).Error; err != nil {
|
|
return nil, err
|
|
}
|
|
return &perm, nil
|
|
}
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &perm, nil
|
|
}
|
|
|
|
// UpdateUserPermissions upserts the permission map for a user.
|
|
func UpdateUserPermissions(db *gorm.DB, userID string, perms PermissionMap) error {
|
|
var perm UserPermission
|
|
err := db.Where("user_id = ?", userID).First(&perm).Error
|
|
if err == gorm.ErrRecordNotFound {
|
|
perm = UserPermission{
|
|
ID: uuid.New().String(),
|
|
UserID: userID,
|
|
Permissions: perms,
|
|
}
|
|
return db.Create(&perm).Error
|
|
}
|
|
if err != nil {
|
|
return err
|
|
}
|
|
perm.Permissions = perms
|
|
return db.Save(&perm).Error
|
|
}
|
|
|
|
// HasFeatureAccess returns true if the user is an admin or has the given feature enabled.
|
|
// When a feature key is absent from the user's permission map, it checks whether the
|
|
// feature defaults to ON (API features) or OFF (agent features) for backward compatibility.
|
|
func HasFeatureAccess(db *gorm.DB, user *User, feature string) bool {
|
|
if user == nil {
|
|
return false
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
return true
|
|
}
|
|
perm, err := GetUserPermissions(db, user.ID)
|
|
if err != nil {
|
|
return false
|
|
}
|
|
val, exists := perm.Permissions[feature]
|
|
if !exists {
|
|
return isDefaultOnFeature(feature)
|
|
}
|
|
return val
|
|
}
|
|
|
|
// GetPermissionMapForUser returns the effective permission map for a user.
|
|
// Admins get all features as true (virtual).
|
|
// For regular users, absent keys are filled with their defaults so the
|
|
// UI/API always returns a complete picture.
|
|
func GetPermissionMapForUser(db *gorm.DB, user *User) PermissionMap {
|
|
if user == nil {
|
|
return PermissionMap{}
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
m := PermissionMap{}
|
|
for _, f := range AllFeatures {
|
|
m[f] = true
|
|
}
|
|
return m
|
|
}
|
|
perm, err := GetUserPermissions(db, user.ID)
|
|
if err != nil {
|
|
return PermissionMap{}
|
|
}
|
|
// Fill in defaults for absent keys
|
|
effective := PermissionMap{}
|
|
for _, f := range AllFeatures {
|
|
val, exists := perm.Permissions[f]
|
|
if exists {
|
|
effective[f] = val
|
|
} else {
|
|
effective[f] = isDefaultOnFeature(f)
|
|
}
|
|
}
|
|
return effective
|
|
}
|
|
|
|
// GetModelAllowlist returns the model allowlist for a user.
|
|
func GetModelAllowlist(db *gorm.DB, userID string) ModelAllowlist {
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return ModelAllowlist{}
|
|
}
|
|
return perm.AllowedModels
|
|
}
|
|
|
|
// UpdateModelAllowlist updates the model allowlist for a user.
|
|
func UpdateModelAllowlist(db *gorm.DB, userID string, allowlist ModelAllowlist) error {
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
perm.AllowedModels = allowlist
|
|
return db.Save(perm).Error
|
|
}
|
|
|
|
// IsModelAllowed returns true if the user is allowed to use the given model.
|
|
// Admins always have access. If the allowlist is not enabled, all models are allowed.
|
|
func IsModelAllowed(db *gorm.DB, user *User, modelName string) bool {
|
|
if user == nil {
|
|
return false
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
return true
|
|
}
|
|
allowlist := GetModelAllowlist(db, user.ID)
|
|
if !allowlist.Enabled {
|
|
return true
|
|
}
|
|
for _, m := range allowlist.Models {
|
|
if m == modelName {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|