mirror of
https://github.com/mudler/LocalAI.git
synced 2026-04-30 03:55:58 -04:00
* feat(voice-recognition): add /v1/voice/{verify,analyze,embed} + speaker-recognition backend
Audio analog to face recognition. Adds three gRPC RPCs
(VoiceVerify / VoiceAnalyze / VoiceEmbed), their Go service and HTTP
layers, a new FLAG_SPEAKER_RECOGNITION capability flag, and a Python
backend scaffold under backend/python/speaker-recognition/ wrapping
SpeechBrain ECAPA-TDNN with a parallel OnnxDirectEngine for
WeSpeaker / 3D-Speaker ONNX exports.
The kokoros Rust backend gets matching unimplemented trait stubs —
tonic's async_trait has no defaults, so adding an RPC without Rust
stubs breaks the build (same regression fixed by eb01c772 for face).
Swagger, /api/instructions, and the auth RouteFeatureRegistry /
APIFeatures list are updated so the endpoints surface everywhere a
client or admin UI looks.
Assisted-by: Claude:claude-opus-4-7
* feat(voice-recognition): add 1:N identify + register/forget endpoints
Mirrors the face-recognition register/identify/forget surface. New
package core/services/voicerecognition/ carries a Registry interface
and a local-store-backed implementation (same in-memory vector-store
plumbing facerecognition uses, separate instance so the embedding
spaces stay isolated).
Handlers under /v1/voice/{register,identify,forget} reuse
backend.VoiceEmbed to compute the probe vector, then delegate the
nearest-neighbour search to the registry. Default cosine-distance
threshold is tuned for ECAPA-TDNN on VoxCeleb (0.25, EER ~1.9%).
As with the face registry, the current backing is in-memory only — a
pgvector implementation is a future constructor-level swap.
Assisted-by: Claude:claude-opus-4-7
* feat(voice-recognition): gallery, docs, CI and e2e coverage
- backend/index.yaml: speaker-recognition backend entry + CPU and
CUDA-12 image variants (plus matching development variants).
- gallery/index.yaml: speechbrain-ecapa-tdnn (default) and
wespeaker-resnet34 model entries. The WeSpeaker SHA-256 is a
deliberate placeholder — the HF URI must be curl'd and its hash
filled in before the entry installs.
- docs/content/features/voice-recognition.md: API reference + quickstart,
mirrors the face-recognition docs.
- React UI: CAP_SPEAKER_RECOGNITION flag export (consumers follow face's
precedent — no dedicated tab yet).
- tests/e2e-backends: voice_embed / voice_verify / voice_analyze specs.
Helper resolveFaceFixture is reused as-is — the only thing face/voice
share is "download a file into workDir", so no need for a new helper.
- Makefile: docker-build-speaker-recognition + test-extra-backend-
speaker-recognition-{ecapa,all} targets. Audio fixtures default to
VCTK p225/p226 samples from HuggingFace.
- CI: test-extra.yml grows a tests-speaker-recognition-grpc job
mirroring insightface. backend.yml matrix gains CPU + CUDA-12 image
build entries — scripts/changed-backends.js auto-picks these up.
Assisted-by: Claude:claude-opus-4-7
* feat(voice-recognition): wire a working /v1/voice/analyze head
Adds AnalysisHead: a lazy-loading age / gender / emotion inference
wrapper that plugs into both SpeechBrainEngine and OnnxDirectEngine.
Defaults to two open-licence HuggingFace checkpoints:
- audeering/wav2vec2-large-robust-24-ft-age-gender (Apache 2.0) —
age regression + 3-way gender (female / male / child).
- superb/wav2vec2-base-superb-er (Apache 2.0) — 4-way emotion.
Both are optional and degrade gracefully when transformers or the
model can't be loaded — the engine raises NotImplementedError so the
gRPC layer returns 501 instead of a generic 500.
Emotion classes pass through from the model (neutral/happy/angry/sad
on the default checkpoint); the e2e test now accepts any non-empty
dominant gender so custom age_gender_model overrides don't fail it.
Adds transformers to the backend's CPU and CUDA-12 requirements.
Assisted-by: Claude:claude-opus-4-7
* fix(voice-recognition): pin real WeSpeaker ResNet34 ONNX SHA-256
Replaces the placeholder hash in gallery/index.yaml with the actual
SHA-256 (7bb2f06e…) of the upstream
Wespeaker/wespeaker-voxceleb-resnet34-LM ONNX at ~25MB. `local-ai
models install wespeaker-resnet34` now succeeds.
Assisted-by: Claude:claude-opus-4-7
* fix(voice-recognition): soundfile loader + honest analyze default
Two issues surfaced on first end-to-end smoke with the actual backend
image:
1. torchaudio.load in torchaudio 2.8+ requires the torchcodec package
for audio decoding. Switch SpeechBrainEngine._load_waveform to the
already-present soundfile (listed in requirements.txt) plus a numpy
linear resample to 16kHz. Drops a heavy ffmpeg-linked dep and the
codepath we never exercise (torchaudio's ffmpeg backend).
2. The AnalysisHead was defaulting to audeering/wav2vec2-large-robust-
24-ft-age-gender, but AutoModelForAudioClassification silently
mangles that checkpoint — it reports the age head weights as
UNEXPECTED and re-initialises the classifier head with random
values, so the "gender" output is noise and there is no age output
at all. Make age/gender opt-in instead (empty default; users wire
a cleanly-loadable Wav2Vec2ForSequenceClassification checkpoint via
age_gender_model: option). Emotion keeps its working Superb default.
Also broaden _infer_age_gender's tensor-shape handling and catch
runtime exceptions so a dodgy age/gender head never takes down the
whole analyze call.
Docs and README updated to match the new policy.
Verified with the branch-scoped gallery on localhost:
- voice/embed → 192-d ECAPA-TDNN vector
- voice/verify → same-clip dist≈6e-08 verified=true; cross-speaker
dist 0.76–0.99 verified=false (as expected)
- voice/register/identify/forget → round-trip works, 404 on unknown id
- voice/analyze → emotion populated, age/gender omitted (opt-in)
Assisted-by: Claude:claude-opus-4-7
* fix(voice-recognition): real CI audio fixtures + fixture-agnostic verify spec
Two issues surfaced after CI actually ran the speaker-recognition e2e
target (I'd curl-tested against a running server but hadn't run the
make target locally):
1. The default BACKEND_TEST_VOICE_AUDIO_* URLs pointed at
huggingface.co/datasets/CSTR-Edinburgh/vctk paths that return 404
(the dataset is gated). Swap them for the speechbrain test samples
served from github.com/speechbrain/speechbrain/raw/develop/ —
public, no auth, correct 16kHz mono format.
2. The VoiceVerify spec required d(file1,file2) < 0.4, assuming
file1/file2 were same-speaker. The speechbrain samples are three
different speakers (example1/2/5), and there is no easy un-gated
source of true same-speaker audio pairs (VoxCeleb/VCTK/LibriSpeech
are all license- or size-gated for CI use). Replace the ceiling
check with a relative-ordering assertion: d(pair) > d(same-clip)
for both file2 and file3 — that's enough to prove the embeddings
encode speaker info, and it works with any three non-identical
clips. Actual speaker ordering d(1,2) vs d(1,3) is logged but not
asserted.
Local run: 4/4 voice specs pass (Health, LoadModel, VoiceEmbed,
VoiceVerify) on the built backend image. 12 non-voice specs skipped
as expected.
Assisted-by: Claude:claude-opus-4-7
* fix(ci): checkout with submodules in the reusable backend_build workflow
The kokoros Rust backend build fails with
failed to read .../sources/Kokoros/kokoros/Cargo.toml: No such file
because the reusable backend_build.yml workflow's actions/checkout
step was missing `submodules: true`. Dockerfile.rust does `COPY .
/LocalAI`, and without the submodule files the subsequent `cargo
build` can't find the vendored Kokoros crate.
The bug pre-dates this PR — scripts/changed-backends.js only triggers
the kokoros image job when something under backend/rust/kokoros or
the shared proto changes, so master had been coasting past it. The
voice-recognition proto addition re-broke it.
Other checkouts in backend.yml (llama-cpp-darwin) and test-extra.yml
(insightface, kokoros, speaker-recognition) already pass
`submodules: true`; this brings the shared backend image builder in
line.
Assisted-by: Claude:claude-opus-4-7
222 lines
6.5 KiB
Go
222 lines
6.5 KiB
Go
package auth
|
|
|
|
import (
|
|
"github.com/google/uuid"
|
|
"github.com/labstack/echo/v4"
|
|
"gorm.io/gorm"
|
|
)
|
|
|
|
const contextKeyPermissions = "auth_permissions"
|
|
|
|
// GetCachedUserPermissions returns the user's permission record, using a
|
|
// request-scoped cache stored in the echo context. This avoids duplicate
|
|
// DB lookups when multiple middlewares (RequireRouteFeature, RequireModelAccess)
|
|
// both need permissions in the same request.
|
|
func GetCachedUserPermissions(c echo.Context, db *gorm.DB, userID string) (*UserPermission, error) {
|
|
if perm, ok := c.Get(contextKeyPermissions).(*UserPermission); ok && perm != nil {
|
|
return perm, nil
|
|
}
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
c.Set(contextKeyPermissions, perm)
|
|
return perm, nil
|
|
}
|
|
|
|
// Feature name constants — all code must use these, never bare strings.
|
|
const (
|
|
// Agent features (default OFF for new users)
|
|
FeatureAgents = "agents"
|
|
FeatureSkills = "skills"
|
|
FeatureCollections = "collections"
|
|
FeatureMCPJobs = "mcp_jobs"
|
|
|
|
// General features (default OFF for new users)
|
|
FeatureFineTuning = "fine_tuning"
|
|
FeatureQuantization = "quantization"
|
|
|
|
// API features (default ON for new users)
|
|
FeatureChat = "chat"
|
|
FeatureImages = "images"
|
|
FeatureAudioSpeech = "audio_speech"
|
|
FeatureAudioTranscription = "audio_transcription"
|
|
FeatureVAD = "vad"
|
|
FeatureDetection = "detection"
|
|
FeatureVideo = "video"
|
|
FeatureEmbeddings = "embeddings"
|
|
FeatureSound = "sound"
|
|
FeatureRealtime = "realtime"
|
|
FeatureRerank = "rerank"
|
|
FeatureTokenize = "tokenize"
|
|
FeatureMCP = "mcp"
|
|
FeatureStores = "stores"
|
|
FeatureFaceRecognition = "face_recognition"
|
|
FeatureVoiceRecognition = "voice_recognition"
|
|
)
|
|
|
|
// AgentFeatures lists agent-related features (default OFF).
|
|
var AgentFeatures = []string{FeatureAgents, FeatureSkills, FeatureCollections, FeatureMCPJobs}
|
|
|
|
// GeneralFeatures lists general features (default OFF).
|
|
var GeneralFeatures = []string{FeatureFineTuning, FeatureQuantization}
|
|
|
|
// APIFeatures lists API endpoint features (default ON).
|
|
var APIFeatures = []string{
|
|
FeatureChat, FeatureImages, FeatureAudioSpeech, FeatureAudioTranscription,
|
|
FeatureVAD, FeatureDetection, FeatureVideo, FeatureEmbeddings, FeatureSound,
|
|
FeatureRealtime, FeatureRerank, FeatureTokenize, FeatureMCP, FeatureStores,
|
|
FeatureFaceRecognition, FeatureVoiceRecognition,
|
|
}
|
|
|
|
// AllFeatures lists all known features (used by UI and validation).
|
|
var AllFeatures = append(append(append([]string{}, AgentFeatures...), GeneralFeatures...), APIFeatures...)
|
|
|
|
// defaultOnFeatures is the set of features that default to ON when absent from a user's permission map.
|
|
var defaultOnFeatures = func() map[string]bool {
|
|
m := map[string]bool{}
|
|
for _, f := range APIFeatures {
|
|
m[f] = true
|
|
}
|
|
return m
|
|
}()
|
|
|
|
// isDefaultOnFeature returns true if the feature defaults to ON when not explicitly set.
|
|
func isDefaultOnFeature(feature string) bool {
|
|
return defaultOnFeatures[feature]
|
|
}
|
|
|
|
// GetUserPermissions returns the permission record for a user, creating a default
|
|
// (empty map = all disabled) if none exists.
|
|
func GetUserPermissions(db *gorm.DB, userID string) (*UserPermission, error) {
|
|
var perm UserPermission
|
|
err := db.Where("user_id = ?", userID).First(&perm).Error
|
|
if err == gorm.ErrRecordNotFound {
|
|
perm = UserPermission{
|
|
ID: uuid.New().String(),
|
|
UserID: userID,
|
|
Permissions: PermissionMap{},
|
|
}
|
|
if err := db.Create(&perm).Error; err != nil {
|
|
return nil, err
|
|
}
|
|
return &perm, nil
|
|
}
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &perm, nil
|
|
}
|
|
|
|
// UpdateUserPermissions upserts the permission map for a user.
|
|
func UpdateUserPermissions(db *gorm.DB, userID string, perms PermissionMap) error {
|
|
var perm UserPermission
|
|
err := db.Where("user_id = ?", userID).First(&perm).Error
|
|
if err == gorm.ErrRecordNotFound {
|
|
perm = UserPermission{
|
|
ID: uuid.New().String(),
|
|
UserID: userID,
|
|
Permissions: perms,
|
|
}
|
|
return db.Create(&perm).Error
|
|
}
|
|
if err != nil {
|
|
return err
|
|
}
|
|
perm.Permissions = perms
|
|
return db.Save(&perm).Error
|
|
}
|
|
|
|
// HasFeatureAccess returns true if the user is an admin or has the given feature enabled.
|
|
// When a feature key is absent from the user's permission map, it checks whether the
|
|
// feature defaults to ON (API features) or OFF (agent features) for backward compatibility.
|
|
func HasFeatureAccess(db *gorm.DB, user *User, feature string) bool {
|
|
if user == nil {
|
|
return false
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
return true
|
|
}
|
|
perm, err := GetUserPermissions(db, user.ID)
|
|
if err != nil {
|
|
return false
|
|
}
|
|
val, exists := perm.Permissions[feature]
|
|
if !exists {
|
|
return isDefaultOnFeature(feature)
|
|
}
|
|
return val
|
|
}
|
|
|
|
// GetPermissionMapForUser returns the effective permission map for a user.
|
|
// Admins get all features as true (virtual).
|
|
// For regular users, absent keys are filled with their defaults so the
|
|
// UI/API always returns a complete picture.
|
|
func GetPermissionMapForUser(db *gorm.DB, user *User) PermissionMap {
|
|
if user == nil {
|
|
return PermissionMap{}
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
m := PermissionMap{}
|
|
for _, f := range AllFeatures {
|
|
m[f] = true
|
|
}
|
|
return m
|
|
}
|
|
perm, err := GetUserPermissions(db, user.ID)
|
|
if err != nil {
|
|
return PermissionMap{}
|
|
}
|
|
// Fill in defaults for absent keys
|
|
effective := PermissionMap{}
|
|
for _, f := range AllFeatures {
|
|
val, exists := perm.Permissions[f]
|
|
if exists {
|
|
effective[f] = val
|
|
} else {
|
|
effective[f] = isDefaultOnFeature(f)
|
|
}
|
|
}
|
|
return effective
|
|
}
|
|
|
|
// GetModelAllowlist returns the model allowlist for a user.
|
|
func GetModelAllowlist(db *gorm.DB, userID string) ModelAllowlist {
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return ModelAllowlist{}
|
|
}
|
|
return perm.AllowedModels
|
|
}
|
|
|
|
// UpdateModelAllowlist updates the model allowlist for a user.
|
|
func UpdateModelAllowlist(db *gorm.DB, userID string, allowlist ModelAllowlist) error {
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
perm.AllowedModels = allowlist
|
|
return db.Save(perm).Error
|
|
}
|
|
|
|
// IsModelAllowed returns true if the user is allowed to use the given model.
|
|
// Admins always have access. If the allowlist is not enabled, all models are allowed.
|
|
func IsModelAllowed(db *gorm.DB, user *User, modelName string) bool {
|
|
if user == nil {
|
|
return false
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
return true
|
|
}
|
|
allowlist := GetModelAllowlist(db, user.ID)
|
|
if !allowlist.Enabled {
|
|
return true
|
|
}
|
|
for _, m := range allowlist.Models {
|
|
if m == modelName {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|