mirror of
https://github.com/mudler/LocalAI.git
synced 2026-06-22 07:39:02 -04:00
* feat(ced): sketch sound-classification backend (CED audio tagger) Wires ced.cpp (CED, 527-class AudioSet sound-event tagger; baby cry, footsteps, glass, alarms, dog bark) into LocalAI as a Go/purego backend. SKETCH (backend skeleton real; core REST wiring + CI/gallery is a checklist in DESIGN.md): - backend/backend.proto: new SoundDetection rpc + SoundClass messages (run `make protogen-go` to regenerate pkg/grpc/proto). - backend/go/ced: main.go (purego dlopen libced.so + ced_capi.h), goced.go (Ced gRPC backend: Load + SoundDetection), Makefile (clone-at-pin CED_VERSION, ggml static-PIC shared build), run.sh, package.sh, .gitignore. - DESIGN.md: REST /v1/audio/classification wiring (handler/route/capability registration checklist), gallery/index + CI registration, and a scoping note for the realtime/websocket live-recognition path (sliding-window classify over the existing ws transport + voicegate; the ced C-API per-PCM entry point is already window-friendly). Backend code does not compile until protogen-go regenerates the pb types and a libced.so is built (Makefile clones+builds it). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): REST /v1/audio/classification endpoint + capability registration Wires the ced sound-event classification backend (AudioSet audio tagger) end to end through the REST surface, mirroring the transcription path. - Handler: core/http/endpoints/openai/sound_classification.go parses the multipart audio upload, temp-files it, resolves the model config and calls the SoundDetection RPC; returns {model, detections[]} JSON. - Backend wrapper: core/backend/sound_classification.go (ModelSoundDetection) loads the model and normalizes the proto response into schema types. - Schema: core/schema/sound_classification.go (SoundClassificationResult). - gRPC layer: SoundDetection wired through the LocalAI wrapper (interface, Backend client, Client, embed, server, base default) so the loader-typed client exposes the RPC; proto regenerated via make protogen-go. - Route: POST /v1/audio/classification (+ /audio/classification alias) with the audio/multipart default-model middleware in routes/openai.go. - Capability surfaces: swagger @Tags/@Router on the handler; FLAG_SOUND_ CLASSIFICATION usecase flag + UsecaseSoundClassification + UsecaseInfoMap + GuessUsecases + ModalityGroups + GetAllModelConfigUsecases; meta usecase option; /api/instructions audio area updated; auth RouteFeatureRegistry + FeatureAudioClassification (APIFeatures, default ON) + FeatureMetas; UI usecaseFilters, capabilities.js CAP_SOUND_CLASSIFICATION, Models.jsx filter + i18n; docs page features/audio-classification.md + whats-new + crosslink. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): realtime sound-event detection over the websocket API When a realtime pipeline configures a sound-classification model, each VAD-committed utterance (the same window the transcription path produces) is also run through the CED sound-event classifier and the scored AudioSet tags are emitted as a new server event. No new backend rpc is needed: the SoundDetection gRPC method already exists on this branch. - config: add Pipeline.SoundDetection (yaml/json sound_detection,omitempty) beside Transcription/VAD. - realtime: add Model.SoundDetection(ctx, audio, topK, threshold) to the ModelInterface; implement it on wrappedModel and transcriptOnlyModel by calling backend.ModelSoundDetection with the session's sound-classification model config (mirrors how Transcribe dispatches). Load the optional config in newModel / newTranscriptionOnlyModel; nil config keeps it additive. - types: add ConversationItemSoundDetectionEvent (item_id, content_index, detections[]{label,score,index}) with type conversation.item.sound_detection, its ServerEventType constant and MarshalJSON, mirroring the transcription completed event. - realtime: add emitSoundDetection (unary path: classify the committed window, build the event, t.SendEvent) and wire it at the utterance-commit hook right after emitTranscription; gated on session.SoundDetectionEnabled (resolved from Pipeline.SoundDetection at session setup, defaults top_k=5, threshold=0). Its error is logged via xlog but never aborts the turn. - test: Ginkgo specs for emitSoundDetection (tags emitted, empty detections, classifier error) plus a SoundDetection method on the fakeModel double. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): implement SoundDetection in nodes backend test doubles The SoundDetection method added to the grpc backend interface left two test doubles (fakeBackendClient, fakeGRPCBackend) incomplete, so core/services/nodes failed to compile under `go vet`/`go test` (go build missed it: the doubles live in _test.go). Add the method to both, mirroring their existing Detect mock. Repairs CI for the nodes package. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): decouple realtime sound detection from VAD (sound-only sessions) Sound-event detection must activate on sounds, not speech, so it no longer runs through the voice VAD/transcription path. A sound-detection-only pipeline (sound_detection set, no transcription/LLM) now: - is accepted by prepareRealtimeConfig (sound_detection counts as a pipeline stage), - builds a lightweight model via newSoundDetectionOnlyModel (no VAD/STT/LLM/TTS loaded), and - defaults the session to turn_detection none (no VAD) with no transcription stage, so the client drives windowing via input_audio_buffer.commit (option A: client-side sliding window). The per-PCM C-API already supports arbitrary windows. commitUtterance gains a sound-only branch: it emits the conversation.item.sound_detection event (scored AudioSet tags) and stops - no transcription, no LLM response. generateResponse is now guarded on a transcription stage being present, so a sound-only turn never invokes the LLM. Existing transcription/VAD sessions are unchanged (additive). Added a commitUtterance sound-only Ginkgo spec asserting it emits the sound event and neither transcribes nor generates a response. go vet + golangci-lint (new-from-merge-base) clean; openai suite green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): register sound-classification backend in gallery + CI Mechanical backend-image registration for the ced sound-event classifier, mirroring the parakeet-cpp Go/purego backend everywhere it is wired up. - .github/backend-matrix.yml: add the ced build matrix, field-for-field copies of the parakeet-cpp entries (cpu amd64/arm64, cublas cuda 12/13 amd64, l4t cuda-13 arm64, l4t-jetpack cuda-12 arm64, sycl f32/f16, vulkan amd64/arm64, rocm hipblas, and the metal darwin entry), changing only backend and tag-suffix. dockerfile stays ./backend/Dockerfile.golang. - backend/index.yaml: add the &ced meta anchor (capabilities map per platform) plus ced-development and the per-arch image entries, each uri/mirror tag-suffix matching the matrix exactly. The model gallery (GGUF) entry is intentionally deferred pending the HuggingFace publish (TODO note inline). - scripts/changed-backends.js: add an explicit item.backend === "ced" branch in inferBackendPath mapping to backend/go/ced/, same mechanism and ordering as the parakeet-cpp branch (before the generic golang fallthrough). - .github/workflows/bump_deps.yaml: register mudler/ced.cpp -> CED_VERSION in backend/go/ced/Makefile so the daily bot bumps the pin. - swagger/{docs.go,swagger.json,swagger.yaml}: regenerated via make swagger so the existing /v1/audio/classification annotations land in the generated spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): server-side windowing for realtime sound detection (option B) Adds an optional server-driven sliding-window classifier so a sound-only realtime client only has to stream audio (no input_audio_buffer.commit): - Pipeline.sound_detection_window_ms / sound_detection_hop_ms config knobs. When both > 0 on a sound-only session, the server classifies the last window of streamed audio every hop and emits a conversation.item.sound_ detection event; the input buffer is trimmed to one window so a long stream stays bounded. When unset, the session stays client-driven (option A). Runs independent of VAD (sound events are not speech). - handleSoundWindow (ticker) + classifySoundWindow (one tick, extracted so it is unit-testable) + writeWindowWAV, which declares the true InputSampleRate (NewWAVHeaderWithRate) so the classifier resamples correctly. Goroutine is started after toggleVAD and torn down with the session (close + wg.Wait). - Register pipeline.sound_detection (+window_ms/hop_ms) in the config meta registry; the earlier realtime commit added pipeline.sound_detection without a registry entry, failing TestAllFieldsHaveRegistryEntries. This fixes that and covers the two new knobs. Tests: classifySoundWindow emits an event + trims the buffer to one window, no-ops on too-little audio; writeWindowWAV declares the given sample rate. go build/vet + golangci-lint (new-from-merge-base) clean; config + openai suites green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add ced-base GGUF model gallery entries (f16 + q8_0) The ced-base weights are now published at mudler/ced-base-gguf (Apache-2.0, converted from mispeech/ced-base). Adds gallery/ced.yaml (backend: ced + known_usecases: sound_classification) and two gallery/index.yaml entries (ced-base-f16 default, ced-base-q8 smallest) with sha256-pinned files, and removes the now-resolved TODO from backend/index.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add tiny/mini/small GGUF model gallery entries Publishes the rest of the CED family (same architecture, metadata-driven port verified end-to-end on ced-tiny) to mudler/ced-{tiny,mini,small}-gguf and adds their f16 + q8_0 gallery entries: ced-tiny (5.5M, edge/Pi-class) f16 11MB / q8_0 6MB ced-mini (9.6M) f16 19MB / q8_0 11MB ced-small (22M) f16 42MB / q8_0 23MB All sha256-pinned. ced-base remains the accuracy default. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): point gallery entries at the consolidated mudler/ced-gguf repo All CED quantizations (tiny/mini/small/base, f16/q8_0) now live in a single HuggingFace repo, mudler/ced-gguf, instead of per-model repos. Repoint the 8 gallery model entries' urls + file uris accordingly. sha256 and filenames are unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): bump CED_VERSION to the short-clip fix Pin the ced backend to ced.cpp 99c6ed3, which fixes a crash on any clip shorter than target_length (~10.11s): time_pos_embed was added at its full 63-frame grid instead of being sliced to the clip's actual time grid, tripping ggml_can_repeat in ggml_add. Surfaced by the live realtime e2e (sub-10s windows) and gated with a short-clip parity test upstream. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(ced): list ced.cpp as a LocalAI-team engine + backend-guide directive - README.md: add ced.cpp to the "native C/C++/GGML engines developed and maintained by the LocalAI project" table. - docs/content/features/backends.md: add a Sound Classification backend category (sound-event classification / audio tagging) listing ced.cpp. - .agents/adding-backends.md: add a "Documenting the backend" section and two verification-checklist items requiring new backends to be documented in the backends.md category list, and in-house native engines to be added to the README maintained-engines table. This directive was missing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): repin CED_VERSION to the v0.1.0 release commit ced.cpp history was squashed into a single release commit (tagged v0.1.0), so the previous pin (99c6ed3) no longer exists upstream. Pin to c04ac14, the v0.1.0 release commit, so the backend builds against a commit that exists. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): silence gosec G304/G103 + govet unsafeptr on audited paths - sound_classification.go: os.Create(dst) where dst = temp dir + path.Base of the upload (no traversal). #nosec G304, matching the depth-anything-cpp handler. - goced.go: reading a NUL-terminated C string from a libced-owned buffer. #nosec G103 (gosec) + //nolint:govet (golangci-lint's unsafeptr check), since the uintptr is a C-owned malloc'd buffer, not Go-GC memory. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
232 lines
7.1 KiB
Go
232 lines
7.1 KiB
Go
package auth
|
|
|
|
import (
|
|
"github.com/google/uuid"
|
|
"github.com/labstack/echo/v4"
|
|
"gorm.io/gorm"
|
|
)
|
|
|
|
const contextKeyPermissions = "auth_permissions"
|
|
|
|
// GetCachedUserPermissions returns the user's permission record, using a
|
|
// request-scoped cache stored in the echo context. This avoids duplicate
|
|
// DB lookups when multiple middlewares (RequireRouteFeature, RequireModelAccess)
|
|
// both need permissions in the same request.
|
|
func GetCachedUserPermissions(c echo.Context, db *gorm.DB, userID string) (*UserPermission, error) {
|
|
if perm, ok := c.Get(contextKeyPermissions).(*UserPermission); ok && perm != nil {
|
|
return perm, nil
|
|
}
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
c.Set(contextKeyPermissions, perm)
|
|
return perm, nil
|
|
}
|
|
|
|
// Feature name constants — all code must use these, never bare strings.
|
|
const (
|
|
// Agent features (default OFF for new users)
|
|
FeatureAgents = "agents"
|
|
FeatureSkills = "skills"
|
|
FeatureCollections = "collections"
|
|
FeatureMCPJobs = "mcp_jobs"
|
|
FeatureLocalAIAssistant = "localai_assistant"
|
|
|
|
// General features (default OFF for new users)
|
|
FeatureFineTuning = "fine_tuning"
|
|
FeatureQuantization = "quantization"
|
|
|
|
// API features (default ON for new users)
|
|
FeatureChat = "chat"
|
|
FeatureImages = "images"
|
|
FeatureAudioSpeech = "audio_speech"
|
|
FeatureAudioTranscription = "audio_transcription"
|
|
FeatureAudioDiarization = "audio_diarization"
|
|
FeatureAudioClassification = "audio_classification"
|
|
FeatureVAD = "vad"
|
|
FeatureDetection = "detection"
|
|
FeatureVideo = "video"
|
|
FeatureEmbeddings = "embeddings"
|
|
FeatureSound = "sound"
|
|
FeatureRealtime = "realtime"
|
|
FeatureRerank = "rerank"
|
|
FeatureTokenize = "tokenize"
|
|
FeatureMCP = "mcp"
|
|
FeatureStores = "stores"
|
|
FeatureFaceRecognition = "face_recognition"
|
|
FeatureVoiceRecognition = "voice_recognition"
|
|
FeatureAudioTransform = "audio_transform"
|
|
// FeaturePIIFilter gates the synchronous PII analyze/redact service
|
|
// (POST /api/pii/{analyze,redact}). Default ON like the other API
|
|
// features; the admin-only events log is gated separately in-handler.
|
|
FeaturePIIFilter = "pii_filter"
|
|
)
|
|
|
|
// AgentFeatures lists agent-related features (default OFF).
|
|
var AgentFeatures = []string{FeatureAgents, FeatureSkills, FeatureCollections, FeatureMCPJobs, FeatureLocalAIAssistant}
|
|
|
|
// GeneralFeatures lists general features (default OFF).
|
|
var GeneralFeatures = []string{FeatureFineTuning, FeatureQuantization}
|
|
|
|
// APIFeatures lists API endpoint features (default ON).
|
|
var APIFeatures = []string{
|
|
FeatureChat, FeatureImages, FeatureAudioSpeech, FeatureAudioTranscription,
|
|
FeatureAudioDiarization, FeatureAudioClassification,
|
|
FeatureVAD, FeatureDetection, FeatureVideo, FeatureEmbeddings, FeatureSound,
|
|
FeatureRealtime, FeatureRerank, FeatureTokenize, FeatureMCP, FeatureStores,
|
|
FeatureFaceRecognition, FeatureVoiceRecognition, FeatureAudioTransform,
|
|
FeaturePIIFilter,
|
|
}
|
|
|
|
// AllFeatures lists all known features (used by UI and validation).
|
|
var AllFeatures = append(append(append([]string{}, AgentFeatures...), GeneralFeatures...), APIFeatures...)
|
|
|
|
// defaultOnFeatures is the set of features that default to ON when absent from a user's permission map.
|
|
var defaultOnFeatures = func() map[string]bool {
|
|
m := map[string]bool{}
|
|
for _, f := range APIFeatures {
|
|
m[f] = true
|
|
}
|
|
return m
|
|
}()
|
|
|
|
// isDefaultOnFeature returns true if the feature defaults to ON when not explicitly set.
|
|
func isDefaultOnFeature(feature string) bool {
|
|
return defaultOnFeatures[feature]
|
|
}
|
|
|
|
// GetUserPermissions returns the permission record for a user, creating a default
|
|
// (empty map = all disabled) if none exists.
|
|
func GetUserPermissions(db *gorm.DB, userID string) (*UserPermission, error) {
|
|
var perm UserPermission
|
|
err := db.Where("user_id = ?", userID).First(&perm).Error
|
|
if err == gorm.ErrRecordNotFound {
|
|
perm = UserPermission{
|
|
ID: uuid.New().String(),
|
|
UserID: userID,
|
|
Permissions: PermissionMap{},
|
|
}
|
|
if err := db.Create(&perm).Error; err != nil {
|
|
return nil, err
|
|
}
|
|
return &perm, nil
|
|
}
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &perm, nil
|
|
}
|
|
|
|
// UpdateUserPermissions upserts the permission map for a user.
|
|
func UpdateUserPermissions(db *gorm.DB, userID string, perms PermissionMap) error {
|
|
var perm UserPermission
|
|
err := db.Where("user_id = ?", userID).First(&perm).Error
|
|
if err == gorm.ErrRecordNotFound {
|
|
perm = UserPermission{
|
|
ID: uuid.New().String(),
|
|
UserID: userID,
|
|
Permissions: perms,
|
|
}
|
|
return db.Create(&perm).Error
|
|
}
|
|
if err != nil {
|
|
return err
|
|
}
|
|
perm.Permissions = perms
|
|
return db.Save(&perm).Error
|
|
}
|
|
|
|
// HasFeatureAccess returns true if the user is an admin or has the given feature enabled.
|
|
// When a feature key is absent from the user's permission map, it checks whether the
|
|
// feature defaults to ON (API features) or OFF (agent features) for backward compatibility.
|
|
func HasFeatureAccess(db *gorm.DB, user *User, feature string) bool {
|
|
if user == nil {
|
|
return false
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
return true
|
|
}
|
|
perm, err := GetUserPermissions(db, user.ID)
|
|
if err != nil {
|
|
return false
|
|
}
|
|
val, exists := perm.Permissions[feature]
|
|
if !exists {
|
|
return isDefaultOnFeature(feature)
|
|
}
|
|
return val
|
|
}
|
|
|
|
// GetPermissionMapForUser returns the effective permission map for a user.
|
|
// Admins get all features as true (virtual).
|
|
// For regular users, absent keys are filled with their defaults so the
|
|
// UI/API always returns a complete picture.
|
|
func GetPermissionMapForUser(db *gorm.DB, user *User) PermissionMap {
|
|
if user == nil {
|
|
return PermissionMap{}
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
m := PermissionMap{}
|
|
for _, f := range AllFeatures {
|
|
m[f] = true
|
|
}
|
|
return m
|
|
}
|
|
perm, err := GetUserPermissions(db, user.ID)
|
|
if err != nil {
|
|
return PermissionMap{}
|
|
}
|
|
// Fill in defaults for absent keys
|
|
effective := PermissionMap{}
|
|
for _, f := range AllFeatures {
|
|
val, exists := perm.Permissions[f]
|
|
if exists {
|
|
effective[f] = val
|
|
} else {
|
|
effective[f] = isDefaultOnFeature(f)
|
|
}
|
|
}
|
|
return effective
|
|
}
|
|
|
|
// GetModelAllowlist returns the model allowlist for a user.
|
|
func GetModelAllowlist(db *gorm.DB, userID string) ModelAllowlist {
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return ModelAllowlist{}
|
|
}
|
|
return perm.AllowedModels
|
|
}
|
|
|
|
// UpdateModelAllowlist updates the model allowlist for a user.
|
|
func UpdateModelAllowlist(db *gorm.DB, userID string, allowlist ModelAllowlist) error {
|
|
perm, err := GetUserPermissions(db, userID)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
perm.AllowedModels = allowlist
|
|
return db.Save(perm).Error
|
|
}
|
|
|
|
// IsModelAllowed returns true if the user is allowed to use the given model.
|
|
// Admins always have access. If the allowlist is not enabled, all models are allowed.
|
|
func IsModelAllowed(db *gorm.DB, user *User, modelName string) bool {
|
|
if user == nil {
|
|
return false
|
|
}
|
|
if user.Role == RoleAdmin {
|
|
return true
|
|
}
|
|
allowlist := GetModelAllowlist(db, user.ID)
|
|
if !allowlist.Enabled {
|
|
return true
|
|
}
|
|
for _, m := range allowlist.Models {
|
|
if m == modelName {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|