mirror of
https://github.com/mudler/LocalAI.git
synced 2026-06-22 15:49:12 -04:00
600dafd20b
* feat(ced): sketch sound-classification backend (CED audio tagger) Wires ced.cpp (CED, 527-class AudioSet sound-event tagger; baby cry, footsteps, glass, alarms, dog bark) into LocalAI as a Go/purego backend. SKETCH (backend skeleton real; core REST wiring + CI/gallery is a checklist in DESIGN.md): - backend/backend.proto: new SoundDetection rpc + SoundClass messages (run `make protogen-go` to regenerate pkg/grpc/proto). - backend/go/ced: main.go (purego dlopen libced.so + ced_capi.h), goced.go (Ced gRPC backend: Load + SoundDetection), Makefile (clone-at-pin CED_VERSION, ggml static-PIC shared build), run.sh, package.sh, .gitignore. - DESIGN.md: REST /v1/audio/classification wiring (handler/route/capability registration checklist), gallery/index + CI registration, and a scoping note for the realtime/websocket live-recognition path (sliding-window classify over the existing ws transport + voicegate; the ced C-API per-PCM entry point is already window-friendly). Backend code does not compile until protogen-go regenerates the pb types and a libced.so is built (Makefile clones+builds it). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): REST /v1/audio/classification endpoint + capability registration Wires the ced sound-event classification backend (AudioSet audio tagger) end to end through the REST surface, mirroring the transcription path. - Handler: core/http/endpoints/openai/sound_classification.go parses the multipart audio upload, temp-files it, resolves the model config and calls the SoundDetection RPC; returns {model, detections[]} JSON. - Backend wrapper: core/backend/sound_classification.go (ModelSoundDetection) loads the model and normalizes the proto response into schema types. - Schema: core/schema/sound_classification.go (SoundClassificationResult). - gRPC layer: SoundDetection wired through the LocalAI wrapper (interface, Backend client, Client, embed, server, base default) so the loader-typed client exposes the RPC; proto regenerated via make protogen-go. - Route: POST /v1/audio/classification (+ /audio/classification alias) with the audio/multipart default-model middleware in routes/openai.go. - Capability surfaces: swagger @Tags/@Router on the handler; FLAG_SOUND_ CLASSIFICATION usecase flag + UsecaseSoundClassification + UsecaseInfoMap + GuessUsecases + ModalityGroups + GetAllModelConfigUsecases; meta usecase option; /api/instructions audio area updated; auth RouteFeatureRegistry + FeatureAudioClassification (APIFeatures, default ON) + FeatureMetas; UI usecaseFilters, capabilities.js CAP_SOUND_CLASSIFICATION, Models.jsx filter + i18n; docs page features/audio-classification.md + whats-new + crosslink. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): realtime sound-event detection over the websocket API When a realtime pipeline configures a sound-classification model, each VAD-committed utterance (the same window the transcription path produces) is also run through the CED sound-event classifier and the scored AudioSet tags are emitted as a new server event. No new backend rpc is needed: the SoundDetection gRPC method already exists on this branch. - config: add Pipeline.SoundDetection (yaml/json sound_detection,omitempty) beside Transcription/VAD. - realtime: add Model.SoundDetection(ctx, audio, topK, threshold) to the ModelInterface; implement it on wrappedModel and transcriptOnlyModel by calling backend.ModelSoundDetection with the session's sound-classification model config (mirrors how Transcribe dispatches). Load the optional config in newModel / newTranscriptionOnlyModel; nil config keeps it additive. - types: add ConversationItemSoundDetectionEvent (item_id, content_index, detections[]{label,score,index}) with type conversation.item.sound_detection, its ServerEventType constant and MarshalJSON, mirroring the transcription completed event. - realtime: add emitSoundDetection (unary path: classify the committed window, build the event, t.SendEvent) and wire it at the utterance-commit hook right after emitTranscription; gated on session.SoundDetectionEnabled (resolved from Pipeline.SoundDetection at session setup, defaults top_k=5, threshold=0). Its error is logged via xlog but never aborts the turn. - test: Ginkgo specs for emitSoundDetection (tags emitted, empty detections, classifier error) plus a SoundDetection method on the fakeModel double. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): implement SoundDetection in nodes backend test doubles The SoundDetection method added to the grpc backend interface left two test doubles (fakeBackendClient, fakeGRPCBackend) incomplete, so core/services/nodes failed to compile under `go vet`/`go test` (go build missed it: the doubles live in _test.go). Add the method to both, mirroring their existing Detect mock. Repairs CI for the nodes package. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): decouple realtime sound detection from VAD (sound-only sessions) Sound-event detection must activate on sounds, not speech, so it no longer runs through the voice VAD/transcription path. A sound-detection-only pipeline (sound_detection set, no transcription/LLM) now: - is accepted by prepareRealtimeConfig (sound_detection counts as a pipeline stage), - builds a lightweight model via newSoundDetectionOnlyModel (no VAD/STT/LLM/TTS loaded), and - defaults the session to turn_detection none (no VAD) with no transcription stage, so the client drives windowing via input_audio_buffer.commit (option A: client-side sliding window). The per-PCM C-API already supports arbitrary windows. commitUtterance gains a sound-only branch: it emits the conversation.item.sound_detection event (scored AudioSet tags) and stops - no transcription, no LLM response. generateResponse is now guarded on a transcription stage being present, so a sound-only turn never invokes the LLM. Existing transcription/VAD sessions are unchanged (additive). Added a commitUtterance sound-only Ginkgo spec asserting it emits the sound event and neither transcribes nor generates a response. go vet + golangci-lint (new-from-merge-base) clean; openai suite green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): register sound-classification backend in gallery + CI Mechanical backend-image registration for the ced sound-event classifier, mirroring the parakeet-cpp Go/purego backend everywhere it is wired up. - .github/backend-matrix.yml: add the ced build matrix, field-for-field copies of the parakeet-cpp entries (cpu amd64/arm64, cublas cuda 12/13 amd64, l4t cuda-13 arm64, l4t-jetpack cuda-12 arm64, sycl f32/f16, vulkan amd64/arm64, rocm hipblas, and the metal darwin entry), changing only backend and tag-suffix. dockerfile stays ./backend/Dockerfile.golang. - backend/index.yaml: add the &ced meta anchor (capabilities map per platform) plus ced-development and the per-arch image entries, each uri/mirror tag-suffix matching the matrix exactly. The model gallery (GGUF) entry is intentionally deferred pending the HuggingFace publish (TODO note inline). - scripts/changed-backends.js: add an explicit item.backend === "ced" branch in inferBackendPath mapping to backend/go/ced/, same mechanism and ordering as the parakeet-cpp branch (before the generic golang fallthrough). - .github/workflows/bump_deps.yaml: register mudler/ced.cpp -> CED_VERSION in backend/go/ced/Makefile so the daily bot bumps the pin. - swagger/{docs.go,swagger.json,swagger.yaml}: regenerated via make swagger so the existing /v1/audio/classification annotations land in the generated spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): server-side windowing for realtime sound detection (option B) Adds an optional server-driven sliding-window classifier so a sound-only realtime client only has to stream audio (no input_audio_buffer.commit): - Pipeline.sound_detection_window_ms / sound_detection_hop_ms config knobs. When both > 0 on a sound-only session, the server classifies the last window of streamed audio every hop and emits a conversation.item.sound_ detection event; the input buffer is trimmed to one window so a long stream stays bounded. When unset, the session stays client-driven (option A). Runs independent of VAD (sound events are not speech). - handleSoundWindow (ticker) + classifySoundWindow (one tick, extracted so it is unit-testable) + writeWindowWAV, which declares the true InputSampleRate (NewWAVHeaderWithRate) so the classifier resamples correctly. Goroutine is started after toggleVAD and torn down with the session (close + wg.Wait). - Register pipeline.sound_detection (+window_ms/hop_ms) in the config meta registry; the earlier realtime commit added pipeline.sound_detection without a registry entry, failing TestAllFieldsHaveRegistryEntries. This fixes that and covers the two new knobs. Tests: classifySoundWindow emits an event + trims the buffer to one window, no-ops on too-little audio; writeWindowWAV declares the given sample rate. go build/vet + golangci-lint (new-from-merge-base) clean; config + openai suites green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add ced-base GGUF model gallery entries (f16 + q8_0) The ced-base weights are now published at mudler/ced-base-gguf (Apache-2.0, converted from mispeech/ced-base). Adds gallery/ced.yaml (backend: ced + known_usecases: sound_classification) and two gallery/index.yaml entries (ced-base-f16 default, ced-base-q8 smallest) with sha256-pinned files, and removes the now-resolved TODO from backend/index.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add tiny/mini/small GGUF model gallery entries Publishes the rest of the CED family (same architecture, metadata-driven port verified end-to-end on ced-tiny) to mudler/ced-{tiny,mini,small}-gguf and adds their f16 + q8_0 gallery entries: ced-tiny (5.5M, edge/Pi-class) f16 11MB / q8_0 6MB ced-mini (9.6M) f16 19MB / q8_0 11MB ced-small (22M) f16 42MB / q8_0 23MB All sha256-pinned. ced-base remains the accuracy default. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): point gallery entries at the consolidated mudler/ced-gguf repo All CED quantizations (tiny/mini/small/base, f16/q8_0) now live in a single HuggingFace repo, mudler/ced-gguf, instead of per-model repos. Repoint the 8 gallery model entries' urls + file uris accordingly. sha256 and filenames are unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): bump CED_VERSION to the short-clip fix Pin the ced backend to ced.cpp 99c6ed3, which fixes a crash on any clip shorter than target_length (~10.11s): time_pos_embed was added at its full 63-frame grid instead of being sliced to the clip's actual time grid, tripping ggml_can_repeat in ggml_add. Surfaced by the live realtime e2e (sub-10s windows) and gated with a short-clip parity test upstream. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(ced): list ced.cpp as a LocalAI-team engine + backend-guide directive - README.md: add ced.cpp to the "native C/C++/GGML engines developed and maintained by the LocalAI project" table. - docs/content/features/backends.md: add a Sound Classification backend category (sound-event classification / audio tagging) listing ced.cpp. - .agents/adding-backends.md: add a "Documenting the backend" section and two verification-checklist items requiring new backends to be documented in the backends.md category list, and in-house native engines to be added to the README maintained-engines table. This directive was missing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): repin CED_VERSION to the v0.1.0 release commit ced.cpp history was squashed into a single release commit (tagged v0.1.0), so the previous pin (99c6ed3) no longer exists upstream. Pin to c04ac14, the v0.1.0 release commit, so the backend builds against a commit that exists. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): silence gosec G304/G103 + govet unsafeptr on audited paths - sound_classification.go: os.Create(dst) where dst = temp dir + path.Base of the upload (no traversal). #nosec G304, matching the depth-anything-cpp handler. - goced.go: reading a NUL-terminated C string from a libced-owned buffer. #nosec G103 (gosec) + //nolint:govet (golangci-lint's unsafeptr check), since the uintptr is a C-owned malloc'd buffer, not Go-GC memory. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
196 lines
7.1 KiB
Go
196 lines
7.1 KiB
Go
package auth
|
|
|
|
// RouteFeature maps a route pattern + HTTP method to a required feature.
|
|
type RouteFeature struct {
|
|
Method string // "POST", "GET", "*" (any)
|
|
Pattern string // Echo route pattern, e.g. "/v1/chat/completions"
|
|
Feature string // Feature constant, e.g. FeatureChat
|
|
}
|
|
|
|
// RouteFeatureRegistry is the single source of truth for endpoint -> feature mappings.
|
|
// To gate a new endpoint, add an entry here -- no other file changes needed.
|
|
var RouteFeatureRegistry = []RouteFeature{
|
|
// Chat / Completions
|
|
{"POST", "/v1/chat/completions", FeatureChat},
|
|
{"POST", "/chat/completions", FeatureChat},
|
|
{"POST", "/v1/completions", FeatureChat},
|
|
{"POST", "/completions", FeatureChat},
|
|
{"POST", "/v1/engines/:model/completions", FeatureChat},
|
|
{"POST", "/v1/edits", FeatureChat},
|
|
{"POST", "/edits", FeatureChat},
|
|
|
|
// Anthropic
|
|
{"POST", "/v1/messages", FeatureChat},
|
|
{"POST", "/messages", FeatureChat},
|
|
|
|
// Open Responses
|
|
{"POST", "/v1/responses", FeatureChat},
|
|
{"POST", "/responses", FeatureChat},
|
|
{"GET", "/v1/responses", FeatureChat},
|
|
{"GET", "/responses", FeatureChat},
|
|
|
|
// Embeddings
|
|
{"POST", "/v1/embeddings", FeatureEmbeddings},
|
|
{"POST", "/embeddings", FeatureEmbeddings},
|
|
{"POST", "/v1/engines/:model/embeddings", FeatureEmbeddings},
|
|
|
|
// Images
|
|
{"POST", "/v1/images/generations", FeatureImages},
|
|
{"POST", "/images/generations", FeatureImages},
|
|
{"POST", "/v1/images/inpainting", FeatureImages},
|
|
{"POST", "/images/inpainting", FeatureImages},
|
|
|
|
// Audio transcription
|
|
{"POST", "/v1/audio/transcriptions", FeatureAudioTranscription},
|
|
{"POST", "/audio/transcriptions", FeatureAudioTranscription},
|
|
|
|
// Audio diarization (speaker turns)
|
|
{"POST", "/v1/audio/diarization", FeatureAudioDiarization},
|
|
{"POST", "/audio/diarization", FeatureAudioDiarization},
|
|
|
|
// Audio classification (sound-event tagging)
|
|
{"POST", "/v1/audio/classification", FeatureAudioClassification},
|
|
{"POST", "/audio/classification", FeatureAudioClassification},
|
|
|
|
// Audio speech / TTS
|
|
{"POST", "/v1/audio/speech", FeatureAudioSpeech},
|
|
{"POST", "/audio/speech", FeatureAudioSpeech},
|
|
{"POST", "/tts", FeatureAudioSpeech},
|
|
{"POST", "/v1/text-to-speech/:voice-id", FeatureAudioSpeech},
|
|
|
|
// VAD
|
|
{"POST", "/vad", FeatureVAD},
|
|
{"POST", "/v1/vad", FeatureVAD},
|
|
|
|
// Detection
|
|
{"POST", "/v1/detection", FeatureDetection},
|
|
|
|
// Face recognition
|
|
{"POST", "/v1/face/verify", FeatureFaceRecognition},
|
|
{"POST", "/v1/face/analyze", FeatureFaceRecognition},
|
|
{"POST", "/v1/face/embed", FeatureFaceRecognition},
|
|
{"POST", "/v1/face/register", FeatureFaceRecognition},
|
|
{"POST", "/v1/face/identify", FeatureFaceRecognition},
|
|
{"POST", "/v1/face/forget", FeatureFaceRecognition},
|
|
|
|
// Voice (speaker) recognition
|
|
{"POST", "/v1/voice/verify", FeatureVoiceRecognition},
|
|
{"POST", "/v1/voice/analyze", FeatureVoiceRecognition},
|
|
{"POST", "/v1/voice/embed", FeatureVoiceRecognition},
|
|
{"POST", "/v1/voice/register", FeatureVoiceRecognition},
|
|
{"POST", "/v1/voice/identify", FeatureVoiceRecognition},
|
|
{"POST", "/v1/voice/forget", FeatureVoiceRecognition},
|
|
|
|
// Audio transform (echo cancellation, noise suppression, voice conversion, etc.)
|
|
{"POST", "/audio/transformations", FeatureAudioTransform},
|
|
{"POST", "/audio/transform", FeatureAudioTransform},
|
|
{"GET", "/audio/transformations/stream", FeatureAudioTransform},
|
|
|
|
// Video
|
|
{"POST", "/video", FeatureVideo},
|
|
|
|
// Sound generation
|
|
{"POST", "/v1/sound-generation", FeatureSound},
|
|
|
|
// Realtime
|
|
{"GET", "/v1/realtime", FeatureRealtime},
|
|
{"POST", "/v1/realtime/sessions", FeatureRealtime},
|
|
{"POST", "/v1/realtime/transcription_session", FeatureRealtime},
|
|
{"POST", "/v1/realtime/calls", FeatureRealtime},
|
|
|
|
// MCP
|
|
{"POST", "/v1/mcp/chat/completions", FeatureMCP},
|
|
{"POST", "/mcp/v1/chat/completions", FeatureMCP},
|
|
{"POST", "/mcp/chat/completions", FeatureMCP},
|
|
|
|
// Tokenize
|
|
{"POST", "/v1/tokenize", FeatureTokenize},
|
|
|
|
// Rerank
|
|
{"POST", "/v1/rerank", FeatureRerank},
|
|
|
|
// Stores
|
|
{"POST", "/stores/set", FeatureStores},
|
|
{"POST", "/stores/delete", FeatureStores},
|
|
{"POST", "/stores/get", FeatureStores},
|
|
{"POST", "/stores/find", FeatureStores},
|
|
|
|
// Fine-tuning
|
|
{"POST", "/api/fine-tuning/jobs", FeatureFineTuning},
|
|
{"GET", "/api/fine-tuning/jobs", FeatureFineTuning},
|
|
{"GET", "/api/fine-tuning/jobs/:id", FeatureFineTuning},
|
|
{"POST", "/api/fine-tuning/jobs/:id/stop", FeatureFineTuning},
|
|
{"DELETE", "/api/fine-tuning/jobs/:id", FeatureFineTuning},
|
|
{"GET", "/api/fine-tuning/jobs/:id/progress", FeatureFineTuning},
|
|
{"GET", "/api/fine-tuning/jobs/:id/checkpoints", FeatureFineTuning},
|
|
{"POST", "/api/fine-tuning/jobs/:id/export", FeatureFineTuning},
|
|
{"GET", "/api/fine-tuning/jobs/:id/download", FeatureFineTuning},
|
|
{"POST", "/api/fine-tuning/datasets", FeatureFineTuning},
|
|
|
|
// PII analyze/redact service (the events log stays admin-gated in-handler)
|
|
{"POST", "/api/pii/analyze", FeaturePIIFilter},
|
|
{"POST", "/api/pii/redact", FeaturePIIFilter},
|
|
|
|
// Quantization
|
|
{"POST", "/api/quantization/jobs", FeatureQuantization},
|
|
{"GET", "/api/quantization/jobs", FeatureQuantization},
|
|
{"GET", "/api/quantization/jobs/:id", FeatureQuantization},
|
|
{"POST", "/api/quantization/jobs/:id/stop", FeatureQuantization},
|
|
{"DELETE", "/api/quantization/jobs/:id", FeatureQuantization},
|
|
{"GET", "/api/quantization/jobs/:id/progress", FeatureQuantization},
|
|
{"POST", "/api/quantization/jobs/:id/import", FeatureQuantization},
|
|
{"GET", "/api/quantization/jobs/:id/download", FeatureQuantization},
|
|
}
|
|
|
|
// FeatureMeta describes a feature for the admin API/UI.
|
|
type FeatureMeta struct {
|
|
Key string `json:"key"`
|
|
Label string `json:"label"`
|
|
DefaultValue bool `json:"default"`
|
|
}
|
|
|
|
// AgentFeatureMetas returns metadata for agent features.
|
|
func AgentFeatureMetas() []FeatureMeta {
|
|
return []FeatureMeta{
|
|
{FeatureAgents, "Agents", false},
|
|
{FeatureSkills, "Skills", false},
|
|
{FeatureCollections, "Collections", false},
|
|
{FeatureMCPJobs, "MCP CI Jobs", false},
|
|
{FeatureLocalAIAssistant, "LocalAI Assistant", false},
|
|
}
|
|
}
|
|
|
|
// GeneralFeatureMetas returns metadata for general features.
|
|
func GeneralFeatureMetas() []FeatureMeta {
|
|
return []FeatureMeta{
|
|
{FeatureFineTuning, "Fine-Tuning", false},
|
|
{FeatureQuantization, "Quantization", false},
|
|
}
|
|
}
|
|
|
|
// APIFeatureMetas returns metadata for API endpoint features.
|
|
func APIFeatureMetas() []FeatureMeta {
|
|
return []FeatureMeta{
|
|
{FeatureChat, "Chat Completions", true},
|
|
{FeatureImages, "Image Generation", true},
|
|
{FeatureAudioSpeech, "Audio Speech / TTS", true},
|
|
{FeatureAudioTranscription, "Audio Transcription", true},
|
|
{FeatureAudioDiarization, "Audio Diarization", true},
|
|
{FeatureAudioClassification, "Audio Classification", true},
|
|
{FeatureVAD, "Voice Activity Detection", true},
|
|
{FeatureDetection, "Detection", true},
|
|
{FeatureVideo, "Video Generation", true},
|
|
{FeatureEmbeddings, "Embeddings", true},
|
|
{FeatureSound, "Sound Generation", true},
|
|
{FeatureRealtime, "Realtime", true},
|
|
{FeatureRerank, "Rerank", true},
|
|
{FeatureTokenize, "Tokenize", true},
|
|
{FeatureMCP, "MCP", true},
|
|
{FeatureStores, "Stores", true},
|
|
{FeatureFaceRecognition, "Face Recognition", true},
|
|
{FeatureVoiceRecognition, "Voice Recognition", true},
|
|
{FeatureAudioTransform, "Audio Transform", true},
|
|
{FeaturePIIFilter, "PII Analyze / Redact", true},
|
|
}
|
|
}
|