72 Commits

Include renames

Author	SHA1	Message	Date
Richard Palethorpe	d10374f849	feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652) ... * feat(router): make KNN a first-class classifier with a persisted, curated corpus Add `classifier: knn` — similarity-weighted voting over labelled example prompts. Unlike score/colbert it needs no classifier model: label knowledge lives in a corpus seeded and curated through the admin API, so routing decisions are deterministic, auditable, and grounded in graded experience rather than a model's opinion. Epistemic gate: corpus entries below knn.similarity_threshold cannot vote; when none clears it the classifier activates no labels and the router uses the fallback — a prompt unlike all labelled experience is treated as undecidable, not guessed. Decisions record nearest_similarity (also on fallback rows) so admins can see how far the nearest labelled experience was; the Routing tab explains out-of-corpus fallbacks and shows per-label corpus counts. Persistence: one JSONL file per router under <data path>/router-corpus (text, labels, vector, embedder fingerprint). The file is the source of truth; the local-store index is rebuilt from it at classifier build time and stays a pure in-memory index. Entries recorded under a different embedding model re-embed on load. Also corrects the docs' false claim that local-store collections persist — the embedding cache never survived restarts (and still doesn't); the corpus does. Corpus input is API-only by design (entries may contain example user content): POST /api/router/{name}/corpus seeds (labels validated against declared policies, embedded server-side, indexed immediately), GET .../corpus/stats inspects — label counts only, entry texts are never returned by any surface — DELETE .../corpus wipes. Admin-gated like the sibling router endpoints, and exposed as MCP tools (seed_router_corpus / get_router_corpus_stats / clear_router_corpus) in both the httpapi and inproc clients with coverage-test route mappings. Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1); local-store gets InsertBatch/Delete as optional fast paths; RouterConfig gains a knn block (embedding_model, k, similarity_threshold, vote_threshold, store_name) with meta-registry fields; the classifier dropdown now offers knn and the previously-missing colbert; embedding_cache is ignored (with a warning) for knn — it IS an embedding-KNN lookup; the stale /api/instructions intelligent-routing entry is rewritten (it described a classifier that no longer exists); swagger regenerated. Tests: KNN vote/gate specs with hand-computed vote shares, corpus manager suite (restart reload without re-embedding, fingerprint re-embed, dedupe, hostile store names), middleware specs (corpus routing, gate fallback, config validation, cache-wrap refusal), corpus endpoint specs pinning the texts-never-returned contract, MCP catalog + route-mapping gates, and a Playwright spec for corpus stats and the out-of-corpus decision detail. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(router): name consulted corpus neighbours in knn decisions Every knn decision (decision log rows and the /api/router/decide response) now carries neighbors: the K retrieved corpus entries by descending similarity - including ones below the epistemic gate, which is what makes fallback decisions diagnosable - each as {id, similarity, labels}. The id is the entry's content hash (first 8 bytes of the SHA-256 of its text, hex): stable across reseeds and re-embeds, and text-free, so an external platform that seeded the corpus can recompute text->id on its own copy and bucket decisions by corpus region (per- region reliability accounting) without corpus text ever leaving the server. A corrupt index payload surfaces as an id-less neighbour at a real similarity instead of disappearing. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor(router): deduplicate knn plumbing and cut corpus hot-path waste Post-review cleanup of the knn-first-class-router branch; no behaviour changes on the API surface. Reuse/altitude: - RouterKNNConfig.ResolvedStoreName is now the single source of the router-corpus-<name> default (was hand-derived in four files). - corpus.ResolveKNNRouter + corpus.Seed carry the shared model resolution and seed validation; the REST endpoints and the assistant MCP client are thin transport adapters over them, with sentinel errors mapped to HTTP statuses at the echo boundary. - middleware.NewClassifierDeps assembles the classifier dependency set once for all five entry points (OpenAI, Anthropic, realtime, decide, corpus) instead of five hand-copied literals. - router.AllClassifiers feeds both the status endpoint and the unknown-classifier error, ending the classifier-list drift. - Per-classifier requirements moved out of validateRouterPolicies into their buildClassifier arms; the knn arm owns its embedding_cache opt-out instead of a name-check in the shared wrap tail. - adminOnly replaces four inline copies of the admin gate in the middleware routes. - localVectorStore.Search delegates to SearchK (identical traces). Efficiency: - Manager.Add embeds outside the manager mutex and appends to the JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a crash mid-append is tolerated on read and repaired on next write. - Stats memoises per store keyed on the file's stat fingerprint and no longer takes the manager mutex, so the 5s status poll stops parsing vector-laden JSONL and stops blocking behind seeds. - KNN Classify decodes each neighbour payload once (was twice) and builds refs and votes in a single pass with one fallback return. - Corpus file writes fsync before rename/close. - The corpus manager is built eagerly in newApplication (sync.Once dropped); test helper dead branch removed. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(mcp): align corpus tool prompts and the mutating-tool safety list Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(proto,backend): report embedding shape from the llama-cpp backend Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(middleware): name the failing fields when post-merge validation 400s An intermittent post-merge validation failure surfaced as an opaque 400 during integration (pooling scheme mismatch that no client had sent). Log the model, the request's pooling override, and the merged config's pooling fields at the failure point so the next occurrence identifies whether the request or the stored config carried the bad value. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(embeddings): scheme override must not inherit the config's half-life A model config defaulting to decayed_mean pooling carries pooling_half_life_tokens; a request overriding the scheme to mean/last without its own half-life inherited that value, and post-merge validation rejected the pair the server itself had assembled. Zero the inherited half-life when the overridden scheme is not decayed_mean; a request that explicitly pairs a half-life with a non-decayed scheme still 400s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix embedding pooling validation and router bounds Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: run local-store integration tests Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-08-18 09:37:43 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	9a6156d808	feat(nemo-speech-cpp): add the NVIDIA NeMo-Speech.cpp backend (#11406) ... * feat(nemo-speech-cpp): scaffold the backend and upstream build Adds the backend skeleton and the NeMo-Speech.cpp build, pinned at 2e12e2def8a98ed06666f7ee3ca94e7193e04be4. The Go side is deliberately a stub: it dlopens the runtime and starts the gRPC server, later work fills in the symbol table and the model logic. Three details of the upstream layout differ from what the plan assumed, and the build reflects the real tree: * The TTS C ABI ships as libnemo_speech_tts, not libnemo_speech_tts_c. Upstream compiles c_api.cpp straight into the implementation library and only aliases the nemo_speech_tts_c CMake target, so no _c object exists on disk. ASR and NMT do build a real _c shim. * Shared objects land in build/bin, since upstream points CMAKE_LIBRARY_OUTPUT_DIRECTORY at ${CMAKE_BINARY_DIR}/bin. * The ASR and NMT _c shims carry a DT_NEEDED on libnemo_speech_asr and libnemo_speech_nmt, so those are staged and packaged alongside them. Otherwise dlopen fails at startup. The ggml patch step uses an order-only prerequisite. cmake writes into the checkout and bumps its mtime past the sentinel, which would otherwise re-run git apply over an already-patched tree and break every incremental build. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): make 'build' produce the package and bundle the ITN stack Addresses the review of the scaffold commit. backend/Dockerfile.golang runs 'make -C backend/go/$(BACKEND) build' and then copies package/ into the final image, so 'build' has to end with a populated package/. It only staged shared objects, which would have shipped an image with no binary and no libraries at all. The old staging recipe is now stage-libs and the chain is stage-libs, nemo-speech-cpp-grpc, package, build, matching every sibling Go backend. Text normalization was packaged incorrectly. nemo_speech_text_normalization is STATIC but links sparrowhawk, fstfar and fst PUBLIC, so they land as DT_NEEDED on libnemo_speech_asr.so, and they live in a project-local prefix that nothing else provides. WITH_NORM stays ON by default on Linux, since normalization is a wanted feature. Instead stage_libs now copies .deps/itn/lib when WITH_NORM=ON, and package.sh bundles it. Staging that prefix is still not enough on its own: Sparrowhawk drags in protobuf, re2 and absl, which neither build_itn_deps.sh nor package-system-libs.sh provides. Rather than hard-code another hand-maintained list, package.sh now walks the DT_NEEDED entries of everything staged and copies whatever is unresolved, skipping the core set and the GPU set that the shared scripts already own. It fails at package time, not at first dlopen, when something cannot be resolved. On a WITH_NORM=OFF build the closure is already complete and it copies nothing. Restore CGO_ENABLED=0 on the Go build to match whisper, parakeet-cpp and omnivoice-cpp. Note that purego reaches dlopen through fakecgo, so the binary is dynamically linked either way; what the flag changes is the NEEDED set, and lib/ld.so routing in run.sh exists precisely because the binary is not static. Replace the hand-rolled .patched sentinel with upstream's scripts/apply-ggml-patches.sh. It applies the series in filename order, exits non-zero when a patch does not apply, and detects "already applied" by comparing the full-series tree hash rather than an mtime, so it is safe to run every time and there is no sentinel left to go stale or to wedge the build when deleted. It is wired as an order-only prerequisite so running it does not force a relink. Also: correct the package.sh header, which claimed three shared objects when there are five and none of the TTS ones carry a _c suffix; give 'make test' the LD_LIBRARY_PATH the dlopen tests will need; document that a NEMO_SPEECH_VERSION bump needs 'make purge'; and extend 'clean' to remove package/ and the ITN libraries. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): make the closure guard fail closed and give CI its toolchain Addresses the second review round. The dependency-closure guard failed open. Its glob expands once per pass, so each pass advanced the closure by exactly one level, and the fixed count of five passes then fell out of the loop without checking whether anything remained. An eight-deep chain packaged six libraries, exited zero and reported success. That is the case the guard was written for: asr to sparrowhawk to protobuf to absl already runs several levels deep, so a WITH_NORM build could ship missing its deepest libraries and fail at first dlopen. The loop now runs until the staged set stops growing, and exhausting the bound is a hard error rather than a silent exit. For the same reason, a build image with neither readelf nor objdump no longer warns and skips. It cannot show the package is complete, so it refuses to ship it. The guard is entered only when there is something to check, so an empty package cannot trip the new error. Dockerfile.golang installed ninja-build only in the Vulkan branch while this Makefile runs cmake -G Ninja unconditionally, so the CPU, cuBLAS and L4T images could not configure at all. ninja-build moves to the common apt list; it does not change CMake's default generator, so it is inert for the other backends. gcc-12 was nowhere in the tree, yet WITH_NORM defaults ON and build_itn_deps.sh needs it, so the committed default was unbuildable in CI. Install it, with the protobuf, absl, re2 and autotools that Sparrowhawk and OpenFST need, gated on BACKEND so the other Go images do not carry it. The list follows upstream's own docker/Dockerfile, trimmed of the gRPC, portaudio and python entries a BUILD_GRPC=OFF build does not use. Text normalization stays ON: downgrading it silently would ship a backend advertising a feature it lacks. Also: make test depend on stage-libs, so LD_LIBRARY_PATH is not an empty directory on a clean tree, and add an engine target so Dockerfile.golang's cacheable prebuild layer is not skipped and a CUDA build stops recompiling all of upstream on every Go-side change. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): pin protoc for ITN and make the norm stack its own target Addresses the third review round. Dockerfile.golang installs protoc 27.1 into /usr/local/bin, ahead of /usr/bin, while libprotobuf-dev is the distro's 3.21 on noble and 3.12 on jammy. Sparrowhawk resolves protoc from PATH at make time (configure.ac uses AC_CHECK_PROG, so PROTOC substitutes to the bare word, and src/proto/Makefile.am invokes it) and commits no pregenerated stubs, so the rule always runs. Code generated by 27.1 includes google/protobuf/runtime_version.h and a PROTOBUF_VERSION guard the older headers lack, so the WITH_NORM build could not complete. Pin PROTOC to the apt one for that step; configure documents that a pre-set value wins. The apt protoc and libprotobuf-dev come from one source package at one version, which is the property that makes this correct. The text-normalization stack is now a target keyed on a file build_itn_deps.sh actually produces, rather than a side effect of the runtime library rule. As a side effect make could not see whether it existed, so once the library was up to date the script could never run again: a tree built WITH_NORM=OFF could not move to ON, and make test hard-failed with no escape but a full 345 MB clean. It is now built on demand and reachable on its own as 'make itn'. Staging keys on the prefix existing rather than on WITH_NORM, so it stages what the tree actually built, and package.sh's closure guard remains the backstop. An already-configured build tree also now wins over the platform default, so a tree built WITH_NORM=OFF is not silently reconfigured to ON by a bare make test, which is what demanded gcc-12 from developers who chose not to have it. An explicit WITH_NORM= on the command line still overrides both, and the ITN rule preflights for gcc-12 with an error that names the alternative. Move ninja-build out of the shared apt layer into the existing BACKEND-gated block. Dockerfile.golang serves 225 matrix entries and only this backend configures with -G Ninja, so the common list is byte-identical to master again and no other image loses its cache. Drop libabsl-dev and correct the comment that justified it. No base image here ships protobuf 25, so nothing needs the absl split, and the cmake glob looks in /usr/lib rather than the multiarch directory Ubuntu actually uses, so the package could never have contributed anything. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): move the backend apt gate below the expensive layers Addresses the fourth review round. The nemo-speech-cpp apt block sat immediately after the shared apt layer, above the Vulkan SDK build, the CUDA and ROCm installs, the Go toolchain and the protoc download. Docker keys each layer on its parent, so inserting a step there re-keys everything below it: a byte-identical shared layer is not enough, and merging as it stood would have forced all of those to re-execute once for every Go backend image. Move it down beside the existing opus, crispasr and sherpa-onnx gates, which sit after those layers for the same reason. Checked the ordering both ways before moving. Nothing between the two positions uses these packages: the Vulkan and opus blocks install their own ninja and pkg-config, go install protoc-gen-go needs the Go toolchain rather than protoc, and the protoc 27.1 step is a release-binary download that needs neither protobuf-compiler nor libprotobuf-dev. Nothing in the block needs anything those layers provide; it uses only apt, and the mirror rewrite from the first RUN persists in the image. It also runs no update-alternatives, so the default compiler stays untouched for later layers. The diff against master is now a single additive hunk with no shared layer touched. Also preflight ITN_PROTOC. configure gates a preset PROTOC on test -n alone, so a path that does not exist is accepted and the error surfaces much later as a bare "No such file or directory" from inside make -C src/proto. The pin introduced that failure on a box whose only protoc is in /usr/local/bin, which worked before. Check it alongside the gcc-12 check and name the ITN_PROTOC= override in the message. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): parse model options Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): guard the empty option value and warn on a bad gpu index An empty value like "vad_model:" must stay empty, since callers read the empty string as "unset". That branch of resolve() had no spec: dropping the guard left every spec green while parseOptions started returning the models directory itself. Add the spec that fails without the guard. A known key with an unparseable value is a typo, not a config from a newer backend, and "gpu:banna" failed expensively: the model loaded, produced correct output, and ran on CPU with no signal anywhere. Log it. Unknown keys stay silently ignored, which is what keeps configs forward compatible. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): detect model family and discover TTS assets Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): bind the C ABI with layout assertions purego binds by name at runtime and the config structs are passed by pointer, so both a renamed symbol and a mismatched struct layout would otherwise survive a green build. registerSymbols names the failing symbol, and the layout specs compare each Go mirror against the size the library reports for itself, against the offsets a C compiler produces for the installed headers, and against the default values upstream writes into the structs it returns. Two of the bindings differ from the plan because the headers do. The plan's nemo_speech_diar_segments signature omits the segmentation-config pointer that diar.h declares as the second parameter, which would have shifted the output buffer, the capacity and the count pointer one position each. And nemo_speech_diar_stream_push_f32 was missing from the symbol table although standalone diarization cannot work without it. Also close the two panic and equality gaps left in family.go: ValueString panics on a mistyped general.architecture, and the self-codec guard compared a Cleaned candidate path against an uncleaned one, so a doubled separator let the primary GGUF be selected as its own codec. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nemo-speech-cpp): run the ABI specs in CI and refuse to skip them The layout assertions were inert. TEST_PATHS does not cover this backend and the per-backend list in test-extra had no entry for it, so nothing invoked the package's tests. Add it next to depth-anything-cpp, supertonic and vllm-cpp, the group whose own test target carries its build prerequisites; stage-libs already pulls the native build chain, so no prepare-test-extra entry is needed. The skip guard was also loader-inconsistent: librariesPresent stats bare filenames relative to the working directory while openLibraries resolves them through the loader search path, so any invocation other than make test skipped every library-backed spec and still reported green. NEMO_SPEECH_REQUIRE_LIBS=1 turns that into a failure naming the directory and the remedy, and the Makefile test target sets it. Unset, the plain skip survives so a developer without a build can still run the pure-Go layer specs. Trim the default-value fingerprint from roughly forty assertions to eight. It was pinning tunables such as threads and flush_partial_chunk, so a legitimate pin bump would have failed with a message reading like a layout error. What survives is only header-documented contract: the lone non-zero max_alternatives, the run of -1 sentinels and the zero that witnesses where it stops. Verified the narrowed spec still catches a mirror and offset table corrupted in lockstep, which is the one class only this layer sees. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): select the family at load and gate RPCs on it Load sniffs the GGUF architecture, maps it to a family and dispatches to the family's loader. requireFamily gates every other RPC, returning Unimplemented naming both the loaded and the wanted family so a misconfigured model YAML produces a message a user can act on. The family is committed only once its loader has succeeded. A load that fails part way through would otherwise leave the gate open on a handle that was never created. cstr uses runtime.Pinner rather than an ordinary Go allocation. The address crosses the ABI as a uintptr, which the collector does not trace, so incidental reachability through the release closure is not a guarantee: a caller discarding that closure could have the bytes collected before the create call reads them. Pinning is the sanctioned mechanism, makes the release function do real work, and turns a dropped release into a loud leaked-Pinner panic instead of silent corruption. Free overrides the base no-op to destroy the handle and reset the family. Every family owns C memory only its own destroy entry point can release, so without this an unloaded model leaks an acoustic model. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): pin the load ordering, close the engineMu race Three review items, plus a defect the race detector turned up. The spec covering "no family selected after a failed load" wrote junk to a .gguf, so Load returned at ggufArchitecture before a family was ever chosen and the assertion was vacuous. Generalised the GGUF test helper to take a string architecture, and added a spec that loads a magpietts GGUF with no sibling codec, so familyFor succeeds and discoverTTSAssets then fails. It self-guards on ggufArchitecture so it cannot degrade back into the earlier path. requireFamily read n.fam unlocked while Free wrote it under engineMu, which the race detector confirms is a real race. pkg/grpc/server.go calls Free without the backend lock every other RPC holds, so teardown can land mid-request. withEngine now takes the lock, checks the family and runs the body under one acquisition; two would leave a window for Free to destroy the handle between check and use. The locking protocol is stated in both directions for the RPCs still to be written. Running -race also enables checkptr, which aborts on cstr's pointer being read back by goString: converting a uintptr to a pointer is fatal whenever the address lands in a Go allocation, so a pinned Go buffer can never be dereferenced from Go. The pointer is for C alone. Both helpers now document the one-way contract, and goString is tested against a real C-owned string by rebinding the version symbol to return a raw char*. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): implement offline transcription Create the ASR recognizer in loadASR and serve AudioTranscription. Segment times are int64 nanoseconds, not seconds: the proto field is an int64 that core/backend reads straight into a time.Duration, while the runtime reports word offsets in milliseconds. Words are grouped into one segment per consecutive speaker run, with the 1-based speaker tag carried through and 0 (untagged) left unlabelled. The whole RPC body runs inside withEngine so the family check and the C calls happen under one acquisition of engineMu. Free runs without the backend lock, so checking the family and then relocking would let a teardown destroy the handle in the gap. The audio decode is inside the closure too, which costs nothing: base.SingleThread already serialises this backend's RPCs. recognizeF32 guards zero-length PCM. &pcm[0] panics on an empty slice, so Go never reaches the C side's own "empty audio" rejection, and a silent clip or a truncated upload is ordinary input. pkg/utils has no WAV decode helper, only the ffmpeg normalisation, so audio.go pairs AudioToWav with go-audio the way parakeet-cpp does. It returns the sample rate rather than a duration, since the C API resamples off that number. Also closes the write-side half of the race Task 5 fixed on the read side: Load now holds engineMu across the family switch and the n.fam commit, matching Free. The loaders still must not take it. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): implement streaming and live transcription AudioTranscriptionStream drives a whole clip through the cache-aware streaming API in 100 ms pushes, emitting each finalized utterance as a delta and closing with the assembled result. AudioTranscriptionLive serves the bidirectional RPC over the same session: config first, a ready ack, deltas with word timings as utterances land, and a terminal result when the caller closes its send side. Both wrap their body in withEngine, so a stream holds engineMu for its whole life and Free waits on it rather than destroying the recognizer underneath a half-finished stream. That makes the way out load-bearing: the file loop honours the request context between pushes, and the live loop ends when the host closes the request channel, so a disconnected client cannot pin the model against unload. Only finals become deltas. The runtime applies punctuation and inverse text normalization on finals only, so a final rewrites the utterance rather than extending its interim, and delta on the wire is newly-finalized text that consumers concatenate. Forwarding interims would duplicate and mispunctuate every utterance. The four streaming entry points sit behind an asrSession interface. No NeMo GGUF is small enough to keep in the tree, so without that seam the need-more-audio drain would have no test at all: nemo_speech_asr_stream_next reports OK with a NULL handle when it wants more audio, which is a pause rather than an end, and reading it either way round drops results or spins forever. Also folds in three items from the offline transcription review: - empty audio is now refused before anything crosses the ABI, not inside recognizeF32. The added integration spec caught the old ordering panicking on an unbound entry point instead of failing; - an undecodable sample rate is an error rather than 0, which this runtime reads as "already at the model rate" and would have made a wrong rate silently pitch-shift the audio; - AudioTranscription guards its result pointer instead of relying on an unstated invariant. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): make live deltas concatenate and fill segment words runLive wrote the inter-utterance separator into the accumulated transcript but emitted the delta without it, so a two-utterance turn sent "one." and "two." while the terminal result read "one. two.". The live consumer is the one that really concatenates: the realtime semantic-VAD path joins the accumulated deltas with the empty string and clears them only at a turn reset, never at an endpoint, so the running caption read "one.two.". The separator now goes into the delta, as it already did on the file path, and the terminal text is the verbatim concatenation rather than a trimmed rebuild. TranscriptSegment.Words was never populated, so a request asking for timestamp_granularities ["word"] came back with no words at all even though the timings were decoded. wordsToSegments now attaches them, gated on the granularity the same way parakeet-cpp gates it, so a transcript that did not ask for word timestamps does not pay for them. Also: the final that comes back from the tail flush no longer claims an end-of-utterance. It is the end of the stream, not a user yielding the turn, and eou is what the realtime turn detector acts on. The comment explaining why interims are suppressed led with the runtime's postprocessing. The wire contract is the stronger reason and now comes first: consumers concatenate deltas, so forwarding a growing hypothesis assembles to "hehellhelloHello.". The postprocessing only explains why no diffing trick would rescue them. It is also ITN and strip_formatting rather than punctuation, which is off by default here. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): implement standalone diarization loadDiarizer creates the Sortformer diarizer and Diarize serves the RPC over a diarization stream: decode, chunked push, finish, then the count-then-fill segments protocol. nemo_speech_diar_segment carries start_time and end_time in SECONDS already, not frame indices, so no conversion happens on the way to DiarizeSegment.start/end and the model's seconds-per-frame is not involved at all. The speaker label is the runtime's 1-based tag as a decimal string, matching what wordsToSegments emits on the ASR path, so the same speaker reads the same way whether a caller diarized a file or transcribed it. The six frame-geometry overrides are written as -1 rather than left zero. c_api.cpp applies left_context_frames when it is >= 0 while every other override needs > 0, so a zeroed config would silently pin the left context to zero and change the model's streaming geometry. nemo_speech_diar_segments writes *count before it rejects a buffer that is too small, so a rejected fill still reports the size to retry with. collectSegments uses that rather than truncating, bounded at four attempts because the RPC holds engineMu for its whole body and an unbounded retry would block an unload behind it. Two DiarizeRequest knobs map onto the segmentation config, and the proto and header names cross over: min_duration_on is the C min_duration_sec and min_duration_off is the C min_gap_sec. Six fields have no equivalent in this pipeline and are logged rather than dropped in silence: num_speakers, min_speakers and max_speakers (Sortformer's capacity is fixed by the checkpoint), clustering_threshold (there is no clustering stage), include_text (no ASR here) and threads. The empty-PCM guard fires before the stream is opened, so a silent clip never reaches a purego entry point that would dereference &pcm[0]. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): pin the diarizer geometry sentinels and cap the segment buffer The six frame-geometry overrides were written as -1 with nothing asserting it. c_api.cpp applies left_context_frames at >= 0 while the other five need > 0, so a dropped sentinel there pins the model's left context to zero, and the struct keeps exactly the same shape, which is all the layout assertions can see. Extracting diarModelConfig makes the values assertable: five specs now pin all six frame fields, the device index, the declared size and the NULL preset, each frame field on its own line so a missing sentinel names itself. distinctSpeakers had a spec with three segments over three distinct labels, which len(segs) satisfies just as well as the real thing. Four segments over three labels makes it a spec that can fail. collectSegments sized its buffer straight from a count the C side reported, and make() panics rather than erroring on a length it cannot satisfy, so an uninitialised size_t coming back across the ABI killed the backend process instead of failing one request. A ceiling of 2^22 segments, upwards of 93 hours of audio at one 80 ms frame each, turns that into a diagnosable error. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): implement TTS and streaming TTS The PCM callback is compiled once per process behind a sync.Once, not once per request and not once per load. purego.NewCallback writes into a fixed table of 2000 entries (purego/syscall_sysv.go) and never releases one, so a per-request callback panics the backend process on the 2001st synthesis, and a per-load one reaches the same ceiling on a server that swaps models. Synthesis is routed through that single callback plus a user_data id: engineMu is per-model, one process holds several models, so a single current-sink pointer would be overwritten by two TTS models synthesizing at once. Deviations from the brief, all verified against the real headers and proto: - TTS is TTS(*pb.TTSRequest) error and TTSStream is TTSStream(*pb.TTSRequest, chan []byte) error, per pkg/grpc/interface.go. The brief's context/pb.Result and server-stream forms do not implement the interface. The channel is closed on every path, including the family rejection, because pkg/grpc/server.go blocks on its drain goroutine and an unclosed channel hangs the RPC with the backend lock held. - The callback takes unsafe.Pointer, not uintptr. Converting a uintptr parameter back to a pointer is a checkptr violation that aborts under -race. - resolveSpeaker refuses to turn a negative number into a speaker index. -1 is the C API's "use the default" sentinel, so the brief's rule would have made a request naming an invalid voice synthesize in the default voice instead of being rejected. temperature and cfg_scale each write their override flag as well: magpietts/runtime.cpp reads the float only when the flag is set, so a temperature without it is silently discarded. Also folds in Task 8's review finding on asr.go: the six bare -1 sentinels in loadASR move to an asrDiarConfig builder reusing diarGeometryDefault, with specs. src/asr/c_api.cpp applies left_context_frames at >= 0, so a dropped sentinel pins the model geometry to 0 and no layout assertion can see it. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): surface NMT translation through Predict nemo_speech_nmt_translate takes explicit source and target languages and has no free-form generation or token-callback entry point, so there is no prompt in the LLM sense. The pair comes from the source_language / target_language model options, with an optional leading [src->tgt] directive as the only per-request override, and PredictStream emits the whole translation as a single chunk because the C API has nothing finer to give it. Both RPCs wrap their body in withEngine so the family check and the C calls that trust the handle share one acquisition of engineMu. PredictStream closes its channel on every path, including the family rejection: this is the legacy streaming contract, and pkg/grpc/server.go blocks on a drain goroutine that only finishes when the channel closes, so leaving it open hangs the RPC rather than failing it. nmtTranslatorConfig is extracted so its four adjacent pointer fields can be asserted against distinct sentinels. Transposing two of them changes neither the struct size nor any field offset, so the layout assertions cannot see it. Also removes goString, which had no production caller: every string-returning symbol in abi.go is bound with a Go string return that purego converts itself. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nemo-speech-cpp): pin the three-segment pair tag in an NMT directive The directive regex allowed an unbounded run of two-letter segments per side, but nothing tested it: narrowing that run back to a single optional segment left every spec green. resolve_tag accepts a ready pair tag in one field with the other empty (src/nmt/langpairs.cc), and those tags run to three segments (en-zh-cn, pt-br-en), so a shorter pattern does not mis-split the tag, it fails to match the directive at all and the whole bracket is handed to the model as text to translate. The justification on the regex was also wrong and is corrected: pt-br and zh-cn are two segments and parse either way. It is the single-field form that needs the run. Renames the NMT handle to n.nmt so it stops sharing a name with the translator interface, following n.synth, which is shortened for the same reason. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): register the backend and give its specs a CI job Registers nemo-speech-cpp across every surface .agents/adding-backends.md requires, and adds the CI job its unit suite never had. backend/index.yaml gets the meta backend (capabilities map, no uri), a development meta and 12 image entries. No amd and no intel capability keys: upstream NeMo-Speech.cpp builds ggml with CUDA, Vulkan or Metal only, and SystemState.Capability falls back to "default", so those hosts get the CPU build rather than a tag that does not exist. The nvidia-cuda-* and nvidia-l4t-cuda-* keys are present because getSystemCapabilities() refines an NVIDIA host to them whenever the CUDA directory exists; without them every modern CUDA host and Jetson would miss the map and quietly run on CPU. .github/backend-matrix.yml gets 7 include rows and 1 includeDarwin row. No hipblas and no sycl rows, for the same upstream reason. cpu and vulkan are per-arch pairs sharing a tag-suffix so backend-merge-jobs builds a multi-arch manifest: an ARM host with no NVIDIA GPU reports "default" and the Jetson image does not cover it. The CI job is the substantive part. make test-extra is dead on master, because prepare-test-extra depends on a protogen-python target that does not exist and no workflow invokes it anyway, so the entry added earlier in this series ran nowhere. abi_test.go asserts the size and field offsets of every Go mirror struct against the C ABI it is dlopened into, and those assertions are the only defence against silent memory corruption after a purego symbol rename or an upstream header change. tests-nemo-speech-cpp in test-extra.yml now executes them on pull_request and on master, gated on the backend's own path filter. The recipe sets NEMO_SPEECH_REQUIRE_LIBS=1, so a missing library fails rather than skips. WITH_NORM=OFF skips the OpenFST leg and costs no coverage: nothing in the four C ABI headers is conditional on it, so the layouts are identical. Also registers the upstream pin with the bump bot, which the backend Makefile already claimed but was never wired up, and adds the BackendCapabilities entry so a hand-written model config gets a real usecase surface. PossibleUsecases is the union of the four families and DefaultUsecases is transcript alone, the audio-cpp pattern. No VoiceCloning key: MagpieTTS synthesizes from baked speaker ids, not a reference clip. No gallery entries: publishing converted GGUFs is a follow-up. ModelIdentity needs no work in this backend. main.go serves through grpc.StartServer, so every RPC lands on pkg/grpc's shared server wrapper first, and checkModelIdentity is the first statement of all seven handlers this backend implements. A second check inside NemoSpeech would be unreachable and would risk diverging from the cross-language sentinel the router matches on. AudioTranscriptionLive stays unguarded because TranscriptLiveRequest carries no ModelIdentity field at all, which is a proto-level gap affecting every backend and needs its own change. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): build the CUDA-13 Jetson image the l4t-cuda-13 key needs The nvidia-l4t-cuda-13 capability pointed at nvidia-l4t-arm64-nemo-speech-cpp, which is built on nvcr.io/nvidia/l4t-jetpack:r36.4.0 and therefore links ggml against CUDA 12. A Jetson whose CUDA 13 runtime is present reports that capability and would have pulled an image with no libcudart.so.12 to dlopen, failing hard at load. That is worse than omitting the key: with no key Capability() falls back to "default" and the host gets a working CPU build. Fixed the way parakeet-cpp and moss-transcribe-cpp already do it, by shipping the second L4T image rather than dropping the key. Nothing prevents building it here: those peers use plain ubuntu:24.04 on ubuntu-24.04-arm with the same Dockerfile.golang as this backend's other rows, and every package in the nemo-speech-cpp apt gate exists on noble arm64. Adds the -nvidia-l4t-cuda-13-arm64-nemo-speech-cpp matrix row and its two index entries, repoints the key on both metas, and rewrites the capability-map comment, which had the reasoning backwards. Also adds the documentary inferBackendPath branch, matching all six sibling *-cpp Go backends. Behaviour is unchanged; the generic golang fallthrough already resolved this backend correctly. The previous commit message said "all seven handlers" of the shared gRPC wrapper. There are eight RPC entry points: seven are guarded by checkModelIdentity and AudioTranscriptionLive is the unguarded eighth, which that message already called out separately. Wording only. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(nemo-speech-cpp): document the backend and list it in the importer Adds docs/content/features/nemo-speech-cpp.md, alongside the audio.cpp page that is its closest sibling, and cross-links it from the speech-to-text, diarization, text-to-speech, backend-type and compatibility-table pages so the backend is reachable from every surface that lists its modalities. The page covers the architecture-to-family table, every option key with a model YAML per family, the translation prefix directive, the acceleration matrix, and the four limitations this backend ships with: Linux-only inverse text normalization, suppressed interim streaming results, the library's default translation context and generation limits, and the absence of gallery entries. knownPrefOnlyBackends gains the backend so it appears in the /import-model dropdown. It stays preference-only and AutoDetect=false: general.architecture lives inside the GGUF where no remote-repo probe can read it, and a translation model carries an ordinary LLM architecture with no NeMo-specific marker. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(nemo-speech-cpp): correct the translation limits, the macOS gap and the TTS conversion Three factual errors found in review, all of them the kind a user would act on. The translation limits were described backwards. Input longer than the 1024-token context is rejected, not truncated: translator.cpp throws "nmt: prompt too long (N tokens) for context 1024", which reaches the caller as a failed request. What is silently cut is the output, by the max_new_tokens loop at 256. The bullet now separates the two and says which one fails quietly. The macOS gap covers TTS text normalization as well. Both directions sit behind the single NEMO_SPEECH_WITH_NORM flag, which the Makefile forces off on Darwin, so tn_dir is as inert there as itn_dir. Neither fails the load: both warn and carry on. pnc_model really is unaffected, since punctuation is compiled in unconditionally. The tn_dir row in the option reference gained the caveat the itn_dir row already had. The TTS conversion procedure produced a model that could not load. It converted MagpieTTS and stopped, leaving no NanoCodec, which the same page lists as required; following it gave "no NanoCodec GGUF found next to ...". Both halves are now there, each with the download that feeds it, so the block runs top to bottom on a clean machine. Also: any negative gpu value pins TTS to the CPU, not only -1, and FLAG_CHAT additionally surfaces the model in the web UI chat picker. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): map every C status, not just the NMT one INVALID_ARGUMENT was translated to codes.InvalidArgument at exactly one of sixteen C call sites. Everywhere else a non-zero status collapsed to codes.Internal, so the same backend answered an unsupported language pair with HTTP 400 and an unknown TTS voice, which is the same class of caller mistake against the same process, with HTTP 500. Status 4 is CANCELLED on the ASR and TTS surfaces and was reported as a backend failure rather than as the consumer having stopped listening. asr.h, tts.h and nmt.h each declare their own status enum and diar.h reuses the ASR one; the values they share agree, and the single divergence is that NMT declares no CANCELLED because nemo_speech_nmt_translate has no callback for a consumer to stop with. That is an absence, not a disagreement, so one table serves all three. status.go carries it, with the header line numbers and a note that a pin bump has to recheck it: purego binds by name and the status crosses as a bare int32, so nothing in the build or the linker can see a drift. New specs cover the whole enum, unknown values, and one real INVALID_ARGUMENT per family driven through the shared objects rather than through the Go mapping asserting against itself. Also add UsecaseChat to this backend's capability entry, which the docs already told operators to set for translation models. chat is a gallery filter key and completion is not, so GET /api/backends/usecases would have greyed the Chat filter out and hidden a Riva-Translate gallery entry from the one filter that fits it. The flag gates no endpoint; it makes the model eligible as the default chat model and puts it in the web UI chat picker, both of which Predict and PredictStream already serve. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): audit the gosec unsafe and file-inclusion sites gosec flags 13 alerts on this backend: one G304 and twelve G103. Each was checked individually rather than blanket-suppressed, and each annotation states what makes that particular site safe. The G304 at audio.go is a false positive. The opened path is filepath.Join of a directory the function just created with os.MkdirTemp and a constant basename; the request-controlled path is the input to AudioToWav and never reaches the open. The twelve G103 sites are the package's three established shapes, and every one was verified against them: cstr and pinPtr take the address of something pinned on the line above and return it one-way (nothing in the package converts either result back, which is what keeps checkptr out of it under -race), and each *Create hands C a stack-local POD config whose uintptr members are cstr allocations or pinPtr addresses held by a pinner the loader unpins only after the call. The two slice-building sites are bounded by construction: DiarSegments is handed exactly len(buf) with the buffer sized under maxDiarSegments and a reported count larger than it rejected rather than sliced to, and the TTS callback copies out a slice whose length is the length the runtime declared for that buffer. Separately, sampleRateOf gets a real fix rather than an annotation. go-audio reads the WAV header's sample rate from an unsigned 32-bit field into an int, so a header claiming more than 2^31-1 passed the "> 0" test and then narrowed to a NEGATIVE rate, which the runtime would take as a resampling ratio. AudioToWav cannot produce one today, but that is a property of another package and this function exists precisely because the rate is read back rather than assumed, so the bound is enforced here and pinned by a spec. The four remaining integer narrowings are annotated with the bound that makes each safe: the WAV payload length is already checked against maxWAVDataBytes, the speaker count is bounded by maxDiarSegments, and the two segment ids are the proto's own int32 wire type. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): skip the CUDA-only ggml patch series on darwin The macOS backend build died in patch-ggml: scripts/apply-ggml-patches.sh: line 56: mapfile: command not found make[1]: *** [patch-ggml] Error 127 mapfile is a bash 4 builtin (and its -d flag needs 4.4). macOS ships bash 3.2.57 as /bin/bash and GitHub's runner images add no newer one, so the bare `bash` the recipe resolves from PATH cannot run upstream's script. Rather than hunt for a capable bash that the runner does not have, drop the step where it does nothing. ggml-patches/ is a CUDA series: every kernel it adds is under src/ggml-cuda/, and its whole footprint outside that directory is an op enum plus prototype in include/ggml.h, the constructor and a name-table entry in src/ggml.c, and two ggml-cpu lines that make the CUDA-only op report unsupported and abort. Nothing it touches is compiled into a Metal kernel or changes a CPU one. The project's own references to patch-only ggml symbols sit behind NEMO_SPEECH_FUSED_RELPOS_ATTN and NEMO_SPEECH_FASTCONFORMER_CUDA_FUSIONS, which cmake already forces OFF without GGML_CUDA, or behind NEMO_SPEECH_GGML_PATCHED itself, which guards a GGML_TENSOR_FLAG_Q8_PLANAR write that a non-CUDA buffer throws before reaching. So passing NEMO_SPEECH_GGML_PATCHED=OFF costs the Metal build nothing, and it is required once the series is skipped: that flag is what stops the ASR sources referencing a tensor flag stock ggml does not define. This is upstream's own Metal configuration. Its metal-* and vulkan-* CMake presets inherit the cpu-* ones, which set NEMO_SPEECH_GGML_PATCHED to OFF; docker/Dockerfile and scripts/windows/build.ps1 do the same for their non-CUDA targets. LocalAI's Makefile never passed the flag at all and so inherited the CUDA default everywhere. Linux is untouched and keeps applying the series, including its idempotency and its hard failure on a patch that does not apply. The gate is the same uname test the WITH_NORM block above already uses, and both branches keep the order-only clone prerequisite, which on a WITH_NORM=OFF tree is the only thing that pulls sources/ in. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): restore std::binary_function for MeCab on libc++ NEMO_SPEECH_TTS_WITH_JA=ON compiles Open JTalk's bundled MeCab, and mecab/src/dictionary.cpp derives a comparator from std::binary_function, which C++17 removed. libstdc++ still ships it as deprecated-but-present under -std=gnu++17, so Linux never notices. libc++ compiles it out and the macOS arm64 build dies with "no template named 'binary_function' in namespace 'std'". This is ours, not an upstream regression: upstream defaults both NEMO_SPEECH_TTS_WITH_JA and NEMO_SPEECH_TTS_WITH_ZH to OFF and the OSS drop carries no CI at all, so that target is never built there. Upstream does already carry the equivalent workaround for MSVC's STL (_HAS_AUTO_PTR_ETC plus /FIfunctional) but has no libc++ branch. libc++ gates the two templates on _LIBCPP_ENABLE_CXX17_REMOVED_UNARY_BINARY_FUNCTION, and has since LLVM 16, older than any clang Xcode still ships. The name is the whole problem: _LIBCPP_ENABLE_CXX17_REMOVED_BINDERS covers bind1st, bind2nd, ptr_fun and mem_fun and not unary_function or binary_function, and the umbrella _LIBCPP_ENABLE_CXX17_REMOVED_FEATURES no longer exists in libcxx at all. A wrong name preprocesses fine and fixes nothing. Applied through CMAKE_CXX_FLAGS rather than to the one target, because the tokenizer CMakeLists is upstream's and sources/ is a pinned checkout. Project-wide is also the safer scope: the macro decides whether libc++'s internal __binary_function alias resolves to std::binary_function or to __binary_function_keep_layout_base, a base class of std::less and friends, so defining it for a subset of translation units would give those class templates two spellings in one binary. Both bases are empty and, at C++17, carry identical members, so the define changes no layout and no ABI. Darwin only. On Linux the branch is unreachable and the macro is not a name libstdc++ knows, so it would be inert even if taken; a Linux configure with the flag forced on puts it on all 23 C++ TUs of nemo_speech_openjtalk_frontend including dictionary.cpp at -std=gnu++17, and on none of the 16 C TUs. Mandarin needs nothing: cppjieba v5.6.7 and limonp have no removed C++17 constructs left (limonp replaced std::not1 and std::bind2nd with lambdas) and cppjieba's own CI builds macos-14 and macos-latest at C++11 through C++20. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): repair OpenFST's FstImpl::operator= for gcc-14 The first WITH_NORM=ON build failed compiling fst_normalizer.cpp against the installed OpenFST 1.8.3 headers: fst.h:690:59: error: no match for 'operator=' (operand types are 'std::unique_ptr<fst::SymbolTable, ...>' and 'fst::SymbolTable*') FstImpl's copy-assignment operator assigns the raw pointer returned by SymbolTable::Copy() straight to a std::unique_ptr member. No C++ standard allows that, so the line is ill-formed everywhere; it survived because nothing instantiates FstImpl::operator= and gcc up to 13 only checks a template member's body when it is instantiated. gcc 14 resolves non-dependent operator expressions at template definition time, so it rejects the line in any translation unit that includes <fst/fst.h>. The CI diagnostic confirms the phase: it reads "In member function", not "In instantiation of", and carries no instantiation backtrace. That is why this surfaces only here. build_itn_deps.sh compiles OpenFST with gcc-12 and upstream's own images build the runtime with gcc-13, so neither compiler reaches the check; backend/Dockerfile.golang installs gcc-14 and promotes it with update-alternatives, and fst_normalizer.cpp is the one translation unit in this backend that includes OpenFST. Fix it in the installed ITN prefix, which is the only copy the cmake build compiles against, using the same .reset() spelling FstImpl::SetInputSymbols already uses for the identical operation. libfst.so is linked before this runs and cannot contain the function, since no compiler could ever have emitted it, so there is no ABI or ODR consequence. The rule is guarded on both sides so a pin bump to a fixed OpenFST fails loudly rather than silently no-opping. Verified with a real gcc 14.2: the CI error reproduces byte for byte from a file whose entire content is '#include <fst/fst.h>', and gcc 14 reports exactly two errors over the whole OpenFST include closure this backend uses, both of them these two lines. After the patch that closure compiles clean under gcc-14 with the target's own flags. The step is reachable only under WITH_NORM=ON, so 'make -n stage-libs WITH_NORM=OFF' mentions neither it nor the ITN build, and darwin, which defaults WITH_NORM to OFF, never evaluates it. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): install cmake 3.31 on bases that ship less than 3.26 The JetPack r36.4.0 row dies on the first line of NeMo-Speech.cpp's CMakeLists.txt: CMake Error at CMakeLists.txt:3 (cmake_minimum_required): -- Configuring incomplete, errors occurred! Upstream opens with cmake_minimum_required(VERSION 3.26). That base image is Ubuntu 22.04 jammy, whose apt cmake is 3.22.1, so configure aborts before it reads a single one of the backend's -D flags. Every other Linux row in this block is noble, which ships 3.28 and clears the bar, so the failure is one base image wide rather than a code problem. Everything before it on that row had already worked, including the OpenFST and Sparrowhawk ITN build. No other Go backend needs this. parakeet-cpp and moss-transcribe-cpp share the same JetPack base and both declare cmake_minimum_required(VERSION 3.18), and nothing in the repo installs a cmake newer than the distro's, so there is no existing pattern to reuse. Nothing depends on jammy's cmake staying 3.22 either: build_itn_deps.sh never invokes cmake at all, since OpenFST and Sparrowhawk are autotools builds. Kitware's release tarball rather than their APT repo or pip. The tarball is a pinned URL with a published checksum, so an upstream release cannot change what lands here. The APT repo does carry jammy arm64, but it serves a moving latest that today is CMake 4.4, and 4.x drops compatibility with cmake_minimum_required below 3.5, which vendored third_party subprojects still declare; pinning it there would mean tracking Kitware's Debian revision string instead of an upstream version. pip would drag a Python toolchain into a backend that has none. 3.31.12 is the last 3.x release, so it clears 3.26 while keeping the CMake 3 policy surface, and it stays close to the 3.28 the green noble rows already use. The binaries need only glibc 2.17 and carry no libstdc++ DT_NEEDED, well under jammy's 2.35. doc/, man/, ccmake and cmake-gui are not extracted; the final image is FROM scratch, but there is no reason to page 100 MB of Qt GUI and docs through the CI cache. Gated on the installed cmake actually being older than 3.26, so the rows that already build green keep configuring with exactly the cmake they use today, and folded into the existing ${BACKEND} block rather than added as a new instruction, so no other Go backend image gains a layer and nothing above the Vulkan SDK, CUDA, Go and protoc layers moves. The symlink lands in /usr/local/bin and shadows apt's cmake. Unlike the protoc shadowing that broke Sparrowhawk earlier in this series that is inert: protoc has to agree with the libprotobuf headers it generates against, whereas cmake links nothing into the product and has no ABI relationship with anything in the image, and it resolves the symlink back to /opt to find its own Modules/ tree, so a 3.31 binary can never read 3.22's modules. The version test avoids $(...) deliberately. BuildKit delivers a RUN heredoc through an outer shell with an unquoted delimiter, so a command substitution runs there, too early, in a container where the files it reads do not exist yet, and its empty output is pasted into the script; the first draft took the install branch on every row because of it. Verified by building the block against nvcr.io/nvidia/l4t-jetpack:r36.4.0 arm64 under qemu, the row's actual base image: cmake 3.22.1 detected, tarball checksum verified, 3.31.12 installed, and a cmake_minimum_required(VERSION 3.26) project configures with -G Ninja and builds, with CMAKE_ROOT resolving to /opt/cmake/share/cmake-3.31. Same on ubuntu:22.04 amd64 and arm64. ubuntu:24.04 skips the install, gains no /opt/cmake and still configures on /usr/share/cmake-3.28. The NeMo-Speech.cpp compile itself on JetPack CUDA 12 is not reproducible here and remains for CI. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-08-07 12:32:39 +02:00
Richard Palethorpeandlocalai-org-maint-bot	9058a2bb46	feat: Add 3d generation UI/API and trellis2cpp backend (#10979) ... * feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint Adds the plumbing for image-conditioned 3D asset generation (binary glTF / GLB output), modeled on the video generation path: - backend.proto: Generate3D RPC + Generate3DRequest (staged image src, glb dst, seed/step/cfg_scale/texture_steps, quality and background enums, params map for backend-specific extras) - pkg/grpc: thread Generate3D through client, server, embed, base and the backend interfaces; connection-evicting and distributed-node wrappers (in-flight tracking + file staging) included - core/config: FLAG_3D usecase (guessed only for the trellis2cpp backend), '3d' canonical usecase string mapped to the Generate3D method, and a '3d' output modality - REST: POST /v1/3d/generations (+ unversioned alias) returning OpenAIResponse with a /generated-3d URL or b64_json; conditioning image accepted as URL, base64, or data URI; quality/background validated at the edge; .glb served as model/gltf-binary - auth: '3d' route feature (default ON); /api/instructions entry Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(trellis2cpp): add the trellis2.cpp image-to-3D backend Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2, pbr-textures branch) as a Go+purego backend, following the stablediffusion-ggml pattern: - backend/go/trellis2cpp: purego bindings to the flat C ABI (v9, asserted at startup), eager pipeline load with model-set validation (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured exactly like the upstream demo), Generate3D via t2_generate + t2_bake_glb writing a binary glTF to dst. Weight-free unit tests cover resolution/validation/param mapping — CI never downloads the multi-GB GGUF set or runs inference. - CPU SIMD variants build into per-variant directories (the shared libggml sonames collide across variants, unlike sd-ggml's flat renamed-.so scheme); run.sh picks one via /proc/cpuinfo. - CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta + latest/master image entries, bump_deps tracking of the pbr-textures branch, changed-backends.js mapping, top-level Makefile targets. - Importer: auto-detects trellis GGUF repos/URIs (registered before llama-cpp so the .gguf match isn't stolen) and expands any trellis URI to the full 10-file component set spanning the three LocalAI-io HF repos. - Gallery: trellis2-4b (full PBR + 1024 cascade) and trellis2-4b-geometry (512 untextured) with verified sha256s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(ui): 3D generation page with native GLB viewer and IndexedDB history Adds a Studio tab + /app/3d page for the new image-to-3D endpoint: - GlbViewer ports the trellis2cpp demo's dependency-free WebGL2 renderer (quaternion trackball, metallic-roughness PBR, ACES, hidden-line wireframe with a bounded index budget) and pairs it with a minimal GLB parser for the two forms t2_bake_glb emits — dense vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as normalized integers) and the opt-in UV-atlas textured form. Parsing happens before any GL so stats and errors render without WebGL2. - use3DHistory stores past generations (params, input thumbnail, and the GLB blob itself) in IndexedDB with keep-newest-20 eviction — GLBs are multi-MB binaries localStorage can't hold — and the page offers a download button for the active GLB. - Wiring: CAP_3D capability constant (FLAG_3D — the exact string /api/models/capabilities serves), threeDApi, router entries, Studio tab, vite dev proxy, en locale keys. - e2e: render-smoke entry plus a focused spec that feeds a real one-triangle vertex-PBR GLB through the parser/viewer and exercises IndexedDB persistence, selection, deletion, and API errors. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(3d): address API correctness and UX issues Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it. Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add previewable print remeshing Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download. Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(trellis2cpp): centralize remesh dependency pins Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement Generate3D stub for new proto RPC The Generate3D RPC added to backend.proto for the trellis2cpp backend made tonic's generated Backend trait require generate3_d, breaking the kokoros-grpc build. Return unimplemented like the other unsupported modalities. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>	2026-07-29 16:15:04 +02:00
3 people	2d889e61a6	feat(backend): add magpie-tts-cpp text-to-speech backend (#11115) ... * feat(backend): add magpie-tts-cpp text-to-speech backend Add a Go + purego backend wrapping the magpie-tts.cpp ggml port of NVIDIA's Magpie TTS Multilingual 357M (encoder + autoregressive decoder over NanoCodec tokens), producing 22.05 kHz mono audio in 5 baked voices (Aria, Jason, John, Leo, Sofia; case-insensitive names or indices 0-4) across 9+ languages from a single self-contained GGUF. Mirrors qwen3-tts-cpp / moss-tts-cpp: dlopen the static-ggml shared library, bind the flat magpie_tts_capi_* C-API via purego (no local C shim needed, the upstream .so exports it directly), and serve the gRPC TTS + TTSStream methods behind base.SingleThread (the C context is not reentrant across synthesize calls). The backend CMakeLists translates the Makefile's -DGGML_{CUDA,METAL,VULKAN,HIP} flags into upstream's MAGPIE_GGML_* toggles (upstream FORCE-overwrites the ggml cache entries from those), pinned to magpie-tts.cpp v0.1.1 (e3f3dd1ebe22b64e7405f93b519f2d1930712568), which statically links ggml into libmagpie-tts.so (ldd shows only system libs). Wires the full registration: backend-matrix.yml (CPU amd64/arm64, CUDA 12/13, Intel SYCL f16/f32, Vulkan amd64/arm64, ROCm, NVIDIA L4T + L4T CUDA 13, and Darwin metal), backend/index.yaml metas and image entries, the root Makefile build targets, the changed-backends backend-filter path mapping, the bump_deps auto-bump matrix, a test-extra per-backend smoke job, the /backends/known pref-only importer entry, the backend capabilities map (TTS + TTSStream, no voice cloning), and the README / compatibility-table docs rows. Verified locally: unit + e2e Ginkgo suites pass against the real q8_0 GGUF (22.05 kHz mono WAV, RMS > 0.01), a live gRPC LoadModel + TTS round-trip returns valid non-silent audio, and the pre-commit gates (make lint, make test-coverage-check) pass, run manually with LOCALAI_TEST_HTTP_PORT overriding the locally-occupied 9090. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * gallery: add magpie-tts-cpp model entries (q8_0 + f16) Add the Magpie TTS Multilingual 357M GGUFs from mudler/magpie-tts.cpp-gguf to the model gallery: q8_0 (~624 MB, near-lossless, fastest decode, recommended) with an f16 (~784 MB) variant, both served by the magpie-tts-cpp backend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * magpie-tts-cpp: bump pin to rewritten upstream v0.1.1 SHA Upstream history was rewritten to purge accidentally committed build artifacts; v0.1.1 now resolves to 6f7696cf. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-25 08:38:48 +02:00
dependabot[bot]	036eccc32d	chore(deps): bump actions/setup-node from 6 to 7 (#10915) ... Bumps [actions/setup-node](https://github.com/actions/setup-node) from 6 to 7. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-07-18 22:36:37 +02:00
LocalAI [bot]andEttore Di Giacinto	fdff114701	ci(vibevoice): skip the ASR transcription e2e on release tag builds (#10567) ... The `tests-vibevoice-cpp-grpc-transcription` job downloads the vibevoice ASR model (`vibevoice-asr-q4_k.gguf`, ~10 GB) and decodes it through the e2e-backends harness. On release tag pushes the detect step forces the full matrix (run-all=true), so this job runs and consistently times out: the inner `go test -timeout 30m` cannot pull a 10 GB file from HuggingFace's throttled Xet CDN within budget (curl --max-time 600 x5 retries overruns the deadline), leaving an orphaned curl and a 30m panic. It has been red on every release (v4.5.3/4/5). Guard the job's `if` with `!startsWith(github.ref, 'refs/tags/')` so it no longer runs on tag/release builds. It still runs on PRs and branch pushes that touch vibevoice-cpp, so real regressions are caught off the release path. A proper fix (a small ASR test GGUF) can re-enable it on tags later. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-28 00:40:21 +02:00
dependabot[bot]	10184b5e28	chore(deps): bump actions/checkout from 6 to 7 (#10451) ... Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-06-22 21:38:37 +02:00
Richard Palethorpe	085fc53bbc	fix(router): production-ready request router + auto-size batch for embedding/rerank (#10104) ... * fix(router): score classifier production-readiness Conversation trimming runs through the classifier model's chat template and trims by exact token count, sized to the model's n_batch which is now scaled to context so long probes can't crash the backend. Missing chat_message templates are a hard error at router build time. Router- facing factories (Embedder/Scorer/Reranker/TokenCounter) re-resolve ModelConfig per call so a model installed post-startup doesn't bind a stub Backend="" config and silently fall into the loader's auto- iterate path. New 'vector_store' backend trace recorded inside localVectorStore on every Search/Insert — including the backend-load-failure path that previously vanished into an xlog.Warn — with outcome tagging (hit/miss/empty_store/backend_load_error/find_error/insert_error/ok). Companion cleanup drops misleading similarity:0 and input_tokens_count:0 from non-hit and text-mode traces. Gallery local-store-development aliases to 'local-store' so the master image satisfies pkg/model.LocalStoreBackend lookups from the embedding cache. Misc: llama-cpp TokenizeString reads the correct 'prompt' JSON key (the original bug); ModelTokenize nil-guard; non-fatal mitm proxy startup; PII 'route_local' renamed to 'allow' with docs/UI in sync; model-editor footer no longer eats the edit area on small screens; several config-editor template/dropdown/section fixes. Tests: e2e router specs (casual/code-hint + long-conversation trim), vector_store trace specs, lazy-factory specs, gallery dev-alias resolution, Playwright trace badge + scroll regression. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(backend): auto-size batch to context for embedding and rerank models Embedding and rerank models pool over the whole input in a single physical batch (n_ubatch). With batch left at the 512 default, the backend rejects longer inputs with "input is too large to process", silently capping a large-context embedder (e.g. 8k/32k) at 512 tokens. Size n_batch to the context for these single-pass usecases, mirroring the existing FLAG_SCORE behaviour; an explicit batch: still wins. Extracts EffectiveContextSize/EffectiveBatchSize from grpcModelOpts so the effective decode window has one home for other callers to reuse. Adds an e2e-aio regression test that embeds a >512-token input. The AIO embedding model is switched to nomic-embed-text-v1.5 (2048 context) because the previous granite model was capped at 512 tokens and could not exercise the larger batch. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(gallery): raise arch-router scoring output cap via parallel:64 Scoring decodes the whole prompt+candidate in a single llama_decode and reads one logit row per candidate token. The vendored llama.cpp server caps causal output rows at n_parallel, so the default of 1 aborts with GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route labels. Set options: [parallel:64] on both arch-router quant entries to lift the cap; kv_unified (the grpc-server default) keeps the full context per sequence, so this does not split the KV cache. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-12 16:21:15 +02:00
LocalAI [bot]andmudler	56cc4f63fc	feat(backend): locate-anything-cpp (open-vocabulary object detection via ggml) (#10264) ... * feat(backend): add locate-anything-cpp backend (open-vocab detection via la_capi) A Go/purego backend wrapping locate-anything.cpp's la_capi C ABI, implementing the gRPC Detect RPC: image + open-vocabulary text prompt -> labeled boxes. Mirrors backend/go/rfdetr-cpp; static-links ggml into a per-CPU-variant .so. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(backend): register locate-anything-cpp in build matrix Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): locate-anything gallery entry + model importer Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(backend): locate-anything-cpp Load+Detect wire test Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add locate-anything-3b model to the gallery index Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(backend): register locate-anything.cpp in bump_deps auto-bump Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: mudler <mudler@localai.io> * ci(test): e2e smoke for locate-anything-cpp in test-extra (loads the 3B + image, runs Detect) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: mudler <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: mudler <mudler@localai.io> Co-authored-by: mudler <mudler@localai.io>	2026-06-12 14:59:07 +02:00
LocalAI [bot]andEttore Di Giacinto	4912c9b73a	feat(parakeet-cpp): add NVIDIA NeMo Parakeet ASR backend (parakeet.cpp) (#10084) ... * feat(parakeet-cpp): L0 backend scaffold, LoadModel + AudioTranscription (text) Add a Go gRPC backend that bridges LocalAI to parakeet.cpp via the flat C-API (parakeet_capi.h), loaded with purego (cgo-less, mirrors the whisper / vibevoice-cpp backends). L0 scope: - main.go: dlopen libparakeet.so (override via PARAKEET_LIBRARY), register the C-API entry points, start the gRPC server. - goparakeetcpp.go: Load (parakeet_capi_load), AudioTranscription (parakeet_capi_transcribe_path, decoder=0 = per-arch default head), Free, serialized through base.SingleThread since the C engine is a thread-unsafe singleton. char* returns are bound as uintptr so the malloc'd buffer is freed via parakeet_capi_free_string after copy. - AudioTranscriptionStream returns a clear "not implemented in L0" error (closes the channel so the server doesn't hang), wired in L2. - Makefile: clone-at-pin + cmake (PARAKEET_VERSION for bump_deps.sh), with a local-symlink dev shortcut; run.sh / package.sh mirror whisper. - Test auto-skips without PARAKEET_BACKEND_TEST_MODEL/_WAV fixtures. Builds clean (CGO_ENABLED=0), gofmt clean, test passes. The single unsafeptr vet note in goStringFromCPtr is documented and matches the whisper backend's tolerated pattern. Word/segment timestamps (L1) and cache-aware streaming (L2) follow. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet-cpp): L1 word/segment timestamps via transcribe_path_json AudioTranscription now calls parakeet_capi_transcribe_path_json and shapes the per-word / per-token timestamps into the TranscriptResult: - Bind parakeet_capi_transcribe_path_json (purego, char* as uintptr like the other returns) and register it in main.go + the test loader. - Parse the JSON document ({"text","words":[{w,start,end,conf}], "tokens":[{id,t,conf}]}) into typed structs. - Synthesise a single whole-clip segment (parakeet emits no native segment boundaries) spanning the first word start to the last word end; token ids populate Segment.Tokens. - Attach word-level timings only when timestamp_granularities=["word"], matching the OpenAI API (segment-level default). secondsToNanos mirrors the whisper backend's nanosecond convention. Verified end-to-end against tdt_ctc-110m (f16): both the default and word-granularity specs pass; builds clean, gofmt clean, vet shows only the one documented unsafeptr note shared with the whisper backend. Cache-aware streaming (L2) follows. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet-cpp): L2 cache-aware streaming with EOU segmentation Wire AudioTranscriptionStream to the streaming RNN-T C-API: - Bind parakeet_capi_stream_{begin,feed,finalize,free}; feed takes 16 kHz mono float PCM ([]float32 via purego) and writes *eou_out on <EOU>/<EOB>. - Decode opts.Dst to 16 kHz mono PCM (utils.AudioToWav + go-audio, same as the whisper backend), feed it in 1 s chunks, and emit each newly-finalized text run as a TranscriptStreamResponse delta. - <EOU>/<EOB> events close the current segment; a closing FinalResult carries the full transcript plus the per-utterance segments (with a whole-clip fallback segment when no EOU fired). - stream_begin returns 0 for non-streaming models, surfaced as a clear error instead of an empty stream. Honours context cancellation between chunks. Frees every malloc'd delta and the session. Verified end-to-end against realtime_eou_120m-v1 (f16): the streamed transcript matches the offline 110m reference word-for-word, deltas reconstruct the final text, and the spec passes alongside the offline specs. Builds clean, gofmt clean, vet shows only the shared documented unsafeptr note. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet-cpp): L3 register backend in build/CI/gallery (whisper parity) Wire the new Go gRPC parakeet-cpp backend (parakeet.cpp ggml port of NVIDIA NeMo Parakeet ASR) into LocalAI's build/CI/gallery surfaces, matching the existing ggml whisper Go backend 1:1. - .github/backend-matrix.yml: add 11 linux entries + 1 darwin entry mirroring every whisper build (cpu amd64/arm64, intel sycl f32/f16, vulkan amd64/arm64, nvidia cuda-12, nvidia cuda-13, nvidia-l4t-arm64, nvidia-l4t-cuda-13-arm64, rocm hipblas, metal-darwin-arm64), all on ./backend/Dockerfile.golang with backend: "parakeet-cpp" and -*-parakeet-cpp tag-suffixes. - scripts/changed-backends.js: explicit inferBackendPath branch resolving parakeet-cpp to backend/go/parakeet-cpp/ before the generic golang branch. - .github/workflows/bump_deps.yaml: track the PARAKEET_VERSION pin in backend/go/parakeet-cpp/Makefile (repo mudler/parakeet.cpp, branch master). - backend/index.yaml: add &parakeetcpp meta + latest/development image entries for every matrix tag-suffix. - Makefile: add backends/parakeet-cpp to .NOTPARALLEL, BACKEND_PARAKEET_CPP definition, docker-build target eval, and test-extra-backend-parakeet-cpp- transcription target (mirrors test-extra-backend-whisper-transcription). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet-cpp): L4 gallery importer for parakeet GGUFs Add ParakeetCppImporter so parakeet.cpp GGUFs auto-detect on /import-model and route to the parakeet-cpp backend (it also surfaces in /backends/known, which drives the import dropdown). - Match is narrow: a .gguf whose name carries a parakeet architecture token (<arch>-<size>-<quant>.gguf, e.g. tdt_ctc-110m-f16.gguf, rnnt-0.6b-q4_k.gguf, realtime_eou_120m-v1-q8_0.gguf), a direct URL to one, or preferences.backend="parakeet-cpp". It deliberately does NOT claim arbitrary llama-style GGUFs, nor the upstream nvidia/parakeet-* NeMo repos (.nemo, not runnable here). - Registered in the ASR batch BEFORE LlamaCPPImporter so its GGUFs aren't swallowed by the generic .gguf importer. - Import nests files under parakeet-cpp/models/<name>/, defaults to the smallest quant (q4_k, near-lossless on parakeet) with a size-ladder fallback, and honours preferences.quantizations / name / description. Tested with synthetic HF details (no network): metadata, positive matches (HF repo, direct URL, preference), narrowness negatives (llama GGUF, NeMo repo), and import (default quant, override, direct URL), 9 specs pass, build/vet/gofmt clean. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(parakeet-cpp): document the parakeet-cpp transcription backend Add parakeet-cpp to the audio-to-text backend list and a dedicated usage section: direct GGUF import (auto-detects to the backend), model YAML, word-level timestamps via timestamp_granularities[]=word, and cache-aware streaming with the realtime_eou model. Points at the mudler/parakeet-cpp-gguf collection repo. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(parakeet-cpp): wire transcription gRPC e2e test into test-extra The L3 commit added the test-extra-backend-parakeet-cpp-transcription Makefile target but never invoked it in CI. Mirror the whisper job: - Add a parakeet-cpp output to detect-changes (emitted by changed-backends.js from the matrix entry). - Add tests-parakeet-cpp-grpc-transcription, gated on the parakeet-cpp path filter / run-all, building the backend image and running the transcription e2e against tdt_ctc-110m + the JFK clip. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(parakeet-cpp): drop em dashes from comments and docs Replace em dashes with plain punctuation in the backend comments, the importer, package.sh, and the audio-to-text docs section (and use "and" instead of the multiplication sign). No behaviour change. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add parakeet-cpp f16 models to the model gallery Add the 10 NVIDIA Parakeet models (f16, the recommended quality/speed default) as gallery entries that install on the parakeet-cpp backend from mudler/parakeet-cpp-gguf: tdt_ctc-110m/1.1b, tdt-0.6b-v2/v3, tdt-1.1b, ctc-0.6b/1.1b, rnnt-0.6b/1.1b, and the cache-aware streaming realtime_eou_120m-v1. Each pins the file sha256 and routes transcript usecases to the backend. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): satisfy govet lint + bump PARAKEET_VERSION - goparakeetcpp.go: //nolint:govet on the C-owned-pointer unsafe.Pointer conversion (golangci-lint reports new-only issues, so unlike the whisper backend's identical line this one is flagged). - Makefile: bump PARAKEET_VERSION to the current parakeet.cpp master commit (the previous pin's commit no longer exists after upstream history was squashed), so the backend image clone/build resolves again. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): pin PARAKEET_VERSION to a tag-stable commit The previous SHA pin was orphaned when parakeet.cpp's single-commit master was amended/force-pushed, so the backend image clone (git fetch <sha>) failed across every build variant. Repoint to 845c29e, which upstream now keeps permanently fetchable via the `localai-backend-pin` tag, so future upstream amends no longer break the backend build. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): init the ggml submodule in the backend image clone The backend Dockerfile clones parakeet.cpp at PARAKEET_VERSION with a shallow fetch + checkout but never initialised submodules, so third_party/ggml was empty and the parakeet.cpp cmake build failed at `add_subdirectory(third_party/ggml)` (CMakeLists.txt:53) on every build variant. Add `git submodule update --init --recursive --depth 1 --single-branch` after checkout, mirroring the whisper backend. Verified locally: clone + submodule + cmake configure now succeeds. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): statically link ggml into libparakeet.so The shared libparakeet.so linked ggml's shared libs (libggml*.so), but the package only ships libparakeet.so, so at runtime dlopen failed with "libggml.so.0: cannot open shared object file" (the e2e transcription test panicked on load). Build ggml static + PIC (BUILD_SHARED_LIBS=OFF, CMAKE_POSITION_INDEPENDENT_CODE=ON) so libparakeet.so embeds ggml and depends only on system libs already present in the runtime image. Verified locally: ldd shows no libggml dependency. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): non-streaming fallback in AudioTranscriptionStream The e2e streaming test ran AudioTranscriptionStream against tdt_ctc-110m (not a cache-aware streaming model), so stream_begin returned 0 and the call errored. Per LocalAI's streaming contract (and the whisper backend), a non-streaming model should fall back to a single offline transcription emitted as one delta plus a closing FinalResult. Do that instead of erroring, so the streaming endpoint works for every parakeet model. Verified locally: the streaming spec passes against the non-streaming 110m model via fallback. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-30 14:46:10 +02:00
LocalAI [bot]andEttore Di Giacinto	7a4ca8f60d	feat(backend): rfdetr-cpp native object detection + segmentation backend (#10028) ... Adds a Go native gRPC backend that dlopens librfdetrcpp.so (built from mudler/rf-detr.cpp at the pinned RFDETR_VERSION) via purego and exposes the rfdetr.cpp inference pipeline through LocalAI's existing Detect RPC. Supports all 5 RF-DETR detection variants (Nano/Small/Base/Medium/Large) and 6 segmentation variants (SegNano/SegSmall/SegMedium/SegLarge/ SegXLarge/Seg2XLarge) with F32/F16/Q8_0/Q4_K quantizations. Pre-built GGUFs ship at mudler/rfdetr-cpp-* on HuggingFace. Detection returns Bbox + class_name + confidence; segmentation also returns PNG-encoded per-detection masks via the rfdetr_capi accessor functions (rfdetr_capi_get_detection_{class_id,box,score,class_name, mask_png}). End-to-end verified through POST /v1/detection: HTTP -> gRPC -> purego dlopen -> rfdetr.cpp -> ggml -> response (9 detections on the detection model, 21 detections + valid PNG masks on the seg-nano model against the kitchen fixture). Wiring: - backend/go/rfdetr-cpp/{main.go,gorfdetrcpp.go,CMakeLists.txt, Makefile,run.sh,package.sh,test.sh,.gitignore} - Top-level Makefile: BACKEND_RFDETR_CPP, docker-build target, .NOTPARALLEL, prepare-test-extra, test-extra - backend/go/rfdetr-cpp/Makefile: `test` target invoked by test-extra - .github/backend-matrix.yml: CPU + CUDA-12/13 + L4T CUDA-12/13 (arm64) + HIP + Vulkan (amd64 + arm64) + SYCL f32/f16 - backend/index.yaml: rfdetr-cpp meta anchor + latest/development image entries for every matrix tag-suffix - .github/workflows/bump_deps.yaml: RFDETR_VERSION pin tracking (mudler/rf-detr.cpp branch main) - gallery/index.yaml: 11 rfdetr-cpp-* entries (nano + 4 detection variants + 6 seg variants), all backed by mudler/rfdetr-cpp-* on HuggingFace with sha256 pinning on the F16 default - core/gallery/importers/rfdetr.go: GGUF auto-routing for HF imports (mudler/rfdetr-cpp-* repos route to rfdetr-cpp, Transformer-format repos stay on the Python rfdetr backend; explicit preferences.backend overrides both heuristics) - core/gallery/importers/rfdetr_test.go: table-driven coverage of the auto-routing + a live mudler/rfdetr-cpp-nano cross-check scripts/changed-backends.js needs no change: the existing Dockerfile.golang -> backend/go/${item.backend}/ branch already routes the 9 rfdetr-cpp matrix entries to the correct backend path. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-27 18:43:57 +02:00
Richard Palethorpe	0245b33eab	feat(realtime): Add Liquid Audio s2s model and assistant mode on talk page (#9801) ... * feat(liquid-audio): add LFM2.5-Audio any-to-any backend + realtime_audio usecase Wires LiquidAI's LFM2.5-Audio-1.5B as a self-contained Realtime API model: single engine handles VAD, transcription, LLM, and TTS in one bidirectional stream — drop-in alternative to a VAD+STT+LLM+TTS pipeline. Backend - backend/python/liquid-audio/ — new Python gRPC backend wrapping the `liquid-audio` package. Modes: chat / asr / tts / s2s, voice presets, Load/Predict/PredictStream/AudioTranscription/TTS/VAD/AudioToAudioStream/ Free and StartFineTune/FineTuneProgress/StopFineTune. Runtime monkey-patch on `liquid_audio.utils.snapshot_download` so absolute local paths from LocalAI's gallery resolve without a HF round-trip. soundfile in place of torchaudio.load/save (torchcodec drags NVIDIA NPP we don't bundle). - backend/backend.proto + pkg/grpc/{backend,client,server,base,embed, interface}.go — new AudioToAudioStream RPC mirroring AudioTransformStream (config/frame/control oneof in; typed event+pcm+meta out). - core/services/nodes/{health_mock,inflight}_test.go — add stubs for the new RPC to the test fakes. Config + capabilities - core/config/backend_capabilities.go — UsecaseRealtimeAudio, MethodAudio ToAudioStream, UsecaseInfoMap entry, liquid-audio BackendCapability row. - core/config/model_config.go — FLAG_REALTIME_AUDIO bitmask, ModalityGroups membership in both speech-input and audio-output groups so a lone flag still reads as multimodal, GetAllModelConfigUsecases entry, GuessUsecases branch. Realtime endpoint - core/http/endpoints/openai/realtime.go — extract prepareRealtimeConfig() so the gate is unit-testable; accept realtime_audio models and self-fill empty pipeline slots with the model's own name (user-pinned slots win). - core/http/endpoints/openai/realtime_gate_test.go — six specs covering nil cfg, empty pipeline, legacy pipeline, self-contained realtime_audio, user-pinned VAD slot, and partial legacy pipeline. UI + endpoints - core/http/routes/ui.go — /api/pipeline-models accepts either a legacy VAD+STT+LLM+TTS pipeline or a realtime_audio model; surfaces a self_contained flag so the Talk page can collapse the four cards. - core/http/routes/ui_api.go — realtime_audio in usecaseFilters. - core/http/routes/ui_pipeline_models_test.go — covers both code paths. - core/http/react-ui/src/pages/Talk.jsx — self-contained badge instead of the four-slot grid; rename Edit Pipeline → Edit Model Config; less pipeline-specific wording. - core/http/react-ui/src/pages/Models.jsx + locales/en/models.json — new realtime_audio filter button + i18n. - core/http/react-ui/src/utils/capabilities.js — CAP_REALTIME_AUDIO. - core/http/react-ui/src/pages/FineTune.jsx — voice + validation-dataset fields, surfaced when backend === liquid-audio, plumbed via extra_options on submit/export/import. Gallery + importer - gallery/liquid-audio.yaml — config template with known_usecases: [realtime_audio, chat, tts, transcript, vad]. - gallery/index.yaml — four model entries (realtime/chat/asr/tts) keyed by mode option. Fixed pre-existing `transcribe` typo on the asr entry (loader silently dropped the unknown string → entry never surfaced as a transcript model). - gallery/lfm.yaml — function block for the LFM2 Pythonic tool-call format `<|tool_call_start|>[name(k="v")]<|tool_call_end|>` matching common_chat_params_init_lfm2 in vendored llama.cpp. - core/gallery/importers/{liquid-audio,liquid-audio_test}.go — detector matches LFM2-Audio HF repos (excludes -gguf mirrors); mode/voice preferences plumbed through to options. - core/gallery/importers/importers.go — register LiquidAudioImporter before LlamaCPPImporter. - pkg/functions/parse_lfm2_test.go — seven specs for the response/argument regex pair on the LFM2 pythonic format. Build matrix - .github/backend-matrix.yml — seven liquid-audio targets (cuda12, cuda13, l4t-cuda-13, hipblas, intel, cpu amd64, cpu arm64). Jetpack r36 cuda-12 is skipped (Ubuntu 22.04 / Python 3.10 incompatible with liquid-audio's 3.12 floor). - backend/index.yaml — anchor + 13 image entries. - Makefile — .NOTPARALLEL, prepare-test-extra, test-extra, docker-build-liquid-audio. Docs - .agents/plans/liquid-audio-integration.md — phased plan; PR-D (real any-to-any wiring via AudioToAudioStream), PR-E (mid-audio tool-call detector), PR-G (GGUF entries once upstream llama.cpp PR #18641 lands) remain. - .agents/api-endpoints-and-auth.md — expand the capability-surface checklist with every place a new FLAG_* needs to be registered. Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): function calling + history cap for any-to-any models Three pieces, all on the realtime_audio path that just landed: 1. liquid-audio backend (backend/python/liquid-audio/backend.py): - _build_chat_state grows a `tools_prelude` arg. - new _render_tools_prelude parses request.Tools (the OpenAI Chat Completions function array realtime.go already serialises) and emits an LFM2 `<|tool_list_start|>…<|tool_list_end|>` system turn ahead of the user history. Mirrors gallery/lfm.yaml's `function:` template so the model sees the same prompt shape whether served via llama-cpp or here. Without this the backend silently dropped tools — function calling was wired end-to-end on the Go side but the model never saw a tool list. 2. Realtime history cap (core/http/endpoints/openai/realtime.go): - Session grows MaxHistoryItems int; default picked by new defaultMaxHistoryItems(cfg) — 6 for realtime_audio models (LFM2.5 1.5B degrades quickly past a handful of turns), 0/unlimited for legacy pipelines composing larger LLMs. - triggerResponse runs conv.Items through trimRealtimeItems before building conversationHistory. Helper walks the cut left if it would orphan a function_call_output, so tool result + call pairs stay intact. - realtime_gate_test.go: specs for defaultMaxHistoryItems and trimRealtimeItems (zero cap, under cap, over cap, tool-call pair preservation). 3. Talk page (core/http/react-ui/src/pages/Talk.jsx): - Reuses the chat page's MCP plumbing — useMCPClient hook, ClientMCPDropdown component, same auto-connect/disconnect effect pattern. No bespoke tool registry, no new REST endpoints; tools come from whichever MCP servers the user toggles on, exactly as on the chat page. - sendSessionUpdate now passes session.tools=getToolsForLLM(); the update re-fires when the active server set changes mid-session. - New response.function_call_arguments.done handler executes via the hook's executeTool (which round-trips through the MCP client SDK), then replies with conversation.item.create {type:function_call_output} + response.create so the model completes its turn with the tool output. Mirrors chat's client-side agentic loop, translated to the realtime wire shape. UI changes require a LocalAI image rebuild (Dockerfile:308-313 bakes react-ui/dist into the runtime image). Backend.py changes can be swapped live in /backends/<id>/backend.py + /backend/shutdown. Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): LocalAI Assistant ("Manage Mode") for the Talk page Mirrors the chat-page metadata.localai_assistant flow so users can ask the realtime model what's loaded / installed / configured. Tools are run server-side via the same in-process MCP holder that powers the chat modality — no transport switch, no proxy, no new wire protocol. Wire: - core/http/endpoints/openai/realtime.go: - RealtimeSessionOptions{LocalAIAssistant,IsAdmin}; isCurrentUserAdmin helper mirrors chat.go's requireAssistantAccess (no-op when auth disabled, else requires auth.RoleAdmin). - Session grows AssistantExecutor mcpTools.ToolExecutor. - runRealtimeSession, when opts.LocalAIAssistant is set: gate on admin, fail closed if DisableLocalAIAssistant or the holder has no tools, DiscoverTools and inject into session.Tools, prepend holder.SystemPrompt() to instructions. - Tool-call dispatch loop: when AssistantExecutor.IsTool(name), run ExecuteTool inproc, append a FunctionCallOutput to conv.Items, skip the function_call_arguments client emit (the client can't execute these — it doesn't know about them). After the loop, if any assistant tool ran, trigger another response so the model speaks the result. Mirrors chat's agentic loop, driven server-side rather than via client round-trip. - core/http/endpoints/openai/realtime_webrtc.go: RealtimeCallRequest gains `localai_assistant` (JSON omitempty). Handshake calls isCurrentUserAdmin and builds RealtimeSessionOptions. - core/http/react-ui/src/pages/Talk.jsx: admin-only "Manage Mode" checkbox under the Tools dropdown; passes localai_assistant: true to realtimeApi.call's body, captured in the connect callback's deps. Mirroring chat's pattern means the in-process MCP tools surface "just works" for the Talk page without exposing a Streamable-HTTP MCP endpoint (which was the alternative). Clients with their own MCP servers can still use the existing ClientMCPDropdown path in parallel; the realtime handler distinguishes them by AssistantExecutor.IsTool() at dispatch time. Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): render Manage Mode tool calls in the Talk transcript Previously the realtime endpoint only emitted response.output_item.added for the FunctionCall item, and Talk.jsx's switch ignored the event — so server-side tool runs were invisible in the UI. The model would speak the result but the user had no way to see what tool was actually called. realtime.go: after executing an assistant tool inproc, emit a second output_item.added/.done pair for the FunctionCallOutput item. Mirrors the way the chat page displays tool_call + tool_result blocks. Talk.jsx: handle both response.output_item.added and .done. Render FunctionCall (with arguments) and FunctionCallOutput (pretty-printed JSON when possible) as two transcript entries — `tool_call` with the wrench icon, `tool_result` with the clipboard icon, both in mono-space secondary-colour. Resets streamingRef after the result so the next assistant text delta starts a fresh transcript entry instead of appending to the previous turn. Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor(realtime): bound the Manage Mode tool-loop + preserve assistant tools Fallout from a review pass on the Manage Mode patches: - Bound the server-side agentic loop. triggerResponse used to recurse on executedAssistantTool with no cap — a model that kept calling tools would blow the goroutine stack. New maxAssistantToolTurns = 10 (mirrors useChat.js's maxToolTurns). Public triggerResponse is now a thin shim over triggerResponseAtTurn(toolTurn int); recursion increments the counter and stops at the cap with an xlog.Warn. - Preserve Manage Mode tools across client session.update. The handler used to blindly overwrite session.Tools, so toggling a client MCP server mid-session silently wiped the in-process admin tools. Session now caches the original AssistantTools slice at session creation and the session.update handler merges them back in (client names win on collision — the client is explicit). - strconv.ParseBool for the localai_assistant query param instead of hand-rolled "1" || "true". Mirrors LocalAIAssistantFromMetadata. - Talk.jsx: render both tool_call and tool_result on response.output_item.done instead of splitting them across .added and .done. The server's event pairing (added → done) stays correct; the UI just doesn't need to inspect both phases of the same item. One switch case instead of two, no behavioural change. Out of scope (noted for follow-ups): extract a shared assistant-tools helper between chat.go and realtime.go (duplication is small enough that two parallel implementations stay readable for now), and an i18n key for the Manage Mode helper text (Talk.jsx doesn't use i18n anywhere else yet). Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci(test-extra): wire liquid-audio backend smoke test The backend ships test.py + a `make test` target and is listed in backend-matrix.yml, so scripts/changed-backends.js already writes a `liquid-audio=true|false` output when files under backend/python/liquid-audio/ change. The workflow just wasn't reading it. - Expose the `liquid-audio` output on the detect-changes job - Add a tests-liquid-audio job that runs `make` + `make test` in backend/python/liquid-audio, gated on the per-backend detect flag The smoke covers Health() and LoadModel(mode:finetune); fine-tune mode short-circuits before any HuggingFace download (backend.py:192), so the job needs neither weights nor a GPU. The full-inference path remains gated on LIQUID_AUDIO_MODEL_ID, which CI doesn't set. The four new Go test files (core/gallery/importers/liquid-audio_test.go, core/http/endpoints/openai/realtime_gate_test.go, core/http/routes/ui_pipeline_models_test.go, pkg/functions/parse_lfm2_test.go) are already picked up by the existing test.yml workflow via `make test` → `ginkgo -r ./pkg/... ./core/...`; their packages all carry RunSpecs entries. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-13 21:57:27 +02:00
LocalAI [bot]andEttore Di Giacinto	19d59102d5	feat(whisper-cpp): implement streaming transcription (#9751) ... * test(whisper): wire e2e streaming transcription target Adds test-extra-backend-whisper-transcription, mirroring the existing llama-cpp / sherpa-onnx / vibevoice-cpp targets. The generic AudioTranscriptionStream spec at tests/e2e-backends/backend_test.go:644 fails today because backend/go/whisper has no streaming impl - this target is the failing TDD gate that the next phase makes pass. Confirmed RED locally: 3 Passed (health, load, offline transcription), 1 Failed (streaming spec hits its 300s context deadline because the base implementation returns 'unimplemented' but doesn't close the result channel, leaving the gRPC stream open until the client times out). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(whisper-cpp): expose new_segment_callback to the Go side Adds set_new_segment_callback() and a C-side trampoline that whisper.cpp invokes once per new text segment during whisper_full(). The trampoline dispatches (idx_first, n_new, user_data) to a Go function pointer registered via purego.NewCallback - text and timings are pulled by Go through the existing get_segment_text/get_segment_t0/get_segment_t1 getters. Wires the hook only when streaming is actually requested, to avoid a per-segment function-pointer dispatch on the offline path. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(whisper-cpp): implement AudioTranscriptionStream Wires whisper.cpp's new_segment_callback through purego back to Go so the streaming transcription RPC produces real, time-correlated deltas while whisper_full() is still decoding. Each segment becomes one TranscriptStreamResponse{Delta}; whisper_full's return is the TranscriptStreamResponse{FinalResult} carrying the full segment list, language, and duration. Per-call state is tracked in a sync.Map keyed by an atomic counter; the Go callback registered via purego.NewCallback is a singleton, dispatched through user_data. SingleThread today means only one entry is ever live, but the map shape matches the sherpa-onnx TTS callback pattern. The streaming path's final.Text is the literal concat of every emitted delta (a strings.Builder accumulated by onNewSegment) so the e2e invariant `final.Text == concat(deltas)` holds exactly. The first delta has no leading space; subsequent deltas are space-prefixed. The offline AudioTranscription path is unchanged. Closes the gap with sherpa-onnx, vibevoice-cpp, llama-cpp, and tinygrad, which already implement AudioTranscriptionStream. Verified GREEN locally: make test-extra-backend-whisper-transcription passes 4/4 specs (3 Passed initially under RED, +1 streaming spec now). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(whisper-cpp): assert progressive multi-segment streaming Drives AudioTranscriptionStream against a real long-audio fixture and asserts len(deltas) >= 2. The generic e2e spec at tests/e2e-backends/backend_test.go:644 only checks len(deltas) >= 1 which is satisfied by both real and faked streaming - this spec is the guardrail that a future "fake" impl can't sneak past. Skipped by default (env-gated, like the cancellation spec); set WHISPER_LIBRARY, WHISPER_MODEL_PATH, and WHISPER_AUDIO_PATH to a 30+ second clip to run. Verified locally with a 55s 5x-JFK concat against ggml-base.en.bin: 1 Passed in 7.3s, deltas >= 2, finalSegmentCount >= 2, concat(deltas) == final.Text. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(whisper-cpp): add transcription gRPC e2e job Mirrors tests-sherpa-onnx-grpc-transcription / tests-llama-cpp-grpc-transcription. Runs make test-extra-backend-whisper-transcription whenever the whisper backend or the run-all switch fires, so a pin-bump or refactor that breaks streaming transcription gets caught before merge. The whisper output on detect-changes is already emitted by scripts/changed-backends.js (it iterates allBackendPaths); this PR just exposes it as a workflow output and consumes it. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(whisper-cpp): silence errcheck on AudioTranscriptionStream defers golangci-lint runs with new-from-merge-base=origin/master, so the identical defer patterns in the existing offline AudioTranscription path are grandfathered while the new ones in AudioTranscriptionStream trip errcheck. Wrap both defers in `func() { _ = ... }()` to match what errcheck wants without altering behavior. The errors from os.RemoveAll and *os.File.Close are not actionable inside a defer here (we're already returning), matching the offline path's contract. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-10 23:11:46 +02:00
Ettore Di Giacinto	5a12392570	ci(concurrency): make cancel-in-progress event-aware, group by sha on push ... Yesterday two PRs (#9724 llama.cpp bump, #9731 llama-cpp-darwin consolidation) merged 11 seconds apart. Both shared the same backend.yml concurrency group (ci-backends-refs/heads/master-...) due to "${{ github.head_ref || github.ref }}" — empty head_ref on push events falls through to the static refs/heads/master. With cancel-in-progress: true that meant the second merge cancelled the first's in-flight backend builds. The first PR's CI never finished; the second PR only touched CI files so its run was a no-op. Two changes per workflow: - group: replace "${{ github.head_ref || github.ref }}" with "${{ github.event.pull_request.number || github.sha }}". On PRs this groups by PR number (same as before, just keyed on number not branch name); on push events it groups per-commit, so two master pushes never share a group. - cancel-in-progress: gate on github.event_name == 'pull_request' so rapid pushes to a PR still cancel old runs (newer push wins) but master pushes never cancel each other. Trade-off vs alternatives: - Merge queue would also solve this and additionally test the merged commit before it lands. Heavier process change; out of scope here. - Allowing per-commit master concurrency means two simultaneous master runs may overlap and race on tag pushes, but each commit's manifest digest is unique and the registry is last-writer-wins on tags — newer commit's tag overwrites older. Applied to 11 workflows that share the same concurrency pattern: backend.yml, backend_pr.yml, image.yml, image-pr.yml, lint.yml, test.yml, test-extra.yml, tests-e2e.yml, tests-aio.yml, tests-ui-e2e.yml, generate_intel_image.yaml. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-09 08:30:55 +00:00
Richard Palethorpe	bb033b16a9	feat: add LocalVQE backend and audio transformations UI (#9640) ... feat(audio-transform): add LocalVQE backend, bidi gRPC RPC, Studio UI Introduce a generic "audio transform" capability for any audio-in / audio-out operation (echo cancellation, noise suppression, dereverberation, voice conversion, etc.) and ship LocalVQE as the first backend implementation. Backend protocol: - Two new gRPC RPCs in backend.proto: unary AudioTransform for batch and bidirectional AudioTransformStream for low-latency frame-by-frame use. This is the first bidi stream in the proto; per-frame unary at LocalVQE's 16 ms hop would be RTT-bound. Wire it through pkg/grpc/{client,server, embed,interface,base} with paired-channel ergonomics. LocalVQE backend (backend/go/localvqe/): - Go-Purego wrapper around upstream liblocalvqe.so. CMake builds the upstream shared lib + its libggml-cpu-*.so runtime variants directly — no MODULE wrapper needed because LocalVQE handles CPU feature selection internally via GGML_BACKEND_DL. - Sets GGML_NTHREADS from opts.Threads (or runtime.NumCPU()-1) — without it LocalVQE runs single-threaded at ~1× realtime instead of the documented ~9.6×. - Reference-length policy: zero-pad short refs, truncate long ones (the trailing portion can't have leaked into a mic that wasn't recording). - Ginkgo test suite (9 always-on specs + 2 model-gated). HTTP layer: - POST /audio/transformations (alias /audio/transform): multipart batch endpoint, accepts audio + optional reference + params[*]=v form fields. Persists inputs alongside the output in GeneratedContentDir/audio so the React UI history can replay past (audio, reference, output) triples. - GET /audio/transformations/stream: WebSocket bidi, 16 ms PCM frames (interleaved stereo mic+ref in, mono out). JSON session.update envelope for config; constants hoisted in core/schema/audio_transform.go. - ffmpeg-based input normalisation to 16 kHz mono s16 WAV via the existing utils.AudioToWav (with passthrough fast-path), so the user can upload any format / rate without seeing the model's strict 16 kHz constraint. - BackendTraceAudioTransform integration so /api/backend-traces and the Traces UI light up with audio_snippet base64 and timing. - Routes registered under routes/localai.go (LocalAI extension; OpenAI has no /audio/transformations endpoint), traced via TraceMiddleware. Auth + capability + importer: - FLAG_AUDIO_TRANSFORM (model_config.go), FeatureAudioTransform (default-on, in APIFeatures), three RouteFeatureRegistry rows. - localvqe added to knownPrefOnlyBackends with modality "audio-transform". - Gallery entry localvqe-v1-1.3m (sha256-pinned, hosted on huggingface.co/LocalAI-io/LocalVQE). React UI: - New /app/transform page surfaced via a dedicated "Enhance" sidebar section (sibling of Tools / Biometrics) — the page is enhancement, not generation, so it lives outside Studio. Two AudioInput components (Upload + Record tabs, drag-drop, mic capture). - Echo-test button: records mic while playing the loaded reference through the speakers — the mic naturally picks up speaker bleed, giving a real (mic, ref) pair for AEC testing without leaving the UI. - Reusable WaveformPlayer (canvas peaks + click-to-seek + audio controls) and useAudioPeaks hook (shared module-scoped AudioContext to avoid hitting browser context limits with three players on one page); migrated TTS, Sound, Traces audio blocks to use it. - Past runs saved in localStorage via useMediaHistory('audio-transform') — the history entry stores all three URLs so clicking re-renders the full triple, not just the output. Build + e2e: - 11 matrix entries removed from .github/workflows/backend.yml (CUDA, ROCm, SYCL, Metal, L4T): upstream supports only CPU + Vulkan, so we ship those two and let GPU-class hardware route through Vulkan in the gallery capabilities map. - tests-localvqe-grpc-transform job in test-extra.yml (gated on detect-changes.outputs.localvqe). - New audio_transform capability + 4 specs in tests/e2e-backends. - Playwright spec suite in core/http/react-ui/e2e/audio-transform.spec.js (8 specs covering tabs, file upload, multipart shape, history, errors). Docs: - New docs/content/features/audio-transform.md covering the (audio, reference) mental model, batch + WebSocket wire formats, LocalVQE param keys, and a YAML config example. Cross-links from text-to-audio and audio-to-text feature pages. Assisted-by: Claude:claude-opus-4-7 [Bash Read Edit Write Agent TaskCreate] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-04 22:07:11 +02:00
Ettore Di Giacinto	fe6eb57082	feat(vibevoice-cpp): add purego TTS+ASR backend (#9610) ... * feat(vibevoice-cpp): add purego TTS+ASR backend Wire up Microsoft VibeVoice via the vibevoice.cpp C ABI as a new purego-based Go backend that serves both Backend.TTS and Backend.AudioTranscription from a single gRPC binary. Mirrors the qwen3-tts-cpp / sherpa-onnx pattern so the variant matrix (cpu/cuda12/cuda13/metal/rocm/sycl-f16/f32/vulkan/l4t) and the e2e-backends gRPC harness reuse existing infrastructure. - backend/go/vibevoice-cpp/ - Makefile, CMakeLists, purego shim, gRPC Backend with model-dir auto-detection, closed-loop TTS->ASR smoke test - backend/index.yaml - &vibevoicecpp meta + 18 image entries - Makefile - .NOTPARALLEL, BACKEND_VIBEVOICE_CPP, docker-build wiring, test-extra-backend-vibevoice-cpp-{tts,transcription} e2e wrappers - .github/workflows/backend.yml - matrix entries for all variants - .github/workflows/test-extra.yml - per-backend smoke + 2 gRPC e2e jobs * feat(vibevoice-cpp): drop hardcoded glob detection, add gallery entries Refactor backend Load() to follow the standard Options[] convention used by sherpa-onnx and the rest of the multi-role backends: ModelFile is the primary gguf, supplementary paths come through opts.Options[] as key=value (or key:value for Make-target compat), resolved against opts.ModelPath. type=asr/tts decides the role of ModelFile when neither tts_model nor asr_model is set explicitly. Add gallery/index.yaml entries: - vibevoice-cpp - realtime 0.5B Q8_0 TTS + tokenizer + Carter voice - vibevoice-cpp-asr - long-form ASR Q8_0 + tokenizer Both pull from huggingface://mudler/vibevoice.cpp-models with sha256 verification. parameters.model + Options[] paths are siblings under {models_dir} per the qwen3-tts-cpp convention. Update Makefile e2e wrappers to pass BACKEND_TEST_OPTIONS comma+colon style, and tighten the per-backend Go closed-loop test to use the explicit Options API. * fix(vibevoice-cpp): force whole-archive link so vv_capi_* exports survive libvibevoice is a STATIC archive linked into the MODULE library. Without --whole-archive (or -force_load on Apple, /WHOLEARCHIVE on MSVC), the linker garbage-collects symbols not referenced from this translation unit - which means dlopen+RegisterLibFunc panics with 'undefined symbol: vv_capi_load' at backend startup, since purego looks them up by name and our cpp/govibevoicecpp.cpp doesn't call them directly. * test(vibevoice-cpp): rewrite suite with Ginkgo v2 Match the convention used by backend/go/sherpa-onnx/backend_test.go. The suite now covers backend semantics that don't need purego (Locking, empty-ModelFile rejection, TTS/ASR-without-loaded-model errors) on top of the gRPC lifecycle specs (Health, Load, closed-loop TTS->ASR). Model-dependent specs Skip() when VIBEVOICE_MODEL_DIR is unset, so `go test ./backend/go/vibevoice-cpp/` is green on a clean checkout and runs the heavyweight closed-loop spec when test.sh has staged the bundle. * fix(vibevoice-cpp): implement TTSStream + AudioTranscriptionStream The gRPC server's stream handlers (pkg/grpc/server.go) spawn a goroutine that ranges over a chan; the only thing closing that chan is the backend's own *Stream method. With the default Base stub returning 'unimplemented' and never touching the chan, the server goroutine hangs forever and the client hits DeadlineExceeded - which is exactly what the e2e harness saw in the test-extra-backend-vibevoice-cpp-tts matrix run. TTSStream synthesizes via vv_capi_tts to a tempfile, then emits a streaming WAV header (chunk sizes 0xFFFFFFFF so HTTP clients can start playback before the full PCM lands) followed by the PCM body in 64 KB slices. The header + >=2 PCM frames satisfy the harness's 'expected >=2 chunks' assertion and give a real progressive stream. AudioTranscriptionStream runs the offline transcription, emits each segment as a delta, and closes with a final_result whose Text equals the concatenated deltas (the harness asserts those match). Two new Ginkgo specs guard the close-channel-on-error path so the deadline-exceeded regression can't come back silently. * fix(vibevoice-cpp): silence errcheck on cleanup paths Lint flagged six unchecked Close()/Remove()/RemoveAll() calls along purely-cleanup deferred paths. Wrap each in '_ = ...' (or a closure for defers that take args) - matches what the rest of the LocalAI backend/go/* tree already does for these callsites. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vibevoice-cpp): closed-loop slot fill + modelRoot-relative path resolution Two bugs the test-extra-backend-vibevoice-cpp-* CI matrix surfaced: 1. Closed-loop Load with ModelFile=tts.gguf + Options[asr_model=...] left v.ttsModel empty, because the default-fill block only ran when BOTH slots were empty. vv_capi_load then got tts="" + a voice and the C side rejected it with rc=-3 'TTS model required to load a voice'. Fix: ModelFile fills the *primary* role-slot (decided by 'type=' in Options, defaulting to tts) independently of the secondary, so ModelFile + asr_model resolves to both. 2. resolvePath stat'd CWD before falling back to relTo. With LocalAI launched from a directory that happens to contain a same-named file, supplementary Options[] paths could leak away from the models dir. Drop the CWD probe entirely - relative paths now *always* join onto opts.ModelPath (the gallery convention). New Ginkgo coverage: * 'ModelFile slot resolution' (4 specs) - asr_model+ModelFile, type=asr, explicit tts_model override, key:value variant. * 'resolvePath (relative-to-modelRoot)' (5 specs) - join, abs passthrough, empty input, empty relTo, and the CWD-trap regression test. * 'Load resolves relative Options paths against opts.ModelPath' - end- to-end gallery layout round-trip. Verified locally: 19/19 specs pass (with model bundle, including the closed-loop TTS->ASR; without bundle, 17 pass + 2 model-dependent skip). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(vibevoice-cpp): use gallery convention in closed-loop spec The 'loads the realtime TTS model' / closed-loop specs were passing already-prefixed paths into Options[]: Options: ['tokenizer=' + filepath.Join(modelDir, 'tokenizer.gguf')] Combined with no ModelPath set on the request, the backend's modelRoot fell back to filepath.Dir(ModelFile) = modelDir, then resolvePath joined the prefixed Options path on top of it - producing 'vibevoice-models/vibevoice-models/tokenizer.gguf' when the CI's VIBEVOICE_MODEL_DIR is the relative './vibevoice-models'. The fix is to mirror the gallery contract LocalAI core actually sends in production: ModelPath is the models root (absolute), ModelFile is a name *under* it, every Options[] path is relative to ModelPath. Uses filepath.Base() to get bare filenames. Verified locally with both VIBEVOICE_MODEL_DIR=/tmp/vv-bundle (abs) and VIBEVOICE_MODEL_DIR=vibevoice-models (the relative shape that broke CI). Both: 19/19 specs pass, ~55-60s. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(vibevoice-cpp): switch ASR to Q4_K + bump transcription timeout The Q8_0 ASR gguf is ~14 GB - too big to fit alongside the runner image, the docker build cache, and the test artifacts on a free ubuntu-latest GHA runner; 'test-extra-backend-vibevoice-cpp-transcription' was getting SIGTERM'd at 90 min before the model could finish loading. Switch to Q4_K (~10 GB on disk, slightly faster CPU decode) for: * the e2e harness Make target * the gallery 'vibevoice-cpp-asr' entry (parameters + files block) * the per-backend test.sh auto-download list Bump tests-vibevoice-cpp-grpc-transcription's timeout-minutes from 90 to 150 - even with Q4_K, the 30 s JFK clip on a CPU runner needs runway above the previous 90 min cap. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(vibevoice-cpp): drop transcription gRPC e2e job - too heavy for free runners The vibevoice ASR is a 7B-parameter model. Even on Q4_K (~10 GB on disk) a single 30 s transcription saturates the per-test 30 min timeout in the e2e-backends harness on a 4-core ubuntu-latest, and the 10 GB download + Docker layer + working space leaves no headroom on the runner's free disk. Two attempts in CI got SIGTERM'd at the LoadModel boundary - the bottleneck isn't tunable from the workflow side without a paid-tier runner. The per-backend tests-vibevoice-cpp job already runs the same AudioTranscription path via a closed-loop TTS->ASR Ginkgo spec - same gRPC contract, same model, single process - so the standalone tests-vibevoice-cpp-grpc-transcription job was redundant on top of the disk/CPU pressure. The Makefile target test-extra-backend-vibevoice-cpp-transcription stays for local invocation on workstations that can afford it - useful when developing the streaming codepaths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(vibevoice-cpp): restore transcription gRPC e2e on bigger-runner Switch tests-vibevoice-cpp-grpc-transcription from ubuntu-latest to the self-hosted 'bigger-runner' label that GPU image builds in backend.yml use, plus the documented Free-disk-space prep step (purge dotnet / ghc / android / CodeQL caches) the disabled vllm/sglang entries in this file describe. That gives the 7B-param Q4_K ASR model the disk + CPU runway it needs. Keep timeout-minutes: 150 - even on a beefier runner the 30 s JFK decode plus 10 GB download has to fit comfortably. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(vibevoice-cpp): apt-get install make on bigger-runner before transcription e2e bigger-runner is a self-hosted bare runner without the standard ubuntu image's preinstalled build tools, so the previous job died at the very first command with 'make: command not found' (exit 127). Add the Dependencies step that the disabled vllm/sglang entries in this file already document - apt-get installs make + build-essential + curl + unzip + ca-certificates + git + tar before the make target runs. Mirrors how every other 'runs-on: bigger-runner' entry in backend.yml prepares the runner. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-29 22:22:14 +02:00
Ettore Di Giacinto	a0317d9926	refactor(tests): split app_test.go, move real-backend coverage to e2e-backends ... core/http/app_test.go had grown to 1495 lines exercising three concerns at once: HTTP-layer integration, real-backend inference (llama-gguf, tts, stablediffusion, transformers embeddings, whisper), and service logic that already has unit-level coverage. Each PR paid for 6 backend builds plus real-model downloads to satisfy a single suite. Reorg per layer: - app_test.go (1495 -> 1003 lines) drives the mock-backend binary only. Kept: auth, routing, gallery API, file:// import, /system, agent-jobs HTTP plumbing, config-file model loading. Deleted real-inference specs (llama-gguf chat, ggml completions/streaming, logprobs, logit_bias, transcription, embeddings, External-gRPC, Stores duplicate, Model gallery Context). Lifted Agent Jobs out of the deleted Stores Context. - tests/e2e-backends/backend_test.go gains logprobs, logit_bias, and no-first-token-dup specs (the latter folded into PredictStream). Two new caps gate them so non-LLM backends opt out. - tests/e2e-aio/e2e_test.go gains a streaming smoke under Context("text") to catch container-level streaming regressions. - tests/models_fixtures/ removed; all fixtures referenced testmodel.ggml. app_test.go now writes per-Context inline mock-model YAMLs. CI: - test.yml + tests-e2e.yml gain paths-ignore (docs/, examples/, *.md, backend/) so docs and backend-only PRs skip them. test.yml drops the 6-backend Build step plus TRANSFORMER_BACKEND/GO_TAGS=tts; tests-apple drops the llama-cpp-darwin build. - New tests-aio.yml runs the AIO container nightly + on workflow_dispatch + master/tags. The tests-e2e-container job moved out of test.yml so PRs no longer pay AIO cost. - New tests-llama-cpp-smoke job in test-extra.yml runs on every PR with no detect-changes gate; pulls quay.io/go-skynet/local-ai-backends: master-cpu-llama-cpp (no build on PR) and exercises predict/stream/ logprobs/logit_bias against Qwen3-0.6B. This is the PR-acceptance real-backend gate after AIO moved to nightly. The path-gated heavy test-extra-backend-llama-cpp wrapper appends the same caps so it exercises the moved specs when the backend actually changes. Makefile: - Deleted test-models/testmodel.ggml (the wget chain), test-llama-gguf, test-tts, test-stablediffusion, test-realtime-models. test target drops --label-filter, HUGGINGFACE_GRPC, TRANSFORMER_BACKEND, TEST_DIR, FIXTURES, CONFIG_FILE, MODELS_PATH, BACKENDS_PATH; depends on build-mock-backend. test-stores keeps a focused entry point and depends on backends/local-store. clean-tests also clears the mock-backend binary. Net per typical Go-side PR: ~25min (6 backend builds + tests + AIO) + ~8min e2e drops to ~5min mock-backend test + ~8min e2e + ~5-10min llama-cpp-smoke (image pulled). Docs and backend-only PRs skip the always-on workflows entirely. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: claude-code:claude-opus-4-7 [Edit] [Write] [Bash]	2026-04-27 23:09:20 +00:00
Richard Palethorpe	13734ae9fa	feat: Add Sherpa ONNX backend for ASR and TTS (#8523) ... feat(backend): Add Sherpa ONNX backend and Omnilingual ASR Adds a new Go backend wrapping sherpa-onnx via purego (no cgo). Same approach as opus/stablediffusion-ggml/whisper — a thin C shim (csrc/shim.c + shim.h → libsherpa-shim.so) wraps the bits purego can't reach directly: nested struct config writes, result-struct field reads, and the streaming TTS callback trampoline. The Go side uses opaque uintptr handles and purego.NewCallback for the TTS callback. Supports: - VAD via sherpa-onnx's Silero VAD - Offline ASR: Whisper, Paraformer, SenseVoice, Omnilingual CTC - Online/streaming ASR: zipformer transducer with endpoint detection (AudioTranscriptionStream emits delta events during decode) - Offline TTS: VITS (LJS, etc.) - Streaming TTS: sherpa-onnx's callback API → PCM chunks on a channel, prefixed by a streaming WAV header Gallery entries: omnilingual-0.3b-ctc-q8-sherpa (1600-language offline ASR), streaming-zipformer-en-sherpa (low-latency streaming ASR), silero-vad-sherpa, vits-ljs-sherpa. E2E coverage: tests/e2e-backends for offline + streaming ASR, tests/e2e for the full realtime pipeline (VAD + STT + TTS). Assisted-by: claude-opus-4-7-1M [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-04-24 14:40:06 +02:00
Ettore Di Giacinto	181ebb6df4	feat: voice recognition (#9500) ... * feat(voice-recognition): add /v1/voice/{verify,analyze,embed} + speaker-recognition backend Audio analog to face recognition. Adds three gRPC RPCs (VoiceVerify / VoiceAnalyze / VoiceEmbed), their Go service and HTTP layers, a new FLAG_SPEAKER_RECOGNITION capability flag, and a Python backend scaffold under backend/python/speaker-recognition/ wrapping SpeechBrain ECAPA-TDNN with a parallel OnnxDirectEngine for WeSpeaker / 3D-Speaker ONNX exports. The kokoros Rust backend gets matching unimplemented trait stubs — tonic's async_trait has no defaults, so adding an RPC without Rust stubs breaks the build (same regression fixed by eb01c772 for face). Swagger, /api/instructions, and the auth RouteFeatureRegistry / APIFeatures list are updated so the endpoints surface everywhere a client or admin UI looks. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): add 1:N identify + register/forget endpoints Mirrors the face-recognition register/identify/forget surface. New package core/services/voicerecognition/ carries a Registry interface and a local-store-backed implementation (same in-memory vector-store plumbing facerecognition uses, separate instance so the embedding spaces stay isolated). Handlers under /v1/voice/{register,identify,forget} reuse backend.VoiceEmbed to compute the probe vector, then delegate the nearest-neighbour search to the registry. Default cosine-distance threshold is tuned for ECAPA-TDNN on VoxCeleb (0.25, EER ~1.9%). As with the face registry, the current backing is in-memory only — a pgvector implementation is a future constructor-level swap. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): gallery, docs, CI and e2e coverage - backend/index.yaml: speaker-recognition backend entry + CPU and CUDA-12 image variants (plus matching development variants). - gallery/index.yaml: speechbrain-ecapa-tdnn (default) and wespeaker-resnet34 model entries. The WeSpeaker SHA-256 is a deliberate placeholder — the HF URI must be curl'd and its hash filled in before the entry installs. - docs/content/features/voice-recognition.md: API reference + quickstart, mirrors the face-recognition docs. - React UI: CAP_SPEAKER_RECOGNITION flag export (consumers follow face's precedent — no dedicated tab yet). - tests/e2e-backends: voice_embed / voice_verify / voice_analyze specs. Helper resolveFaceFixture is reused as-is — the only thing face/voice share is "download a file into workDir", so no need for a new helper. - Makefile: docker-build-speaker-recognition + test-extra-backend- speaker-recognition-{ecapa,all} targets. Audio fixtures default to VCTK p225/p226 samples from HuggingFace. - CI: test-extra.yml grows a tests-speaker-recognition-grpc job mirroring insightface. backend.yml matrix gains CPU + CUDA-12 image build entries — scripts/changed-backends.js auto-picks these up. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): wire a working /v1/voice/analyze head Adds AnalysisHead: a lazy-loading age / gender / emotion inference wrapper that plugs into both SpeechBrainEngine and OnnxDirectEngine. Defaults to two open-licence HuggingFace checkpoints: - audeering/wav2vec2-large-robust-24-ft-age-gender (Apache 2.0) — age regression + 3-way gender (female / male / child). - superb/wav2vec2-base-superb-er (Apache 2.0) — 4-way emotion. Both are optional and degrade gracefully when transformers or the model can't be loaded — the engine raises NotImplementedError so the gRPC layer returns 501 instead of a generic 500. Emotion classes pass through from the model (neutral/happy/angry/sad on the default checkpoint); the e2e test now accepts any non-empty dominant gender so custom age_gender_model overrides don't fail it. Adds transformers to the backend's CPU and CUDA-12 requirements. Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): pin real WeSpeaker ResNet34 ONNX SHA-256 Replaces the placeholder hash in gallery/index.yaml with the actual SHA-256 (7bb2f06e…) of the upstream Wespeaker/wespeaker-voxceleb-resnet34-LM ONNX at ~25MB. `local-ai models install wespeaker-resnet34` now succeeds. Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): soundfile loader + honest analyze default Two issues surfaced on first end-to-end smoke with the actual backend image: 1. torchaudio.load in torchaudio 2.8+ requires the torchcodec package for audio decoding. Switch SpeechBrainEngine._load_waveform to the already-present soundfile (listed in requirements.txt) plus a numpy linear resample to 16kHz. Drops a heavy ffmpeg-linked dep and the codepath we never exercise (torchaudio's ffmpeg backend). 2. The AnalysisHead was defaulting to audeering/wav2vec2-large-robust- 24-ft-age-gender, but AutoModelForAudioClassification silently mangles that checkpoint — it reports the age head weights as UNEXPECTED and re-initialises the classifier head with random values, so the "gender" output is noise and there is no age output at all. Make age/gender opt-in instead (empty default; users wire a cleanly-loadable Wav2Vec2ForSequenceClassification checkpoint via age_gender_model: option). Emotion keeps its working Superb default. Also broaden _infer_age_gender's tensor-shape handling and catch runtime exceptions so a dodgy age/gender head never takes down the whole analyze call. Docs and README updated to match the new policy. Verified with the branch-scoped gallery on localhost: - voice/embed → 192-d ECAPA-TDNN vector - voice/verify → same-clip dist≈6e-08 verified=true; cross-speaker dist 0.76–0.99 verified=false (as expected) - voice/register/identify/forget → round-trip works, 404 on unknown id - voice/analyze → emotion populated, age/gender omitted (opt-in) Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): real CI audio fixtures + fixture-agnostic verify spec Two issues surfaced after CI actually ran the speaker-recognition e2e target (I'd curl-tested against a running server but hadn't run the make target locally): 1. The default BACKEND_TEST_VOICE_AUDIO_* URLs pointed at huggingface.co/datasets/CSTR-Edinburgh/vctk paths that return 404 (the dataset is gated). Swap them for the speechbrain test samples served from github.com/speechbrain/speechbrain/raw/develop/ — public, no auth, correct 16kHz mono format. 2. The VoiceVerify spec required d(file1,file2) < 0.4, assuming file1/file2 were same-speaker. The speechbrain samples are three different speakers (example1/2/5), and there is no easy un-gated source of true same-speaker audio pairs (VoxCeleb/VCTK/LibriSpeech are all license- or size-gated for CI use). Replace the ceiling check with a relative-ordering assertion: d(pair) > d(same-clip) for both file2 and file3 — that's enough to prove the embeddings encode speaker info, and it works with any three non-identical clips. Actual speaker ordering d(1,2) vs d(1,3) is logged but not asserted. Local run: 4/4 voice specs pass (Health, LoadModel, VoiceEmbed, VoiceVerify) on the built backend image. 12 non-voice specs skipped as expected. Assisted-by: Claude:claude-opus-4-7 * fix(ci): checkout with submodules in the reusable backend_build workflow The kokoros Rust backend build fails with failed to read .../sources/Kokoros/kokoros/Cargo.toml: No such file because the reusable backend_build.yml workflow's actions/checkout step was missing `submodules: true`. Dockerfile.rust does `COPY . /LocalAI`, and without the submodule files the subsequent `cargo build` can't find the vendored Kokoros crate. The bug pre-dates this PR — scripts/changed-backends.js only triggers the kokoros image job when something under backend/rust/kokoros or the shared proto changes, so master had been coasting past it. The voice-recognition proto addition re-broke it. Other checkouts in backend.yml (llama-cpp-darwin) and test-extra.yml (insightface, kokoros, speaker-recognition) already pass `submodules: true`; this brings the shared backend image builder in line. Assisted-by: Claude:claude-opus-4-7	2026-04-23 12:07:14 +02:00
Ettore Di Giacinto	20baec77ab	feat(face-recognition): add insightface/onnx backend for 1:1 verify, 1:N identify, embedding, detection, analysis (#9480) ... * feat(face-recognition): add insightface backend for 1:1 verify, 1:N identify, embedding, detection, analysis Adds face recognition as a new first-class capability in LocalAI via the `insightface` Python backend, with a pluggable two-engine design so non-commercial (insightface model packs) and commercial-safe (OpenCV Zoo YuNet + SFace) models share the same gRPC/HTTP surface. New gRPC RPCs (backend/backend.proto): * FaceVerify(FaceVerifyRequest) returns FaceVerifyResponse * FaceAnalyze(FaceAnalyzeRequest) returns FaceAnalyzeResponse Existing Embedding and Detect RPCs are reused (face image in PredictOptions.Images / DetectOptions.src) for face embedding and face detection respectively. New HTTP endpoints under /v1/face/: * verify — 1:1 image pair same-person decision * analyze — per-face age + gender (emotion/race reserved) * register — 1:N enrollment; stores embedding in vector store * identify — 1:N recognition; detect → embed → StoresFind * forget — remove a registered face by opaque ID Service layer (core/services/facerecognition/) introduces a `Registry` interface with one in-memory `storeRegistry` impl backed by LocalAI's existing local-store gRPC vector backend. HTTP handlers depend on the interface, not on StoresSet/StoresFind directly, so a persistent PostgreSQL/pgvector implementation can be slotted in via a single constructor change in core/application (TODO marker in the package doc). New usecase flag FLAG_FACE_RECOGNITION; insightface is also wired into FLAG_DETECTION so /v1/detection works for face bounding boxes. Gallery (backend/index.yaml) ships three entries: * insightface-buffalo-l — SCRFD-10GF + ArcFace R50 + genderage (~326MB pre-baked; non-commercial research use only) * insightface-opencv — YuNet + SFace (~40MB pre-baked; Apache 2.0) * insightface-buffalo-s — SCRFD-500MF + MBF (runtime download; non-commercial) Python backend (backend/python/insightface/): * engines.py — FaceEngine protocol with InsightFaceEngine and OnnxDirectEngine; resolves model paths relative to the backend directory so the same gallery config works in docker-scratch and in the e2e-backends rootfs-extraction harness. * backend.py — gRPC servicer implementing Health, LoadModel, Status, Embedding, Detect, FaceVerify, FaceAnalyze. * install.sh — pre-bakes buffalo_l + OpenCV YuNet/SFace inside the backend directory so first-run is offline-clean (the final scratch image only preserves files under /<backend>/). * test.py — parametrized unit tests over both engines. Tests: * Registry unit tests (go test -race ./core/services/facerecognition/...) — in-memory fake grpc.Backend, table-driven, covers register/ identify/forget/error paths + concurrent access. * tests/e2e-backends/backend_test.go extended with face caps (face_detect, face_embed, face_verify, face_analyze); relative ordering + configurable verifyCeiling per engine. * Makefile targets: test-extra-backend-insightface-buffalo-l, -opencv, and the -all aggregate. * CI: .github/workflows/test-extra.yml gains tests-insightface-grpc, auto-triggered by changes under backend/python/insightface/. Docs: * docs/content/features/face-recognition.md — feature page with license table, quickstart (defaults to the commercial-safe model), models matrix, API reference, 1:N workflow, storage caveats. * Cross-refs in object-detection.md, stores.md, embeddings.md, and whats-new.md. * Contributor README at backend/python/insightface/README.md. Verified end-to-end: * buffalo_l: 6/6 specs (health, load, face_detect, face_embed, face_verify, face_analyze). * opencv: 5/5 specs (same minus face_analyze — SFace has no demographic head; correctly skipped via BACKEND_TEST_CAPS). Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): move engine selection to model gallery, collapse backend entries The previous commit put engine/model_pack options on backend gallery entries (`backend/index.yaml`). That was wrong — `GalleryBackend` (core/gallery/backend_types.go:32) has no `options` field, so the YAML decoder silently dropped those keys and all three "different insightface-*" backend entries resolved to the same container image with no distinguishing configuration. Correct split: * `backend/index.yaml` now has ONE `insightface` backend entry shipping the CPU + CUDA 12 container images. The Python backend bundles both the non-commercial insightface model packs (buffalo_l / buffalo_s) and the commercial-safe OpenCV Zoo weights (YuNet + SFace); the active engine is selected at LoadModel time via `options: ["engine:..."]`. * `gallery/index.yaml` gains three model entries — `insightface-buffalo-l`, `insightface-opencv`, `insightface-buffalo-s` — each setting the appropriate `overrides.backend` + `overrides.options` so installing one actually gives the user the intended engine. This matches how `rfdetr-base` lives in the model gallery against the `rfdetr` backend. The earlier e2e tests passed despite this bug because the Makefile targets pass `BACKEND_TEST_OPTIONS` directly to LoadModel via gRPC, bypassing any gallery resolution entirely. No code changes needed. Assisted-by: Claude:claude-opus-4-7 * feat(face-recognition): cover all supported models in the gallery + drop weight baking Follows up on the model-gallery split: adds entries for every model configuration either engine actually supports, and switches weight delivery from image-baked to LocalAI's standard gallery mechanism. Gallery now has seven `insightface-*` model entries (gallery/index.yaml): insightface (family) — non-commercial research use • buffalo-l (326MB) — SCRFD-10GF + ResNet50 + genderage, default • buffalo-m (313MB) — SCRFD-2.5GF + ResNet50 + genderage • buffalo-s (159MB) — SCRFD-500MF + MBF + genderage • buffalo-sc (16MB) — SCRFD-500MF + MBF, recognition only (no landmarks, no demographics — analyze returns empty attributes) • antelopev2 (407MB) — SCRFD-10GF + ResNet100@Glint360K + genderage OpenCV Zoo family — Apache 2.0 commercial-safe • opencv — YuNet + SFace fp32 (~40MB) • opencv-int8 — YuNet + SFace int8 (~12MB, ~3x smaller, faster on CPU) Model weights are no longer baked into the backend image. The image now ships only the Python runtime + libraries (~275MB content size, ~1.18GB disk vs ~1.21GB when weights were baked). Weights flow through LocalAI's gallery mechanism: * OpenCV variants list `files:` with ONNX URIs + SHA-256, so `local-ai models install insightface-opencv` pulls them into the models directory exactly like any other gallery-managed model. * insightface packs (upstream distributes .zip archives only, not individual ONNX files) auto-download on first LoadModel via FaceAnalysis' built-in machinery, rooted at the LocalAI models directory so they live alongside everything else — same pattern `rfdetr` uses with `inference.get_model()`. Backend changes (backend/python/insightface/): * backend.py — LoadModel propagates `ModelOptions.ModelPath` (the LocalAI models directory) to engines via a `_model_dir` hint. This replaces the earlier ModelFile-dirname approach; ModelPath is the canonical "models directory" variable set by the Go loader (pkg/model/initializers.go:144) and is always populated. * engines.py::_resolve_model_path — picks up `model_dir` and searches it (plus basename-in-model-dir) before falling back to the dev script-dir. This is how OnnxDirectEngine finds gallery-downloaded YuNet/SFace files by filename only. * engines.py::_flatten_insightface_pack — new helper that works around an upstream packaging inconsistency: buffalo_l/s/sc zips expand flat, but buffalo_m and antelopev2 zips wrap their ONNX files in a redundant `<name>/` directory. insightface's own loader looks one level too shallow and fails. We call `ensure_available()` explicitly, flatten if nested, then hand to FaceAnalysis. * engines.py::InsightFaceEngine.prepare — root-resolution order now includes the `_model_dir` hint so packs download into the LocalAI models directory by default. * install.sh — no longer pre-downloads any weights. Everything is gallery-managed now. * smoke.py (new) — parametrized smoke test that iterates over every gallery configuration, simulating the LocalAI install flow (creates a models dir, fetches OpenCV files with checksum verification, lets insightface auto-download its packs), then runs detect + embed + verify (+ analyze where supported) through the in-process BackendServicer. * test.py — OnnxDirectEngineTest no longer hardcodes `/models/opencv/` paths; downloads ONNX files to a temp dir at setUpClass time and passes ModelPath accordingly. Registry change (core/services/facerecognition/store_registry.go): * `dim=0` in NewStoreRegistry now means "accept whatever dimension arrives" — needed because the backend supports 512-d ArcFace/MBF and 128-d SFace via the same Registry. A non-zero dim still fails fast with ErrDimensionMismatch. * core/application plumbs `faceEmbeddingDim = 0`, explaining the rationale in the comment. Backend gallery description updated to reflect that the image carries no weights — it's just Python + engines. Smoke-tested all 7 configurations against the rebuilt image (with the flatten fix applied), exit 0: PASS: insightface-buffalo-l faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-sc faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-s faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-m faces=6 dim=512 same-dist=0.000 PASS: insightface-antelopev2 faces=6 dim=512 same-dist=0.000 PASS: insightface-opencv faces=6 dim=128 same-dist=0.000 PASS: insightface-opencv-int8 faces=6 dim=128 same-dist=0.000 7/7 passed Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): pre-fetch OpenCV ONNX for e2e target; drop stale pre-baked claim CI regression from the previous commit: I moved OpenCV Zoo weight delivery to LocalAI's gallery `files:` mechanism, but the test-extra-backend-insightface-opencv target was still passing relative paths `detector_onnx:models/opencv/yunet.onnx` in BACKEND_TEST_OPTIONS. The e2e suite drives LoadModel directly over gRPC without going through the gallery, so those relative paths resolved to nothing and OpenCV's ONNXImporter failed: LoadModel failed: Failed to load face engine: OpenCV(4.13.0) ... Can't read ONNX file: models/opencv/yunet.onnx Fix: add an `insightface-opencv-models` prerequisite target that fetches the two ONNX files (YuNet + SFace) to a deterministic host cache at /tmp/localai-insightface-opencv-cache/, verifies SHA-256, and skips the download on re-runs. The opencv test target depends on it and passes absolute paths in BACKEND_TEST_OPTIONS, so the backend finds the files via its normal absolute-path resolution branch. Also refresh the buffalo_l comment: it no longer says "pre-baked" (nothing is — the pack auto-downloads from upstream's GitHub release on first LoadModel, same as in CI). Locally verified: `make test-extra-backend-insightface-opencv` passes 5/5 specs (health, load, face_detect, face_embed, face_verify). Assisted-by: Claude:claude-opus-4-7 * feat(face-recognition): add POST /v1/face/embed + correct /v1/embeddings docs The docs promised that /v1/embeddings returns face vectors when you send an image data-URI. That was never true: /v1/embeddings is OpenAI-compatible and text-only by contract — its handler goes through `core/backend/embeddings.go::ModelEmbedding`, which sets `predictOptions.Embeddings = s` (a string of TEXT to embed) and never populates `predictOptions.Images[]`. The Python backend's Embedding gRPC method does handle Images[] (that's how /v1/face/register reaches it internally via `backend.FaceEmbed`), but the HTTP embeddings endpoint wasn't wired to populate it. Rather than overload /v1/embeddings with image-vs-text detection — messy, and the endpoint is OpenAI-compatible by design — add a dedicated /v1/face/embed endpoint that wraps `backend.FaceEmbed` (already used internally by /v1/face/register and /v1/face/identify). Matches LocalAI's convention of a dedicated path per non-standard flow (/v1/rerank, /v1/detection, /v1/face/verify etc.). Response: { "embedding": [<dim> floats, L2-normed], "dim": int, // 512 for ArcFace R50 / MBF, 128 for SFace "model": "<name>" } Live-tested on the opencv engine: returns a 128-d L2-normalized vector (sum(x^2) = 1.0000). Sentinel in docs updated to note /v1/embeddings is text-only and point image users at /v1/face/embed instead. Assisted-by: Claude:claude-opus-4-7 * fix(http): map malformed image input + gRPC status codes to proper 4xx Image-input failures on LocalAI's single-image endpoints (/v1/detection, /v1/face/{verify,analyze,embed,register,identify}) have historically returned 500 — even when the client was the one who sent garbage. Classic example: you POST an "image" that isn't a URL, isn't a data-URI, and isn't a valid JPEG/PNG — the server shouldn't claim that's its fault. Two helpers land in core/http/endpoints/localai/images.go and every single-image handler is switched over: * decodeImageInput(s) Wraps utils.GetContentURIAsBase64 and turns any failure (invalid URL, not a data-URI, download error, etc.) into echo.NewHTTPError(400, "invalid image input: ..."). * mapBackendError(err) Inspects the gRPC status on a backend call error and maps: INVALID_ARGUMENT → 400 Bad Request NOT_FOUND → 404 Not Found FAILED_PRECONDITION → 412 Precondition Failed Unimplemented → 501 Not Implemented All other codes fall through unchanged (still 500). Before, my 1×1 PNG error-path test returned: HTTP 500 "rpc error: code = InvalidArgument desc = failed to decode one or both images" After: HTTP 400 "failed to decode one or both images" Scope-limited to the LocalAI single-image endpoints. The multi-modal paths (middleware/request.go, openresponses/responses.go, openai/realtime.go) intentionally log-and-skip individual media parts when decoding fails — different design intent (graceful degradation of a multi-part message), not a 400-worthy failure. Left untouched. Live-verified: every error case in /tmp/face_errors.py now returns 4xx with a meaningful message; the "image with no face (1x1 PNG)" case specifically went from 500 → 400. Assisted-by: Claude:claude-opus-4-7 * refactor(face-recognition): insightface packs go through gallery files:, drop FaceAnalysis Follows up on the discovery that LocalAI's gallery `files:` mechanism handles archives (zip, tar.gz, …) via mholt/archiver/v3 — the rhasspy piper voices use exactly this pattern. Insightface packs are zip archives, so we can now deliver them the same way every other gallery-managed model gets delivered: declaratively, checksum-verified, through LocalAI's standard download+extract pipeline. Two changes: 1. Gallery (gallery/index.yaml) — every insightface-* entry gains a `files:` list with the pack zip's URI + SHA-256. `local-ai models install insightface-buffalo-l` now fetches the zip, verifies the hash, and extracts it into the models directory. No more reliance on insightface's library-internal `ensure_available()` auto-download or its hardcoded `BASE_REPO_URL`. 2. InsightFaceEngine (backend/python/insightface/engines.py) — drops the FaceAnalysis wrapper and drives insightface's `model_zoo` directly. The ~50 lines FaceAnalysis provides — glob ONNX files, route each through `model_zoo.get_model()`, build a `{taskname: model}` dict, loop per-face at inference — are reimplemented in `InsightFaceEngine`. The actual inference classes (RetinaFace, ArcFaceONNX, Attribute, Landmark) are still insightface's — we only replicate the glue, so drift risk against upstream is minimal. Why drop FaceAnalysis: it hard-codes a `<root>/models/<name>/*.onnx` layout that doesn't match what LocalAI's zip extraction produces. LocalAI unpacks archives flat into `<models_dir>`. Upstream packs are inconsistent — buffalo_l/s/sc ship ONNX at the zip root (lands at `<models_dir>/*.onnx`), buffalo_m/antelopev2 wrap in a redundant `<name>/` dir (lands at `<models_dir>/<name>/*.onnx`). The new `_locate_insightface_pack` helper searches both locations plus legacy paths and returns whichever has ONNX files. Replaces the earlier `_flatten_insightface_pack` helper (which tried to fight FaceAnalysis's layout expectations; now we just find the files wherever they are). Net effect for users: install once via LocalAI's managed flow, weights live alongside every other model, progress shows in the jobs endpoint, no first-load network call. Same API surface, cleaner plumbing. Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): CI's insightface e2e path needs the pack pre-fetched The e2e suite drives LoadModel over gRPC without going through LocalAI's gallery flow, so the engine's `_model_dir` option (normally populated from ModelPath) is empty. Previously the insightface target relied on FaceAnalysis auto-download to paper over this, but we dropped FaceAnalysis in favor of direct model_zoo calls — so the buffalo_l target started failing at LoadModel with "no insightface pack found". Mirror the opencv target's pre-fetch pattern: download buffalo_sc.zip (same SHA as the gallery entry), extract it on the host, and pass `root:<dir>` so the engine locates the pack without needing ModelPath. Switched to buffalo_sc (smallest pack, ~16MB) to keep CI fast; it covers the same insightface engine code path as buffalo_l. Face analyze cap dropped since buffalo_sc has no age/gender head. Assisted-by: Claude:claude-opus-4-7[1m] * feat(face-recognition): surface face-recognition in advertised feature maps The six /v1/face/* endpoints were missing from every place LocalAI advertises its feature surface to clients: * api_instructions — the machine-readable capability index at GET /api/instructions. Added `face-recognition` as a dedicated instruction area with an intro that calls out the in-memory registry caveat and the /v1/face/embed vs /v1/embeddings split. * auth/permissions — added FeatureFaceRecognition constant, routed all six face endpoints through it so admins can gate them per-user like any other API feature. Default ON (matches the other API features). * React UI capabilities — CAP_FACE_RECOGNITION symbol mapped to FLAG_FACE_RECOGNITION. Declared only for now; the Face page is a follow-up (noted in the plan). Instruction count bumped 9 → 10; test updated. Assisted-by: Claude:claude-opus-4-7[1m] * docs(agents): capture advertising-surface steps in the endpoint guide Before this change, adding a new /v1/* endpoint reliably missed one or more of: the swagger @Tags annotation, the /api/instructions registry, the auth RouteFeatureRegistry, and the React UI CAP_* symbol. The endpoint would work but be invisible to API consumers, admins, and the UI — and nothing in the existing docs said to look in those places. Extend .agents/api-endpoints-and-auth.md with a new "Advertising surfaces" section covering all four surfaces (swagger tags, /api/ instructions, capabilities.js, docs/), and expand the closing checklist so it's impossible to ship a feature without visiting each one. Hoist a one-liner reminder into AGENTS.md's Quick Reference so agents skim it before diving in. Assisted-by: Claude:claude-opus-4-7[1m]	2026-04-22 21:55:41 +02:00
Ettore Di Giacinto	b4e30692a2	feat(backends): add sglang (#9359) ... * feat(backends): add sglang Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(sglang): force AVX-512 CXXFLAGS and disable CI e2e job sgl-kernel's shm.cpp uses __m512 AVX-512 intrinsics unconditionally; -march=native fails on CI runners without AVX-512 in /proc/cpuinfo. Force -march=sapphirerapids so the build always succeeds, matching sglang upstream's docker/xeon.Dockerfile recipe. The resulting binary still requires an AVX-512 capable CPU at runtime, so disable tests-sglang-grpc in test-extra.yml for the same reason tests-vllm-grpc is disabled. Local runs with make test-extra-backend-sglang still work on hosts with the right SIMD baseline. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(sglang): patch CMakeLists.txt instead of CXXFLAGS for AVX-512 CXXFLAGS with -march=sapphirerapids was being overridden by add_compile_options(-march=native) in sglang's CPU CMakeLists.txt, since CMake appends those flags after CXXFLAGS. Sed-patch the CMakeLists.txt directly after cloning to replace -march=native. --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-16 22:40:56 +02:00
Ettore Di Giacinto	95efb8a562	feat(backend): add turboquant llama.cpp-fork backend (#9355) ... * feat(backend): add turboquant llama.cpp-fork backend turboquant is a llama.cpp fork (TheTom/llama-cpp-turboquant, branch feature/turboquant-kv-cache) that adds a TurboQuant KV-cache scheme. It ships as a first-class backend reusing backend/cpp/llama-cpp sources via a thin wrapper Makefile: each variant target copies ../llama-cpp into a sibling build dir and invokes llama-cpp's build-llama-cpp-grpc-server with LLAMA_REPO/LLAMA_VERSION overridden to point at the fork. No duplication of grpc-server.cpp — upstream fixes flow through automatically. Wires up the full matrix (CPU, CUDA 12/13, L4T, L4T-CUDA13, ROCm, SYCL f32/f16, Vulkan) in backend.yml and the gallery entries in index.yaml, adds a tests-turboquant-grpc e2e job driven by BACKEND_TEST_CACHE_TYPE_K/V=q8_0 to exercise the KV-cache config path (backend_test.go gains dedicated env vars wired into ModelOptions.CacheTypeKey/Value — a generic improvement usable by any llama.cpp-family backend), and registers a nightly auto-bump PR in bump_deps.yaml tracking feature/turboquant-kv-cache. scripts/changed-backends.js gets a special-case so edits to backend/cpp/llama-cpp/ also retrigger the turboquant CI pipeline, since the wrapper reuses those sources. * feat(turboquant): carry upstream patches against fork API drift turboquant branched from llama.cpp before upstream commit 66060008 ("server: respect the ignore eos flag", #21203) which added the `logit_bias_eog` field to `server_context_meta` and a matching parameter to `server_task::params_from_json_cmpl`. The shared backend/cpp/llama-cpp/grpc-server.cpp depends on that field, so building it against the fork unmodified fails. Cherry-pick that commit as a patch file under backend/cpp/turboquant/patches/ and apply it to the cloned fork sources via a new apply-patches.sh hook called from the wrapper Makefile. Simplifies the build flow too: instead of hopping through llama-cpp's build-llama-cpp-grpc-server indirection, the wrapper now drives the copied Makefile directly (clone -> patch -> build). Drop the corresponding patch whenever the fork catches up with upstream — the build fails fast if a patch stops applying, which is the signal to retire it. * docs: add turboquant backend section + clarify cache_type_k/v Document the new turboquant (llama.cpp fork with TurboQuant KV-cache) backend alongside the existing llama-cpp / ik-llama-cpp sections in features/text-generation.md: when to pick it, how to install it from the gallery, and a YAML example showing backend: turboquant together with cache_type_k / cache_type_v. Also expand the cache_type_k / cache_type_v table rows in advanced/model-configuration.md to spell out the accepted llama.cpp quantization values and note that these fields apply to all llama.cpp-family backends, not just vLLM. * feat(turboquant): patch ggml-rpc GGML_OP_COUNT assertion The fork adds new GGML ops bringing GGML_OP_COUNT to 97, but ggml/include/ggml-rpc.h static-asserts it equals 96, breaking the GGML_RPC=ON build paths (turboquant-grpc / turboquant-rpc-server). Carry a one-line patch that updates the expected count so the assertion holds. Drop this patch whenever the fork fixes it upstream. * feat(turboquant): allow turbo* KV-cache types and exercise them in e2e The shared backend/cpp/llama-cpp/grpc-server.cpp carries its own allow-list of accepted KV-cache types (kv_cache_types[]) and rejects anything outside it before the value reaches llama.cpp's parser. That list only contains the standard llama.cpp types — turbo2/turbo3/turbo4 would throw "Unsupported cache type" at LoadModel time, meaning nothing the LocalAI gRPC layer accepted was actually fork-specific. Add a build-time augmentation step (patch-grpc-server.sh, called from the turboquant wrapper Makefile) that inserts GGML_TYPE_TURBO2_0/3_0/4_0 into the allow-list of the *copied* grpc-server.cpp under turboquant-<flavor>-build/. The original file under backend/cpp/llama-cpp/ is never touched, so the stock llama-cpp build keeps compiling against vanilla upstream which has no notion of those enum values. Switch test-extra-backend-turboquant to set BACKEND_TEST_CACHE_TYPE_K=turbo3 / _V=turbo3 so the e2e gRPC suite actually runs the fork's TurboQuant KV-cache code paths (turbo3 also auto-enables flash_attention in the fork). Picking q8_0 here would only re-test the standard llama.cpp path that the upstream llama-cpp backend already covers. Refresh the docs (text-generation.md + model-configuration.md) to list turbo2/turbo3/turbo4 explicitly and call out that you only get the TurboQuant code path with this backend + a turbo* cache type. * fix(turboquant): rewrite patch-grpc-server.sh in awk, not python3 The builder image (ubuntu:24.04 stage-2 in Dockerfile.turboquant) does not install python3, so the python-based augmentation step errored with `python3: command not found` at make time. Switch to awk, which ships in coreutils and is already available everywhere the rest of the wrapper Makefile runs. * Apply suggestion from @mudler Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> --------- Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2026-04-15 01:25:04 +02:00
Ettore Di Giacinto	87e6de1989	feat: wire transcription for llama.cpp, add streaming support (#9353) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-14 16:13:40 +02:00
Ettore Di Giacinto	d67623230f	feat(vllm): parity with llama.cpp backend (#9328) ... * fix(schema): serialize ToolCallID and Reasoning in Messages.ToProto The ToProto conversion was dropping tool_call_id and reasoning_content even though both proto and Go fields existed, breaking multi-turn tool calling and reasoning passthrough to backends. * refactor(config): introduce backend hook system and migrate llama-cpp defaults Adds RegisterBackendHook/runBackendHooks so each backend can register default-filling functions that run during ModelConfig.SetDefaults(). Migrates the existing GGUF guessing logic into hooks_llamacpp.go, registered for both 'llama-cpp' and the empty backend (auto-detect). Removes the old guesser.go shim. * feat(config): add vLLM parser defaults hook and importer auto-detection Introduces parser_defaults.json mapping model families to vLLM tool_parser/reasoning_parser names, with longest-pattern-first matching. The vllmDefaults hook auto-fills tool_parser and reasoning_parser options at load time for known families, while the VLLMImporter writes the same values into generated YAML so users can review and edit them. Adds tests covering MatchParserDefaults, hook registration via SetDefaults, and the user-override behavior. * feat(vllm): wire native tool/reasoning parsers + chat deltas + logprobs - Use vLLM's ToolParserManager/ReasoningParserManager to extract structured output (tool calls, reasoning content) instead of reimplementing parsing - Convert proto Messages to dicts and pass tools to apply_chat_template - Emit ChatDelta with content/reasoning_content/tool_calls in Reply - Extract prompt_tokens, completion_tokens, and logprobs from output - Replace boolean GuidedDecoding with proper GuidedDecodingParams from Grammar - Add TokenizeString and Free RPC methods - Fix missing `time` import used by load_video() * feat(vllm): CPU support + shared utils + vllm-omni feature parity - Split vllm install per acceleration: move generic `vllm` out of requirements-after.txt into per-profile after files (cublas12, hipblas, intel) and add CPU wheel URL for cpu-after.txt - requirements-cpu.txt now pulls torch==2.7.0+cpu from PyTorch CPU index - backend/index.yaml: register cpu-vllm / cpu-vllm-development variants - New backend/python/common/vllm_utils.py: shared parse_options, messages_to_dicts, setup_parsers helpers (used by both vllm backends) - vllm-omni: replace hardcoded chat template with tokenizer.apply_chat_template, wire native parsers via shared utils, emit ChatDelta with token counts, add TokenizeString and Free RPCs, detect CPU and set VLLM_TARGET_DEVICE - Add test_cpu_inference.py: standalone script to validate CPU build with a small model (Qwen2.5-0.5B-Instruct) * fix(vllm): CPU build compatibility with vllm 0.14.1 Validated end-to-end on CPU with Qwen2.5-0.5B-Instruct (LoadModel, Predict, TokenizeString, Free all working). - requirements-cpu-after.txt: pin vllm to 0.14.1+cpu (pre-built wheel from GitHub releases) for x86_64 and aarch64. vllm 0.14.1 is the newest CPU wheel whose torch dependency resolves against published PyTorch builds (torch==2.9.1+cpu). Later vllm CPU wheels currently require torch==2.10.0+cpu which is only available on the PyTorch test channel with incompatible torchvision. - requirements-cpu.txt: bump torch to 2.9.1+cpu, add torchvision/torchaudio so uv resolves them consistently from the PyTorch CPU index. - install.sh: add --index-strategy=unsafe-best-match for CPU builds so uv can mix the PyTorch index and PyPI for transitive deps (matches the existing intel profile behaviour). - backend.py LoadModel: vllm >= 0.14 removed AsyncLLMEngine.get_model_config so the old code path errored out with AttributeError on model load. Switch to the new get_tokenizer()/tokenizer accessor with a fallback to building the tokenizer directly from request.Model. * fix(vllm): tool parser constructor compat + e2e tool calling test Concrete vLLM tool parsers override the abstract base's __init__ and drop the tools kwarg (e.g. Hermes2ProToolParser only takes tokenizer). Instantiating with tools= raised TypeError which was silently caught, leaving chat_deltas.tool_calls empty. Retry the constructor without the tools kwarg on TypeError — tools aren't required by these parsers since extract_tool_calls finds tool syntax in the raw model output directly. Validated with Qwen/Qwen2.5-0.5B-Instruct + hermes parser on CPU: the backend correctly returns ToolCallDelta{name='get_weather', arguments='{"location": "Paris, France"}'} in ChatDelta. test_tool_calls.py is a standalone smoke test that spawns the gRPC backend, sends a chat completion with tools, and asserts the response contains a structured tool call. * ci(backend): build cpu-vllm container image Add the cpu-vllm variant to the backend container build matrix so the image registered in backend/index.yaml (cpu-vllm / cpu-vllm-development) is actually produced by CI. Follows the same pattern as the other CPU python backends (cpu-diffusers, cpu-chatterbox, etc.) with build-type='' and no CUDA. backend_pr.yml auto-picks this up via its matrix filter from backend.yml. * test(e2e-backends): add tools capability + HF model name support Extends tests/e2e-backends to cover backends that: - Resolve HuggingFace model ids natively (vllm, vllm-omni) instead of loading a local file: BACKEND_TEST_MODEL_NAME is passed verbatim as ModelOptions.Model with no download/ModelFile. - Parse tool calls into ChatDelta.tool_calls: new "tools" capability sends a Predict with a get_weather function definition and asserts the Reply contains a matching ToolCallDelta. Uses UseTokenizerTemplate with OpenAI-style Messages so the backend can wire tools into the model's chat template. - Need backend-specific Options[]: BACKEND_TEST_OPTIONS lets a test set e.g. "tool_parser:hermes,reasoning_parser:qwen3" at LoadModel time. Adds make target test-extra-backend-vllm that: - docker-build-vllm - loads Qwen/Qwen2.5-0.5B-Instruct - runs health,load,predict,stream,tools with tool_parser:hermes Drops backend/python/vllm/test_{cpu_inference,tool_calls}.py — those standalone scripts were scaffolding used while bringing up the Python backend; the e2e-backends harness now covers the same ground uniformly alongside llama-cpp and ik-llama-cpp. * ci(test-extra): run vllm e2e tests on CPU Adds tests-vllm-grpc to the test-extra workflow, mirroring the llama-cpp and ik-llama-cpp gRPC jobs. Triggers when files under backend/python/vllm/ change (or on run-all), builds the local-ai vllm container image, and runs the tests/e2e-backends harness with BACKEND_TEST_MODEL_NAME=Qwen/Qwen2.5-0.5B-Instruct, tool_parser:hermes, and the tools capability enabled. Uses ubuntu-latest (no GPU) — vllm runs on CPU via the cpu-vllm wheel we pinned in requirements-cpu-after.txt. Frees disk space before the build since the docker image + torch + vllm wheel is sizeable. * fix(vllm): build from source on CI to avoid SIGILL on prebuilt wheel The prebuilt vllm 0.14.1+cpu wheel from GitHub releases is compiled with SIMD instructions (AVX-512 VNNI/BF16 or AMX-BF16) that not every CPU supports. GitHub Actions ubuntu-latest runners SIGILL when vllm spawns the model_executor.models.registry subprocess for introspection, so LoadModel never reaches the actual inference path. - install.sh: when FROM_SOURCE=true on a CPU build, temporarily hide requirements-cpu-after.txt so installRequirements installs the base deps + torch CPU without pulling the prebuilt wheel, then clone vllm and compile it with VLLM_TARGET_DEVICE=cpu. The resulting binaries target the host's actual CPU. - backend/Dockerfile.python: accept a FROM_SOURCE build-arg and expose it as an ENV so install.sh sees it during `make`. - Makefile docker-build-backend: forward FROM_SOURCE as --build-arg when set, so backends that need source builds can opt in. - Makefile test-extra-backend-vllm: call docker-build-vllm via a recursive $(MAKE) invocation so FROM_SOURCE flows through. - .github/workflows/test-extra.yml: set FROM_SOURCE=true on the tests-vllm-grpc job. Slower but reliable — the prebuilt wheel only works on hosts that share the build-time SIMD baseline. Answers 'did you test locally?': yes, end-to-end on my local machine with the prebuilt wheel (CPU supports AVX-512 VNNI). The CI runner CPU gap was not covered locally — this commit plugs that gap. * ci(vllm): use bigger-runner instead of source build The prebuilt vllm 0.14.1+cpu wheel requires SIMD instructions (AVX-512 VNNI/BF16) that stock ubuntu-latest GitHub runners don't support — vllm.model_executor.models.registry SIGILLs on import during LoadModel. Source compilation works but takes 30-40 minutes per CI run, which is too slow for an e2e smoke test. Instead, switch tests-vllm-grpc to the bigger-runner self-hosted label (already used by backend.yml for the llama-cpp CUDA build) — that hardware has the required SIMD baseline and the prebuilt wheel runs cleanly. FROM_SOURCE=true is kept as an opt-in escape hatch: - install.sh still has the CPU source-build path for hosts that need it - backend/Dockerfile.python still declares the ARG + ENV - Makefile docker-build-backend still forwards the build-arg when set Default CI path uses the fast prebuilt wheel; source build can be re-enabled by exporting FROM_SOURCE=true in the environment. * ci(vllm): install make + build deps on bigger-runner bigger-runner is a bare self-hosted runner used by backend.yml for docker image builds — it has docker but not the usual ubuntu-latest toolchain. The make-based test target needs make, build-essential (cgo in 'go test'), and curl/unzip (the Makefile protoc target downloads protoc from github releases). protoc-gen-go and protoc-gen-go-grpc come via 'go install' in the install-go-tools target, which setup-go makes possible. * ci(vllm): install libnuma1 + libgomp1 on bigger-runner The vllm 0.14.1+cpu wheel ships a _C C++ extension that dlopens libnuma.so.1 at import time. When the runner host doesn't have it, the extension silently fails to register its torch ops, so EngineCore crashes on init_device with: AttributeError: '_OpNamespace' '_C_utils' object has no attribute 'init_cpu_threads_env' Also add libgomp1 (OpenMP runtime, used by torch CPU kernels) to be safe on stripped-down runners. * feat(vllm): bundle libnuma/libgomp via package.sh The vllm CPU wheel ships a _C extension that dlopens libnuma.so.1 at import time; torch's CPU kernels in turn use libgomp.so.1 (OpenMP). Without these on the host, vllm._C silently fails to register its torch ops and EngineCore crashes with: AttributeError: '_OpNamespace' '_C_utils' object has no attribute 'init_cpu_threads_env' Rather than asking every user to install libnuma1/libgomp1 on their host (or every LocalAI base image to ship them), bundle them into the backend image itself — same pattern fish-speech and the GPU libs already use. libbackend.sh adds ${EDIR}/lib to LD_LIBRARY_PATH at run time so the bundled copies are picked up automatically. - backend/python/vllm/package.sh (new): copies libnuma.so.1 and libgomp.so.1 from the builder's multilib paths into ${BACKEND}/lib, preserving soname symlinks. Runs during Dockerfile.python's 'Run backend-specific packaging' step (which already invokes package.sh if present). - backend/Dockerfile.python: install libnuma1 + libgomp1 in the builder stage so package.sh has something to copy (the Ubuntu base image otherwise only has libgomp in the gcc dep chain). - test-extra.yml: drop the workaround that installed these libs on the runner host — with the backend image self-contained, the runner no longer needs them, and the test now exercises the packaging path end-to-end the way a production host would. * ci(vllm): disable tests-vllm-grpc job (heterogeneous runners) Both ubuntu-latest and bigger-runner have inconsistent CPU baselines: some instances support the AVX-512 VNNI/BF16 instructions the prebuilt vllm 0.14.1+cpu wheel was compiled with, others SIGILL on import of vllm.model_executor.models.registry. The libnuma packaging fix doesn't help when the wheel itself can't be loaded. FROM_SOURCE=true compiles vllm against the actual host CPU and works everywhere, but takes 30-50 minutes per run — too slow for a smoke test on every PR. Comment out the job for now. The test itself is intact and passes locally; run it via 'make test-extra-backend-vllm' on a host with the required SIMD baseline. Re-enable when: - we have a self-hosted runner label with guaranteed AVX-512 VNNI/BF16, or - vllm publishes a CPU wheel with a wider baseline, or - we set up a docker layer cache that makes FROM_SOURCE acceptable The detect-changes vllm output, the test harness changes (tests/ e2e-backends + tools cap), the make target (test-extra-backend-vllm), the package.sh and the Dockerfile/install.sh plumbing all stay in place.	2026-04-13 11:00:29 +02:00
Ettore Di Giacinto	9ca03cf9cc	feat(backends): add ik-llama-cpp (#9326) ... * feat(backends): add ik-llama-cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: add grpc e2e suite, hook to CI, update README Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Apply suggestion from @mudler Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> * Apply suggestion from @mudler Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2026-04-12 13:51:28 +02:00
Ettore Di Giacinto	7a0e6ae6d2	feat(qwen3tts.cpp): add new backend (#9316) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-11 23:14:26 +02:00
Richard Palethorpe	ea6e850809	feat: Add Kokoros backend (#9212) ... Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-04-08 19:23:16 +02:00
Richard Palethorpe	e0eb2fd734	chore(ci): Scope tests extras backend tests (#9170) ... Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-03-30 17:46:07 +00:00
Ettore Di Giacinto	f7e8d9e791	feat(quantization): add quantization backend (#9096) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-22 00:56:34 +01:00
Ettore Di Giacinto	a738f8b0e4	feat(backends): add ace-step.cpp (#8965) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-12 18:56:26 +01:00
Ettore Di Giacinto	7dc691c171	feat: add fish-speech backend (#8962) ... * feat: add fish-speech backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * drop portaudio Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-12 07:48:23 +01:00
Ettore Di Giacinto	bf5a1dd840	feat(voxtral): add voxtral backend (#8451) ... * feat(voxtral): add voxtral backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * simplify Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-02-09 09:12:05 +01:00
3 people	3370d807c2	feat(nemo): add Nemo (only asr for now) backend (#8436) ... * feat(nemo): add Nemo (only asr for now) backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo): add Nemo backend without Python version pins (#8438) * Initial plan * Remove Python version pins from nemo backend install.sh Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> * Pin pyarrow to 20.0.0 in nemo requirements Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>	2026-02-07 08:19:37 +01:00
Ettore Di Giacinto	1e08e02598	feat(qwen-asr): add support to qwen-asr (#8281) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-29 21:50:35 +01:00
Ettore Di Giacinto	9b973b79f6	feat: add VoxCPM tts backend (#8109) ... * feat: add VoxCPM tts backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Disable voxcpm on arm64 cpu Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-28 14:44:04 +01:00
Ettore Di Giacinto	ec1598868b	feat(vibevoice): add ASR support (#8222) ... * feat(vibevoice): add ASR support Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(tests): download voice files Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to run on bigger runner Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * debug Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * CI can't hold vibevoice Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-27 20:19:22 +01:00
Ettore Di Giacinto	923ebbb344	feat(qwen-tts): add Qwen-tts backend (#8163) ... * feat(qwen-tts): add Qwen-tts backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Update intel deps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop flash-attn for cuda13 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-23 15:18:41 +01:00
Ettore Di Giacinto	a6ff354c86	feat(tts): add pocket-tts backend (#8018) ... * feat(pocket-tts): add new backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add to the gallery Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Update docs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-13 23:35:19 +01:00
Ettore Di Giacinto	b964b3d53e	feat(backends): add moonshine backend for faster transcription (#7833) ... * feat(backends): add moonshine backend for faster transcription Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add backend to CI, update AGENTS.md from this exercise Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-07 21:44:35 +01:00
dependabot[bot]	91248da09e	chore(deps): bump actions/checkout from 5 to 6 (#7339) ... Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2025-11-24 21:18:15 +01:00
dependabot[bot]	0ca1765c17	chore(deps): bump actions/checkout from 4 to 5 (#6014) ... Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v5) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2025-08-12 18:54:39 +02:00
Ettore Di Giacinto	d5c9c717b5	feat(chatterbox): add new backend (#5524) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-30 10:52:55 +02:00
Ettore Di Giacinto	6e1c93d84f	fix(ci): comment out vllm tests ... Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2025-05-01 10:01:22 +02:00
Wyatt Neal	4076ea0494	fix: vllm missing logprobs (#5279) ... * working to address missing items referencing #3436, #2930 - if i could test it, this might show that the output from the vllm backend is processed and returned to the user Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com> * adding in vllm tests to test-extras Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com> * adding in tests to pipeline for execution Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com> * removing todo block, test via pipeline Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com> --------- Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com>	2025-04-30 12:55:07 +00:00
Ettore Di Giacinto	eef80b9880	chore(ci): cleanup tests ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-23 10:02:57 +01:00
Ettore Di Giacinto	1e9bf19c8d	feat(transformers): merge sentencetransformers backend (#4624) ... * merge sentencetransformers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add alias to silently redirect sentencetransformers to transformers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add alias also for transformers-musicgen Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop from makefile Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Move tests from sentencetransformers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Remove sentencetransformers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Remove tests from CI (part of transformers) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Do not always try to load the tokenizer Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Adapt tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix typo Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Tiny adjustments Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-18 18:30:30 +01:00
Ettore Di Giacinto	8027fdf1c7	feat(transformers): merge musicgen functionalities to a single backend (#4620) ... * feat(transformers): merge musicgen functionalities to a single backend So we optimize space Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * specify type in tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Some adaptations for the MusicgenForConditionalGeneration type Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-17 17:01:16 +01:00
Ettore Di Giacinto	7d0ac1ea3f	chore(vall-e-x): Drop backend (#4619) ... There are many new architectures that are SOTA and replaces vall-e-x nowadays. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-17 09:35:10 +01:00
Ettore Di Giacinto	3ad920b50a	fix(parler-tts): pin protobuf (#3963) ... * fix(parler-tts): pin protobuf Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * debug Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Re-apply workaround Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-25 23:50:12 +02:00
Ettore Di Giacinto	9475a6fa05	chore: drop petals (#3316) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-08-20 10:01:38 +02:00