23 Commits

Include renames

Author	SHA1	Message	Date
Ettore Di Giacinto	2383726d6d	Revert "chore(tests): Avoid network, sleep and more during tests" (#11601) ... Revert "chore(tests): Avoid network, sleep and more during tests (#11050)" This reverts commit cb3bf7af3f.	2026-08-19 16:39:39 +02:00
Richard Palethorpeandlocalai-org-maint-bot	cb3bf7af3f	chore(tests): Avoid network, sleep and more during tests (#11050) ... * test: make coverage failures observable Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: parallelize coverage without remote fixtures Assisted-by: Codex:gpt-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: add offline resource infrastructure Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: enforce offline resource replay Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: harden offline resource refresh Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: expose slow coverage waits Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: eliminate avoidable wall-clock waits Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: remove repeated fixture startup waits Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fix offline resource CI portability Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: cache Go modules before offline tests Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: drop the static network lint in favour of real isolation The offline test suite already prevents tests from reaching the network twice over: run-test-linux-offline.sh puts the test process in a cgroup and REJECTs egress outside the private ranges, and HardenedTransport installs testnetwork.LocalGuard to refuse dials that resolve to a public address. Both fail the test with a precise error at the moment of the dial. test-network-lint.sh added neither. Its diff stage defaulted to a HEAD base, so on a clean checkout it compared the tree against itself and inspected nothing; the branch's own commits were never examined. It only produced output when an earlier job step dirtied the tree, and then it matched a bare https?:// against whatever changed. make react-ui runs npm install rather than npm ci, so CI rewrote core/http/react-ui/package-lock.json and the lint reported an npm registry URL as forbidden test network access: + "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz", Its fingerprint stage was self-defeating in a quieter way: hashing the whole tree's network-mechanism inventory meant every rebase onto a master that touched any _test.go needed a manual baseline bump, so the check mostly caught its own staleness. Remove the script, its make target and the two prerequisite edges, along with the test-network: fixture markers that existed only to suppress it. The isolation itself is untouched. Assisted-by: Claude:claude-opus-5 [go vet] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: keep hidden files in the offline test bundle artifact Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives under .cache/, which actions/upload-artifact skips by default, so the Linux job packed an artifact missing the very file the next step restores. The other half of 15a37b0ac moved test-network-lint out of the `test` and `test-coverage` prerequisite lists into a recipe line, so parallel make could not fingerprint the tree while generated fixtures were still changing. That is dropped: the preceding commit removes the lint entirely, and the race it worked around is one more reason a whole-tree fingerprint was the wrong mechanism. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor: share bounded exponential backoff Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: mirror Jetson Python wheels Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * docs(agents): index the Jetson wheels mirror Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: add defensive build network proxy Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement updated backend trait Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): clear recovered proxy errors Do not mark a request failed when a later safe retry succeeds. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: require HTTPS build interception Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): preserve system trust in unproxied builds Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds. Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): persist build proxy trust Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): trust proxy in nested build scripts Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): use HTTPS apt sources for Bonsai Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(privacy-filter): trust build proxy CA Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fail on hidden offline egress Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): preserve system CA trust Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>	2026-08-19 10:59:31 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	9a6156d808	feat(nemo-speech-cpp): add the NVIDIA NeMo-Speech.cpp backend (#11406) ... * feat(nemo-speech-cpp): scaffold the backend and upstream build Adds the backend skeleton and the NeMo-Speech.cpp build, pinned at 2e12e2def8a98ed06666f7ee3ca94e7193e04be4. The Go side is deliberately a stub: it dlopens the runtime and starts the gRPC server, later work fills in the symbol table and the model logic. Three details of the upstream layout differ from what the plan assumed, and the build reflects the real tree: * The TTS C ABI ships as libnemo_speech_tts, not libnemo_speech_tts_c. Upstream compiles c_api.cpp straight into the implementation library and only aliases the nemo_speech_tts_c CMake target, so no _c object exists on disk. ASR and NMT do build a real _c shim. * Shared objects land in build/bin, since upstream points CMAKE_LIBRARY_OUTPUT_DIRECTORY at ${CMAKE_BINARY_DIR}/bin. * The ASR and NMT _c shims carry a DT_NEEDED on libnemo_speech_asr and libnemo_speech_nmt, so those are staged and packaged alongside them. Otherwise dlopen fails at startup. The ggml patch step uses an order-only prerequisite. cmake writes into the checkout and bumps its mtime past the sentinel, which would otherwise re-run git apply over an already-patched tree and break every incremental build. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): make 'build' produce the package and bundle the ITN stack Addresses the review of the scaffold commit. backend/Dockerfile.golang runs 'make -C backend/go/$(BACKEND) build' and then copies package/ into the final image, so 'build' has to end with a populated package/. It only staged shared objects, which would have shipped an image with no binary and no libraries at all. The old staging recipe is now stage-libs and the chain is stage-libs, nemo-speech-cpp-grpc, package, build, matching every sibling Go backend. Text normalization was packaged incorrectly. nemo_speech_text_normalization is STATIC but links sparrowhawk, fstfar and fst PUBLIC, so they land as DT_NEEDED on libnemo_speech_asr.so, and they live in a project-local prefix that nothing else provides. WITH_NORM stays ON by default on Linux, since normalization is a wanted feature. Instead stage_libs now copies .deps/itn/lib when WITH_NORM=ON, and package.sh bundles it. Staging that prefix is still not enough on its own: Sparrowhawk drags in protobuf, re2 and absl, which neither build_itn_deps.sh nor package-system-libs.sh provides. Rather than hard-code another hand-maintained list, package.sh now walks the DT_NEEDED entries of everything staged and copies whatever is unresolved, skipping the core set and the GPU set that the shared scripts already own. It fails at package time, not at first dlopen, when something cannot be resolved. On a WITH_NORM=OFF build the closure is already complete and it copies nothing. Restore CGO_ENABLED=0 on the Go build to match whisper, parakeet-cpp and omnivoice-cpp. Note that purego reaches dlopen through fakecgo, so the binary is dynamically linked either way; what the flag changes is the NEEDED set, and lib/ld.so routing in run.sh exists precisely because the binary is not static. Replace the hand-rolled .patched sentinel with upstream's scripts/apply-ggml-patches.sh. It applies the series in filename order, exits non-zero when a patch does not apply, and detects "already applied" by comparing the full-series tree hash rather than an mtime, so it is safe to run every time and there is no sentinel left to go stale or to wedge the build when deleted. It is wired as an order-only prerequisite so running it does not force a relink. Also: correct the package.sh header, which claimed three shared objects when there are five and none of the TTS ones carry a _c suffix; give 'make test' the LD_LIBRARY_PATH the dlopen tests will need; document that a NEMO_SPEECH_VERSION bump needs 'make purge'; and extend 'clean' to remove package/ and the ITN libraries. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): make the closure guard fail closed and give CI its toolchain Addresses the second review round. The dependency-closure guard failed open. Its glob expands once per pass, so each pass advanced the closure by exactly one level, and the fixed count of five passes then fell out of the loop without checking whether anything remained. An eight-deep chain packaged six libraries, exited zero and reported success. That is the case the guard was written for: asr to sparrowhawk to protobuf to absl already runs several levels deep, so a WITH_NORM build could ship missing its deepest libraries and fail at first dlopen. The loop now runs until the staged set stops growing, and exhausting the bound is a hard error rather than a silent exit. For the same reason, a build image with neither readelf nor objdump no longer warns and skips. It cannot show the package is complete, so it refuses to ship it. The guard is entered only when there is something to check, so an empty package cannot trip the new error. Dockerfile.golang installed ninja-build only in the Vulkan branch while this Makefile runs cmake -G Ninja unconditionally, so the CPU, cuBLAS and L4T images could not configure at all. ninja-build moves to the common apt list; it does not change CMake's default generator, so it is inert for the other backends. gcc-12 was nowhere in the tree, yet WITH_NORM defaults ON and build_itn_deps.sh needs it, so the committed default was unbuildable in CI. Install it, with the protobuf, absl, re2 and autotools that Sparrowhawk and OpenFST need, gated on BACKEND so the other Go images do not carry it. The list follows upstream's own docker/Dockerfile, trimmed of the gRPC, portaudio and python entries a BUILD_GRPC=OFF build does not use. Text normalization stays ON: downgrading it silently would ship a backend advertising a feature it lacks. Also: make test depend on stage-libs, so LD_LIBRARY_PATH is not an empty directory on a clean tree, and add an engine target so Dockerfile.golang's cacheable prebuild layer is not skipped and a CUDA build stops recompiling all of upstream on every Go-side change. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): pin protoc for ITN and make the norm stack its own target Addresses the third review round. Dockerfile.golang installs protoc 27.1 into /usr/local/bin, ahead of /usr/bin, while libprotobuf-dev is the distro's 3.21 on noble and 3.12 on jammy. Sparrowhawk resolves protoc from PATH at make time (configure.ac uses AC_CHECK_PROG, so PROTOC substitutes to the bare word, and src/proto/Makefile.am invokes it) and commits no pregenerated stubs, so the rule always runs. Code generated by 27.1 includes google/protobuf/runtime_version.h and a PROTOBUF_VERSION guard the older headers lack, so the WITH_NORM build could not complete. Pin PROTOC to the apt one for that step; configure documents that a pre-set value wins. The apt protoc and libprotobuf-dev come from one source package at one version, which is the property that makes this correct. The text-normalization stack is now a target keyed on a file build_itn_deps.sh actually produces, rather than a side effect of the runtime library rule. As a side effect make could not see whether it existed, so once the library was up to date the script could never run again: a tree built WITH_NORM=OFF could not move to ON, and make test hard-failed with no escape but a full 345 MB clean. It is now built on demand and reachable on its own as 'make itn'. Staging keys on the prefix existing rather than on WITH_NORM, so it stages what the tree actually built, and package.sh's closure guard remains the backstop. An already-configured build tree also now wins over the platform default, so a tree built WITH_NORM=OFF is not silently reconfigured to ON by a bare make test, which is what demanded gcc-12 from developers who chose not to have it. An explicit WITH_NORM= on the command line still overrides both, and the ITN rule preflights for gcc-12 with an error that names the alternative. Move ninja-build out of the shared apt layer into the existing BACKEND-gated block. Dockerfile.golang serves 225 matrix entries and only this backend configures with -G Ninja, so the common list is byte-identical to master again and no other image loses its cache. Drop libabsl-dev and correct the comment that justified it. No base image here ships protobuf 25, so nothing needs the absl split, and the cmake glob looks in /usr/lib rather than the multiarch directory Ubuntu actually uses, so the package could never have contributed anything. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): move the backend apt gate below the expensive layers Addresses the fourth review round. The nemo-speech-cpp apt block sat immediately after the shared apt layer, above the Vulkan SDK build, the CUDA and ROCm installs, the Go toolchain and the protoc download. Docker keys each layer on its parent, so inserting a step there re-keys everything below it: a byte-identical shared layer is not enough, and merging as it stood would have forced all of those to re-execute once for every Go backend image. Move it down beside the existing opus, crispasr and sherpa-onnx gates, which sit after those layers for the same reason. Checked the ordering both ways before moving. Nothing between the two positions uses these packages: the Vulkan and opus blocks install their own ninja and pkg-config, go install protoc-gen-go needs the Go toolchain rather than protoc, and the protoc 27.1 step is a release-binary download that needs neither protobuf-compiler nor libprotobuf-dev. Nothing in the block needs anything those layers provide; it uses only apt, and the mirror rewrite from the first RUN persists in the image. It also runs no update-alternatives, so the default compiler stays untouched for later layers. The diff against master is now a single additive hunk with no shared layer touched. Also preflight ITN_PROTOC. configure gates a preset PROTOC on test -n alone, so a path that does not exist is accepted and the error surfaces much later as a bare "No such file or directory" from inside make -C src/proto. The pin introduced that failure on a box whose only protoc is in /usr/local/bin, which worked before. Check it alongside the gcc-12 check and name the ITN_PROTOC= override in the message. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): parse model options Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): guard the empty option value and warn on a bad gpu index An empty value like "vad_model:" must stay empty, since callers read the empty string as "unset". That branch of resolve() had no spec: dropping the guard left every spec green while parseOptions started returning the models directory itself. Add the spec that fails without the guard. A known key with an unparseable value is a typo, not a config from a newer backend, and "gpu:banna" failed expensively: the model loaded, produced correct output, and ran on CPU with no signal anywhere. Log it. Unknown keys stay silently ignored, which is what keeps configs forward compatible. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): detect model family and discover TTS assets Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): bind the C ABI with layout assertions purego binds by name at runtime and the config structs are passed by pointer, so both a renamed symbol and a mismatched struct layout would otherwise survive a green build. registerSymbols names the failing symbol, and the layout specs compare each Go mirror against the size the library reports for itself, against the offsets a C compiler produces for the installed headers, and against the default values upstream writes into the structs it returns. Two of the bindings differ from the plan because the headers do. The plan's nemo_speech_diar_segments signature omits the segmentation-config pointer that diar.h declares as the second parameter, which would have shifted the output buffer, the capacity and the count pointer one position each. And nemo_speech_diar_stream_push_f32 was missing from the symbol table although standalone diarization cannot work without it. Also close the two panic and equality gaps left in family.go: ValueString panics on a mistyped general.architecture, and the self-codec guard compared a Cleaned candidate path against an uncleaned one, so a doubled separator let the primary GGUF be selected as its own codec. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nemo-speech-cpp): run the ABI specs in CI and refuse to skip them The layout assertions were inert. TEST_PATHS does not cover this backend and the per-backend list in test-extra had no entry for it, so nothing invoked the package's tests. Add it next to depth-anything-cpp, supertonic and vllm-cpp, the group whose own test target carries its build prerequisites; stage-libs already pulls the native build chain, so no prepare-test-extra entry is needed. The skip guard was also loader-inconsistent: librariesPresent stats bare filenames relative to the working directory while openLibraries resolves them through the loader search path, so any invocation other than make test skipped every library-backed spec and still reported green. NEMO_SPEECH_REQUIRE_LIBS=1 turns that into a failure naming the directory and the remedy, and the Makefile test target sets it. Unset, the plain skip survives so a developer without a build can still run the pure-Go layer specs. Trim the default-value fingerprint from roughly forty assertions to eight. It was pinning tunables such as threads and flush_partial_chunk, so a legitimate pin bump would have failed with a message reading like a layout error. What survives is only header-documented contract: the lone non-zero max_alternatives, the run of -1 sentinels and the zero that witnesses where it stops. Verified the narrowed spec still catches a mirror and offset table corrupted in lockstep, which is the one class only this layer sees. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): select the family at load and gate RPCs on it Load sniffs the GGUF architecture, maps it to a family and dispatches to the family's loader. requireFamily gates every other RPC, returning Unimplemented naming both the loaded and the wanted family so a misconfigured model YAML produces a message a user can act on. The family is committed only once its loader has succeeded. A load that fails part way through would otherwise leave the gate open on a handle that was never created. cstr uses runtime.Pinner rather than an ordinary Go allocation. The address crosses the ABI as a uintptr, which the collector does not trace, so incidental reachability through the release closure is not a guarantee: a caller discarding that closure could have the bytes collected before the create call reads them. Pinning is the sanctioned mechanism, makes the release function do real work, and turns a dropped release into a loud leaked-Pinner panic instead of silent corruption. Free overrides the base no-op to destroy the handle and reset the family. Every family owns C memory only its own destroy entry point can release, so without this an unloaded model leaks an acoustic model. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): pin the load ordering, close the engineMu race Three review items, plus a defect the race detector turned up. The spec covering "no family selected after a failed load" wrote junk to a .gguf, so Load returned at ggufArchitecture before a family was ever chosen and the assertion was vacuous. Generalised the GGUF test helper to take a string architecture, and added a spec that loads a magpietts GGUF with no sibling codec, so familyFor succeeds and discoverTTSAssets then fails. It self-guards on ggufArchitecture so it cannot degrade back into the earlier path. requireFamily read n.fam unlocked while Free wrote it under engineMu, which the race detector confirms is a real race. pkg/grpc/server.go calls Free without the backend lock every other RPC holds, so teardown can land mid-request. withEngine now takes the lock, checks the family and runs the body under one acquisition; two would leave a window for Free to destroy the handle between check and use. The locking protocol is stated in both directions for the RPCs still to be written. Running -race also enables checkptr, which aborts on cstr's pointer being read back by goString: converting a uintptr to a pointer is fatal whenever the address lands in a Go allocation, so a pinned Go buffer can never be dereferenced from Go. The pointer is for C alone. Both helpers now document the one-way contract, and goString is tested against a real C-owned string by rebinding the version symbol to return a raw char*. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): implement offline transcription Create the ASR recognizer in loadASR and serve AudioTranscription. Segment times are int64 nanoseconds, not seconds: the proto field is an int64 that core/backend reads straight into a time.Duration, while the runtime reports word offsets in milliseconds. Words are grouped into one segment per consecutive speaker run, with the 1-based speaker tag carried through and 0 (untagged) left unlabelled. The whole RPC body runs inside withEngine so the family check and the C calls happen under one acquisition of engineMu. Free runs without the backend lock, so checking the family and then relocking would let a teardown destroy the handle in the gap. The audio decode is inside the closure too, which costs nothing: base.SingleThread already serialises this backend's RPCs. recognizeF32 guards zero-length PCM. &pcm[0] panics on an empty slice, so Go never reaches the C side's own "empty audio" rejection, and a silent clip or a truncated upload is ordinary input. pkg/utils has no WAV decode helper, only the ffmpeg normalisation, so audio.go pairs AudioToWav with go-audio the way parakeet-cpp does. It returns the sample rate rather than a duration, since the C API resamples off that number. Also closes the write-side half of the race Task 5 fixed on the read side: Load now holds engineMu across the family switch and the n.fam commit, matching Free. The loaders still must not take it. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): implement streaming and live transcription AudioTranscriptionStream drives a whole clip through the cache-aware streaming API in 100 ms pushes, emitting each finalized utterance as a delta and closing with the assembled result. AudioTranscriptionLive serves the bidirectional RPC over the same session: config first, a ready ack, deltas with word timings as utterances land, and a terminal result when the caller closes its send side. Both wrap their body in withEngine, so a stream holds engineMu for its whole life and Free waits on it rather than destroying the recognizer underneath a half-finished stream. That makes the way out load-bearing: the file loop honours the request context between pushes, and the live loop ends when the host closes the request channel, so a disconnected client cannot pin the model against unload. Only finals become deltas. The runtime applies punctuation and inverse text normalization on finals only, so a final rewrites the utterance rather than extending its interim, and delta on the wire is newly-finalized text that consumers concatenate. Forwarding interims would duplicate and mispunctuate every utterance. The four streaming entry points sit behind an asrSession interface. No NeMo GGUF is small enough to keep in the tree, so without that seam the need-more-audio drain would have no test at all: nemo_speech_asr_stream_next reports OK with a NULL handle when it wants more audio, which is a pause rather than an end, and reading it either way round drops results or spins forever. Also folds in three items from the offline transcription review: - empty audio is now refused before anything crosses the ABI, not inside recognizeF32. The added integration spec caught the old ordering panicking on an unbound entry point instead of failing; - an undecodable sample rate is an error rather than 0, which this runtime reads as "already at the model rate" and would have made a wrong rate silently pitch-shift the audio; - AudioTranscription guards its result pointer instead of relying on an unstated invariant. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): make live deltas concatenate and fill segment words runLive wrote the inter-utterance separator into the accumulated transcript but emitted the delta without it, so a two-utterance turn sent "one." and "two." while the terminal result read "one. two.". The live consumer is the one that really concatenates: the realtime semantic-VAD path joins the accumulated deltas with the empty string and clears them only at a turn reset, never at an endpoint, so the running caption read "one.two.". The separator now goes into the delta, as it already did on the file path, and the terminal text is the verbatim concatenation rather than a trimmed rebuild. TranscriptSegment.Words was never populated, so a request asking for timestamp_granularities ["word"] came back with no words at all even though the timings were decoded. wordsToSegments now attaches them, gated on the granularity the same way parakeet-cpp gates it, so a transcript that did not ask for word timestamps does not pay for them. Also: the final that comes back from the tail flush no longer claims an end-of-utterance. It is the end of the stream, not a user yielding the turn, and eou is what the realtime turn detector acts on. The comment explaining why interims are suppressed led with the runtime's postprocessing. The wire contract is the stronger reason and now comes first: consumers concatenate deltas, so forwarding a growing hypothesis assembles to "hehellhelloHello.". The postprocessing only explains why no diffing trick would rescue them. It is also ITN and strip_formatting rather than punctuation, which is off by default here. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): implement standalone diarization loadDiarizer creates the Sortformer diarizer and Diarize serves the RPC over a diarization stream: decode, chunked push, finish, then the count-then-fill segments protocol. nemo_speech_diar_segment carries start_time and end_time in SECONDS already, not frame indices, so no conversion happens on the way to DiarizeSegment.start/end and the model's seconds-per-frame is not involved at all. The speaker label is the runtime's 1-based tag as a decimal string, matching what wordsToSegments emits on the ASR path, so the same speaker reads the same way whether a caller diarized a file or transcribed it. The six frame-geometry overrides are written as -1 rather than left zero. c_api.cpp applies left_context_frames when it is >= 0 while every other override needs > 0, so a zeroed config would silently pin the left context to zero and change the model's streaming geometry. nemo_speech_diar_segments writes *count before it rejects a buffer that is too small, so a rejected fill still reports the size to retry with. collectSegments uses that rather than truncating, bounded at four attempts because the RPC holds engineMu for its whole body and an unbounded retry would block an unload behind it. Two DiarizeRequest knobs map onto the segmentation config, and the proto and header names cross over: min_duration_on is the C min_duration_sec and min_duration_off is the C min_gap_sec. Six fields have no equivalent in this pipeline and are logged rather than dropped in silence: num_speakers, min_speakers and max_speakers (Sortformer's capacity is fixed by the checkpoint), clustering_threshold (there is no clustering stage), include_text (no ASR here) and threads. The empty-PCM guard fires before the stream is opened, so a silent clip never reaches a purego entry point that would dereference &pcm[0]. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): pin the diarizer geometry sentinels and cap the segment buffer The six frame-geometry overrides were written as -1 with nothing asserting it. c_api.cpp applies left_context_frames at >= 0 while the other five need > 0, so a dropped sentinel there pins the model's left context to zero, and the struct keeps exactly the same shape, which is all the layout assertions can see. Extracting diarModelConfig makes the values assertable: five specs now pin all six frame fields, the device index, the declared size and the NULL preset, each frame field on its own line so a missing sentinel names itself. distinctSpeakers had a spec with three segments over three distinct labels, which len(segs) satisfies just as well as the real thing. Four segments over three labels makes it a spec that can fail. collectSegments sized its buffer straight from a count the C side reported, and make() panics rather than erroring on a length it cannot satisfy, so an uninitialised size_t coming back across the ABI killed the backend process instead of failing one request. A ceiling of 2^22 segments, upwards of 93 hours of audio at one 80 ms frame each, turns that into a diagnosable error. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): implement TTS and streaming TTS The PCM callback is compiled once per process behind a sync.Once, not once per request and not once per load. purego.NewCallback writes into a fixed table of 2000 entries (purego/syscall_sysv.go) and never releases one, so a per-request callback panics the backend process on the 2001st synthesis, and a per-load one reaches the same ceiling on a server that swaps models. Synthesis is routed through that single callback plus a user_data id: engineMu is per-model, one process holds several models, so a single current-sink pointer would be overwritten by two TTS models synthesizing at once. Deviations from the brief, all verified against the real headers and proto: - TTS is TTS(*pb.TTSRequest) error and TTSStream is TTSStream(*pb.TTSRequest, chan []byte) error, per pkg/grpc/interface.go. The brief's context/pb.Result and server-stream forms do not implement the interface. The channel is closed on every path, including the family rejection, because pkg/grpc/server.go blocks on its drain goroutine and an unclosed channel hangs the RPC with the backend lock held. - The callback takes unsafe.Pointer, not uintptr. Converting a uintptr parameter back to a pointer is a checkptr violation that aborts under -race. - resolveSpeaker refuses to turn a negative number into a speaker index. -1 is the C API's "use the default" sentinel, so the brief's rule would have made a request naming an invalid voice synthesize in the default voice instead of being rejected. temperature and cfg_scale each write their override flag as well: magpietts/runtime.cpp reads the float only when the flag is set, so a temperature without it is silently discarded. Also folds in Task 8's review finding on asr.go: the six bare -1 sentinels in loadASR move to an asrDiarConfig builder reusing diarGeometryDefault, with specs. src/asr/c_api.cpp applies left_context_frames at >= 0, so a dropped sentinel pins the model geometry to 0 and no layout assertion can see it. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): surface NMT translation through Predict nemo_speech_nmt_translate takes explicit source and target languages and has no free-form generation or token-callback entry point, so there is no prompt in the LLM sense. The pair comes from the source_language / target_language model options, with an optional leading [src->tgt] directive as the only per-request override, and PredictStream emits the whole translation as a single chunk because the C API has nothing finer to give it. Both RPCs wrap their body in withEngine so the family check and the C calls that trust the handle share one acquisition of engineMu. PredictStream closes its channel on every path, including the family rejection: this is the legacy streaming contract, and pkg/grpc/server.go blocks on a drain goroutine that only finishes when the channel closes, so leaving it open hangs the RPC rather than failing it. nmtTranslatorConfig is extracted so its four adjacent pointer fields can be asserted against distinct sentinels. Transposing two of them changes neither the struct size nor any field offset, so the layout assertions cannot see it. Also removes goString, which had no production caller: every string-returning symbol in abi.go is bound with a Go string return that purego converts itself. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nemo-speech-cpp): pin the three-segment pair tag in an NMT directive The directive regex allowed an unbounded run of two-letter segments per side, but nothing tested it: narrowing that run back to a single optional segment left every spec green. resolve_tag accepts a ready pair tag in one field with the other empty (src/nmt/langpairs.cc), and those tags run to three segments (en-zh-cn, pt-br-en), so a shorter pattern does not mis-split the tag, it fails to match the directive at all and the whole bracket is handed to the model as text to translate. The justification on the regex was also wrong and is corrected: pt-br and zh-cn are two segments and parse either way. It is the single-field form that needs the run. Renames the NMT handle to n.nmt so it stops sharing a name with the translator interface, following n.synth, which is shortened for the same reason. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nemo-speech-cpp): register the backend and give its specs a CI job Registers nemo-speech-cpp across every surface .agents/adding-backends.md requires, and adds the CI job its unit suite never had. backend/index.yaml gets the meta backend (capabilities map, no uri), a development meta and 12 image entries. No amd and no intel capability keys: upstream NeMo-Speech.cpp builds ggml with CUDA, Vulkan or Metal only, and SystemState.Capability falls back to "default", so those hosts get the CPU build rather than a tag that does not exist. The nvidia-cuda-* and nvidia-l4t-cuda-* keys are present because getSystemCapabilities() refines an NVIDIA host to them whenever the CUDA directory exists; without them every modern CUDA host and Jetson would miss the map and quietly run on CPU. .github/backend-matrix.yml gets 7 include rows and 1 includeDarwin row. No hipblas and no sycl rows, for the same upstream reason. cpu and vulkan are per-arch pairs sharing a tag-suffix so backend-merge-jobs builds a multi-arch manifest: an ARM host with no NVIDIA GPU reports "default" and the Jetson image does not cover it. The CI job is the substantive part. make test-extra is dead on master, because prepare-test-extra depends on a protogen-python target that does not exist and no workflow invokes it anyway, so the entry added earlier in this series ran nowhere. abi_test.go asserts the size and field offsets of every Go mirror struct against the C ABI it is dlopened into, and those assertions are the only defence against silent memory corruption after a purego symbol rename or an upstream header change. tests-nemo-speech-cpp in test-extra.yml now executes them on pull_request and on master, gated on the backend's own path filter. The recipe sets NEMO_SPEECH_REQUIRE_LIBS=1, so a missing library fails rather than skips. WITH_NORM=OFF skips the OpenFST leg and costs no coverage: nothing in the four C ABI headers is conditional on it, so the layouts are identical. Also registers the upstream pin with the bump bot, which the backend Makefile already claimed but was never wired up, and adds the BackendCapabilities entry so a hand-written model config gets a real usecase surface. PossibleUsecases is the union of the four families and DefaultUsecases is transcript alone, the audio-cpp pattern. No VoiceCloning key: MagpieTTS synthesizes from baked speaker ids, not a reference clip. No gallery entries: publishing converted GGUFs is a follow-up. ModelIdentity needs no work in this backend. main.go serves through grpc.StartServer, so every RPC lands on pkg/grpc's shared server wrapper first, and checkModelIdentity is the first statement of all seven handlers this backend implements. A second check inside NemoSpeech would be unreachable and would risk diverging from the cross-language sentinel the router matches on. AudioTranscriptionLive stays unguarded because TranscriptLiveRequest carries no ModelIdentity field at all, which is a proto-level gap affecting every backend and needs its own change. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): build the CUDA-13 Jetson image the l4t-cuda-13 key needs The nvidia-l4t-cuda-13 capability pointed at nvidia-l4t-arm64-nemo-speech-cpp, which is built on nvcr.io/nvidia/l4t-jetpack:r36.4.0 and therefore links ggml against CUDA 12. A Jetson whose CUDA 13 runtime is present reports that capability and would have pulled an image with no libcudart.so.12 to dlopen, failing hard at load. That is worse than omitting the key: with no key Capability() falls back to "default" and the host gets a working CPU build. Fixed the way parakeet-cpp and moss-transcribe-cpp already do it, by shipping the second L4T image rather than dropping the key. Nothing prevents building it here: those peers use plain ubuntu:24.04 on ubuntu-24.04-arm with the same Dockerfile.golang as this backend's other rows, and every package in the nemo-speech-cpp apt gate exists on noble arm64. Adds the -nvidia-l4t-cuda-13-arm64-nemo-speech-cpp matrix row and its two index entries, repoints the key on both metas, and rewrites the capability-map comment, which had the reasoning backwards. Also adds the documentary inferBackendPath branch, matching all six sibling *-cpp Go backends. Behaviour is unchanged; the generic golang fallthrough already resolved this backend correctly. The previous commit message said "all seven handlers" of the shared gRPC wrapper. There are eight RPC entry points: seven are guarded by checkModelIdentity and AudioTranscriptionLive is the unguarded eighth, which that message already called out separately. Wording only. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(nemo-speech-cpp): document the backend and list it in the importer Adds docs/content/features/nemo-speech-cpp.md, alongside the audio.cpp page that is its closest sibling, and cross-links it from the speech-to-text, diarization, text-to-speech, backend-type and compatibility-table pages so the backend is reachable from every surface that lists its modalities. The page covers the architecture-to-family table, every option key with a model YAML per family, the translation prefix directive, the acceleration matrix, and the four limitations this backend ships with: Linux-only inverse text normalization, suppressed interim streaming results, the library's default translation context and generation limits, and the absence of gallery entries. knownPrefOnlyBackends gains the backend so it appears in the /import-model dropdown. It stays preference-only and AutoDetect=false: general.architecture lives inside the GGUF where no remote-repo probe can read it, and a translation model carries an ordinary LLM architecture with no NeMo-specific marker. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(nemo-speech-cpp): correct the translation limits, the macOS gap and the TTS conversion Three factual errors found in review, all of them the kind a user would act on. The translation limits were described backwards. Input longer than the 1024-token context is rejected, not truncated: translator.cpp throws "nmt: prompt too long (N tokens) for context 1024", which reaches the caller as a failed request. What is silently cut is the output, by the max_new_tokens loop at 256. The bullet now separates the two and says which one fails quietly. The macOS gap covers TTS text normalization as well. Both directions sit behind the single NEMO_SPEECH_WITH_NORM flag, which the Makefile forces off on Darwin, so tn_dir is as inert there as itn_dir. Neither fails the load: both warn and carry on. pnc_model really is unaffected, since punctuation is compiled in unconditionally. The tn_dir row in the option reference gained the caveat the itn_dir row already had. The TTS conversion procedure produced a model that could not load. It converted MagpieTTS and stopped, leaving no NanoCodec, which the same page lists as required; following it gave "no NanoCodec GGUF found next to ...". Both halves are now there, each with the download that feeds it, so the block runs top to bottom on a clean machine. Also: any negative gpu value pins TTS to the CPU, not only -1, and FLAG_CHAT additionally surfaces the model in the web UI chat picker. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): map every C status, not just the NMT one INVALID_ARGUMENT was translated to codes.InvalidArgument at exactly one of sixteen C call sites. Everywhere else a non-zero status collapsed to codes.Internal, so the same backend answered an unsupported language pair with HTTP 400 and an unknown TTS voice, which is the same class of caller mistake against the same process, with HTTP 500. Status 4 is CANCELLED on the ASR and TTS surfaces and was reported as a backend failure rather than as the consumer having stopped listening. asr.h, tts.h and nmt.h each declare their own status enum and diar.h reuses the ASR one; the values they share agree, and the single divergence is that NMT declares no CANCELLED because nemo_speech_nmt_translate has no callback for a consumer to stop with. That is an absence, not a disagreement, so one table serves all three. status.go carries it, with the header line numbers and a note that a pin bump has to recheck it: purego binds by name and the status crosses as a bare int32, so nothing in the build or the linker can see a drift. New specs cover the whole enum, unknown values, and one real INVALID_ARGUMENT per family driven through the shared objects rather than through the Go mapping asserting against itself. Also add UsecaseChat to this backend's capability entry, which the docs already told operators to set for translation models. chat is a gallery filter key and completion is not, so GET /api/backends/usecases would have greyed the Chat filter out and hidden a Riva-Translate gallery entry from the one filter that fits it. The flag gates no endpoint; it makes the model eligible as the default chat model and puts it in the web UI chat picker, both of which Predict and PredictStream already serve. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): audit the gosec unsafe and file-inclusion sites gosec flags 13 alerts on this backend: one G304 and twelve G103. Each was checked individually rather than blanket-suppressed, and each annotation states what makes that particular site safe. The G304 at audio.go is a false positive. The opened path is filepath.Join of a directory the function just created with os.MkdirTemp and a constant basename; the request-controlled path is the input to AudioToWav and never reaches the open. The twelve G103 sites are the package's three established shapes, and every one was verified against them: cstr and pinPtr take the address of something pinned on the line above and return it one-way (nothing in the package converts either result back, which is what keeps checkptr out of it under -race), and each *Create hands C a stack-local POD config whose uintptr members are cstr allocations or pinPtr addresses held by a pinner the loader unpins only after the call. The two slice-building sites are bounded by construction: DiarSegments is handed exactly len(buf) with the buffer sized under maxDiarSegments and a reported count larger than it rejected rather than sliced to, and the TTS callback copies out a slice whose length is the length the runtime declared for that buffer. Separately, sampleRateOf gets a real fix rather than an annotation. go-audio reads the WAV header's sample rate from an unsigned 32-bit field into an int, so a header claiming more than 2^31-1 passed the "> 0" test and then narrowed to a NEGATIVE rate, which the runtime would take as a resampling ratio. AudioToWav cannot produce one today, but that is a property of another package and this function exists precisely because the rate is read back rather than assumed, so the bound is enforced here and pinned by a spec. The four remaining integer narrowings are annotated with the bound that makes each safe: the WAV payload length is already checked against maxWAVDataBytes, the speaker count is bounded by maxDiarSegments, and the two segment ids are the proto's own int32 wire type. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): skip the CUDA-only ggml patch series on darwin The macOS backend build died in patch-ggml: scripts/apply-ggml-patches.sh: line 56: mapfile: command not found make[1]: *** [patch-ggml] Error 127 mapfile is a bash 4 builtin (and its -d flag needs 4.4). macOS ships bash 3.2.57 as /bin/bash and GitHub's runner images add no newer one, so the bare `bash` the recipe resolves from PATH cannot run upstream's script. Rather than hunt for a capable bash that the runner does not have, drop the step where it does nothing. ggml-patches/ is a CUDA series: every kernel it adds is under src/ggml-cuda/, and its whole footprint outside that directory is an op enum plus prototype in include/ggml.h, the constructor and a name-table entry in src/ggml.c, and two ggml-cpu lines that make the CUDA-only op report unsupported and abort. Nothing it touches is compiled into a Metal kernel or changes a CPU one. The project's own references to patch-only ggml symbols sit behind NEMO_SPEECH_FUSED_RELPOS_ATTN and NEMO_SPEECH_FASTCONFORMER_CUDA_FUSIONS, which cmake already forces OFF without GGML_CUDA, or behind NEMO_SPEECH_GGML_PATCHED itself, which guards a GGML_TENSOR_FLAG_Q8_PLANAR write that a non-CUDA buffer throws before reaching. So passing NEMO_SPEECH_GGML_PATCHED=OFF costs the Metal build nothing, and it is required once the series is skipped: that flag is what stops the ASR sources referencing a tensor flag stock ggml does not define. This is upstream's own Metal configuration. Its metal-* and vulkan-* CMake presets inherit the cpu-* ones, which set NEMO_SPEECH_GGML_PATCHED to OFF; docker/Dockerfile and scripts/windows/build.ps1 do the same for their non-CUDA targets. LocalAI's Makefile never passed the flag at all and so inherited the CUDA default everywhere. Linux is untouched and keeps applying the series, including its idempotency and its hard failure on a patch that does not apply. The gate is the same uname test the WITH_NORM block above already uses, and both branches keep the order-only clone prerequisite, which on a WITH_NORM=OFF tree is the only thing that pulls sources/ in. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): restore std::binary_function for MeCab on libc++ NEMO_SPEECH_TTS_WITH_JA=ON compiles Open JTalk's bundled MeCab, and mecab/src/dictionary.cpp derives a comparator from std::binary_function, which C++17 removed. libstdc++ still ships it as deprecated-but-present under -std=gnu++17, so Linux never notices. libc++ compiles it out and the macOS arm64 build dies with "no template named 'binary_function' in namespace 'std'". This is ours, not an upstream regression: upstream defaults both NEMO_SPEECH_TTS_WITH_JA and NEMO_SPEECH_TTS_WITH_ZH to OFF and the OSS drop carries no CI at all, so that target is never built there. Upstream does already carry the equivalent workaround for MSVC's STL (_HAS_AUTO_PTR_ETC plus /FIfunctional) but has no libc++ branch. libc++ gates the two templates on _LIBCPP_ENABLE_CXX17_REMOVED_UNARY_BINARY_FUNCTION, and has since LLVM 16, older than any clang Xcode still ships. The name is the whole problem: _LIBCPP_ENABLE_CXX17_REMOVED_BINDERS covers bind1st, bind2nd, ptr_fun and mem_fun and not unary_function or binary_function, and the umbrella _LIBCPP_ENABLE_CXX17_REMOVED_FEATURES no longer exists in libcxx at all. A wrong name preprocesses fine and fixes nothing. Applied through CMAKE_CXX_FLAGS rather than to the one target, because the tokenizer CMakeLists is upstream's and sources/ is a pinned checkout. Project-wide is also the safer scope: the macro decides whether libc++'s internal __binary_function alias resolves to std::binary_function or to __binary_function_keep_layout_base, a base class of std::less and friends, so defining it for a subset of translation units would give those class templates two spellings in one binary. Both bases are empty and, at C++17, carry identical members, so the define changes no layout and no ABI. Darwin only. On Linux the branch is unreachable and the macro is not a name libstdc++ knows, so it would be inert even if taken; a Linux configure with the flag forced on puts it on all 23 C++ TUs of nemo_speech_openjtalk_frontend including dictionary.cpp at -std=gnu++17, and on none of the 16 C TUs. Mandarin needs nothing: cppjieba v5.6.7 and limonp have no removed C++17 constructs left (limonp replaced std::not1 and std::bind2nd with lambdas) and cppjieba's own CI builds macos-14 and macos-latest at C++11 through C++20. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): repair OpenFST's FstImpl::operator= for gcc-14 The first WITH_NORM=ON build failed compiling fst_normalizer.cpp against the installed OpenFST 1.8.3 headers: fst.h:690:59: error: no match for 'operator=' (operand types are 'std::unique_ptr<fst::SymbolTable, ...>' and 'fst::SymbolTable*') FstImpl's copy-assignment operator assigns the raw pointer returned by SymbolTable::Copy() straight to a std::unique_ptr member. No C++ standard allows that, so the line is ill-formed everywhere; it survived because nothing instantiates FstImpl::operator= and gcc up to 13 only checks a template member's body when it is instantiated. gcc 14 resolves non-dependent operator expressions at template definition time, so it rejects the line in any translation unit that includes <fst/fst.h>. The CI diagnostic confirms the phase: it reads "In member function", not "In instantiation of", and carries no instantiation backtrace. That is why this surfaces only here. build_itn_deps.sh compiles OpenFST with gcc-12 and upstream's own images build the runtime with gcc-13, so neither compiler reaches the check; backend/Dockerfile.golang installs gcc-14 and promotes it with update-alternatives, and fst_normalizer.cpp is the one translation unit in this backend that includes OpenFST. Fix it in the installed ITN prefix, which is the only copy the cmake build compiles against, using the same .reset() spelling FstImpl::SetInputSymbols already uses for the identical operation. libfst.so is linked before this runs and cannot contain the function, since no compiler could ever have emitted it, so there is no ABI or ODR consequence. The rule is guarded on both sides so a pin bump to a fixed OpenFST fails loudly rather than silently no-opping. Verified with a real gcc 14.2: the CI error reproduces byte for byte from a file whose entire content is '#include <fst/fst.h>', and gcc 14 reports exactly two errors over the whole OpenFST include closure this backend uses, both of them these two lines. After the patch that closure compiles clean under gcc-14 with the target's own flags. The step is reachable only under WITH_NORM=ON, so 'make -n stage-libs WITH_NORM=OFF' mentions neither it nor the ITN build, and darwin, which defaults WITH_NORM to OFF, never evaluates it. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nemo-speech-cpp): install cmake 3.31 on bases that ship less than 3.26 The JetPack r36.4.0 row dies on the first line of NeMo-Speech.cpp's CMakeLists.txt: CMake Error at CMakeLists.txt:3 (cmake_minimum_required): -- Configuring incomplete, errors occurred! Upstream opens with cmake_minimum_required(VERSION 3.26). That base image is Ubuntu 22.04 jammy, whose apt cmake is 3.22.1, so configure aborts before it reads a single one of the backend's -D flags. Every other Linux row in this block is noble, which ships 3.28 and clears the bar, so the failure is one base image wide rather than a code problem. Everything before it on that row had already worked, including the OpenFST and Sparrowhawk ITN build. No other Go backend needs this. parakeet-cpp and moss-transcribe-cpp share the same JetPack base and both declare cmake_minimum_required(VERSION 3.18), and nothing in the repo installs a cmake newer than the distro's, so there is no existing pattern to reuse. Nothing depends on jammy's cmake staying 3.22 either: build_itn_deps.sh never invokes cmake at all, since OpenFST and Sparrowhawk are autotools builds. Kitware's release tarball rather than their APT repo or pip. The tarball is a pinned URL with a published checksum, so an upstream release cannot change what lands here. The APT repo does carry jammy arm64, but it serves a moving latest that today is CMake 4.4, and 4.x drops compatibility with cmake_minimum_required below 3.5, which vendored third_party subprojects still declare; pinning it there would mean tracking Kitware's Debian revision string instead of an upstream version. pip would drag a Python toolchain into a backend that has none. 3.31.12 is the last 3.x release, so it clears 3.26 while keeping the CMake 3 policy surface, and it stays close to the 3.28 the green noble rows already use. The binaries need only glibc 2.17 and carry no libstdc++ DT_NEEDED, well under jammy's 2.35. doc/, man/, ccmake and cmake-gui are not extracted; the final image is FROM scratch, but there is no reason to page 100 MB of Qt GUI and docs through the CI cache. Gated on the installed cmake actually being older than 3.26, so the rows that already build green keep configuring with exactly the cmake they use today, and folded into the existing ${BACKEND} block rather than added as a new instruction, so no other Go backend image gains a layer and nothing above the Vulkan SDK, CUDA, Go and protoc layers moves. The symlink lands in /usr/local/bin and shadows apt's cmake. Unlike the protoc shadowing that broke Sparrowhawk earlier in this series that is inert: protoc has to agree with the libprotobuf headers it generates against, whereas cmake links nothing into the product and has no ABI relationship with anything in the image, and it resolves the symlink back to /opt to find its own Modules/ tree, so a 3.31 binary can never read 3.22's modules. The version test avoids $(...) deliberately. BuildKit delivers a RUN heredoc through an outer shell with an unquoted delimiter, so a command substitution runs there, too early, in a container where the files it reads do not exist yet, and its empty output is pasted into the script; the first draft took the install branch on every row because of it. Verified by building the block against nvcr.io/nvidia/l4t-jetpack:r36.4.0 arm64 under qemu, the row's actual base image: cmake 3.22.1 detected, tarball checksum verified, 3.31.12 installed, and a cmake_minimum_required(VERSION 3.26) project configures with -G Ninja and builds, with CMAKE_ROOT resolving to /opt/cmake/share/cmake-3.31. Same on ubuntu:22.04 amd64 and arm64. ubuntu:24.04 skips the install, gains no /opt/cmake and still configures on /usr/share/cmake-3.28. The NeMo-Speech.cpp compile itself on JetPack CUDA 12 is not reproducible here and remains for CI. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-08-07 12:32:39 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	cbc916aec2	fix(ci): build the native engine in a layer the registry cache can restore (#11221) ... Dockerfile.golang builds 215 of the 434 matrix entries: every ggml/C++ engine wrapped in Go. Each of those Makefiles clones an upstream repo at a pinned SHA and compiles it once per SIMD variant (depth-anything-cpp builds four: avx, avx2, avx512, fallback), and those variant targets depend only on the clone. They cannot observe a change anywhere else in the LocalAI tree. The compile sat below `COPY . /LocalAI`, so any edit anywhere invalidated it and recompiled C++ that had not changed. Move it above that COPY, behind a copy of only the backend's own directory. This lands the compile in the part of the image the registry cache already restores. Measured on two real CI builds of this Dockerfile (jobs 90551029008 and 90551028904, both fresh runners): 13 of 17 layers CACHED from quay.io/go-skynet/ci-cache. The uncached tail is exactly `COPY . /LocalAI`, the git-config RUN and the build RUN. Putting the engine above the COPY moves it from the uncached tail into the cached region. Local measurement, depth-anything-cpp CPU, rebuild after editing a Go file outside the backend: master 78s, 216 C++ objects compiled this change 25s, 0 C++ objects compiled (67% faster) Note what this deliberately is not. An earlier attempt wired a --mount=type=cache ccache into the same RUN. BuildKit does not export cache mounts to a registry cache, so that measured well locally and is a no-op in CI (see the ccache section of .agents/ci-caching.md). This change relies only on ordinary layer caching, which the 13-of-17 figure above shows already works here. The layer copies the backend's whole directory rather than just the Makefile: the CMake targets also need CMakeLists.txt and the file list differs per backend. The cost is that editing a backend's own Go sources invalidates its engine layer. The expensive cases are unaffected, since a shared-build-input or backend.proto change, the weekly full-matrix cron and a tag push all rebuild every backend while touching none of their directories. Scoped to one backend for now: only depth-anything-cpp gains the `engine` target. The other 27 fall through the `make -n engine` guard and build exactly as before, verified against local-store and silero-vad. Rolling the target out to the remaining 12 backends that define VARIANT_TARGETS is mechanical once this is confirmed against the registry cache on master. One caveat on merge: inserting layers shifts the cache keys, so the first build of each entry after this lands is a full miss. It pays for itself on the second. Assisted-by: Claude:opus-5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-30 14:39:13 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	2ecf893c6c	fix(sherpa-onnx): install cuDNN in the CUDA builder so the package can bundle it (#11145) ... sherpa-onnx links onnxruntime's CUDA execution provider, and libonnxruntime_providers_cuda.so carries cuDNN as a hard DT_NEEDED. The onnxruntime GPU tarball ships no cuDNN of its own, and Dockerfile.golang only installs libcudnn9 on the arm64 + CUDA 13 branch, so the amd64 CUDA builders have none at all. Since #10946 added the packaging guard, that combination is fatal rather than silent: package-gpu-libs.sh reports 'cuDNN: venv=absent system=absent -> bundle=detect', correctly detects the reference, finds nothing to copy and refuses to emit the package. Both -gpu-nvidia-cuda-12-sherpa-onnx and -gpu-nvidia-cuda-13-sherpa-onnx have failed to build since 2026-07-19, so neither image has been published. Before the guard existed they shipped without cuDNN and failed at load time instead. Install the runtime package for this backend only. The auto-detection bundles solely what a package references, so no other backend would grow, but every Go CUDA builder would pay ~1.1 GB of layer and registry cache for a library ggml never calls. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-27 22:39:37 +02:00
LocalAI [bot]andEttore Di Giacinto	de2ec2f136	feat(backends): add voice-detect + face-detect ggml backends (replace Python insightface/speaker-recognition) (#10441) ... * feat(voice-detect): add Go purego backend for voice-detect.cpp Add backend/go/voice-detect implementing the Backend gRPC voice subset (VoiceEmbed/VoiceVerify/VoiceAnalyze) over libvoicedetect.so via purego, mirroring the parakeet-cpp / omnivoice-cpp backends. The flat voicedetect_capi C ABI is dlopen'd cgo-less; malloc'd string and float-vector returns are owned by Go and released through the matching capi free functions, with the per-ctx last error surfaced into Go errors. Calls are serialized via base.SingleThread since the C context is not reentrant. Proto field mapping: - VoiceEmbed: VoiceEmbedRequest.audio (path) -> embed_path -> Embedding+Model. - VoiceVerify: audio1/audio2 + threshold (<=0 falls back to the verify_threshold option, default 0.25) -> verify_paths -> verified/distance/ threshold/confidence/model/processing_time_ms. - VoiceAnalyze: audio (path) -> analyze_path_json; the JSON age/gender/emotion document maps to a single VoiceAnalysis segment (start/end 0; gender "label" -> dominant_gender with the remaining float scores as the gender map; emotion label/scores -> dominant_emotion/emotion). The Makefile pins voice-detect.cpp to 47546430, clones+builds libvoicedetect.so with ggml static-linked (PIC, GGML_NATIVE off) so dlopen needs no external libggml/libvoicedetect; ldd on the artifact shows only system libs. Ginkgo tests cover option parsing and analyze-JSON mapping; embed/verify smoke specs gate on VOICEDETECT_BACKEND_TEST_MODEL + VOICEDETECT_BACKEND_TEST_WAV. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(voice-detect): wire backend into index, gallery and build Register the voice-detect.cpp speaker-recognition + voice-analysis backend (added in Voice-INT-A) into LocalAI's distribution surfaces, mirroring the ced backend (the closest mudler C++/ggml audio analogue): - backend/index.yaml: add the &voicedetect meta-backend (capabilities platform map, no top-level uri) plus the full set of concrete per-arch image entries (cpu/cuda12/cuda13/metal/rocm/sycl/vulkan/l4t and the -development variants). Referential integrity audited - every alias target resolves. - gallery/index.yaml: add 5 model entries on backend voice-detect - ECAPA-TDNN, WeSpeaker ResNet34, 3D-Speaker ERes2Net, CAM++ and the wav2vec2 age/gender/emotion analyze model. The engine architecture is read from GGUF metadata (voicedetect.arch) at load. GGUF artifacts are not yet published: each files: entry points at the intended mudler/voice-detect-gguf location with a TODO to fill sha256 after upload (no fabricated hashes). - .github/backend-matrix.yml: add the linux build matrix block + the darwin metal entry mirroring ced. - .github/workflows/bump_deps.yaml: track mudler/voice-detect.cpp via VOICEDETECT_VERSION (pin 47546430, = 4754643). - core/config/backend_capabilities.go: register voice-detect in the backend capability map (VoiceVerify/VoiceEmbed/VoiceAnalyze -> speaker_recognition), mirroring speaker-recognition. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(face-detect): add purego Go backend for face-detect.cpp Add the LocalAI Go backend that dlopens libfacedetect.so (the flat facedetect_capi_* C-ABI) via purego, mirroring the sibling voice-detect backend. Implements the Face subset of the Backend gRPC service: - Embeddings(PredictOptions): Images[0] base64 -> temp file -> embed_path -> L2-normalized ArcFace embedding. - Detect(DetectOptions): src -> detect_path_json -> Detection boxes (class_name "face", [x1,y1,x2,y2] -> x/y/w/h). - FaceVerify(FaceVerifyRequest): two images + threshold + anti_spoof -> verify_paths; best-effort img areas via detect. - FaceAnalyze(FaceAnalyzeRequest): img -> analyze_path_json -> per-face age + gender ("M"/"F" normalized to "Man"/"Woman"). The Makefile pins face-detect.cpp to 636a1963 and builds the shared lib with ggml + vendored libjpeg-turbo static (PIC), so the .so is ldd-clean (no libggml) and exports only facedetect_capi_* (no jpeg_ symbols). Gated Ginkgo e2e mirrors voice-detect. Note for the gallery-wiring task: backend registration (index.yaml, gallery, core/config/backend_capabilities.go) is intentionally not touched here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(voice-detect): replace em dashes in net-new descriptions Project style forbids em/en dashes. Replace the three U+2014 chars introduced by the voice-detect gallery/index wiring with `-`/`:`. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(face-detect): wire backend into index, gallery and build Register the face-detect.cpp face detection / embedding / verification / analysis backend (added in Face-INT-A) into LocalAI's distribution surfaces, mirroring the voice-detect wiring (the closest mudler C++/ggml recognition analogue): - backend/index.yaml: add the &facedetect meta-backend (capabilities platform map, no top-level uri to avoid the meta-backend gotcha) plus the full set of concrete per-arch image entries (cpu/cuda12/cuda13/ metal/rocm/sycl-f16/sycl-f32/vulkan/l4t and the -development variants), 22 entries. Referential integrity audited: every alias target resolves. - gallery/index.yaml: add 4 model entries on backend face-detect - face-detect-buffalo-l/m/s (insightface SCRFD + ArcFace/MBF, NON-COMMERCIAL) and face-detect-yunet-sface (OpenCV-Zoo YuNet + SFace, APACHE-2.0, the commercial-friendly alternative). The detector/embedder architecture is read from GGUF metadata (facedetect.arch) at load; only the real verify_threshold option is set (0.35 buffalo, 0.363 sface). GGUF artifacts are not yet published: each files: entry points at the intended mudler/face-detect-gguf location with a TODO to fill sha256 after upload (no fabricated hashes). - core/config/backend_capabilities.go: register face-detect in the backend capability map (Embedding/Detect/FaceVerify/FaceAnalyze -> face_recognition), mirroring insightface. - .github/backend-matrix.yml: add the linux build matrix block + the darwin metal entry mirroring voice-detect. - .github/workflows/bump_deps.yaml: track mudler/face-detect.cpp via FACEDETECT_VERSION (pin 636a1963). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(recon): voice-detect metal build branch + face-detect gallery usecases Add the missing metal BUILD_TYPE branch to the voice-detect Makefile forwarding -DVOICEDETECT_GGML_METAL=ON, mirroring face-detect, so the darwin metal CI artifact is built with the Metal backend instead of CPU-only. Expand the 4 face-detect gallery models' known_usecases to [face_recognition, detection, embeddings] to match the backend capabilities map and the mirrored insightface-buffalo entries, so auto-selection for /v1/detect and /embeddings works. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * docs(recon): document voice-detect and face-detect ggml backends Document the new standalone C++/ggml biometric backends as the recommended/default option for face and voice recognition, keeping the existing Python insightface / speaker-recognition backends framed as the legacy path. - features/face-recognition.md: add a face-detect (ggml) backend section with the gallery entries (buffalo-l/m/s non-commercial, yunet-sface Apache-2.0), licensing, and verify/detect/analyze quickstart. - features/voice-recognition.md: add a voice-detect (ggml) backend section with the gallery entries (ecapa-tdnn, wespeaker-resnet34, eres2net, campplus speaker recognizers; emotion-wav2vec2 non-commercial analyze head) and quickstart. - reference/compatibility-table.md: add face-detect.cpp and voice-detect.cpp rows to the Vision, Detection & Recognition table. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(gallery): publish recon backend GGUF uris + sha256 Fill in the published HuggingFace GGUF uris and verified sha256 for the 9 recon gallery entries (voice-detect-* and face-detect-*), and remove the TODO publish markers. Correct the eres2net, campplus, and emotion-wav2vec2 uris to the actual published filenames. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(gallery): re-embed buffalo anti-spoof + add audeering age/gender voice model Update the 3 buffalo face-detect GGUF sha256 (anti-spoof ensemble now embedded and re-uploaded under the same filenames/uris) and note the FaceVerify anti_spoof request flag in each description. Add a new voice-detect-age-gender-wav2vec2 gallery entry mirroring the emotion model. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(gallery): add face-detect-buffalo-sc and antelopev2 packs Add gallery entries for two newly-published insightface face packs on the face-detect backend: buffalo_sc (smallest pack, SCRFD-500M + small ArcFace) and antelopev2 (higher-accuracy, SCRFD-10G + ArcFace glint360k R100, 512-d). Both are non-commercial research-only. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(recon): honor LocalAI per-model threads in voice/face-detect backends LocalAI spawns one backend process per model and serves requests concurrently, so the engines' own min(hardware_concurrency, 8) default can oversubscribe cores. Forward the per-model Threads value from the gRPC LoadModel options into the engine via VOICEDETECT_THREADS / FACEDETECT_THREADS (read at backend construction) before the capi load. A non-positive Threads is treated as unset, leaving the engine default. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump backend pins to CPU-optimized engine commits voice-detect.cpp -> 0d9c1b3 (radix-2 FFT FBank, threads, flash attn + cached pos-conv); face-detect.cpp -> 523aee1 (thread-gated direct conv, threads). Brings the CPU optimizations into the LocalAI backend builds. GGUF format and parity unchanged, so the published HF GGUFs remain valid. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump backend pins to round-2 CPU-optimized engines voice-detect.cpp -> fe7e6a3 (ERes2Net 1x1->mul_mat, CAM++ layout+context, wav2vec2 conv-LN, ECAPA capture-drop, AVX512 dispatch opt-in); face-detect.cpp -> 9c8adb7 (AVX2 Winograd F(2x2,3x3) for SCRFD/ArcFace 3x3 convs, ArcFace BN-fold). Parity unchanged (cosine=1.0); GGUF format unchanged, HF GGUFs valid. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump backend pins to round-3 Winograd engines voice-detect.cpp -> 45122ec (Winograd F(2x2,3x3) for WeSpeaker/ERes2Net 3x3 convs, -22%/-20% @8t); face-detect.cpp -> cd5c962 (Winograd F(4x4,3x3) for SCRFD large maps, -22% @1t on top of F(2x2), more load-stable). Parity held (cosine=1.0); GGUF format unchanged, HF GGUFs valid. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump backend pins to round-4 Winograd engines (CPU opt complete) voice-detect.cpp -> d2839ca (CAM++ FCM 2D convs through Winograd, -15.5%/-10.3%); face-detect.cpp -> c1db23d (AVX2-vectorized Winograd tile transforms, SCRFD detect -14%/-9.6%). Final CPU optimization round; the conv-kernel lever class is now exhausted (parity held cosine=1.0; GGUF/parity unchanged, HF GGUFs valid). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump face-detect pin to deep-kernel engine (7ae5c4d) face-detect.cpp -> 7ae5c4d: register-blocked winograd-domain GEMM microkernel (2.8x isolated GFLOP/s), AVX-512 zmm evolution behind runtime CPUID dispatch (ship-safe, AVX2 fallback bit-identical), bias/relu fused into the winograd output transform, and SFace Conv+BN fold + bias/PReLU fusion. SCRFD detect ~1.4x faster end-to-end vs the round-4 baseline; parity bit-exact; portable single binary (function-multiversioned, no global -mavx512f). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump voice-detect pin to ECAPA operand-order win (e9c56ae) voice-detect.cpp -> e9c56ae: weight-as-src0 mul_mat order in ECAPA's F32 conv1d_same (routes through tinyBLAS sgemm); ECAPA embed 1.67x @1t / ~1.3x @8t, parity cosine=1.0. Isolated to encoder.cpp (ECAPA-only); ERes2Net/CAM++/WeSpeaker do not call conv1d_same so are provably unaffected. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to FMA-throughput engines (voice f7b9f89, face 2d2d5f0) face -> 2d2d5f0: route ArcFace 3x3 body convs through the AVX-512 winograd microkernel (kWinoMinSize 80->14); ArcFace 1.62x @1t, SCRFD detect to 0.966 of MLAS @1t, no regression. voice -> f7b9f89: runtime-CPUID-dispatched AVX-512 winograd-GEMM microkernel (ship-safe, AVX2 fallback bit-identical); WeSpeaker 1.90x @1t. Parity cosine=1.0 throughout; portable single binaries. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to MLAS-class direct-conv engines (voice 7ecfd07, face be22d67) Hand-tuned nChw16c AVX-512 register-tiled direct-conv microkernel (~263 GFLOP/s, within 6-7% of MLAS per-op efficiency), runtime-CPUID-dispatched + AVX2 fallback, fused bias/relu. voice 7ecfd07: default 3x3-s1 kernel for WeSpeaker (+37%/+32%) + ERes2Net, CAM++ pinned to Winograd. face be22d67: shape-gated to the ArcFace recognizer body (+25-27% @8t); SCRFD detector stays on Winograd (no regression). Parity cosine=1.0 / detect <=1px on AVX-512 + AVX2 paths. Portable single binaries. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump voice pin to Phase-A blocked backbone (f4e7eef) WeSpeaker ResNet34 runs as one nChw16c blocked island (2 reorders/forward vs ~60) on AVX-512, default; per-conv directconv fallback on AVX2. +2.9% @1t / +17-19% @8t vs per-conv directconv, parity cosine=1.0. The conv microkernel is already FMA-bound near peak (~0.86-0.98x MLAS-implied); residual to MLAS is sub-peak edge + non-conv tail, documented in docs/cpu-optimization.md. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to breadth blocked-backbone (voice 7f66871, face d80092b) voice 7f66871: AVX2-vectorized (ymm) blocked island - AVX2-only hosts now run the blocked backbone for WeSpeaker (2.3x over per-conv-AVX2, cosine=1.0); ERes2Net stays per-conv (blocked regresses, opt-in only); CAM++ Winograd-pinned. face d80092b: ArcFace recognizer blocked island, AVX-512 default (-13% @8t, ~0.90x MLAS, the closest conv result), auto per-conv on AVX2; SCRFD untouched on Winograd (0 island invocations during detect). Parity cosine=1.0 / detect <=1px throughout. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to small-spatial + stem conv kernels (voice 99b1804, face 47fdab6) Measured-gap-driven conv kernels: small-spatial (fill the register tile when output width <= tile width) + small-IC stem + strided-1x1/downsample recovery. ArcFace recognizer 0.57 -> 0.70x MLAS @1t (the closest conv model), WeSpeaker 0.65 -> 0.79x @1t. Parity cosine=1.0 / detect <=1px. The OC-block-sharing lever was a measured dead-end (deep stride-1 is L3-weight-bandwidth bound, not read-port bound) and was NOT shipped. Kernel ceiling reached; further gap needs an algorithm-class change (cache-blocked weight-stationary GEMM, or q8 weights). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to GPU persistent-graph + multi-model-safe cache (voice 45d2e6b, face 0a4799a) GPU wins (CUDA/ggml backend, no CPU-path change): persistent per-shape graph+context cache in Backend::compute() eliminates the per-call cudaGraph re-instantiation churn -> wav2vec2 emotion+age-gender now AT GPU parity with torch-cuDNN on GB10 (0.97-0.98x), CAM++ -5.7ms; bit-identical parity. Cache hardened multi-model-safe (invalidate-on-free keyed by the ModelLoader weights buffer) so LocalAI multi-model hosting cannot stale-hit. Conv models still trail cuDNN (im2col-materialization-bound) - cuDNN implicit-GEMM lever next. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to cuDNN-conv-capable engines (voice b6e4356, face 6107a24) Adds the opt-in cuDNN implicit-GEMM conv path (VOICEDETECT_GGML_CUDNN / FACEDETECT_GGML_CUDNN, DEFAULT OFF -> zero build/runtime dep until enabled). On GPU it kills the im2col-materialization bottleneck and reaches torch-cuDNN parity on the spill-bound convs: SCRFD detect 14.8->6.4ms (2.3x, ~parity), WeSpeaker ~parity, ERes2Net beats torch (1.10x); ArcFace/CAM++ neutral (no spill). Parity exact (SCRFD <=1px, cosine=1.0). To USE it in LocalAI, the CUDA backend build must enable the flag AND bundle libcudnn - deferred until a cuDNN-bundled GPU image; flag stays OFF here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(recon): enable cuDNN conv path on arm64+CUDA13 recon backends The voice-detect.cpp / face-detect.cpp engines have an opt-in cuDNN implicit-GEMM conv path behind VOICEDETECT_GGML_CUDNN / FACEDETECT_GGML_CUDNN (default OFF) that kills im2col on the GPU and reaches torch-cuDNN parity (SCRFD 2.3x, WeSpeaker/ERes2Net parity), measured on the GB10 (arm64, CUDA 13, sm_121a). Enable it for the CUDA build, but only where cuDNN actually ships: the arm64 + CUDA 13 image (GB10/Jetson/L4T). x86 CUDA images carry no cuDNN, so flipping it on globally for BUILD_TYPE=cublas would be a link failure. The Makefiles gate on CUDA_MAJOR_VERSION=13 + arch (TARGETARCH from the matrix/Docker build, uname -m fallback for local builds). backend/Dockerfile.golang already installs the runtime libcudnn9-cuda-13 in the arm64+CUDA13 apt block; add the matching libcudnn9-dev-cuda-13 so the build-time link resolves. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump voice-detect pin to ERes2Net blocked-default (30beecd) Defaults VD_ERES2NET_BLOCKED ON: routes the ERes2Net Res2Net body through the blocked nChw16c AVX-512 directconv island instead of the 1x1 mul_mat fast path (CONT-transpose + skinny low-K GEMM). On the shipped GGML_NATIVE=OFF build (ggml mul_mat is AVX2-only) this wins ~2x at every thread count (2.07x@1t, 2.2x@4t, 2.05x@8t); pure-AVX2 fallback still 1.3-1.62x. Parity exact (cosine=1.000000 vs golden), so registered voices + verify/identify thresholds are unaffected. The prior default-OFF rested on a stale comment whose 23pct regression only held on the non-shipping GGML_NATIVE=ON build. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * docs(readme): announce native voice-detect + face-detect backends in Latest News Add a Latest News entry for the new from-scratch C++/ggml biometric backends (voice-detect.cpp + face-detect.cpp) that replace the Python insightface and speaker-recognition backends: no Python/onnxruntime at inference, self-contained GGUF, bit-exact parity, GPU cuDNN parity. Mirrors the parakeet.cpp / locate-anything.cpp native-backend news entries. Refs PR #10441. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): re-pin to the squashed engine release commits The voice-detect.cpp and face-detect.cpp histories were squashed to a single release commit, which orphaned the previous pins (voice 30beecd, face 6107a24). Re-pin to the new single-commit SHAs (voice 3d51077, face 06914b0); the tree is identical, so the backend build is unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-28 09:29:08 +02:00
Richard Palethorpe	606128e4e9	feat(vulkan): make Vulkan backends self-contained on the GPU (#10404) ... Vulkan backends bundled their own loader and ICD manifests but neither the Mesa driver the manifests point at nor a way to make the loader find them, so on a runtime base image without Mesa the loader enumerated zero devices and the GPU silently fell back to CPU (only NVIDIA worked, since its ICD is injected by the container toolkit). - scripts/build/package-gpu-libs.sh: for each installed ICD manifest, bundle the driver .so its library_path names — no hard-coded, platform-dependent soname list — plus that driver's ldd dependencies, skipping manifests whose driver isn't installed. Rewrite each library_path to a bare soname so the bundled driver resolves via the LD_LIBRARY_PATH run.sh already sets. - .docker/install-base-deps.sh, backend/Dockerfile.golang, backend/Dockerfile.python: install mesa-vulkan-drivers in every Vulkan builder so the driver + manifests exist to be packaged (the LunarG SDK ships only the loader and shader tooling). - pkg/model/process.go: when a backend ships vulkan/icd.d/, point the loader at it via VK_DRIVER_FILES/VK_ICD_FILENAMES at launch (no-op otherwise). Covered by pkg/model/process_vulkan_test.go. - backend/go/parakeet-cpp/package.sh: complete the L0 stub (was missing the libc-family ldd walk + GPU-lib packaging) by mirroring whisper, so the vulkan-parakeet image actually bundles its GPU runtime. Assisted-by: Claude Code:claude-opus-4-8 Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-19 17:16:33 +02:00
LocalAI [bot]andEttore Di Giacinto	50dea8c983	feat(crispasr): bundle espeak-ng and add piper TTS voices to the gallery (#10283) ... CrispASR's piper backend phonemizes non-English text via espeak-ng (dlopen, the MIT-clean path; English uses a built-in G2P). The FROM scratch crispasr image shipped none of it, so non-English piper voices loaded but failed synthesis with "phonemization failed". Bundle the espeak-ng runtime so they work: - Dockerfile.golang: install espeak-ng-data + libespeak-ng1 and its libpcaudio0 / libsonic0 deps in the crispasr builder (espeak's dlopen fails without the latter two). - package.sh: copy libespeak-ng.so.1, libpcaudio.so.0, libsonic.so.0 into package/lib/ and the espeak-ng-data dir into the package root. - run.sh: export CRISPASR_ESPEAK_DATA_PATH so the bundled data is found. Add 9 single-speaker piper voices (de/en/it, incl. Italian paola + riccardo) to the gallery, run through backend:piper, hosted at LocalAI-Community/piper-voices-GGUF (converted from rhasspy/piper-voices with CrispASR's convert-piper-to-gguf.py). Only single-speaker low/medium voices are included; the engine does not yet support multi-speaker or high-quality piper decoders. All 9 verified end-to-end: each synthesizes a WAV at the model's native sample rate using only the image-bundled espeak payload. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-12 23:10:30 +02:00
Ettore Di Giacinto	e86ade54a6	feat(api): add /v1/audio/diarization endpoint with sherpa-onnx + vibevoice.cpp (#9654) ... * feat(api): add /v1/audio/diarization endpoint with sherpa-onnx + vibevoice.cpp Closes #1648. OpenAI-style multipart endpoint that returns "who spoke when". Single endpoint instead of the issue's three-endpoint sketch (refactor /vad, /vad/embedding, /diarization) — the typical client wants one call, and embeddings can land later as a sibling without breaking this surface. Response shape borrows from Pyannote/Deepgram: segments carry a normalised SPEAKER_NN id (zero-padded, stable across the response) plus the raw backend label, optional per-segment text when the backend bundles ASR, and a speakers summary in verbose_json. response_format also accepts rttm so consumers can pipe straight into pyannote.metrics / dscore. Backends: * vibevoice-cpp — Diarize() reuses the existing vv_capi_asr pass. vibevoice's ASR prompt asks the model to emit [{Start,End,Speaker,Content}] natively, so diarization is a by-product of the same pass; include_text=true preserves the transcript per segment, otherwise we drop it. * sherpa-onnx — wraps the upstream SherpaOnnxOfflineSpeakerDiarization C API (pyannote segmentation + speaker-embedding extractor + fast clustering). libsherpa-shim grew config builders, a SetClustering wrapper for per-call num_clusters/threshold overrides, and a segment_at accessor (purego can't read field arrays out of SherpaOnnxOfflineSpeakerDiarizationSegment[] directly). Plumbing: new Diarize gRPC RPC + DiarizeRequest / DiarizeSegment / DiarizeResponse messages, threaded through interface.go, base, server, client, embed. Default Base impl returns unimplemented. Capability surfaces all updated: FLAG_DIARIZATION usecase, FeatureAudioDiarization permission (default-on), RouteFeatureRegistry entries for /v1/audio/diarization and /audio/diarization, audio instruction-def description widened, CAP_DIARIZATION JS symbol, swagger regenerated, /api/instructions discovery map updated. Tests: * core/backend: speaker-label normalisation (first-seen → SPEAKER_NN, per-speaker totals, nil-safety, fallback to backend NumSpeakers when no segments). * core/http/endpoints/openai: RTTM rendering (file-id basename, negative duration clamping, fallback id). * tests/e2e: mock-backend grew a deterministic Diarize that emits raw labels "5","2","5" so the e2e suite verifies SPEAKER_NN remapping, verbose_json speakers summary + transcript pass-through (gated by include_text), RTTM bytes content-type, and rejection of unknown response_format. mock-diarize model config registered with known_usecases=[FLAG_DIARIZATION] to bypass the backend-name guard. Docs: new features/audio-diarization.md (request/response, RTTM example, sherpa-onnx + vibevoice setup), cross-link from audio-to-text.md, entry in whats-new.md. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * fix(diarization): correct sherpa-onnx symbol name + lint cleanup CI failures on #9654: * sherpa-onnx-grpc-{tts,transcription} and sherpa-onnx-realtime panicked at backend startup with `undefined symbol: SherpaOnnxDestroyOfflineSpeakerDiarizationResult`. Upstream's actual symbol is SherpaOnnxOfflineSpeakerDiarizationDestroyResult (Destroy in the middle, not the prefix); the rest of the diarization surface follows the same naming pattern. The mismatched name made purego.RegisterLibFunc fail at dlopen time and crashed the gRPC server before the BeforeAll could probe Health, taking down every sherpa-onnx test job — not just the diarization-related ones. * golangci-lint flagged 5 errcheck violations on new defer cleanups (os.RemoveAll / Close / conn.Close); wrap each in a `defer func() { _ = X() }()` closure (matches the pattern other LocalAI files use for new code, since pre-existing bare defers are grandfathered in via new-from-merge-base). * golangci-lint also flagged forbidigo violations: the new diarization_test.go files used testing.T-style `t.Errorf` / `t.Fatalf`, which are forbidden by the project's coding-style policy (.agents/coding-style.md). Convert both files to Ginkgo/Gomega Describe/It with Expect(...) — they get picked up by the existing TestBackend / TestOpenAI suites, no new suite plumbing needed. * modernize linter: tightened the diarization segment loop to `for i := range int(numSegments)` (Go 1.22+ idiom). Verified locally: golangci-lint with new-from-merge-base=origin/master reports 0 issues across all touched packages, and the four mocked diarization e2e specs in tests/e2e/mock_backend_test.go still pass. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * fix(vibevoice-cpp): convert non-WAV input via ffmpeg + raise ASR token budget Confirmed end-to-end against a real LocalAI instance with vibevoice-asr-q4_k loaded and the multi-speaker MP3 sample at vibevoice.cpp/samples/2p_argument.mp3: both /v1/audio/transcriptions and /v1/audio/diarization now succeed and return correctly attributed speaker turns for the full clip. Two latent issues surfaced once the diarization endpoint actually exercised the backend with a non-trivial input: 1. vv_capi_asr only accepts WAV via load_wav_24k_mono. The previous code passed the uploaded path straight through, so anything that wasn't already a 24 kHz mono s16le WAV failed at the C side with rc=-8 and the very unhelpful "vv_capi_asr failed". prepareWavInput shells out to ffmpeg ("-ar 24000 -ac 1 -acodec pcm_s16le") in a per-call temp dir, matching the rate the model was trained on; both AudioTranscription and Diarize now route through it. This is the same shape sherpa-onnx uses (utils.AudioToWav), but vibevoice needs 24 kHz rather than 16 kHz so we don't reuse that helper. 2. The C ABI's max_new_tokens defaults to 256 when 0 is passed. That's fine for a five-second clip but not for anything past ~10 s — vibevoice stops mid-JSON, the parse fails, and the caller sees a hard error. Pass a much larger budget (16 384 ≈ ~9 minutes of speech at the model's ~30 tok/s rate); generation stops at EOS so this is a cap rather than a target. 3. As a defensive belt-and-braces, mirror AudioTranscription's existing "fall back to a single segment if the model emits non-JSON text" pattern in Diarize, so partial / unusual model output never produces a 500. This kept the endpoint usable while diagnosing (1) and (2), and is the right behaviour to keep. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * fix(vibevoice-cpp): pass valid WAVs through directly so ffmpeg is not required at runtime Spotted by tests-e2e-backend (1.25.x): the previous fix forced every incoming audio file through `ffmpeg -ar 24000 ...`, which meant the backend container — which does not ship ffmpeg — failed even for the existing happy path where the caller already uploads a WAV. The container-side error was: rpc error: code = Unknown desc = vibevoice-cpp: ffmpeg convert to 24k mono wav: exec: "ffmpeg": executable file not found in $PATH Reading vibevoice.cpp's audio_io.cpp, `load_wav_24k_mono` uses drwav and already accepts any PCM/IEEE-float WAV at any sample rate, downmixes multi-channel input to mono, and resamples to 24 kHz internally. So the only inputs that genuinely need an external converter are non-WAV formats (MP3, OGG, FLAC, ...). Detect WAVs by RIFF/WAVE magic at bytes 0..3 / 8..11 and pass them straight through with a no-op cleanup; everything else still goes through ffmpeg with the same 24 kHz mono s16le target. The result: * Container builds without ffmpeg keep working for WAV uploads (the e2e-backends fixture is jfk.wav at 16 kHz mono s16le). * MP3 and other non-WAV inputs still get the new ffmpeg conversion path so the diarization endpoint stays useful. * If the caller uploads a non-WAV but ffmpeg isn't on PATH, the surfaced error is still descriptive enough to act on. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * fix(ci): make gcc-14 install in Dockerfile.golang best-effort for jammy bases The LocalVQE PR (bb033b16) made `gcc-14 g++-14` an unconditional apt install in backend/Dockerfile.golang and pointed update-alternatives at them. That works on the default `BASE_IMAGE=ubuntu:24.04` (noble has gcc-14 in main), but every Go backend that builds on `nvcr.io/nvidia/l4t-jetpack:r36.4.0` — jammy under the hood — now fails at the apt step: E: Unable to locate package gcc-14 This blocked unrelated jobs: backend-jobs(*-nvidia-l4t-arm64-{stablediffusion-ggml, sam3-cpp, whisper, acestep-cpp, qwen3-tts-cpp, vibevoice-cpp}). LocalVQE itself is only matrix-built on ubuntu:24.04 (CPU + Vulkan), so it doesn't actually need gcc-14 anywhere else. Make the gcc-14 install conditional on the package being available in the configured apt repos. On noble: identical behaviour to today (gcc-14 installed, update-alternatives points at it). On jammy: skip the gcc-14 stanza entirely and let build-essential's default gcc take over, which is what the other Go backends compile with anyway. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-05 15:10:13 +02:00
Richard Palethorpe	bb033b16a9	feat: add LocalVQE backend and audio transformations UI (#9640) ... feat(audio-transform): add LocalVQE backend, bidi gRPC RPC, Studio UI Introduce a generic "audio transform" capability for any audio-in / audio-out operation (echo cancellation, noise suppression, dereverberation, voice conversion, etc.) and ship LocalVQE as the first backend implementation. Backend protocol: - Two new gRPC RPCs in backend.proto: unary AudioTransform for batch and bidirectional AudioTransformStream for low-latency frame-by-frame use. This is the first bidi stream in the proto; per-frame unary at LocalVQE's 16 ms hop would be RTT-bound. Wire it through pkg/grpc/{client,server, embed,interface,base} with paired-channel ergonomics. LocalVQE backend (backend/go/localvqe/): - Go-Purego wrapper around upstream liblocalvqe.so. CMake builds the upstream shared lib + its libggml-cpu-*.so runtime variants directly — no MODULE wrapper needed because LocalVQE handles CPU feature selection internally via GGML_BACKEND_DL. - Sets GGML_NTHREADS from opts.Threads (or runtime.NumCPU()-1) — without it LocalVQE runs single-threaded at ~1× realtime instead of the documented ~9.6×. - Reference-length policy: zero-pad short refs, truncate long ones (the trailing portion can't have leaked into a mic that wasn't recording). - Ginkgo test suite (9 always-on specs + 2 model-gated). HTTP layer: - POST /audio/transformations (alias /audio/transform): multipart batch endpoint, accepts audio + optional reference + params[*]=v form fields. Persists inputs alongside the output in GeneratedContentDir/audio so the React UI history can replay past (audio, reference, output) triples. - GET /audio/transformations/stream: WebSocket bidi, 16 ms PCM frames (interleaved stereo mic+ref in, mono out). JSON session.update envelope for config; constants hoisted in core/schema/audio_transform.go. - ffmpeg-based input normalisation to 16 kHz mono s16 WAV via the existing utils.AudioToWav (with passthrough fast-path), so the user can upload any format / rate without seeing the model's strict 16 kHz constraint. - BackendTraceAudioTransform integration so /api/backend-traces and the Traces UI light up with audio_snippet base64 and timing. - Routes registered under routes/localai.go (LocalAI extension; OpenAI has no /audio/transformations endpoint), traced via TraceMiddleware. Auth + capability + importer: - FLAG_AUDIO_TRANSFORM (model_config.go), FeatureAudioTransform (default-on, in APIFeatures), three RouteFeatureRegistry rows. - localvqe added to knownPrefOnlyBackends with modality "audio-transform". - Gallery entry localvqe-v1-1.3m (sha256-pinned, hosted on huggingface.co/LocalAI-io/LocalVQE). React UI: - New /app/transform page surfaced via a dedicated "Enhance" sidebar section (sibling of Tools / Biometrics) — the page is enhancement, not generation, so it lives outside Studio. Two AudioInput components (Upload + Record tabs, drag-drop, mic capture). - Echo-test button: records mic while playing the loaded reference through the speakers — the mic naturally picks up speaker bleed, giving a real (mic, ref) pair for AEC testing without leaving the UI. - Reusable WaveformPlayer (canvas peaks + click-to-seek + audio controls) and useAudioPeaks hook (shared module-scoped AudioContext to avoid hitting browser context limits with three players on one page); migrated TTS, Sound, Traces audio blocks to use it. - Past runs saved in localStorage via useMediaHistory('audio-transform') — the history entry stores all three URLs so clicking re-renders the full triple, not just the output. Build + e2e: - 11 matrix entries removed from .github/workflows/backend.yml (CUDA, ROCm, SYCL, Metal, L4T): upstream supports only CPU + Vulkan, so we ship those two and let GPU-class hardware route through Vulkan in the gallery capabilities map. - tests-localvqe-grpc-transform job in test-extra.yml (gated on detect-changes.outputs.localvqe). - New audio_transform capability + 4 specs in tests/e2e-backends. - Playwright spec suite in core/http/react-ui/e2e/audio-transform.spec.js (8 specs covering tabs, file upload, multipart shape, history, errors). Docs: - New docs/content/features/audio-transform.md covering the (audio, reference) mental model, batch + WebSocket wire formats, LocalVQE param keys, and a YAML config example. Cross-links from text-to-audio and audio-to-text feature pages. Assisted-by: Claude:claude-opus-4-7 [Bash Read Edit Write Agent TaskCreate] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-04 22:07:11 +02:00
Ettore Di Giacinto	8edac61e57	feat(ci): allow routing apt traffic through an alternate Ubuntu mirror (#9650) ... * feat(ci): allow routing apt traffic through an alternate Ubuntu mirror Adds opt-in APT_MIRROR / APT_PORTS_MIRROR knobs to all Dockerfiles, the Makefile, and CI workflows so we can fail over to a non-canonical Ubuntu mirror when archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com are degraded (recently observed: multi-day DDoS against the default pool). Defaults are empty everywhere — behavior is unchanged unless a mirror is configured. To enable in CI, set the repo-level GitHub Actions variables APT_MIRROR (and APT_PORTS_MIRROR for arm64 builds). Locally: make docker APT_MIRROR=http://azure.archive.ubuntu.com A small POSIX-sh helper in .docker/apt-mirror.sh rewrites both DEB822 (/etc/apt/sources.list.d/ubuntu.sources, Ubuntu 24.04+) and the legacy /etc/apt/sources.list before the first apt-get update. Dockerfile stages load it via RUN --mount=type=bind, so there is no extra layer and no cache invalidation when the script is unchanged. Reusable workflows also rewrite the runner's own /etc/apt sources before any sudo apt-get call. Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apt-mirror): default to the Azure mirror, visible in the workflow source Bakes Azure (http://azure.archive.ubuntu.com / http://azure.ports.ubuntu.com) in as the default for both Docker builds and runner-side apt — rather than hiding the URL behind a GitHub Actions repo variable that's not visible from the source tree. A new composite action at .github/actions/configure-apt-mirror is the single source of truth for runner-side rewrites. Five standalone workflows (build-test, release, tests-e2e, tests-ui-e2e, update_swagger) just `uses: ./.github/actions/configure-apt-mirror`. Three workflows (image_build, backend_build, checksum_checker) keep an inline bash rewrite, because they install/upgrade git via apt *before* the checkout step (so the local composite action isn't loadable yet). The Azure URL is visible in those files too. The `apt-mirror` / `apt-ports-mirror` inputs of the reusable workflows keep their now-Azure defaults — they still feed the Docker build-args block in addition to the inline runner-side rewrite. Callers (image.yml, image-pr.yml, backend.yml, backend_pr.yml) drop the previous `vars.APT_MIRROR` plumbing and rely on those defaults. Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apt-mirror): drop Force Install GIT, consolidate on the composite action The PPA git upgrade ran add-apt-repository ppa:git-core/ppa, which talks to api.launchpad.net — also part of Canonical's infrastructure and currently returning HTTP 504. The Azure mirror only covers archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com, not PPAs. The system git that ubuntu-latest already ships is sufficient for actions/checkout and the build pipeline, so just drop the upgrade. With that gone, the apt-before-checkout constraint disappears too — all three holdouts (image_build, backend_build, checksum_checker) can now switch to ./.github/actions/configure-apt-mirror like the other five. Net: 0 inline apt-mirror blocks, all 8 workflows route through the composite action. Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-03 23:50:13 +02:00
Russell Sim	18e039f305	fix(ci): fix AMDGPU_TARGETS empty-string bypass in hipblas builds (#9626) ... * fix(ci): fix AMDGPU_TARGETS empty-string bypass in hipblas builds 399c1dec wired amdgpu-targets through the backend_build workflow_call interface, intending the input's default value to cover matrix entries that don't specify targets. However, GitHub Actions only applies a workflow_call input default when the caller omits the input entirely. When backend.yml passes `amdgpu-targets: ${{ matrix.amdgpu-targets }}` and the matrix entry has no amdgpu-targets key, the expression evaluates to an empty string, which is treated as an explicit value — bypassing the default. The result is Docker receiving AMDGPU_TARGETS="" which in turn causes Make's ?= default to be skipped (since the variable is already set in the environment, even to empty), and cmake gets -DAMDGPU_TARGETS= with no targets, so the HIP backend compiles for an indeterminate target rather than the intended GPU list. Fix this at two levels: 1. backend.yml: use a || fallback in the expression so that an undefined matrix.amdgpu-targets never reaches the reusable workflow as an empty string. The target list is the canonical default and lives here. 2. backend_build.yml: remove the now-misleading default value from the input declaration. The default never fired due to the above bug, so keeping it implied a guarantee that didn't exist. 3. backend/cpp/llama-cpp/Makefile: add an explicit $(error ...) guard after the ?= assignment so that if AMDGPU_TARGETS is empty (whether from environment or any future CI wiring mistake) the build fails immediately with a clear message rather than silently producing a binary compiled for an unknown GPU target. Assisted-by: Claude Code:claude-sonnet-4-6 Signed-off-by: Russell Sim <rsl@simopolis.xyz> * fix(build): plumb AMDGPU_TARGETS through to Docker builds The docker-build-backend Makefile macro and Dockerfile.golang did not pass AMDGPU_TARGETS to the inner make invocation, so hipblas builds always used the backend Makefile's hardcoded default GPU targets regardless of what was specified via environment or CI inputs. Signed-off-by: Russell Sim <rsl@simopolis.xyz> --------- Signed-off-by: Russell Sim <rsl@simopolis.xyz>	2026-05-02 15:53:14 +02:00
Andreas Egli	1d0de757c3	fix: add hipblaslt library (#9541) ... Signed-off-by: Andreas Egli <github@kharan.ch>	2026-04-24 18:50:03 +02:00
Richard Palethorpe	f9a850c02a	feat(realtime): WebRTC support (#8790) ... * feat(realtime): WebRTC support Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(tracing): Show full LLM opts and deltas Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-03-13 21:37:15 +01:00
Ettore Di Giacinto	bf5a1dd840	feat(voxtral): add voxtral backend (#8451) ... * feat(voxtral): add voxtral backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * simplify Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-02-09 09:12:05 +01:00
Ettore Di Giacinto	7891c33cb1	chore(vulkan): bump vulkan-sdk to 1.4.335.0 (#7981) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-12 07:51:26 +01:00
Ettore Di Giacinto	917c7aa9f3	chore(ci): roll back l4t-cuda12 configurations (#7935) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-08 23:04:33 +01:00
Copilotandmudler	b2ff1cea2a	feat: enable Vulkan arm64 image builds (#7912) ... * Initial plan * Add arm64 support for Vulkan builds in Dockerfiles and workflows Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>	2026-01-07 21:49:50 +01:00
Richard PalethorpeandAlessandro Sturniolo	e6ba26c3e7	chore: Update to Ubuntu24.04 (cont #7423) (#7769) ... * ci(workflows): bump GitHub Actions images to Ubuntu 24.04 Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): remove CUDA 11.x support from GitHub Actions (incompatible with ubuntu:24.04) Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): bump GitHub Actions CUDA support to 12.9 Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(docker): bump base image to ubuntu:24.04 and adjust Vulkan SDK/packages Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * fix(backend): correct context paths for Python backends in workflows, Makefile and Dockerfile Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(make): disable parallel backend builds to avoid race conditions Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(make): export CUDA_MAJOR_VERSION and CUDA_MINOR_VERSION for override Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(backend): update backend Dockerfiles to Ubuntu 24.04 Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(backend): add ROCm env vars and default AMDGPU_TARGETS for hipBLAS builds Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(chatterbox): bump ROCm PyTorch to 2.9.1+rocm6.4 and update index URL; align hipblas requirements Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore: add local-ai-launcher to .gitignore Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): fix backends GitHub Actions workflows after rebase Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(docker): use build-time UBUNTU_VERSION variable Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(docker): remove libquadmath0 from requirements-stage base image Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(make): add backends/vllm to .NOTPARALLEL to prevent parallel builds Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * fix(docker): correct CUDA installation steps in backend Dockerfiles Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(backend): update ROCm to 6.4 and align Python hipblas requirements Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): switch GitHub Actions runners to Ubuntu-24.04 for CUDA on arm64 builds Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(docker): update base image and backend Dockerfiles for Ubuntu 24.04 compatibility on arm64 Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(backend): increase timeout for uv installs behind slow networks on backend/Dockerfile.python Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): switch GitHub Actions runners to Ubuntu-24.04 for vibevoice backend Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): fix failing GitHub Actions runners Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * fix: Allow FROM_SOURCE to be unset, use upstream Intel images etc. Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(build): rm all traces of CUDA 11 Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(build): Add Ubuntu codename as an argument Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com>	2026-01-06 15:26:42 +01:00
Ettore Di Giacinto	edcbf82b31	chore(ci): add wget	2025-12-04 10:01:34 +01:00
Ettore Di Giacinto	6558caca85	chore(ci): adapt also golang-based backends docker images	2025-12-04 09:14:08 +01:00
Richard Palethorpe	c07bc55fee	fix(intel): Set GPU vendor on Intel images and cleanup (#5945) ... Signed-off-by: Richard Palethorpe <io@richiejp.com>	2025-07-31 19:44:46 +02:00
Dave	9cecf5e7ac	fix: rename Dockerfile.go --> Dockerfile.golang to avoid IDE errors (#5892) ... extract up and out Dockerfile.go --> Dockerfile.golang rename. Prevents syntax highlighting and IDE errors Signed-off-by: Dave Lee <dave@gray101.com>	2025-07-23 21:33:26 +02:00