mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-22 06:04:55 -04:00
test/distributed-e2e-ci
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
fffb4d9aec |
feat(distributed): stop a backend on one route, whatever the worker is
nodes.<id>.backend.stop was the last worker-facing NATS subject, and it existed only because ONE publisher had not moved. An agent worker already mounted workerctl.PathBackendStop on the tunnel it holds, and a backend worker already took its stop there, so RemoteUnloaderAdapter branched on NodeType to pick a carrier for a verb both kinds of worker served the same way. The branch is gone, and with it nodeTypeOf and its NodeTypeBackend default, which removes one of the ten NodeType branches left to sweep. The adapter loses its messaging.MessagingClient outright rather than keeping an unused field: it now holds no publisher, so re-routing any verb back onto the bus is a change to the struct and to every caller of the constructor, and does not compile until all of them agree. messaging.SubjectNodeBackendStop and subjectNodePrefix are deleted, the agent worker's subscription with them. pkg/natsauth drops the per-node backend.stop grant from the agent SUB list. That is a narrowing of eleven entries to ten, never to nothing: NATS reads an EMPTY allow list as unrestricted, so the coverage spec asserts both that the retired subject is no longer covered and that the queue subjects an agent worker lives on still are. The e2e half proves it against a real enforcing server: one spec subscribes successfully on an agent-minted JWT, the next is refused the retired subject on a JWT minted the same way. Both halves of the old split were pinned, so both pins are re-aimed rather than deleted, and the two node types are asserted separately rather than as one parameterised case, because only two cases can show that the two used to differ. Three assertions that the adapter published nothing are deleted instead: with no publisher to hold, no change could ever redden them. The CLI's handler set moves into agentWorkerControlHandlers so a spec can stand it up and post to it. That wiring was a bare literal no spec pinned, and deleting the subscription made it the ONLY carrier for backend.stop: a dropped field would have been a 404 the frontend reads as a worker too old to serve the verb, and nothing in the repo would have noticed. Mutations: the agent branch restored off the control route reddens two specs; the backend branch restored, separately, reddens five; PathBackendDelete in place of PathBackendStop reddens nine across both node types; dropping the CLI wiring line reddens the new wiring table; re-adding the allow-list entry reddens the unit spec and the JWT e2e spec; and restoring the publisher for real does not compile. Four comments this change falsified are fixed, in core/cli, pkg/model and the distributed-mode docs, which now say both kinds of worker serve POST /v1/control/backend/stop and what each does with it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
5759b4584b |
fix(distributed): let a worker's own refusal be evidence about its backend
A worker that refuses a stream has answered, and cluster.Dial keeps the three tunnelproto sentinels out of the ErrNoRoute umbrella precisely so a consumer can act on that. No consumer did. Since workers stopped listening, a backend process that crashed on a healthy worker is no longer a dead listener's codes.Unavailable: the worker refuses the stream with ErrStreamTargetUnavailable, gRPC flattens it into Unavailable anyway, and nodes.unroutable reported the whole thing as "this frontend has no route". Every reap path then answered ProbeUnknown and left the row, so the replica slot never freed and at the default MaxReplicasPerModel=1 the only cleanup left was LRU eviction of models that were working. isWorkerAnswer is exported as cluster.IsWorkerAnswer, so the errors the dialer keeps out of the umbrella are by construction the errors the consumers treat as the worker answering. nodes.unroutable and pkg/model's transportFailure both use it; ConnectionEvictingClient, the site reached during inference, goes through transportFailure rather than asking the transport directly. A reply code this frontend does not recognise is still not an answer, so a newer worker's vocabulary costs a retry and not a replica. The reap guards keep the allow-list rather than requiring ErrNoRoute: an unrecognised dial error must mean "no route", never "the backend is gone". Also in this final pass over the branch: - Docs: recommend upgrading FRONTENDS first, with the symptom of each order. Workers-first fails now that a 4xx registration is a verdict rather than an outage, so an old frontend's "address is required for backend workers" makes each restarted worker exit and drains the fleet a node per restart. - Docs: LOCALAI_WORKER_TUNNEL=false is a fatal startup error, not a degraded mode, in both places that described it; and a frontend rollback needs every worker restarted, because re-registration force-clears the address columns. - A replica with no advertised address now says so every five minutes and names the workers only it can reach, instead of one startup warning for a cost paid for the life of the process. - callerRanOut's rule now holds at all three siblings, so an expired caller deadline stops reading as a broken tunnel; probeHealth's withdrawn reason for using the raw client is corrected; the dead DoOrCached is deleted and its coverage kept on DoOrCachedResult; sweepLeakedInFlight enumerates the outcomes that reach it. - The peer route's self-declared id is recorded as a phase-3 deferral, in the handler, in the isolation claim it narrows, and in the operator docs. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d7002b89c5 |
test(nodes): make the joiner spec deterministic, and gate the wrapper shape in lint
Re-review round 2. One blocking item, and it was a spec I wrote: eight goroutines raced at the probe cache and nothing made them coalesce, so a straggler that missed the flight re-entered the probe and double-closed a channel. It panicked about one run in three and took the four-suite race block down. The green verification I reported was not reproducible, which means one green run was never evidence for a spec that coordinates goroutines. Its comment claimed the probe blocked until every goroutine was inside flight.Do, and that gap was exactly the panic: the comment described the design intended rather than the one written. It is deterministic now rather than tolerant. singleflight.DoChan registers its channel on an in-flight call under the group's own mutex and returns without running its function, so calling it while the leader is provably parked inside the probe joins that exact flight with no window and no dependence on the scheduler. The spec asserts the join really happened, that the joiner got the reason and not only the answer, and that the probe ran once; the entered channel is sent on rather than closed so a second probe fails an assertion instead of panicking. Twenty runs green under race against the committed code, five out of five red on the mutation back to a closed-over variable. The future-decorator gap is closed in the lint gate, but not the way the review suggested, and the reason is worth recording. HasMethod rejects inline signatures outright, its method-reference form needs a package ruleguard's own typechecker can import and that typechecker cannot import this module, and Implements tests the value method set while every Unwrap is on a pointer receiver, so it fired on all three wrappers that already had one. So the safe shape is structural instead. grpc.WrappedBackend gives the same pass-through method set plus Unwrap on a value receiver, and a decorator that embeds it is transparent by construction; forgetting stops being expressible rather than merely discouraged, which is the move loopbackService already makes in the worker. FileStagingClient and ConnectionEvictingClient embed it and their hand-written Unwrap methods are gone. The ruleguard rule then only has to catch the raw embedding, needs no type filter, and cannot misfire. It was verified to fire on a throwaway wrapper and stay silent on a correct one, and reports nothing across core and pkg with the baseline disabled. InFlightTrackingClient is the one exception and says why in a nolint: it embeds ControlBackend deliberately so that leaving an inference method unwrapped breaks the build, and WrappedBackend embeds the full interface, so adopting it would silently restore pass-through for every inference method and delete that guarantee. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
34f9b14b67 |
fix(grpc): let the transport answer through the wrappers, not only past gRPC
Re-review round 2. One blocking defect, and it was the concern I filed myself last round and mis-scoped as a future trap. It was live, and it sat on the most destructive reaping path of the five. RouteResult.Client is an InFlightTrackingClient, over a FileStagingClient when a stager is configured. model_router puts that on the cached remote model and pkg/model's checkIsLoaded asks IT whether the transport failed. Both wrappers embed grpc.Backend, which does not declare LastDialError, so the type assertion read nil and the guard added last round fell straight through to the old eviction. That eviction sends backend.stop over NATS to every node holding the model and deletes every replica row, where the other sites delete one. The spec covering it built a bare client by hand, which is why it passed while production did not. This is the third time in this task a correct fix was disarmed one layer out, so the fix is a mechanism rather than two methods. BackendUnwrapper is one line per decorator, LastDialErrorOf walks the chain, and both consumers now call it instead of each keeping its own assertion. One implementation, no per-caller policy to get wrong. Sweeping every type that embeds or holds a grpc.Backend found a third decorator the review had not named, and it is itself a reaping consumer of the same collapsed signal. ConnectionEvictingClient is built for remote models in initializers.go and its evict callback runs ShutdownModel; it fires during INFERENCE rather than on a health check, so a tunnel blip mid-request was enough to stop a model that was loaded and serving. It consults the transport first now. A locally spawned backend has no custom transport, so that path is unchanged byte for byte. Everything else touching a Backend is a consumer rather than a decorator; there is no fourth. The probe cache joiner shape is pinned. It was the right design last round with nothing holding it: the mutation back to a closed-over variable passed all 602 specs in the package. Eight goroutines coalesced on a probe that blocks on a channel now assert every joiner gets the leader's REASON and not just its answer, which is the difference between a leader declining to reap and its seven joiners reaping on the leader's own observation. The LastDialError scope note claimed an exactness it does not have at checkIsLoaded, which reads a shared long-lived client after releasing opMutex. It now says which caller is not exact, why the imprecision is accepted there, and what making it exact would cost. The four-outcome table in the docs still said a worker with no live owner is treated as absent and rescheduled, contradicting the code and the paragraph nine lines below it. None of those outcomes is absence any more, and the table says so, names the fifth, and points at the heartbeat as the thing that does decide presence. Five mutations, each reddening named specs, including the two the reviewer found surviving. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
2facfc0d88 |
feat: Add kimodo.cpp and 3D animation API/UI (#12095)
* fix(vulkan): preserve host ICD discovery for packaged backends Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add Kimodo CPU and Vulkan animation backend Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing. Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history. Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(kimodocpp): adopt monolithic encoders and resident inference Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support. Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin. Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Richard Palethorpeandlocalai-org-maint-bot
|
9058a2bb46 |
feat: Add 3d generation UI/API and trellis2cpp backend (#10979)
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint Adds the plumbing for image-conditioned 3D asset generation (binary glTF / GLB output), modeled on the video generation path: - backend.proto: Generate3D RPC + Generate3DRequest (staged image src, glb dst, seed/step/cfg_scale/texture_steps, quality and background enums, params map for backend-specific extras) - pkg/grpc: thread Generate3D through client, server, embed, base and the backend interfaces; connection-evicting and distributed-node wrappers (in-flight tracking + file staging) included - core/config: FLAG_3D usecase (guessed only for the trellis2cpp backend), '3d' canonical usecase string mapped to the Generate3D method, and a '3d' output modality - REST: POST /v1/3d/generations (+ unversioned alias) returning OpenAIResponse with a /generated-3d URL or b64_json; conditioning image accepted as URL, base64, or data URI; quality/background validated at the edge; .glb served as model/gltf-binary - auth: '3d' route feature (default ON); /api/instructions entry Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(trellis2cpp): add the trellis2.cpp image-to-3D backend Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2, pbr-textures branch) as a Go+purego backend, following the stablediffusion-ggml pattern: - backend/go/trellis2cpp: purego bindings to the flat C ABI (v9, asserted at startup), eager pipeline load with model-set validation (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured exactly like the upstream demo), Generate3D via t2_generate + t2_bake_glb writing a binary glTF to dst. Weight-free unit tests cover resolution/validation/param mapping — CI never downloads the multi-GB GGUF set or runs inference. - CPU SIMD variants build into per-variant directories (the shared libggml sonames collide across variants, unlike sd-ggml's flat renamed-.so scheme); run.sh picks one via /proc/cpuinfo. - CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta + latest/master image entries, bump_deps tracking of the pbr-textures branch, changed-backends.js mapping, top-level Makefile targets. - Importer: auto-detects trellis GGUF repos/URIs (registered before llama-cpp so the .gguf match isn't stolen) and expands any trellis URI to the full 10-file component set spanning the three LocalAI-io HF repos. - Gallery: trellis2-4b (full PBR + 1024 cascade) and trellis2-4b-geometry (512 untextured) with verified sha256s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(ui): 3D generation page with native GLB viewer and IndexedDB history Adds a Studio tab + /app/3d page for the new image-to-3D endpoint: - GlbViewer ports the trellis2cpp demo's dependency-free WebGL2 renderer (quaternion trackball, metallic-roughness PBR, ACES, hidden-line wireframe with a bounded index budget) and pairs it with a minimal GLB parser for the two forms t2_bake_glb emits — dense vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as normalized integers) and the opt-in UV-atlas textured form. Parsing happens before any GL so stats and errors render without WebGL2. - use3DHistory stores past generations (params, input thumbnail, and the GLB blob itself) in IndexedDB with keep-newest-20 eviction — GLBs are multi-MB binaries localStorage can't hold — and the page offers a download button for the active GLB. - Wiring: CAP_3D capability constant (FLAG_3D — the exact string /api/models/capabilities serves), threeDApi, router entries, Studio tab, vite dev proxy, en locale keys. - e2e: render-smoke entry plus a focused spec that feeds a real one-triangle vertex-PBR GLB through the parser/viewer and exercises IndexedDB persistence, selection, deletion, and API errors. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(3d): address API correctness and UX issues Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it. Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add previewable print remeshing Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download. Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(trellis2cpp): centralize remesh dependency pins Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement Generate3D stub for new proto RPC The Generate3D RPC added to backend.proto for the trellis2cpp backend made tonic's generated Backend trait require generate3_d, breaking the kokoros-grpc build. Return unimplemented like the other unsupported modalities. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
294170d3ed |
feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery (#10352)
* feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery Mirrors the locate-anything-cpp backend to register a new depth-anything backend that wraps the Depth Anything 3 ggml port (depth-anything.cpp) via purego (cgo-less, no Python at inference). - backend/go/depth-anything-cpp/: gRPC backend (Load + Predict + GenerateImage), purego binding to the da_capi_* C ABI, CMake/Makefile/run/package/test scripts building depth-anything.cpp's DA_SHARED static .so per CPU variant. - backend/index.yaml: depth-anything backend meta + all hardware-variant capability entries (cpu/cuda12/cuda13/intel-sycl-f32+f16/vulkan/nvidia-l4t). - gallery/index.yaml: 8 Depth Anything 3 GGUF models (base q4_k/q8_0/f16/f32, small, large, giant, mono-large). - .github/backend-matrix.yml: one build entry per hardware variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): typed Depth RPC + REST endpoint exposing full DA3 data Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to e0b6814 (ABI 3 dense C-API) The Depth RPC handler calls da_capi_depth_dense / da_capi_points (C-API ABI 3); pin the native build to the commit that exports them. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to v0.1.0 release (b515c31) Repoint the native version from the now-orphaned e0b6814 to the b515c31 release commit, kept alive by the upstream v0.1.0 tag. C-API is unchanged (da_capi_abi_version == 3). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): wire depth-anything-cpp into build, CI bump, and importer The backend dir, gallery index, and CI build-matrix were present but the backend was never wired into the integration points that adding-backends.md requires: - root Makefile: add to .NOTPARALLEL, the test-extra chain, a BACKEND_* definition, the docker-build target eval, and docker-build-backends (mirrors parakeet-cpp; the backend's own Makefile already documented that its `test` target is driven by test-extra). - bump_deps.yaml: register the DEPTHANYTHING_VERSION pin so the daily auto-bump bot tracks mudler/depth-anything.cpp master (it cannot see an unregistered Makefile pin). - import form: add a preference-only KnownBackend entry so depth-anything is selectable at /import-model (mirrors sam3-cpp; no reliable GGUF auto-detect signal, so pref-only per the doc's default). changed-backends.js needs no entry: the generic golang suffix branch already resolves backend/go/depth-anything-cpp/. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): auto-detect importer for depth-anything GGUFs Replace the preference-only entry with a real auto-detect importer (mirrors parakeet-cpp / locate-anything): - DepthAnythingImporter matches a .gguf whose name carries a depth-anything token (depth-anything-<size>-<quant>.gguf), so /import-model recognises mudler/depth-anything.cpp-gguf repos and direct GGUF URLs without an explicit backend preference. preferences.backend= "depth-anything" still forces it. - Registered before LlamaCPPImporter so its GGUF bundles aren't claimed by the generic .gguf importer; the narrow name match means it cannot claim arbitrary llama GGUFs or the upstream safetensors PyTorch repos. - Multi-quant repos pick the smallest quant by default (q4_k -> ... -> f32, depth stays >0.998 corr even at q4_k); quantizations preference overrides. - Drops the now-redundant knownPrefOnlyBackends entry (importer-backed backends are not listed there, matching parakeet-cpp). - Table-driven Ginkgo test covers detection, negative cases (llama GGUF, upstream safetensors), default/override/fallback quant pick, and direct URL import. 10/10 specs pass. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): check conn.Close error in grpc Depth client (errcheck) The new Depth() client method used a bare `defer conn.Close()`. golangci-lint runs with new-from-merge-base, so although the 39 sibling methods use the same bare form (grandfathered), the newly added line trips errcheck. Drop the result explicitly to satisfy the linter. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.1 (embeddable CMake) v0.1.0 (b515c31) used ${CMAKE_SOURCE_DIR} for its include dirs, which points at the parent project when built via add_subdirectory() as this backend does, so the container build failed with missing stb_image.h / da_gguf_keys.h. v0.1.1 (2d42897) switches to project-relative paths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): resolve gosec findings in the backend wrapper The code-scanning gate flagged three new failure-level alerts in godepthanythingcpp.go (gosec runs with -no-fail; GitHub gates on new alerts): - G301: export dirs were created with 0o755. Tighten to 0o750 (no world access needed for backend-written export output). - G304: writeDepthPNG creates req.GetDst(). That path is chosen by the LocalAI core as the intended output destination (same pattern every image backend uses), not attacker input, so annotate with #nosec G304 and document why. The remaining G103 "audit unsafe" notes on the unsafe.Slice C-buffer copies are warning-level (the same purego interop whisper/parakeet use) and do not gate the check, per the supertonic exclusion precedent in secscan.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.2 (CUDA cross-build arch) v0.1.1 forced CMAKE_CUDA_ARCHITECTURES=native, which breaks the GPU-less l4t/cublas CI builds (nvcc "Unsupported gpu architecture 'compute_'" on CMake 3.22). v0.1.2 (442eea4) drops the override and lets ggml pick its default cross-build arch list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
6a80e23733 |
feat(middleware): Model routing, PII filtering, Cloud model proxies (#9802)
Add a routing middleware stack and a cloud-proxy backend. * cloud-proxy: a Go gRPC backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers, with an optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response) and full tool-calling support. * routing: admission control, content-aware model routing (embedding cache + classifier + rerank + Arch-Router score), PII detection/redaction (regex + NER) with streaming filter and OpenAI/Anthropic adapters, and a per-user/per-key billing recorder backed by GORM or in-memory storage. * middleware: UsageMiddleware records usage via the billing recorder, plus admission, route-model, usage-stamp and trace middlewares. * observability: BackendTrace ring buffer stores full request bodies (capped), MITM proxy emits structured trace events, and router classifier decisions surface at /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0). * UI: cloud-proxy model-editor fields, classifier system-prompt and score-normalization config, and a Traces page rendering request bodies. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Ettore Di Giacinto
|
87e6de1989 |
feat: wire transcription for llama.cpp, add streaming support (#9353)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
510d6759fe |
fix(nodes): better detection if nodes goes down or model is not available (#9274)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |