mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 17:18:49 -04:00
2676dcafe8e543ff5135b685458a09bcb901bbc3
61
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
6d5c9600d7 |
feat(localai-proxy): add a backend that serves text APIs from a remote LocalAI
Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
8dbe7a0e3f |
chore(e2e): ignore the cloud-proxy backend binary
make build-cloud-proxy-backend writes it next to the mock backend, which is ignored; the cloud-proxy binary showed up as untracked. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
74b7ea2829 |
feat(ui): replace the gallery and inventory tables with a rail and a detail pane (#11288)
* feat(ui): rename the Install Models nav entry to Discover "Install Models" named the action rather than the destination, and it was the only multi-word entry in a rail of one-word ones (Home, Chat, Studio, Talk, Build, Operate). A bare "Models" was the obvious fix but it collides with the installed-models view under Host, which is a different page for a different job. "Discover" keeps the rhythm and says what the page is for. The icon moves from a download arrow to a compass for the same reason: the page is browsed before it is installed from. Translated in all seven locales rather than left to fall back, so a locale switch does not leave the entry in English. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): replace the gallery table with a rail and a detail pane The eight-column table was not the real problem; the click-to-expand row underneath it was. Variants, files and a VRAM estimate never fitted inside a <tr>, so they were pushed into a drawer that could hold one model at a time, could not be linked to, and had no room to say anything useful. The gallery is now a rail to scan and a pane that answers. The pane has two states and no third: with nothing selected it is the discovery page, and with a model selected it is that model's detail. Selection lives in the URL, so a model is linkable and Back steps out of the detail instead of off the page. The rail groups by capability while browsing and flattens to results the moment a term is typed. That is a rule rather than a toggle: once someone has said what they are looking for, the buckets are between them and the answer, and making the user choose would be handing them our problem. The detail pane plots VRAM against context length with the host's own limit drawn across it. This is new information, not a restyle. A single number invites "so will it run?", and the honest answer is usually "yes, up to a 32k context", which is a shape rather than a number. The estimates were already fetched for every context size, so it costs no new request. Backends that take no context length say so instead of being given a meaningless chart, and a host with no GPU gets no chart at all rather than bars with nothing to compare against. The split-button variant menu goes with the actions column. The pane lists every build with its backend, quantization, size, fit and a details disclosure, each installable, which is what the dropdown was a cramped substitute for. Its tests move onto that list; the three contracts it alone carried (fetch-once caching, the loading state, an unfit build staying installable) are backfilled against the pane. RecommendedModels moves inside the pane, where it has the width to argue for a model instead of listing one, and keeps its own dismissal and collapse. Rail entries carry no description. Two lines is the budget and the second is better spent on whether the thing will run; the stripped-Markdown contract moves to the pane's lede, tooltip included. e2e: 123 passing across models-gallery, navigation, recommended-panel, model-artifact-operation, operations-strip and page-render-smoke. Inline styles in Models.jsx drop from 82 to 41. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * refactor(ui): extract the split view into shared components Discover shipped its rail, pane and detail header as private functions inside Models.jsx. Backends and Host have the same defect and want the same shape, so leaving them there guarantees three rails that drift. SplitView, EntityRail, DetailHeader and StatGrid now live under components/split/. EntityRail is deliberately data-driven: a surface maps its own entity onto { id, name, icon, meta, stripe, groupId } and keeps its vocabulary to itself, which is what stops the rail learning about models, backends and loaded state all at once. The CSS moves with it. What was .discover__rail is .entity-rail, .discover__ pane is .split-view__pane and so on, because a class named after one page is a lie on the next two. Only what is genuinely Discover's stays behind the old prefix: the shelves, the hero and the VRAM-by-context chart. Two additions the shared rail needs and Discover did not: a state stripe, for surfaces read by condition before they are read by name, and an empty label. Discover passes neither. No behaviour change. e2e 100 passing across models-gallery, navigation and models-recommended-panel. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): put the backend gallery on the split view Same defect as the model gallery, so the same shape: a seven-column table over a click-to-expand row that was the only place the repository, licence, tags and links could go. The rail groups backends by the use case they serve, sharing Discover's taxonomy on purpose: a backend is the runtime a use case needs, so "vision" ought to mean the same thing one level down. It flattens on a query for the same reason it does on Discover. The zero state is the one real departure. A backend's fitness is not free memory, it is the accelerator and platform it was built for, so the pane leads with what this host is, then what is not installed yet, then whether anything installed has gone stale. The table listed 37 runtimes and left "which of these can even run here" entirely to the reader. Distribution moves into the pane, which is the one thing a row could never carry: which nodes hold a copy and which do not, with the install-on-more control next to it rather than squeezed against a chip. The distributed and target-node action logic is unchanged, including the guard that keeps a hardware-specific build off the fan-out path. The split-button popover loses its per-row anchoring because there are no rows; one pane, one anchor. Selection lives in ?backend=, preserving the ?target= scope rather than clobbering it. e2e: 139 passing across models-gallery, navigation, backends-management, models-recommended-panel, nodes-per-node-backend-actions, page-render-smoke, operations-strip and model-artifact-operation. The backends spec gains six split-view tests; its three description-cell tests move onto the pane lede. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): put the Host inventory on the split view The last of the three surfaces, and the one that is not a catalog. Both tabs had the same click-to-expand row, so the shell transfers; what does not transfer is the zero state, because there is nothing to discover in your own inventory. With nothing selected the pane reports what is happening: how many models are loaded, what failed, what has an update, and which models are holding VRAM right now. Every number was already on the page. None of them had been assembled into one statement, so "what is going on" was a question the tabs could not answer however long you looked at them. The rail buckets by state rather than capability - Running, Idle, Disabled for models; Update available, Installed for backends - which is the opposite of the galleries and deliberately so: nobody opens Host wondering which of their models does vision. Entries carry a state stripe for the same reason. Load and Stop are promoted out of the kebab, because that is what an operator came for; the rest stays behind the menu rather than diluting it. Adopted, pinned and alias badges follow the model into the pane: they are facts about the thing, not about its state, and the rail line is spent on state. Deliberately NOT done: folding the two tabs into one rail, as the mock had it. It costs five URL parameters, the manage-tab localStorage key and the stat-card shortcuts, all of which are live deep-links today. The tabs stay as the group selector; merging them is a follow-up with its own migration. e2e: full suite 355 passing. New host-split-view spec; alias-template, manage-logs-link, manage-action-menu-position and model-editor-back-nav move off `.table` and the row kebab onto the rail and the pane. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * polish(ui): accessibility and consistency pass over the three split views Findings from a pass over what the previous four commits actually shipped, rather than what they were supposed to. The rail was not a listbox. ARIA lets a listbox contain options and groups, and nothing else, but each group's collapse control is a button that has to sit inside the scroller with the entries it folds. It is now a labelled group of buttons, which is the honest description; selection is announced with aria-current and the arrow keys are unaffected. Every entry was its own tab stop, so tabbing past a forty-entry rail to reach the pane took forty keystrokes. Roving tabindex makes the rail one stop, and arrowing now moves focus with the selection instead of leaving it behind on an entry Tab can no longer reach. The rail rounds its corners with overflow:hidden, which was clipping the focus ring off the first and last entries entirely. Inset outlines fix it. A 30px row is fine under a mouse and too small under a thumb, so coarse pointers get a 44px target without costing density on a desktop. One slot said three different things: "9 models loaded" on Discover, "12 loaded" on Backends, "3 of 9" on Host. All three lists are a page of a larger set, so all three now say so the same way. Also removed: an emptyLabel prop on EntityRail that nothing passed, its dead CSS rule, and MODELS_COLSPAN and ResourceRowDesc, which died with the tables. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): correct three defects only a real gallery exposed Running the branch against a live instance with 1,595 models and 1,017 backends, rather than against mocked fixtures, surfaced three things the e2e suite could not. Grouping did nothing. The rails matched on the use-case keys the filter chips send (`chat`, `tts`, `transcript`), but those are a server-side vocabulary the handler maps onto entries. What entries actually carry is free-form and inconsistent: models come back tagged `llm`, `gguf`, `vision`, `coding`, and backends `LLM`, `text-to-text`, `audio-transcription`. Nothing matched, so every model landed in "Everything else" and the feature was decorative. Grouping now lives in utils/entityGroups.js, shared by both galleries, matching case-insensitively against the vocabulary the API really uses, with the entry's backend as a fallback signal - a backend named `whisper` is a speech backend whatever its tags say. Order is specific before general and that is load-bearing: a vision model is tagged `llm` too, so testing text first would swallow it. The zero state claimed GPU memory on a machine with no GPU. The resources endpoint reports system RAM in the same field when gpu_count is 0, so the hero read "84.4 GB of GPU memory" next to the recommendations panel correctly saying "No GPU detected". The number was never wrong, only its label; it now says system memory unless a GPU is actually present. The page title still said "Install Models" under a nav entry saying Discover. Also: the keyboard test named the model it expected to arrive at, which made it a hostage of the grouping table and broke the moment the buckets were fixed. It now asserts that the selection moves and returns. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): the filters and the rail were fighting over the same job Four things you find odd on Discover, and they turn out to be one mistake seen from four sides. The rail grouped the current page. The listing is paginated at nine rows, so those bucket headers described nine entries out of 1,595, and turning a page reshuffled the sections under the reader. The structure was never stable because it was computed over the wrong set. The chips were redundant for the same reason, seen from the other side. They send tag= and filter all 1,595 server-side. The rail grouped nine of them client-side by the same axis. Two controls for one job, and the weaker one was the one this branch added, so it goes. Grouping stays only on Host, where the list is complete, local, and bucketed by state rather than capability. The search bar felt odd because it sat in a full-width band while the thing it narrowed was a 290px rail below and to the left. The whole band now lives in the rail column: search, backend, use cases, refinements, then the list it narrows. One column to say what you want, one to show what you got. Nineteen chips do not fit at that width, so they fold into a disclosure that states the selection. A disclosure and not a popover, deliberately: picking use cases is multi-select and interleaves with the backend select and the toggles below, and a popover dismisses itself the moment you touch either. The header held two counts and two buttons at arm's length from all of it. The counts were the third statement of the same number on one screen, after the rail's "9 of 1,247" and the pane's own headline, so they go. The buttons move into the pane's zero state, which is the surface that answers "what do I do here". Also: the two first-run empty states wore .loading-center, which is display:flex in the default row direction because it exists to centre one spinner. With four children that put the icon, the heading, the sentence and the buttons on a single line with no gap. They are now a proper full-height empty state. e2e: full suite 353 passing. Grouping tests are replaced by ones asserting the rail stays flat; chip tests open the disclosure first; two filter-layout tests that asserted the old three-band arrangement now assert the column. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * polish(ui): make Discover a full-height view, group the chips, name the refinements Four things, all of them the same complaint: the page read as a document with controls scattered on it rather than as one view. The header is fused. A title block with its own padding, a subtitle and two counts made the split view look like an attachment to a document that happened to sit below it. It is now a slim bar carrying the title, the count and the two page-level actions, and the split fills the rest of the window. Rail and pane scroll independently, so the filters and the pane's headline stay put while a long list moves under them. The chips group. Nineteen in a flat row is a lot to scan even behind a disclosure, and they already belong to the four families the rest of the UI speaks, so they are bucketed by those. "All" sits on its own above them without a heading, because it is a reset rather than a use case. The refinements stop looking dumped. When the band became a column they were three controls left where they landed; they now read as a named section with one control per row. The zero state suggests again. It had decayed into a "Browsing / 9 of 1,247 / select a model" line that restated the count for the third time on one screen. It now offers the four use cases as tiles that set the filter, which is the shelf idea from the mock without inventing curation or paying for a second fetch. Two bugs found by looking at it rather than at the tests: the disclosure was clamped to 190px, which cut it off partway through its third section so two of the five never appeared at all; and the creation actions rendered twice, once in the new bar and once in the pane hero a few pixels away. e2e: full suite 353 passing. The chip-row test now holds its contract across the per-family rows rather than a single one, and additionally asserts every family is present and non-empty. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): pin the split view's height so a long detail scrolls the pane Selecting a model with a long description grew the whole page and dragged the rail down with it, which is the opposite of what "full height" was supposed to buy. The flex chain was right and the ceiling was missing. .app-layout and .main-content are min-height:100dvh, which is a floor: flex distributes free space but nothing caps growth, so a pane taller than the viewport expanded the column, the document scrolled, and the rail stretched to match. height:100% on the pane then resolved against an auto-height parent and did nothing. The chat route already solves this by pinning .main-content to 100dvh. The same treatment now applies to any route containing a .page--app, selected with :has() so the shell does not have to learn which pages happen to be split views. Below the stacking breakpoint the pin is lifted, because two stacked halves in two short scrollers is worse than a page that scrolls. Measured on a live instance: document height stays at the viewport across selection (950px either side) and the pane overflows internally instead. Adds discover-height.spec.js, which asserts the page height and the rail height are unchanged by selection and that the pane is the thing that scrolls. The existing specs could not have caught this: they mock short descriptions, and the bug only appears when the pane has more content than the viewport holds. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): give Backends and Host the full-height view, and fix the Update button Backends now matches Discover: the header fuses into a slim bar carrying the title, the count and the page-level actions, the filters move into the rail column where they narrow the rail and nothing else, and the split fills the window. Its seven chips fit at rail width, so unlike Discover's nineteen they need no disclosure. Host gets the bar and the height; its resource monitor, summary cards and tabs stay above the split, because those are read once while the rail and the pane are worked in. Two things the height change surfaced. The console layout is a flex row with align-items:flex-start, so its body sizes to content. Right for the pages it was built for, wrong for a split view, which needs a ceiling to scroll inside: without it the Backends rail ran past the viewport and over the footer. Pinned with :has() so only split-view routes are affected. The filters vanished when nothing matched. Both galleries swapped the whole shell for an empty state, which took the search box and the chips with it, so the page said "try adjusting your search or filters" while offering neither. The shell now stays and the empty state moves into the pane. Also fixes the Update control on Host, which had no className at all and rendered as bare text, next to a status span that had picked up btn classes and two copies of `fas` and so rendered as a button you cannot press. They have swapped appearances back. e2e: full suite 355 passing. The render-smoke selector learns .view-bar__title, since the pages it checks no longer all use PageHeader. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): keep the view mounted while searching, and bring rail grouping back Searching replaced the whole view with a loader. The search box lives in the rail column, so every debounced refetch unmounted the field being typed into and dropped its focus with it. The list, the filters and the pane went too. The shell now stays and the rail says it is busy: a sweep bar under its header and the stale list dimmed, so the eye knows the answer is being replaced without losing its place. A cold start still gets the skeleton, because there is nothing to keep. The condition for that is "nothing has loaded yet", not "the list is empty". Those differ exactly when someone is editing a query that matched nothing, and getting it wrong there would unmount the view on the keystroke after a no-results search - the worst possible moment. Grouping comes back on both galleries. It was removed because nine rows could not fill five buckets, so a page turn rebuilt the rail's whole structure. That was a symptom of the page size rather than of grouping: the rail now asks for 30 rows instead of 9 (Backends 60 instead of 21), which is enough for the sections to read as structure and turns five times fewer pages. The order of the sections is fixed, so what changes between pages is membership, not arrangement. Grouped while browsing, flat while searching, as before: once a term is typed the buckets stand between the reader and the answer. Also gives GalleryLoader a class and a testid instead of six inline style declarations on a bare div, which is why nothing could select it. e2e: full suite 359 passing, including a new spec asserting the search box keeps its focus and its value across a refetch, and that a cold start still shows the skeleton. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * perf(gallery): stop invalidating the VRAM estimate caches on every request Searching or turning a page felt slow. It was not the search and not the listing: /api/models answers in 3-9ms. It was the VRAM estimate, which the gallery asks for once per row, and which took ~2.3s every single time however often the same model was asked about. pkg/vram already caches what makes that expensive - the remote content-length probes, the GGUF metadata reads and the HF repo sizes. Those caches key on a gallery generation counter, and AvailableGalleryModelsCached triggered a background refresh on every call, with each refresh bumping the counter. One page view is one listing request plus thirty estimate requests, each of which re-read the gallery and started another refresh, so the generation moved constantly and every cache entry was stale before it could ever be read. The caches were dead in production. Three changes, each doing one thing: A refresh interval. The cached list is still served immediately; this only decides how often re-fetching from upstream is worth starting. Five minutes, as a package variable so tests can drive it without waiting. A generation bump only when the gallery actually changed. An unchanged gallery re-fetched on schedule must not throw away work that is still valid, which is the difference between an estimate costing nothing and costing a network round trip. A separate "loaded" flag. The cache engaged on `cached != nil`, so a gallery that legitimately holds nothing read as never-loaded and took the blocking path on every call, bumping the generation each time. Found by the test for the interval, which could not pass while this was true. Measured against a live instance with 1,595 models: one estimate, repeated 2.3s -> 2ms a page of 30, in parallel 10s -> 0.04s A first, genuinely unseen model still costs its remote probe. That is inherent; what changed is that it is now paid once per model per gallery version rather than once per request. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * perf(gallery): warm VRAM estimates at startup, and stop the UI waiting on them Two halves of the same complaint: the gallery stalls on VRAM estimation. Server side, the estimates are now warmed in the background at startup. Estimating an entry nobody has asked about costs a remote probe of its weight files, and the gallery needs one per row, so the first visitor was paying for the whole page. The warm-up walks the gallery in the order the UI lists it, so the first page is ready before anyone reaches it. It is bounded and it never blocks: 300 entries at 4 at a time by default, on its own goroutine, stopping with the server's context. Warming the whole gallery would be thousands of probes on every boot, which is rude to the upstream and slow to finish; warming nothing leaves the first page paying two seconds a row. Anything past the limit still warms itself on first view. LOCALAI_VRAM_WARM_LIMIT=0 turns it off for an air-gapped host, LOCALAI_VRAM_WARM_CONCURRENCY=1 slows it for a metered link. Client side, the page no longer waits on estimates it does not need yet. It fired one request per row at once; a browser allows about six connections per host, so thirty estimates took every slot and the request behind a click - the variant list, an install - queued behind work nobody asked for. That is the freeze: the list was already usable, and the UI was busy fetching sizes. Four at a time leaves room for the interactive request to overtake, and a row whose estimate is still in flight says "sizing…" rather than leaving a blank where a number will appear. buildEstimateInput moves to core/gallery as EstimateInput, since the handler and the warmer both need it. Measured against 1,595 models, from a cold boot: page 1, 30 estimates in parallel 10s -> 0.04s full warm-up (299 of 300 entries) 3m, in the background Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * chore: untrack data/.local_user_id and ignore the runtime data dir `local-ai run` writes its instance state under ./data when started from the repo root, which is exactly what a contributor testing a build does. The identity file ended up committed on this branch by a `git add -A` while verifying the gallery changes against a live instance. Anchored, so it matches the runtime directory at the repo root and not a `data` directory nested inside some package. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
94d5affcea |
feat(website): split the site, move docs to /docs, add a landing page (#11243)
* feat(website): split the site, move docs to /docs, add a landing page The Hugo docs site has always been localai.io itself, which left nowhere to explain what LocalAI is or show what the team builds. This adds a separate marketing site at the root and moves the documentation under /docs/. Docs: The existing site keeps its content tree and its Relearn theme, and now builds with baseURL <root>/docs/. Its _index.md, which held a hand written landing page, becomes a real documentation home. Every previously published URL keeps working. GitHub Pages has no server side rewrites, so .github/ci/gen-redirects.sh walks the built docs output and leaves a meta refresh plus a canonical link at each old root path. It covers bare .html files too, which is what keeps /gallery.html alive, and it never overwrites a path the marketing site already owns. Website: A second Hugo site under website/ with its own layouts and no external theme, so the marketing side does not have to fight Relearn's home rooted menu and asset pipeline. CI builds both and merges them into one Pages artifact. The design is derived from the project logo rather than invented: the navy of the triangle, the cyan of the llama, the purple of the speed bars. Those offset bars became the motion signature. The background renders a real depth-anything.cpp depth map as contour lines and switches to a locate-anything.cpp style detection overlay over the engines section. Also included: an /engines/ index driven entirely by data/engines.yaml, a /blog/ section with five posts written from the release notes and the engine benchmark suites, install.sh and a Kubernetes manifest since the site advertises both, and a rule in .agents/ that release preparation now includes a blog post and demo clips. Every figure on the site is derived from the repository or the GitHub API, not from memory. Correcting them against their sources found one error in README.md: voxtral-tts.c is text to speech, not speech to text. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [Agent] * feat(website): add a star history chart, rewrite the history post in first person The history post read like a changelog written by a committee. It is now in Ettore's voice, first person, with the admissions left in. The numbers paragraph in particular read like a directory listing. It now says what the figures mean rather than which file they came from. Adds an interactive star history chart, built from the GitHub stargazers API rather than embedded from a third party, so the page makes no external request and cannot break when someone else's service is down. The four releases the post is organised around are marked on the curve, and the labels stack into rows because three of them land within two months of each other. The API stops paginating at 40,000 items, so the curve is measured up to December 2025 and the segment from there to today's total is drawn dashed, labelled as an estimate in the caption and in the tooltip. It is a straight line between two known points, and the chart says so rather than implying it is data. Also drops "marketing site" from the README heading and everywhere else it appeared, and calls it the main site instead. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandDaria Korenieva
|
9bfd71387b |
feat(stores): add Valkey Search vector store backend (#11196)
* feat: add Valkey Search vector store backend Add a new built-in Go gRPC store backend 'valkey-store' that implements the four Stores RPCs (Set/Get/Delete/Find) against the Valkey Search module (FT.*) using the pure-Go github.com/valkey-io/valkey-go client. It is selected via the existing per-request 'backend' field on /stores, so there is no proto or HTTP API change, and it mirrors the in-memory local-store while adding persistence across restarts and opt-in HNSW. Each vector is a Valkey HASH keyed by hex(little-endian float32); the index is created lazily on first Set (FLAT+COSINE by default), cosine similarity is derived as 1-distance, and namespaces get a collision-resistant token. Includes unit tests (valkey-go mock) and env-gated integration tests against valkey/valkey-bundle, plus build/matrix/gallery wiring and docs. Assisted-by: Kiro:claude-opus-4.8 golangci-lint Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: recover persisted index dimension, harden Find - Load now recovers the persisted vector DIM from FT.INFO (not just index existence), so a post-restart Set/Find validates against the real DIM instead of silently re-learning a wrong one and dropping mismatched vectors from the index. This also restores Find's dimension check after a restart. - StoresFind treats a dropped/missing index as an empty store (empty result, no error) and clears the stale indexCreated flag, matching local-store's empty-store behaviour. - StoresSet reuses checkDims for its per-key length check so the four RPCs share one dimension-guard implementation. - Add unit tests for FT.INFO dimension recovery, loadIndexState, and the dropped-index Find path. Assisted-by: Kiro:claude-opus-4.8 Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: TLS ServerName/CA, Find nil-check, config fail-fast Addresses external review comments on the valkey-store backend: - StoresFind now rejects a nil/empty query Key before dereferencing it, so a malformed gRPC request can no longer panic the backend. - TLS: derive ServerName (SNI) from the VALKEY_ADDR host so certificate verification works for IP-addressed endpoints, and add VALKEY_TLS_CA_CERT (custom CA bundle) and VALKEY_TLS_SKIP_VERIFY (testing-only) knobs. - Config integer parsing now fails fast on a malformed value (e.g. VALKEY_HNSW_M=1x6) instead of silently defaulting, matching the fail-fast behaviour of the index-algo/distance-metric validation. - Add VALKEY_DB (SELECT n) support for logical-DB isolation. - Cap the human-readable part of a namespace token at 64 chars so a very long model name cannot produce an unbounded key prefix / index name (the appended short hash keeps distinct namespaces collision-free). - Document the KNN-query injection-safety invariant (fields are constants) and why StoresGet uses a single aggregate DoMulti deadline for reads. - Unit tests for the Find nil/empty-key guard, fail-fast HNSW parsing, and VALKEY_DB parsing/validation; docs + .env updated for the new vars. Assisted-by: Kiro:claude-opus-4.8 golangci-lint Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: configure valkey-store via model config richiejp asked that the valkey-store backend take its configuration from a model config rather than process-wide VALKEY_* environment variables, so multiple stores can each have their own Valkey config within one LocalAI process. This removes every env access from the backend and routes config through the model-config seam every other backend uses. - config.go: loadConfig(opts *pb.ModelOptions) now parses the model config `options:` list (key:value strings, split on the first ':') instead of os.Getenv. Option keys mirror the old VALKEY_* names without the prefix (addr, index_algo, distance_metric, ...). Defaults, fail-fast validation and the mandatory client name are unchanged. - store.go: Load threads opts into loadConfig; TLS comments/errors renamed off the VALKEY_* names. - core/backend/stores.go: StoreBackend and NewVectorStore take a *config.ModelConfigLoader, resolve the per-store ModelConfig by store name, and pass its Options (and Backend when unset) to the backend via WithLoadGRPCLoadModelOpts. No config -> default backend + built-in defaults, preserving the zero-config experience. - Endpoints/routes/application: thread the config loader to StoreBackend. - Unit + integration tests: configure via options; the integration test passes addr through the model-config path (VALKEY_ADDR is now only the test harness locating the server). - docs + .env: document the model-config options, drop the env var table. Assisted-by: Kiro:claude-opus-4.8 Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Remove valkey-store informational comment from .env The backend is configured via model config, not env vars — the comment was unnecessary noise in .env. The configuration is already documented in docs/content/features/stores.md. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * feat(valkey-store): gate Load on NamespacePrefix to refuse autoload probing Mirror local-store's pattern: reject model names without store.NamespacePrefix so the model loader's greedy autoload probe cannot bind an arbitrary model name to the vector store backend (the #9287 failure mode). Also adds unit tests for the gate covering: prefixed namespace, prefix alone, unprefixed model name, empty model, and nil opts. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * feat(valkey-store): add username_env/password_env credential indirection Add support for resolving Valkey credentials from environment variables named in the model config, mirroring cloud-proxy's api_key_env pattern. This keeps secrets out of model YAML files and lets distinct store configs each reference their own credentials. Options: username_env / password_env name the env var holding the value. The direct username / password options still work and take precedence when both are set (backward compatible). Includes 5 unit tests and updated stores.md documentation. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * fix: correct rebase artifacts in backend-matrix.yml and Makefile Fix two issues introduced by the conflict-resolution script during the rebase onto master: 1. .github/backend-matrix.yml: valkey-store entries were merged INTO the cloud-proxy entries (duplicate keys in same YAML map items) instead of being separate list items. This broke cloud-proxy Linux builds and the cloud-proxy darwin entry lost its build-type/lang. Fixed by making them standalone entries and restoring cloud-proxy exactly as on master. 2. Makefile: duplicated .NOTPARALLEL and docker-build-backends lines. Collapsed to single lines that are master's current content plus the valkey-store additions. Also adds the three optional pickups from #10801: - /valkey-store in .gitignore (the built binary) - valkey-store row in docs/content/reference/compatibility-table.md - valkey-store line in backend/README.md Signed-off-by: Daria Korenieva <daric2612@gmail.com> --------- Signed-off-by: Daria Korenieva <daric2612@gmail.com> Co-authored-by: Daria Korenieva <daric2612@gmail.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
5c96e097ba |
feat(gallery): fix stale DFlash drafters and add the APEX families as variant ladders (#11027)
* fix(gallery): repoint qwen3-4b/qwen3.5-9b dflash drafters at post-rename GGUFs The drafters both entries referenced were converted from the pre-merge DFlash PR branch and carry dflash.target_layer_ids. llama.cpp reads dflash.target_layers and refuses the load. The stored values are offset by +1 relative to the HF-side field, so the files cannot be repaired by renaming the key and must be replaced. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): add apexentries HuggingFace client Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apexentries): build the HF client via pkg/httpclient The apexentries HuggingFace client was constructed as a raw &http.Client{Timeout: 60s}. The repo convention (documented in .golangci.yml, which cannot express this as a forbidigo pattern) is that all outbound HTTP goes through pkg/httpclient, which refuses redirects by default and sets a TLS 1.2 floor. The std client follows redirects and forwards custom credential headers to the redirect target on a cross-host hop (GHSA-3mj3-57v2-4636). Only a User-Agent is sent today, but this calls an external API and an HF_TOKEN header added later would leak. Switch to httpclient.NewWithTimeout, preserving the 60 second timeout. No behaviour change for the current header set. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): discover APEX tiers by filename suffix Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): resolve unsloth counterparts and sharded quants Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): render APEX child entries with the dflash/mtp tag rule Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apexentries): set backend, known_usecases and cross-repo drafters RenderChild left three gaps against the hand-written gallery entries. The generated entries reference gallery/virtual.yaml, which supplies no backend, so every generated entry named no engine at all. All comparable hand-written entries set backend: llama-cpp in overrides; do the same. Set known_usecases to [chat] alongside it: LocalAI falls back to the backend defaults when it is absent, so this is convention rather than breakage, but generated entries should not read differently from their neighbours. The drafter was also assumed to live in the repo publishing the weights. Speculative pairings routinely cross repos, and a drafter URI built from the weights repo 404s at install time. Add ChildInput.DraftRepo, used for both the drafter URI and its local path, falling back to Repo when empty so pairings that do ship the drafter alongside the weights are unchanged. The dflash/mtp tagging rule is untouched: the tag still follows SpecType and nothing else. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): dedupe generated entries against the existing gallery Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * apexentries: canonicalize HF URIs and dedup the generated batch Merge exists to stop a second gallery entry being added for weights the gallery already ships, but two gaps let duplicates through on a bulk run. The URI key was compared as an exact string while render.go only ever emits https://huggingface.co/{repo}/resolve/main/{file} and the gallery records 1038 of its URIs in huggingface://{repo}/{file} shorthand. A generated unsloth rung whose weights are already shipped in shorthand was therefore not recognised. canonicalURI reduces both spellings to one key and is applied on both sides, taking care that the repo is exactly the first two path segments so sharded quants in a subdirectory still match. A URI in neither form is returned untouched so other hosts dedup on their literal string. Merge also never accounted for entries it had just accepted, so two generated entries sharing a name or a primary URI both landed in add. Several APEX repos share one base model and resolve to the same unsloth counterpart, so the identical rungs are generated twice under the same name. Batch state is tracked locally rather than written back into the caller's ExistingIndex, which a caller may reasonably reuse. Name is still checked before URI: a name collision must block the add regardless of the weights. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): verify variant and tagging invariants in the gallery index Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): scope the apex-entries verifier to what it can actually judge The verifier reported 60 problems against the real gallery, 57 of which were llama.cpp assumptions meeting entries from other backends. A gate that is wrong 57 times out of 60 cannot gate anything. - The weight-count check catches a quant label collision in llama-cpp quant discovery, so it now runs only for overrides.backend: llama-cpp. Entries with no declared backend are skipped because their weights are declared in the referenced url: template, which the verifier never reads. - The dflash/mtp tag check now implements the per-backend table in .agents/adding-gallery-models.md instead of assuming llama.cpp's spec_type: vocabulary. ds4 declares mtp_path:/mtp_draft:; sglang declares speculative_algorithm: in a file this verifier cannot follow, so sglang entries are not judged in either direction. The check stays bidirectional within the backends it does judge. - sha256 is now required on .gguf files only, since every non-GGUF asset in the index belongs to a hand-curated entry outside this generator's scope. Against the current gallery this leaves exactly the three genuine problems: two entries setting spec_type:draft-mtp without the mtp tag, and one entry whose overrides.mmproj names a file it does not download. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apexentries): anchor quant matching and invert the sha256 rule UnaccountedQuants matched files to wanted quants with strings.Contains, which reproduces the substring collision it was written to warn about: Q8_0 is a substring of UD-Q8_0, so a repo publishing only UD-Q8_0 was reported as publishing an unbuilt Q8_0. Subdirectory-sharded UD quants are the normal unsloth layout for large repos, so this fired on realistic input. Match on the quant label as an anchored token instead, the way DiscoverUnslothQuants does, so the diagnostic and the discovery it audits cannot disagree about what a file is. Root-level shards, the layout the diagnostic mainly exists to catch, stay detected. The sha256 requirement was scoped to .gguf, which exempted seven real model weights: wan_2.1_vae.safetensors and clip_vision_h.safetensors across the wan-2.1-*-ggml entries, both load-bearing weights named by gallery/wan-ggml.yaml. Invert the rule so a checksum is required on everything except metadata extensions, which keeps a future weight format covered by default rather than silently exempt. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): wire the apexentries command Adds the generation path to the apexentries command: list the mudler APEX repos, discover each one's quality ladder and its unsloth counterpart's quant rungs from the filenames actually published, render a child entry per build plus a family parent carrying the variants list, dedup against the gallery, and write the additions to -out or append them with -apply. Discovery shortfalls are reported at discovery time rather than left to the verifier. A quant or a tier that discovery drops leaves no trace in a finished gallery file, and because an empty imatrix ladder falls back to the plain one, a repo whose imatrix filenames all fail to match downgrades the whole family silently instead of erroring. Merge's single reused map is split into two reported categories. A URI match means the gallery already ships exactly these weights and referencing the existing entry is correct; a name collision means an unrelated entry owns the name and referencing it would substitute a different build. Multimodal children now declare known_usecases [chat, vision]. An explicit known_usecases suppresses the backend-default fallback, so a chat-only entry carrying an mmproj never matches the vision or multimodal gallery filters. .github/ci is invisible to go list ./..., so a workflow names both generator packages explicitly and their specs finally run on pull requests. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): gather APEX builds under the base model entry The hub for a family is the BASE model entry, never a generated *-apex parent. Somebody looking for qwen3.6-35b-a3b has to find every build of those weights under that one name, so a competing qwen3.6-35b-a3b-apex hub would split the family and leave half of it invisible. When the gallery already ships the base entry, a variants block is spliced into it textually, leaving its description, icon, tags, overrides and files untouched. Only a family whose base model the gallery does not ship gets a new hub, still named for the base model and carrying one of the discovered builds as its own payload so it declares a backend the verifier can judge. The line editing is factored into .github/ci/galleryedit, shared with the variantproposals job, so the two cannot drift apart on where a variants block belongs. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(apexentries): treat an unreadable optional counterpart repo as absent HuggingFace answers 401 Unauthorized, not 404, for a repository that does not exist when the request carries no credentials. FetchRepoFiles treated only 404 as absence, so probing for the OPTIONAL unsloth counterpart hard failed for every family that legitimately has none: 27 of the 45 APEX families are community merges that will never have an unsloth build, and a full run failed all of them. Split the fetch so the two call sites can apply different policies to the same response. The APEX repo itself stays strict: a 401 or 403 on a repo the run requires is a real failure and still errors. Only the optional probe tolerates it, because without a token 401 cannot be told apart from absence. That collapse is lossy in one direction, since a private or gated repo also answers 401, so the skipped candidates are named in the run summary alongside the other silent-shortfall counters instead of being dropped in silence. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apexentries): report full-precision sources as a known exclusion The 45 APEX repos publish their unquantized F16 sources next to the imatrix ladder, flat or sharded. Discovery correctly emits nothing for them, but they were landing in the unclassified total, leaving a permanent baseline of 24 benign lines on every run. That baseline is what the unclassified check exists to prevent: a standing count of known-benign files is exactly what hides the one file that ever genuinely matters. Count full-precision sources separately and give them their own summary line, so unclassified returns to 0 and stays loud when something really is an unknown shape. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(apexentries): namespace local paths by owner and enable MTP builds localPath namespaced downloads by the repo basename alone, so two repos publishing the same filename under different owners collapsed to one local path. LiquidAI/LFM2.5-8B-A1B-GGUF and unsloth/LFM2.5-8B-A1B-GGUF collided that way, and both were offered from the same hub, so installing the second either overwrote the first model's weights or was skipped as already present while recording a sha256 that did not match the bytes on disk. The owner is now its own path segment: owner/repo is globally unique on HuggingFace and neither half can contain a separator, so uniqueness holds by construction. Verify gains a check for the whole class, that no local filename may map to two different upstream URIs. It surfaces seven pre-existing collisions in the gallery, which are left alone here. Entries built from the *-APEX-MTP-GGUF repos now configure MTP rather than shipping the heads inert, matching the pattern the hand-written MTP entries already use: spec_type:draft-mtp with spec_n_max and spec_p_min, tagged mtp, and no draft_model because the heads live in the weights. RenderChild no longer requires a separate drafter file before it will configure a spec type, while the cross-repo drafter path is unchanged. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add the APEX GGUF families as variant ladders Adds the imatrix quality ladder from each mudler/*-APEX-GGUF repo, a fixed subset of unsloth quant rungs where a counterpart repo exists, and the MTP builds, then attaches them to the base model entry so one entry offers every build of the same weights and LocalAI picks the one that fits the hardware. Ten existing base model entries gain a variants list; twenty-seven families that the gallery had no base entry for get one. Builds are discovered from the filenames each repo actually publishes rather than derived from its name, since six repos ship a stem that differs from their repo name. Every file carries a sha256 taken from the HuggingFace API. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
65bdbc4ee3 |
fix(http): make /readyz reflect startup readiness, plus gitignore and coverage-ratchet fixes (#10989)
* fix(http): make /readyz reflect startup readiness instead of always 200 /readyz was registered as a static handler returning 200 unconditionally, so it carried no information: it was green whenever it could be reached at all. Readiness could not distinguish "serving" from "still starting", and any future change that started the HTTP listener earlier would silently turn the probe into a lie. Track startup completion on the Application (atomic flag, flipped at the very end of New() on the success path only) and have the readiness handler consult it per request, returning 503 with a small JSON body while startup is in progress. A nil readiness source fails open so embedders keep the historical behaviour. /healthz is deliberately left readiness-independent. Liveness and readiness answer different questions, and failing liveness during a long preload makes an orchestrator restart the pod mid-download so the preload never finishes. This matters because since #10949 the startup preload materializes HuggingFace artifacts for managed backends: tens of GB for a large model (31 GB observed on a live cluster). Both probes stay in quietPaths and stay exempt from auth. Note the listener is still started only after New() returns, so today the not-ready state is not observable over HTTP. Moving the listener earlier is a separate, deliberate decision and is not made here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(gitignore): anchor the mock-backend pattern so its source dir is traversable The bare `mock-backend` pattern matched the *directory* tests/e2e/mock-backend/, not just the binary built into it. Git will not descend into an ignored directory even for tracked files, so `git add tests/e2e/mock-backend/main.go` required -f. This was hit while working on #10970. Anchor it to the artifact's full path. The built binary stays ignored (it is also covered by tests/e2e/mock-backend/.gitignore) while the source directory becomes traversable again. Verified with `git check-ignore -v`: a new source file under tests/e2e/mock-backend/ is no longer ignored, and the binary produced by `make build-mock-backend` still is. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(coverage): raise the coverage ratchet from 48.5% to 54.2% The committed baseline had drifted well below reality: it still read 48.5% while a full instrumented run measures 54.2%. A stale-low baseline makes the gate meaningless — coverage could regress by more than 5 percentage points and still pass. Raising a ratchet is a deliberate act, not something to fold into an unrelated fix, so it gets its own commit. The headroom was earned by tests landed in #10946, #10947, #10948, #10949, #10956, #10967, #10968, #10970 and #10975. Measured with `make test-coverage` on this branch (the same instrumented run `make test-coverage-baseline` uses: ginkgo over ./pkg and ./core plus the in-process tests/e2e suite, --covermode=atomic, --coverpkg over core/... and pkg/..., generated protobuf excluded). The run completed with exit 0 and zero spec failures; the total was then written with the exact command the test-coverage-baseline target uses: go tool cover -func=coverage/coverage.out \ | awk '/^total:/{gsub(/%/,"",$NF); print $NF}' > coverage-baseline.txt Verified afterwards with scripts/coverage-check.sh, which reports OK. Note the measured figure includes the readiness specs added earlier on this branch, so it is a demonstrated floor rather than an estimate. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
5d0c43ec6e |
feat(realtime): Semantic VAD EOU token (#10444)
* feat(realtime): EOU-driven semantic_vad turn detection Add a `semantic_vad` turn-detection mode to the realtime API that feeds the transcription model live and decides "the user finished speaking" from the `<EOU>` end-of-utterance token rather than from silence alone. When EOU fires the turn commits immediately (~0.3s); otherwise it falls back to an eagerness-scaled silence threshold (low/med/high = 8/4/2s). Plumbing, bottom to top: - proto: `AudioTranscriptionLive` bidirectional RPC (config-first oneof, mono float PCM @16k, ready-ack / Unimplemented degrade signal) plus `TranscriptResult.eou` for the unary retranscribe gate. - pkg/grpc: client/server/base/embed scaffolding for the bidi stream, modeled on AudioTransformStream; release stream conns on terminal Recv. - parakeet-cpp: live transcription RPC with per-C-call engine locking (one live stream per turn, finalize+free at commit); bump parakeet.cpp to ABI v5 — incremental StreamingMel (no more quadratic per-feed mel recompute that delayed EOU on long turns) and the <EOU>/<EOB> split; strip the literal <EOU>/<EOB> from offline text and set Eou. - core/backend: LiveTranscriptionSession wrapper + pipeline `turn_detection:` config block (type/eagerness/retranscribe). - realtime: semantic_vad integration — live input captions streamed as transcription deltas while the user speaks, EOU-immediate commit with eagerness fallback, optional retranscribe gate (batch re-decode must also end in <EOU> to confirm), clause synthesis off the LLM token callback, and per-turn live-transcription / model_load telemetry. - UI: show the realtime pipeline components as a vertical list. Docs and tests included; opt-in via the pipeline YAML or per-session `session.update`. Non-streaming STT backends degrade to silence-only. Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Write] [Bash] Assisted-by: Claude Code:claude-fable-5 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): explicit formally-verified state machines + parakeet streaming driver The realtime API had several implicit state machines whose state was inferred from scattered booleans, channels, and five separate mutexes, leaving illegal/inconsistent states reachable. Make them explicit and keep the implementation in step with a formal design; rework the parakeet streaming backend along the same lines. Realtime state machines (M1-M5). Each is a sealed sum-type State/Event/Effect with a total, pure Next(state,event)->(state,[]effect) behind a single-writer Coordinator: M1 conncoord connection lifecycle: VAD toggle + once-only teardown (replaces vadServerStarted + a `done` channel closed from two sites). M2 turncoord turn detection: collapses speechStarted and the live-stream "turn open" flag into one state, so discardTurn can no longer desync them and suppress the next onset. M3 respcoord response coordination: serializes the dual-writer start/cancel so at most one response is live; one response.done per response.create. M4 compactcoord conversation compaction: single-flight (replaces the `compacting atomic.Bool` CAS). M5 ttscoord TTS pipeline: open->closing->closed, idempotent wait(), rejects enqueue-after-close (was a silent drop). The Coordinator/Sink/Next plumbing — only the sealed types and Next differed per machine — is extracted once into core/http/endpoints/openai/coordinator as a generic Coordinator[S,E,F]; each machine keeps its public API via type aliases, so no sink, call-site, or test moved. Hierarchy. session_lifecycle.fizz models M1 as the parent region with its children (M2/M3/M4) as one statechart and asserts ChildrenDieWithParent (conn torn => all children terminal, none start after teardown). respcoord and compactcoord gain an absorbing Terminated state + Shutdown event; conncoord's teardown drives the children terminal. This closes a compaction teardown gap: a fire-and-forget compaction could outlive a torn session — compactionSink now takes a session-scoped cancellable context + WaitGroup and joins the in-flight summarize+evict on shutdown. Formal verification. formal-verification/ holds one authoritative FizzBee spec per machine plus the composition spec, each with an always-assertion and a documented one-line edit that makes the checker fail (verified non-vacuous). scripts/realtime-conformance.sh is fail-closed: all Go conformance suites under -race AND a model-check of every .fizz spec; a missing FizzBee is a hard error (only the loud REALTIME_CONFORMANCE_SKIP_FIZZBEE=1 bypasses it, never in CI). FizzBee is pinned by sha256 and installed via scripts/install-fizzbee.sh into .tools/ (gitignored). Wired as make test-realtime-conformance, a CI workflow, and a pre-commit path filter. Go conformance tests are Ginkgo/Gomega (per the repo's forbidigo lint): transition tables + fixed-seed property walks + concurrent/-race specs, no rapid dependency. Design map: docs/design/realtime-state-machines.md. Parakeet streaming backend. The same treatment applied to the parakeet-cpp streaming paths: - AudioTranscriptionStream returns codes.Unimplemented for non-streaming models instead of decoding offline and emitting it as one delta + final. A client that asked for streaming learns the model cannot stream rather than receiving a batch result shaped like a stream. New grpcerrors.StreamTranscriptionUnsupported carries that signal; the HTTP /v1/audio/transcriptions stream path surfaces it as an SSE error event. Mirrors AudioTranscriptionLive, which already did this. - utteranceBoundary (boundary.go): a single definition of the end-of-utterance latch, replacing three open-coded finalEou toggles. Modelled as a two-valued type so illegal states are unrepresentable. - Shared decode driver (driver.go): streamFeedResult (one per-feed event) + feedChunk (hides the ABI v4 JSON vs text-only split) + feedSlices + flushTail. The feed loop is written once. - AudioTranscriptionLive becomes a bidi adapter: it streams the per-feed {delta,eou,eob,words} the realtime turn detector consumes and a terminal FinalResult carrying only Text. Segments/duration/eou are offline-only and no longer produced (nor read) on the live path; liveTraceState drops the terminal eou and keeps the per-feed eou_events count. - AudioTranscriptionStream + streamJSON merge into one driver-based function; streamSegmenter is generalized to the unified event with a text-only fallback that preserves the legacy (no-words) library's per-utterance segmentation. Verified: build/vet/gofumpt clean, golangci-lint 0 issues, all coordinator and parakeet packages under -race, the fail-closed conformance gate green, and make test-realtime (12 e2e WS+WebRTC). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
5b3572f8b8 |
feat(macos): sign and notarize the DMG, app, and server binary (#10510)
Produce a Gatekeeper-clean macOS distribution with no user workaround: - Launcher DMG + the LocalAI.app inside it are built via fyne, codesigned with the Developer ID under the hardened runtime, then the DMG is signed, notarized (notarytool) and stapled. Replaces macos-dmg-creator (which had no signing hook) with fyne package + hdiutil so we control the .app before packaging. - The bare local-ai darwin server binary is signed + notarized via GoReleaser's native notarize block (quill backend, runs on Linux). - All signing is gated on secrets being present, so forks/PRs/local builds stay unsigned and green (contrib/macos/sign-and-notarize.sh no-ops). - Add hardened-runtime entitlements and FyneApp.toml for deterministic packaging; update macOS install docs to drop the quarantine workaround. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
95b058e1c5 |
feat(ui): restructure Cluster Nodes view (pulse + panel roster + detail page) (#10447)
* chore: gitignore SDD scratch directory Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(nodes): add GET /api/nodes/models cluster-wide loaded-models endpoint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(ui): add nodesApi.allModels() for cluster-wide model roster Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(ui): move Scheduling to its own page and nav item Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(ui): replace nodes stat-card strip with cluster pulse + attention callout Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(ui): node-panel roster with inline model chips and segmented filter Replace the Nodes table with a full-width node-panel roster that shows each backend node's running-model chips without an expand click, plus an All/Backend/Agent segmented filter. Per-node detail (models, backends, labels, capacity) moves to the node detail page. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(ui): add deep-linkable node detail page at /app/nodes/:id Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(ui): remove em-dash from CapacityEditor comment; align detail spec backend mock Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(ui): nodes page cleanup, hover/chip polish, docs for restructured cluster view Nodes.jsx dead-code sweep confirmed clean (no StatCard/table/expand state/scheduling-form leftovers). Two App.css polish fixes: move the node-panel hover border-color onto the bordered element so hover gives real feedback, and add the missing .model-chip__state rule the ModelChip component already emits. Update distributed-mode docs prose to describe the restructured cluster view (cluster pulse, attention callout, node-panel roster with inline model chips, All/Backend/Agent filter, node detail page at /app/nodes/:id, Scheduling as its own page). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(ui): drop unused gpuVendorLabel export from nodeStatus Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
8d70855ea6 |
test: add Go + React UI coverage gates and fill test gaps (#9989)
- Strict monotonic Go coverage gate (make test-coverage-check, 45% baseline) run in CI; fixes ginkgo dropping all-but-one coverprofile across multiple recursive roots, builds with -tags auth, and folds in the in-process tests/e2e suite via --coverpkg. - React UI e2e coverage (make test-ui-coverage: vite-plugin-istanbul + nyc, nix-provided Chromium) plus e2e specs for 6 previously-untested pages, and a UI coverage gate (make test-ui-coverage-check) with a small tolerance since e2e line coverage jitters ~0.5pp run-to-run. - pre-commit hook: lint + coverage on Go changes, Playwright e2e + UI coverage gate on react-ui changes; install with make install-hooks. - New Go handler tests (settings, branding), hermetic base64 download test. - fix(ui): model editor reads vram_display (snake_case), so the VRAM estimate renders again; covered by a regression test. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Richard Palethorpe
|
6a80e23733 |
feat(middleware): Model routing, PII filtering, Cloud model proxies (#9802)
Add a routing middleware stack and a cloud-proxy backend. * cloud-proxy: a Go gRPC backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers, with an optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response) and full tool-calling support. * routing: admission control, content-aware model routing (embedding cache + classifier + rerank + Arch-Router score), PII detection/redaction (regex + NER) with streaming filter and OpenAI/Anthropic adapters, and a per-user/per-key billing recorder backed by GORM or in-memory storage. * middleware: UsageMiddleware records usage via the billing recorder, plus admission, route-model, usage-stamp and trace middlewares. * observability: BackendTrace ring buffer stores full request bodies (capped), MITM proxy emits structured trace events, and router classifier decisions surface at /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0). * UI: cloud-proxy model-editor fields, classifier system-prompt and score-normalization config, and a Traces page rendering request bodies. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
0b2ae3c6ca |
fix(openai): stream usage non-zero when tools are enabled (#9941)
* chore: ignore local .worktrees directory Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(openai): stream usage non-zero when tools are enabled The streaming chat-completions worker for tool-bearing requests (processTools in core/http/endpoints/openai/chat.go) never forwarded the cumulative TokenUsage from ComputeChoices to the chunks it placed on the responses channel. The outer streaming loop's running usage tracker therefore stayed at the zero value, and the include_usage trailer reported {prompt_tokens:0, completion_tokens:0, total_tokens:0} whenever the request carried a `tools` array. Without tools, the alternative `process` path stamps Usage on every chunk, so that path was unaffected. Forward the final TokenUsage via a usage-only sentinel chunk (empty Choices, populated Usage) emitted right before close(responses). The outer loop's per-chunk Usage capture moves above the empty-Choices skip so the sentinel updates the tracker without ever reaching the wire, keeping the existing OpenAI spec contract (intermediate chunks carry no `usage` field, and the deferred-final-chunk helpers remain Usage-free per the regression test for issue #8546). Adds streamUsageFromTokenUsage, usageSentinelChunk, and applyChunkToUsage helpers with focused Ginkgo coverage plus a flow-level test that mirrors the outer-loop sequence. Fixes #9927 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4-7 [Claude Code] * refactor(openai): return final TokenUsage from stream workers Replace the usage-only sentinel SSE chunk introduced in the previous commit with a plain return value. The streaming workers process and processTools (now extracted as package-level processStream and processStreamWithTools) return (backend.TokenUsage, error); the outer ChatEndpoint loop reads the cumulative counts off the existing `ended` channel (now carrying streamWorkerResult{usage, err}) and builds the include_usage trailer from a normal Go value after the LOOP exits. This drops the empty-Choices "skip but capture Usage" rule from the outer loop and removes the usageSentinelChunk / applyChunkToUsage helpers entirely. The SSE responses channel is back to a single purpose: wire chunks only. processStream and processStreamWithTools move into chat_stream_workers.go so they can be exercised directly from tests. The chat_stream_usage_test.go suite now drives the workers with a mocked backend.ModelInferenceFunc and asserts on the returned TokenUsage. The regression coverage for issue #9927 is therefore behavioral: reverting the fix (discarding ComputeChoices' usage return) makes the assertions fail with concrete count mismatches. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4-7 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
3d9ccd1ddc |
fix(ui): Add tracing inline settings back and create UI tests (#9027)
Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Ettore Di Giacinto
|
5affb747a9 |
chore: drop AIO images (#9004)
AIO images are behind, and takes effort to maintain these. Wizard and installation of models have been semplified massively, so AIO images lost their purpose. This allows us to be more laser focused on main images and reliefes stress from CI. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
f9a850c02a |
feat(realtime): WebRTC support (#8790)
* feat(realtime): WebRTC support Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(tracing): Show full LLM opts and deltas Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Ettore Di Giacinto
|
09ddaf94b2 |
feat(ui): move to React for frontend (#8772)
* feat(ui): move to React Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add import model Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * syntax highlight Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Minor fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
b6459ddd57 |
feat(api): Add transcribe response format request parameter & adjust STT backends (#8318)
* WIP response format implementation for audio transcriptions (cherry picked from commit e271dd764bbc13846accf3beb8b6522153aa276f) Signed-off-by: Andres Smith <andressmithdev@pm.me> * Rework transcript response_format and add more formats (cherry picked from commit 6a93a8f63e2ee5726bca2980b0c9cf4ef8b7aeb8) Signed-off-by: Andres Smith <andressmithdev@pm.me> * Add test and replace go-openai package with official openai go client (cherry picked from commit f25d1a04e46526429c89db4c739e1e65942ca893) Signed-off-by: Andres Smith <andressmithdev@pm.me> * Fix faster-whisper backend and refactor transcription formatting to also work on CLI Signed-off-by: Andres Smith <andressmithdev@pm.me> (cherry picked from commit 69a93977d5e113eb7172bd85a0f918592d3d2168) Signed-off-by: Andres Smith <andressmithdev@pm.me> --------- Signed-off-by: Andres Smith <andressmithdev@pm.me> Co-authored-by: nanoandrew4 <nanoandrew4@gmail.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |
||
|
Richard PalethorpeandAlessandro Sturniolo
|
e6ba26c3e7 |
chore: Update to Ubuntu24.04 (cont #7423) (#7769)
* ci(workflows): bump GitHub Actions images to Ubuntu 24.04 Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): remove CUDA 11.x support from GitHub Actions (incompatible with ubuntu:24.04) Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): bump GitHub Actions CUDA support to 12.9 Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(docker): bump base image to ubuntu:24.04 and adjust Vulkan SDK/packages Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * fix(backend): correct context paths for Python backends in workflows, Makefile and Dockerfile Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(make): disable parallel backend builds to avoid race conditions Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(make): export CUDA_MAJOR_VERSION and CUDA_MINOR_VERSION for override Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(backend): update backend Dockerfiles to Ubuntu 24.04 Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(backend): add ROCm env vars and default AMDGPU_TARGETS for hipBLAS builds Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(chatterbox): bump ROCm PyTorch to 2.9.1+rocm6.4 and update index URL; align hipblas requirements Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore: add local-ai-launcher to .gitignore Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): fix backends GitHub Actions workflows after rebase Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(docker): use build-time UBUNTU_VERSION variable Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(docker): remove libquadmath0 from requirements-stage base image Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(make): add backends/vllm to .NOTPARALLEL to prevent parallel builds Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * fix(docker): correct CUDA installation steps in backend Dockerfiles Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * chore(backend): update ROCm to 6.4 and align Python hipblas requirements Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): switch GitHub Actions runners to Ubuntu-24.04 for CUDA on arm64 builds Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(docker): update base image and backend Dockerfiles for Ubuntu 24.04 compatibility on arm64 Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * build(backend): increase timeout for uv installs behind slow networks on backend/Dockerfile.python Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): switch GitHub Actions runners to Ubuntu-24.04 for vibevoice backend Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * ci(workflows): fix failing GitHub Actions runners Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> * fix: Allow FROM_SOURCE to be unset, use upstream Intel images etc. Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(build): rm all traces of CUDA 11 Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(build): Add Ubuntu codename as an argument Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: Alessandro Sturniolo <alessandro.sturniolo@gmail.com> |
||
|
Ettore Di Giacinto
|
f8a8cf3e95 |
feat(launcher): add LocalAI launcher app (#6127)
* Add launcher (WIP) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Update gomod Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Cleanup, focus on systray Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Separate launcher from main Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add a way to identify the binary version Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Implement save config, and start on boot Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Save installed version as metadata Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Stop LocalAI on quit Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix goreleaser Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Check first if binary is there Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * do not show version if we don't have it Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to build on CI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * use fyne package Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add to release Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fyne.Do Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * show WEBUI button only if LocalAI is started Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Default to localhost Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * CI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Show rel notes Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Update logo Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small improvements and fix tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to fix e2e tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b7b3164736 |
chore: try to speedup build
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
98e5291afc |
feat: refactor build process, drop embedded backends (#5875)
* feat: split remaining backends and drop embedded backends - Drop silero-vad, huggingface, and stores backend from embedded binaries - Refactor Makefile and Dockerfile to avoid building grpc backends - Drop golang code that was used to embed backends - Simplify building by using goreleaser Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(gallery): be specific with llama-cpp backend templates Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(docs): update Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ci): minor fixes Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: drop all ffmpeg references Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: run protogen-go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Always enable p2p mode Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Update gorelease file Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(stores): do not always load Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix linting issues Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Simplify Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Mac OS fixup Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
294f7022f3 |
feat: do not bundle llama-cpp anymore (#5790)
* Build llama.cpp separately Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * WIP Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * WIP Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * WIP Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Start to try to attach some tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add git and small fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: correctly autoload external backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to run AIO tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Slightly update the Makefile helps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Adapt auto-bumper Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to run linux test Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add llama-cpp into build pipelines Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add default capability (for cpu) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop llama-cpp specific logic from the backend loader Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * drop grpc install in ci for tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Pass by backends path for tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Build protogen at start Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tests): set backends path consistently Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Correctly configure the backends path Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to build for darwin Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * WIP Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Compile for metal on arm64/darwin Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to run build off from cross-arch Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add to the backend index nvidia-l4t and cpu's llama-cpp backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Build also darwin-x86 for llama-cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Disable arm64 builds temporary Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Test backend build on PR Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixup build backend reusable workflow Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * pass by skip drivers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Use crane Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Skip drivers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * x86 darwin Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add packaging step for llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix leftover from bark-cpp extraction Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to fix hipblas build Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
58ff47de26 |
feat(bark-cpp): add new bark.cpp backend (#4287)
* feat(bark-cpp): add new bark.cpp backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build on linux only for now Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * track bark.cpp in CI bumps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop old entries from bumper Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * No need to test rwkv specifically, now part of llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2b62260b6d |
feat(models): use rwkv from llama.cpp (#4264)
feat(rwkv): use rwkv from llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Dave
|
9a8a249932 |
feat: devcontainer part 3 (#3318)
* stash initial fixes, attempt to open branch inside container Signed-off-by: Dave Lee <dave@gray101.com> * add yq, from inside DC Signed-off-by: Dave Lee <dave@gray101.com> * stash progress, rebuild container Signed-off-by: Dave Lee <dave@gray101.com> * snap Signed-off-by: Dave Lee <dave@gray101.com> * split builder into builder-sd, will speed up devcontainer build times and potentially help caching in other situations. Signed-off-by: Dave Lee <dave@gray101.com> * fix yq Signed-off-by: Dave Lee <dave@gray101.com> * fix paths Signed-off-by: Dave Lee <dave@gray101.com> * fix paths - new folder to bypass the .dockerignore which _should_ exclude the other files Signed-off-by: Dave Lee <dave@gray101.com> * fix Signed-off-by: Dave Lee <dave@gray101.com> * fix ] Signed-off-by: Dave Lee <dave@gray101.com> --------- Signed-off-by: Dave Lee <dave@gray101.com> |
||
|
Ettore Di Giacinto
|
882556d4db |
feat(gallery): show available models in website, allow local-ai models install to install from galleries (#2555)
* WIP Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gen a static page instead (we force DNS redirects to it) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): install models from CLI, unify install Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Uniform graphic of model page Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Makefile: update targets Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Slightly enhance gallery view Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
ea330d452d |
models(gallery): add mistral-0.3 and command-r, update functions (#2388)
* models(gallery): add mistral-0.3 and command-r, update functions Add also disable_parallel_new_lines to disable newlines in the JSON output when forcing parallel tools. Some models (like mistral) might be very sensible to that when being used for function calling. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * models(gallery): add aya-23-8b Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
cryptk
|
28a421cb1d |
feat: migrate python backends from conda to uv (#2215)
* feat: migrate diffusers backend from conda to uv
- replace conda with UV for diffusers install (prototype for all
extras backends)
- add ability to build docker with one/some/all extras backends
instead of all or nothing
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate autogtpq bark coqui from conda to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: convert exllama over to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate exllama2 to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate mamba to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate parler to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate petals to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: fix tests
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate rerankers to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate sentencetransformers to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: install uv for tests-linux
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: make sure file exists before installing on intel images
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate transformers backend to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate transformers-musicgen to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate vall-e-x to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: migrate vllm to uv
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: add uv install to the rest of test-extra.yml
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: adjust file perms on all install/run/test scripts
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: add missing acclerate dependencies
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: add some more missing dependencies to python backends
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: parler tests venv py dir fix
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: correct filename for transformers-musicgen tests
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: adjust the pwd for valle tests
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: cleanup and optimization work for uv migration
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: add setuptools to requirements-install for mamba
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: more size optimization work
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* feat: make installs and tests more consistent, cleanup some deps
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: cleanup
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: mamba backend is cublas only
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
* fix: uncomment lines in makefile
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
---------
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>
|
||
|
cryptk
|
3411e072ca |
Fix cleanup sonarqube findings (#2106)
* fix: update dockerignore and gitignore to exclude sonarqube work dir Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: remove useless equality check Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: use sonarqube Dockerfile recommendations Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> --------- Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> |
||
|
cryptk
|
1981154f49 |
fix: dont commit generated files to git (#1993)
* fix: initial work towards not committing generated files to the repository Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: improve build docs Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: remove unused folder from .dockerignore and .gitignore Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: attempt to fix extra backend tests Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: attempt to fix other tests Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: more test fixes Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: fix apple tests Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: more extras tests fixes Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: add GOBIN to PATH in docker build Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: extra tests and Dockerfile corrections Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: remove build dependency checks Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: add golang protobuf compilers to tests-linux action Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: ensure protogen is run for extra backend installs Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: use newer protobuf Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: more missing protoc binaries Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: missing dependencies during docker build Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: don't install grpc compilers in the final stage if they aren't needed Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: python-grpc-tools in 22.04 repos is too old Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: add a couple of extra build dependencies to Makefile Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: unbreak container rebuild functionality Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> --------- Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> |
||
|
Dave
|
1c312685aa |
refactor: move remaining api packages to core (#1731)
* core 1 * api/openai/files fix * core 2 - core/config * move over core api.go and tests to the start of core/http * move over localai specific endpoints to core/http, begin the service/endpoint split there * refactor big chunk on the plane * refactor chunk 2 on plane, next step: port and modify changes to request.go * easy fixes for request.go, major changes not done yet * lintfix * json tag lintfix? * gitignore and .keep files * strange fix attempt: rename the config dir? |
||
|
Ettore Di Giacinto
|
db926896bd |
Revert "[Refactor]: Core/API Split" (#1550)
Revert "[Refactor]: Core/API Split (#1506)"
This reverts commit
|
||
|
Dave
|
ab7b4d5ee9 |
[Refactor]: Core/API Split (#1506)
Refactors api folder to core, creates firm split between backend code and api frontend. |
||
|
Dave
|
8b6e601405 |
Feat: new backend: transformers-musicgen (#1387)
Transformers-MusicGen --------- Signed-off-by: Dave <dave@gray101.com> |
||
|
Ettore Di Giacinto
|
3c9544b023 |
refactor: rename llama-stable to llama-ggml (#1287)
* refactor: rename llama-stable to llama-ggml * Makefile: get sources in sources/ Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixup path Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixup sources Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups sd Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * update SD * fixup * fixup: create piper libdir also when not built Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix make target on linux test Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Dave
|
6bd56460de |
Update .gitignore for backend/llama.cpp (#1235)
Signed-off-by: Dave <dave@gray101.com> |
||
|
Dave
|
901f0709c5 | Feat: rwkv improvements: (#937) | ||
|
Dave
|
8cb1061c11 | Usage Features (#863) | ||
|
DaveandEttore Di Giacinto
|
ae58fb8821 |
fix: update gitignore and make clean (#798)
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
e459f114cd |
fix: fix tests, small refactors
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
1d0ed95a54 |
feat: move other backends to grpc
This finally makes everything more consistent Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
ae533cadef |
feat: move gpt4all to a grpc service
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b816009db0 |
feat: add falcon ggllm via grpc client
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2a11f16c0f | fix: copy metal file from build (#564) | ||
|
Ettore Di Giacinto
|
78ad4813df |
feat: Update gpt4all, support multiple implementations in runtime (#472)
Signed-off-by: mudler <mudler@mocaccino.org> |
||
|
Pavel ZloiandEttore Di Giacinto
|
3ba07a5928 |
feat: add LangChainGo Huggingface backend (#446)
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |
||
|
AisukoandEttore Di Giacinto
|
49ce24984c |
feat: Add more test-cases and remove dev container (#433)
Signed-off-by: Aisuko <urakiny@gmail.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
43d3fb3eba | ci: add binary releases pipelines (#358) | ||
|
Ettore Di Giacinto
|
9d051c5d4f | feat: add image generation with ncnn-stablediffusion (#272) |