mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
feat(ui): replace the stacked operations bar with a one-line strip and an Activity page (#11163)
* feat(ui): record finished gallery operations in a bounded history ring The operations panel drops an operation the moment it succeeds, so a user who steps away cannot tell whether an install finished, failed or was never started. OpCache now keeps the last 50 terminal operations, recorded from the point where an op leaves the cache. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): pin the history ring's dedupe, outcome order and start stamp Review of the history ring found four gaps. The dedupe guard and the bounded seen set were unreachable through the exported API and so had no coverage; an in-package spec file now drives opHistory directly. The outcome switch claimed an ordering was load bearing that nothing pinned, so an errored op that never reached Processed now has a spec. Two behaviour fixes come with it. StartedAt was the zero time for ops recovered from the store or replicated from a peer, since neither path stamps a start time, which would have rendered as a two-millennia duration; it now falls back to the finish time. Reusing a cache key with a fresh job ID orphaned the previous stamp, so Set and SetBackend now drop it. The comment on the outcome switch described a state the code cannot be in: CancelOperation sets Cancelled and Processed synchronously before the handler removes the entry, so status.Cancelled already covers the cancel endpoint. The !Processed clause stays for the dismiss endpoint firing on an in-flight op, and the comments now say so. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): record operations that end on a peer replica The NATS end event is the only signal a replica gets for an install another replica ran. Record from applyEnd too, deduped by job ID so the originating replica does not record its own broadcast twice. Three start-stamp defects in the same path go with it. applyEnd now drops the stamp unconditionally, since recordTerminal only cleans up on the path where it found a cache key and an end event can overtake the local Set. applyStart drops the stamp of the job whose cache key it replaces, which a peer-driven retry previously stranded. And recordTerminal reads the stamp once instead of testing Exists and then reading, so a concurrent record for the same job can no longer delete the stamp between the two and let the zero time overwrite the finish-time fallback, which the Activity page would render as a two-millennia run. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): do not guess the outcome of a peer operation with no local status A replica that restarts mid-operation hydrates its OpCache keys from PostgreSQL, but gallery statuses are in-memory only and come back empty. The end broadcast then landed on recordTerminal's nil-status branch, which reads a missing status as queued-and-removed and filed a successful install as cancelled. That reading is right locally and wrong on the peer path, where a missing status means the outcome was never held here. recordTerminal now takes the source of the terminal event and records nothing when the peer path finds no status, restoring what the replica did before the end event started recording. The local path is unchanged. Also move the ApplyEndForTest seam to the conventional export_test.go, and stop the dedupe spec from claiming to guard the ring's seen set: the local delete removes the status keys, so the broadcast that follows returns before reaching it. An in-package spec that calls recordTerminal twice does the pinning. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(api): add GET and DELETE /api/operations/history Admin gated like the rest of the operations API. The live /api/operations payload is unchanged so the one second poll stays small. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): expose operation history through OperationsContext Fetched on demand and when the live list shrinks, never on the one second poll interval. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): detect operation departure by identity and ignore committing ops in the ETA gate Refetching history on a shrinking live count missed a completion that coincided with a start, which is the common case during a batch install. Track the live job IDs instead, so any departure triggers the refetch regardless of how the count moved. An operation that has finished downloading stays live at currentBytes == totalBytes for the whole commit and install phase and can never produce an estimate, so counting it in the all-or-nothing gate blanked every other operation's time remaining for as long as it lasted. Only operations still moving bytes get a vote. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): let only downloading operations gate the time remaining estimate Verifying pins an operation flat below its total for the whole sha256 pass: the AfterDownload hook reports completedBytes plus the finished file against a total summed over every file, then hashes synchronously without emitting progress. Files download sequentially, so a 15 shard model enters that window 14 times, and a byte comparison cannot see it because the counter is genuinely below the total throughout. Gating on phase closes resolving, verifying, committing and persisting in one predicate, so a quiet neighbour no longer blanks every other operation's estimate for minutes at a time. The byte clauses stay: a producer can report downloading with bytes already at the total. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): collapse the operations bar to a single line Four concurrent installs used to take four rows above every page. The strip now shows one operation, failure first, with a counter linking to Activity. The close button hides the strip and no longer cancels an install. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): keep the operations strip from widening the page and from muting a failure A long install error made the strip report a 1600px minimum width, which sized main-content to fit and gave every page under it a horizontal scrollbar. Inline-size containment plus shrinkable detail and bytes cells keep it inside the viewport. Hiding is no longer able to swallow the hidden job's own failure, a completed removal or staging says so instead of claiming an install, a cancelling operation renders as cancelling, and the live region no longer covers the per-second percentage. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): shrink main-content instead of containing the strip, and expose progress min-width on .main-content is what actually lets a long install error shrink, and unlike inline-size containment it has no browser support floor and no latent collapse if the strip ever lands in a shrink-to-fit context. It matches what .app-layout-chat .main-content already does, and it clears pre-existing horizontal overflow on narrow viewports as a side effect. The progress track is now a labelled progressbar, so assistive tech can read the value on demand rather than losing it to the aria-hidden that stopped the live region re-announcing every poll. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add the live operation card for the Activity page Carries the detail the one-line strip has to drop: phase, bytes, the per-node breakdown for cluster installs, and a labelled Cancel button. Cancelling is destructive, so it gets a labelled button rather than a glyph. A cancelling operation drops its progress bar and its time estimate, the same call the strip makes: a percentage still climbing under "Cancelling" reads as the cancel not having taken. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): give the operation card a verb, live node disclosure and its per-node detail The card carried no verb, so an install, a removal and a staging op rendered as spinner plus name plus kind tag and were indistinguishable. It now runs the same verb and icon chain as the one-line strip, which is what stops the page that is meant to carry more detail from carrying less. The auto-expand default was evaluated once at mount. An operation is listed as soon as it is admitted but its nodes are filled in only when the fan-out starts reporting, so a card mounted at creation latched on the empty list and stayed collapsed. The default is a live expression now, and state holds only an explicit choice. Also: an optional onRetry gates a Retry button, so the page can own the install reconstruction without the card ever showing a control with nothing behind it; the disclosure moved above the region it controls and gained aria-controls; the toggle is gated at more than one node so the count is never "1 nodes"; an unmapped node status is passed through instead of being relabelled "Queued"; error text is clamped with the full string in the title; and file_name plus the per-node progress bar are rendered again, reviving three CSS rules that had gone dead along with the detail they styled. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add the Activity page Live operations, unacknowledged failures and the record of what finished, at /app/activity in the Operate console. Cancelling an install now lives here behind a labelled button rather than on the strip, and a failed install can be retried: the retry dismisses the failure first so it still reaches the record, then reissues the model, backend or node-scoped backend install. The sidebar Operate entry carries the operation count. The console rail is only rendered on an Operate route and can be collapsed, so a badge there could vanish while operations were still running. Two follow-ups from review fold in here: a failed removal or staging job no longer reports a failed install on either the card or the strip, and the card's error text can shrink so one unbroken token cannot widen the card. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): dismiss operations by job, and stop the Activity page contradicting itself Dismissing resolved the job by display id, but /api/operations strips the "node:<nodeID>:" prefix before emitting, so a local install and a node-scoped install of one backend arrive as two jobs sharing one id. Dismissing by id retired whichever came first. That defeated the guarantee retry was built around: with the wrong job dismissed, the reinstall overwrote the acted-on failure's opcache entry in place, bypassing recordTerminal, while an unrelated failure vanished from Needs attention. dismissFailedOp, the card's dismiss control and the strip now all pass the jobID, which is what the endpoint takes. A filter matching nothing rendered the "nothing has ever run" empty state while the header counted the records the filter had hidden. The empty state is now gated on the All chip and a narrowed view gets its own message plus a way back; the header counts the instance rather than the chip, so selecting Backends no longer reports "Nothing running" over running model installs. Also: the summary drops a zero clause instead of rendering "0 needs attention" on the happy path and pluralises both counts; a record duration is floored at "< 1s" and rejected above a day, so a zero-value start stamp cannot render a span of millennia and a zero span cannot render "installed in" with nothing after it; a deletion cancelled mid-flight reports the cancellation rather than claiming it was removed; and the retry variant comment names the fix instead of calling the gap closed. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: document the Activity page and the operations history endpoints Adds an Activity page under Operations covering the one-line operations strip, the /app/activity sections and filters, per-operation cancel, retry and dismiss, the in-memory 50-entry record, and the sidebar count. Documents GET and DELETE /api/operations/history, and fills the gap in the admin-only endpoint list, which also omitted the pre-existing POST /api/operations/:jobID/dismiss. Corrects the distributed-mode install-watching section: the per-node breakdown now lives on the Activity page rather than on the strip, which rolls a fan-out up into a single phrase. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct nine details in the Activity page documentation The operations strip never renders a file name: its detail line is the error, the node roll-up, the target node, the phase or the queued note. Drops the stale clause in the distributed-mode section, where the per-node bullet is now the only place a file name is described. Scopes the phase vocabulary to artifact-backed gallery models, since a plain GGUF install emits no phase. Corrects the per-node list: the toggle exists for any fan-out of two or more workers and the four-node threshold only governs whether it starts open, while the N nodes tag needs more than one node. Notes that a cancelled operation can sit in the live section reading Cancelling, that cluster staging never reaches the record, and that Clear history appears only when the record has something in it. Names the operations response envelope, with a JSON example, so callers do not index a bare array, and stops describing the icon-only dismiss control as a labelled button. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: drop the unreachable Cancelling state and scope the byte claims An operation can only report isCancelled while it is unprocessed, but every writer of Cancelled sets Processed in the same breath, on the peer path as much as the local one, and the cache evicts cancelled entries before the handler sees them. The state cannot reach the page, so the live section is described again as running or queued operations. Byte counts come from the artifact bridge alone, the same producer as the phase, so a plain GGUF install, a removal and a backend install report none. Scopes both to artifact-backed gallery models and leaves the verb, the name and the percentage as what every operation shows. A worker backend install reports its bytes through fields the operations payload does not carry, so the distributed section now describes the percentage and the node roll-up, with per-file counts pointed at the per-node detail. Also: staging jobs carry no error, so they never reach Needs attention and Retry never had a staging case to exclude; an install that involves workers is no longer called node-scoped, which this page uses for node-targeted installs; and the record timestamps carry nanoseconds. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: state only the verb and the name as unconditional on the strip The percentage is as conditional as the bytes were: it renders only for a running operation that has reported progress, so a queued operation, a failed one and a removal never carry it. A removal in particular sits at progress zero for its whole visible life, since the delete path reports none and its completion is filtered out. Both the strip and the card paragraphs now lead with what always shows and list the rest as conditions. The Cluster chip matches on a node list that finished operations do not carry, so a fan-out install leaves the chip once it reaches the record. Scoped that claim to the live sections. Two more of the same shape, found by re-reading each clause alone: the strip also appears for a failure, which is not running, and the four-second hold only applies when nothing replaces the operation that just finished. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): stop reporting a cancelled install as installed, and make queued real Three defects that all trace to one root cause: `isCancelled: true` is unreachable from /api/operations. Every writer of Cancelled=true also sets Processed=true, the handler skips Processed && Cancelled, and OpCache.GetStatus evicts a cancelled op before the handler iterates it. Cancelling the last running operation put a green "Installed model X" on the strip for four seconds: the completion hold was guarded by `!previous.isCancelled`, which is dead. A cancellation deletes the operation server side, so the strip sees exactly what it sees on a completion, and nothing in the payload separates the two. The signal now comes from the side that issued the cancel: the operations context remembers the job IDs it cancelled (pruned after a minute) and the strip asks before it holds anything. A cancelled operation goes as soon as it stops; the record already reports it as cancelled. isQueued was set only when the gallery status was missing, but markQueued publishes a "queued" status at admission, so a queued op has a status for its whole queued life and the state was unreachable outside a microsecond window. Every operation waiting behind a running install rendered as "Installing model X" with a spinner. The queued phase is now the signal, via an exported PhaseQueued and a nil-safe OpStatus.IsQueued() next to the writer. With those two fixed, the Cancelling state has no way to be entered: cancelling is instantaneous from the API's point of view. Its branches, CSS, locale key and the isCancelled field itself are removed rather than left for a future reader to assume they work. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): keep a removal a removal, and say what an install is doing OpStatus.Deletion was set once, at admission, and lost on the next status write: UpdateStatus replaces the whole status and only carried Nodes forward. Every later writer (the worker's first write, the progress ticks, the failure path) leaves the field at its zero value, so the flag survived only the queued window, and both surfaces test isQueued first. The reachable consequence is that a failed removal reported itself as a failed install, which is exactly the shape the Activity page offers Retry for, and Retry installs: pressing it on a removal that failed re-downloaded the model. A running delete also rendered as "Installing model X" with a spinner, and a successful one as "Installed model X". Carry Deletion forward the way Nodes already is. A job is a delete or an install for its whole life; an unset flag means "no new information", not "this is an install". Pinned by Go specs on both the service and /api/operations: the existing Playwright specs were green only because they stubbed a payload the server could not emit. Also restore the operation's own status message on the Activity card. Phases and byte counters exist only on the managed-artifact path, so a legacy files: gallery model and every backend install rendered a sub-row with nothing in it but the verb. The strip stays terse on purpose. And give the strip's name a min-width floor: overflow: hidden zeroes its automatic minimum, so a long error squeezed the name down to "mod…" and the identity of the thing that broke was the first thing lost. primaryOperation is made module-private: its comment claimed the Activity page selected the same operation, but that page shows all of them, partitioned into failed and running, and never imported it. Assisted-by: Claude Code:Opus 5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(activity): read the operations record from PostgreSQL The Activity page's record of finished installs and removals was a 50-entry in-memory ring per frontend replica. In distributed mode that is the wrong place for it: each replica keeps its own copy, a replica added by a scale-out or a rolling deploy starts empty and never backfills, and "Clear history" clears only the replica that served the request, so the record reappears on the next poll routed elsewhere. The data is already in gallery_operations. Read it from there. GalleryStore gains ListTerminal and ClearTerminal, sharing a lifted terminalStatuses set with CleanOld so there is one definition of "finished". ListTerminal orders by updated_at, when the operation reached its terminal status, because the record reports what finished and when. OpCache.History and ClearHistory dispatch on whether a store is wired, so the HTTP handlers and the OpRecord JSON shape are unchanged and the page needed no change. A failed store read falls back to the local ring rather than blanking the page, and ClearHistory empties the ring as well so a database blip cannot resurrect a record the admin just cleared. The name derivation in recordTerminal is lifted into operationDisplayName and used by both paths, so the ring and the store cannot name the same operation differently. Also fixes a pre-existing bug the store path made visible: the backend channel hardcoded op_type "backend_install" even for a removal, while the model channel derives model_install/model_delete from op.Delete. Both channels carry the same ManagementOp, whose Delete field the backend handler already branches on, so the backend channel now derives backend_delete the same way. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(activity): keep a cancelled operation cancelled, and report a failed clear Review follow-up on the store-backed Activity record. A cancelled install was recorded as a failure. The cancel handler persists "cancelled" synchronously, then the handler goroutine unwinds with the context error and Start hands that to updateError unconditionally, which overwrote the row with "failed: context canceled". The page rendered a cancelled install as a red failure card offering Retry, with a raw context error as the reason. Fixed in GalleryStore rather than in Start, because an operation finishes once and the paths that retire one are not mutually exclusive: UpdateStatus now refuses to rewrite a row that already reached a terminal status. That also pins updated_at to when the operation really finished, which is the key the record is ordered by, and Create's upsert now freezes the same columns so a worker dequeuing an operation the admin cancelled while it was queued cannot reopen it as pending. ClearHistory returned nothing, so a failed delete logged a warning while the handler still answered 200. The admin watched the record clear and come back on the next fetch with nothing said about why. It now returns the error, the DELETE handler answers 500, and the store is cleared before the local ring so a failure leaves the fallback record intact rather than faking an empty one. Hydrate is the only reader that decides from op_type whether an operation is a removal, and it tested for "model_delete" exactly, so the backend_delete added in the previous commit hydrated as an install: a replica restarting during a backend removal rendered "Installing backend X". Both discriminations now go through IsDeleteOpType/IsBackendOpType so a fifth op_type cannot silently read as an install in whichever consumer was missed. Also: the backend channel now persists Cancellable as !op.Delete, matching the model channel; IsBackend falls back to the op_type prefix, since is_backend_op is only written by UpsertCacheKey and the rows needing the name fallback were reporting backend operations as models; and an unrecognized terminal status is logged rather than quietly filed as a success, which is what the comment already claimed. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(activity): keep a reaped operation correctable by its real outcome The terminal-status freeze added in the previous commit was too wide. It froze "failed" alongside "completed" and "cancelled", and the stale reaper writes "failed" onto operations that are still going to run. The gallery worker is a single goroutine consuming both channels serially, so an operation queued behind a large download sits in "pending" with nothing bumping updated_at, and ReapStaleOperations gives up on it after 30 minutes. That used to be self-healing: the worker dequeued it, Create reset the row to "pending", and the operation reported its real outcome. With the freeze the row stayed "failed" forever while the install ran and succeeded underneath it: a red failure card offering Retry for a model that is installed, omitted from ListActive so no replica hydrates it, and no longer deduped cluster-wide by FindDuplicate. Freeze on ("completed", "cancelled") instead. That is all the cancelled-install fix ever needed, and it leaves a failure correctable by what actually happened. The set is separate from terminalStatuses, which ListTerminal, ClearTerminal and CleanOld all still want in full, because the two mean different things: a failure can be superseded by a real outcome, a completion or a cancellation is the real outcome. UpdateStatus now writes the error column unconditionally, so a corrected outcome drops the previous attempt's reason rather than being recorded as completed while still carrying "stale operation reaped" as its error. Also adds the route-level spec for the 500 branch of DELETE /api/operations/history, and trims a comment that credited the persisted cancellable column with more than it survives long enough to do. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(activity): offer Cancel in the phase that can honour it The cancellable flag was set at both ends of an operation's life and was wrong at both, in opposite directions. A queued operation is cancellable whatever it is. EnqueueModelOp and EnqueueBackendOp select on the operation context, so cancelling one that is still waiting releases the delivery goroutine and abandonQueued retires it: the worker never sees it, nothing is downloaded, nothing is deleted. markQueued nevertheless wrote Cancellable: !deletion, so a queued removal reported cancellable: false and the UI hid the Cancel button in the one window where pressing it both works and leaves no trace. A removal queued behind a large install was stuck there until the install finished. A running removal is not cancellable at all. DeleteModel and DeleteBackend take no context, and modelHandler only checks the operation context after the call returns, so a "cancelled" verdict would land after the model was already gone. Both handlers nevertheless wrote Cancellable: true unconditionally at entry, ahead of the op.Delete branch, offering a Cancel button the server cannot honour. So the queued phase is more cancellable than the running phase, which is the reverse of the usual shape. markQueued now reports true unconditionally, and the handler-entry writes report !op.Delete. Both sites carry a comment saying why, because reading either one alone suggests the other is a bug. GalleryStore.Create keeps !op.Delete: it runs at dequeue, so its value already describes the running phase. Its comment now says so. Specs cover queued removal, queued install, running removal and running install through the handlers, plus the queued-removal case through /api/operations where the flag is consumed, plus the behaviour the whole asymmetry rests on: a removal cancelled while queued never reaches the worker and deletes nothing. No existing spec asserted the old values. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(activity): clamp the installer message, and add a real-binary e2e spec Running the page against a real local-ai showed the legacy installer message wrapping to three lines and dominating the card: it embeds an absolute file path, so it is both long and a single unbreakable token. One line, ellipsised, full text in the title, matching what the error string already does. The spec that found it runs with no route stubbing at all. Every other spec here stubs /api/operations, which is how a payload the server cannot emit (isDeletion true on a live operation) stayed green through a full review while the UI rendered a removal as an install. It is skipped unless LOCALAI_REAL_BINARY is set, so CI is unaffected. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]
committed by
GitHub
GitHub
parent
823fc25bb7
commit
0f7186f214
417
core/http/react-ui/e2e/activity-page.spec.js
Normal file
417
core/http/react-ui/e2e/activity-page.spec.js
Normal file
@@ -0,0 +1,417 @@
|
||||
import { test, expect } from './coverage-fixtures.js'
|
||||
|
||||
const stub = (page, { operations = [], history = [] } = {}) => Promise.all([
|
||||
page.route('**/api/operations', (route) => route.fulfill({
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({ operations }),
|
||||
})),
|
||||
page.route('**/api/operations/history', (route) => route.fulfill({
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({ operations: history }),
|
||||
})),
|
||||
])
|
||||
|
||||
test('lists live operations and cancels one from a labelled button', async ({ page }) => {
|
||||
await stub(page, {
|
||||
operations: [{
|
||||
id: 'gemma-3-27b-it',
|
||||
name: 'gemma-3-27b-it',
|
||||
jobID: 'job-gemma',
|
||||
progress: 22,
|
||||
taskType: 'installation',
|
||||
isBackend: false,
|
||||
isQueued: false,
|
||||
isDeletion: false,
|
||||
cancellable: true,
|
||||
phase: 'downloading',
|
||||
}],
|
||||
})
|
||||
|
||||
let cancelledPath = ''
|
||||
await page.route('**/api/operations/job-gemma/cancel', (route) => {
|
||||
cancelledPath = new URL(route.request().url()).pathname
|
||||
return route.fulfill({ contentType: 'application/json', body: '{}' })
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
const card = page.locator('.operation-card').filter({ hasText: 'gemma-3-27b-it' })
|
||||
await expect(card).toBeVisible()
|
||||
await expect(card).toContainText('22%')
|
||||
|
||||
await card.locator('.operation-card__cancel').click()
|
||||
expect(cancelledPath).toBe('/api/operations/job-gemma/cancel')
|
||||
})
|
||||
|
||||
test('separates an unacknowledged failure from the record', async ({ page }) => {
|
||||
await stub(page, {
|
||||
operations: [{
|
||||
id: 'sherpa-onnx',
|
||||
name: 'sherpa-onnx',
|
||||
jobID: 'job-sherpa',
|
||||
progress: 0,
|
||||
taskType: 'installation',
|
||||
isBackend: true,
|
||||
isQueued: false,
|
||||
isDeletion: false,
|
||||
cancellable: false,
|
||||
error: 'no space left on device',
|
||||
}],
|
||||
history: [{
|
||||
id: 'bark-cpp',
|
||||
name: 'bark-cpp',
|
||||
jobID: 'job-bark',
|
||||
isBackend: true,
|
||||
taskType: 'installation',
|
||||
outcome: 'failed',
|
||||
error: 'checksum mismatch',
|
||||
startedAt: '2026-07-28T13:40:00Z',
|
||||
finishedAt: '2026-07-28T13:41:00Z',
|
||||
}],
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
// Live and unacknowledged: a card that needs a decision.
|
||||
await expect(page.locator('.operation-card--error')).toContainText('sherpa-onnx')
|
||||
// Dismissed earlier: a row in the record.
|
||||
await expect(page.locator('.activity-row')).toContainText('bark-cpp')
|
||||
})
|
||||
|
||||
test('a failure never appears in both In progress and Needs attention', async ({ page }) => {
|
||||
// Section membership has to be unambiguous: the same job showing twice makes
|
||||
// the two failure paths (retry / dismiss) impossible to reason about.
|
||||
await stub(page, {
|
||||
operations: [
|
||||
{
|
||||
id: 'model-a',
|
||||
name: 'model-a',
|
||||
jobID: 'job-a',
|
||||
progress: 40,
|
||||
taskType: 'installation',
|
||||
isBackend: false,
|
||||
isQueued: false,
|
||||
isDeletion: false,
|
||||
cancellable: true,
|
||||
},
|
||||
{
|
||||
id: 'sherpa-onnx',
|
||||
name: 'sherpa-onnx',
|
||||
jobID: 'job-sherpa',
|
||||
progress: 0,
|
||||
taskType: 'installation',
|
||||
isBackend: true,
|
||||
isQueued: false,
|
||||
isDeletion: false,
|
||||
cancellable: false,
|
||||
error: 'no space left on device',
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
await expect(page.locator('.operation-card')).toHaveCount(2)
|
||||
await expect(page.locator('.operation-card').filter({ hasText: 'sherpa-onnx' })).toHaveCount(1)
|
||||
})
|
||||
|
||||
test('retrying a failed backend install dismisses it before reinstalling', async ({ page }) => {
|
||||
// Order is load-bearing: a bare reinstall overwrites the opcache entry
|
||||
// without going through recordTerminal, so the failure would never reach the
|
||||
// record.
|
||||
await stub(page, {
|
||||
operations: [{
|
||||
id: 'sherpa-onnx',
|
||||
name: 'sherpa-onnx',
|
||||
fullName: 'localai@sherpa-onnx',
|
||||
jobID: 'job-sherpa',
|
||||
progress: 0,
|
||||
taskType: 'installation',
|
||||
isBackend: true,
|
||||
isQueued: false,
|
||||
isDeletion: false,
|
||||
cancellable: false,
|
||||
error: 'no space left on device',
|
||||
}],
|
||||
})
|
||||
|
||||
const calls = []
|
||||
await page.route('**/api/operations/job-sherpa/dismiss', (route) => {
|
||||
calls.push('dismiss')
|
||||
return route.fulfill({ contentType: 'application/json', body: '{}' })
|
||||
})
|
||||
await page.route('**/api/backends/install/**', (route) => {
|
||||
calls.push(new URL(route.request().url()).pathname)
|
||||
return route.fulfill({ contentType: 'application/json', body: '{}' })
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
await page.locator('.operation-card__retry').click()
|
||||
|
||||
await expect.poll(() => calls).toEqual(['dismiss', '/api/backends/install/localai@sherpa-onnx'])
|
||||
})
|
||||
|
||||
test('retry dismisses the job it was pressed on, not another sharing its id', async ({ page }) => {
|
||||
// /api/operations strips the "node:<id>:" prefix, so a local install and a
|
||||
// node-scoped install of one backend arrive with the same id and different
|
||||
// jobIDs. Dismissing by id retired whichever came first, which both left the
|
||||
// acted-on failure live and silently retired an unrelated one.
|
||||
const failed = (over) => ({
|
||||
id: 'sherpa-onnx',
|
||||
name: 'sherpa-onnx',
|
||||
fullName: 'sherpa-onnx',
|
||||
progress: 0,
|
||||
taskType: 'installation',
|
||||
isBackend: true,
|
||||
isQueued: false,
|
||||
isDeletion: false,
|
||||
cancellable: false,
|
||||
error: 'no space left on device',
|
||||
...over,
|
||||
})
|
||||
await stub(page, {
|
||||
operations: [
|
||||
failed({ jobID: 'job-local' }),
|
||||
failed({ jobID: 'job-node', nodeID: 'node-1' }),
|
||||
],
|
||||
})
|
||||
|
||||
const calls = []
|
||||
await page.route('**/api/operations/*/dismiss', (route) => {
|
||||
calls.push(new URL(route.request().url()).pathname)
|
||||
return route.fulfill({ contentType: 'application/json', body: '{}' })
|
||||
})
|
||||
await page.route('**/api/nodes/*/backends/install', (route) => {
|
||||
calls.push(new URL(route.request().url()).pathname)
|
||||
return route.fulfill({ contentType: 'application/json', body: '{}' })
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
// Nothing on screen tells the two cards apart, which is the point: they
|
||||
// share a name and an id, and only the jobID behind each one differs. The
|
||||
// node-scoped job is second in the payload, so it is the second card.
|
||||
await expect(page.locator('.operation-card')).toHaveCount(2)
|
||||
await page.locator('.operation-card').nth(1).locator('.operation-card__retry').click()
|
||||
|
||||
await expect.poll(() => calls).toEqual([
|
||||
'/api/operations/job-node/dismiss',
|
||||
'/api/nodes/node-1/backends/install',
|
||||
])
|
||||
})
|
||||
|
||||
test('the dismiss control also acts on the job it belongs to', async ({ page }) => {
|
||||
// Same hazard as retry: the card's X passed the display id too.
|
||||
const failed = (over) => ({
|
||||
id: 'sherpa-onnx',
|
||||
name: 'sherpa-onnx',
|
||||
fullName: 'sherpa-onnx',
|
||||
progress: 0,
|
||||
taskType: 'installation',
|
||||
isBackend: true,
|
||||
isQueued: false,
|
||||
isDeletion: false,
|
||||
cancellable: false,
|
||||
error: 'no space left on device',
|
||||
...over,
|
||||
})
|
||||
await stub(page, {
|
||||
operations: [
|
||||
failed({ jobID: 'job-local' }),
|
||||
failed({ jobID: 'job-node', nodeID: 'node-1' }),
|
||||
],
|
||||
})
|
||||
|
||||
const dismissed = []
|
||||
await page.route('**/api/operations/*/dismiss', (route) => {
|
||||
dismissed.push(new URL(route.request().url()).pathname)
|
||||
return route.fulfill({ contentType: 'application/json', body: '{}' })
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
await expect(page.locator('.operation-card')).toHaveCount(2)
|
||||
await page.locator('.operation-card').nth(1).locator('.operation-card__hide').click()
|
||||
|
||||
await expect.poll(() => dismissed).toEqual(['/api/operations/job-node/dismiss'])
|
||||
})
|
||||
|
||||
test('a filter matching nothing does not claim the instance is empty', async ({ page }) => {
|
||||
// Three model records on file: telling the user nothing has ever run, while
|
||||
// the header counts those same three, is simply false.
|
||||
await stub(page, {
|
||||
history: [1, 2, 3].map((n) => ({
|
||||
id: `model-${n}`,
|
||||
name: `model-${n}`,
|
||||
jobID: `job-${n}`,
|
||||
isBackend: false,
|
||||
taskType: 'installation',
|
||||
outcome: 'completed',
|
||||
startedAt: '2026-07-28T13:40:00Z',
|
||||
finishedAt: '2026-07-28T13:40:20Z',
|
||||
})),
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
await expect(page.locator('.activity-row')).toHaveCount(3)
|
||||
|
||||
await page.locator('.activity-chip', { hasText: 'Backends' }).click()
|
||||
|
||||
await expect(page.locator('.activity-empty--filtered')).toBeVisible()
|
||||
await expect(page.locator('.activity-empty')).not.toContainText('No operations since startup')
|
||||
// And the way back out is on screen.
|
||||
await page.locator('.activity-empty--filtered button').click()
|
||||
await expect(page.locator('.activity-row')).toHaveCount(3)
|
||||
})
|
||||
|
||||
test('the summary drops a zero count instead of reporting it', async ({ page }) => {
|
||||
await stub(page, {
|
||||
operations: [{
|
||||
id: 'model-a',
|
||||
name: 'model-a',
|
||||
jobID: 'job-a',
|
||||
progress: 40,
|
||||
taskType: 'installation',
|
||||
isBackend: false,
|
||||
isQueued: false,
|
||||
isDeletion: false,
|
||||
cancellable: true,
|
||||
}],
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
const supporting = page.locator('.page-header__supporting')
|
||||
await expect(supporting).toHaveText('1 operation running.')
|
||||
await expect(supporting).not.toContainText('0')
|
||||
})
|
||||
|
||||
test('a cancelled deletion reports the cancellation, not a removal', async ({ page }) => {
|
||||
await stub(page, {
|
||||
history: [{
|
||||
id: 'model-a',
|
||||
name: 'model-a',
|
||||
jobID: 'job-a',
|
||||
isBackend: false,
|
||||
taskType: 'deletion',
|
||||
outcome: 'cancelled',
|
||||
startedAt: '2026-07-28T13:40:00Z',
|
||||
finishedAt: '2026-07-28T13:40:02Z',
|
||||
}],
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
await expect(page.locator('.activity-row')).toContainText('cancelled')
|
||||
await expect(page.locator('.activity-row')).not.toContainText('removed')
|
||||
})
|
||||
|
||||
test('an implausible or zero duration never reaches the row', async ({ page }) => {
|
||||
await stub(page, {
|
||||
history: [
|
||||
{
|
||||
id: 'zero-span',
|
||||
name: 'zero-span',
|
||||
jobID: 'job-zero',
|
||||
isBackend: false,
|
||||
taskType: 'installation',
|
||||
outcome: 'completed',
|
||||
// recordTerminal seeds StartedAt = FinishedAt and only overwrites it
|
||||
// with a real stamp, so this is an ordinary arrival.
|
||||
startedAt: '2026-07-28T13:40:00Z',
|
||||
finishedAt: '2026-07-28T13:40:00Z',
|
||||
},
|
||||
{
|
||||
id: 'zero-stamp',
|
||||
name: 'zero-stamp',
|
||||
jobID: 'job-stamp',
|
||||
isBackend: false,
|
||||
taskType: 'installation',
|
||||
outcome: 'completed',
|
||||
startedAt: '0001-01-01T00:00:00Z',
|
||||
finishedAt: '2026-07-28T13:41:00Z',
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
const zeroSpan = page.locator('.activity-row').filter({ hasText: 'zero-span' })
|
||||
await expect(zeroSpan).toContainText('installed in < 1s')
|
||||
|
||||
// A zero-value Go stamp is not a duration. The row says what happened and
|
||||
// stops, rather than stating a span of millennia as fact.
|
||||
const zeroStamp = page.locator('.activity-row').filter({ hasText: 'zero-stamp' })
|
||||
await expect(zeroStamp).toContainText('installed')
|
||||
await expect(zeroStamp).not.toContainText('installed in')
|
||||
})
|
||||
|
||||
test('a failed removal offers no retry, because retry only means install', async ({ page }) => {
|
||||
await stub(page, {
|
||||
operations: [{
|
||||
id: 'model-a',
|
||||
name: 'model-a',
|
||||
fullName: 'model-a',
|
||||
jobID: 'job-a',
|
||||
progress: 0,
|
||||
taskType: 'deletion',
|
||||
isBackend: false,
|
||||
isQueued: false,
|
||||
isDeletion: true,
|
||||
cancellable: false,
|
||||
error: 'file is busy',
|
||||
}],
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
await expect(page.locator('.operation-card--error')).toBeVisible()
|
||||
await expect(page.locator('.operation-card__retry')).toHaveCount(0)
|
||||
// And it must not claim an install was attempted.
|
||||
await expect(page.locator('.operation-card--error')).not.toContainText('install')
|
||||
})
|
||||
|
||||
test('filters the record down to backends', async ({ page }) => {
|
||||
await stub(page, {
|
||||
history: [
|
||||
{
|
||||
id: 'gemma-3-27b-it',
|
||||
name: 'gemma-3-27b-it',
|
||||
jobID: 'job-gemma',
|
||||
isBackend: false,
|
||||
taskType: 'installation',
|
||||
outcome: 'completed',
|
||||
startedAt: '2026-07-28T13:40:00Z',
|
||||
finishedAt: '2026-07-28T13:41:30Z',
|
||||
},
|
||||
{
|
||||
id: 'bark-cpp',
|
||||
name: 'bark-cpp',
|
||||
jobID: 'job-bark',
|
||||
isBackend: true,
|
||||
taskType: 'installation',
|
||||
outcome: 'completed',
|
||||
startedAt: '2026-07-28T13:40:00Z',
|
||||
finishedAt: '2026-07-28T13:40:20Z',
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
await page.goto('/app/activity')
|
||||
await expect(page.locator('.activity-row')).toHaveCount(2)
|
||||
|
||||
await page.locator('.activity-chip', { hasText: 'Backends' }).click()
|
||||
|
||||
await expect(page.locator('.activity-row')).toHaveCount(1)
|
||||
await expect(page.locator('.activity-row')).toContainText('bark-cpp')
|
||||
})
|
||||
|
||||
test('shows the empty state when nothing has run', async ({ page }) => {
|
||||
await stub(page)
|
||||
|
||||
await page.goto('/app/activity')
|
||||
|
||||
await expect(page.locator('.page-title')).toBeVisible()
|
||||
await expect(page.locator('.activity-empty')).toBeVisible()
|
||||
})
|
||||
@@ -1,6 +1,6 @@
|
||||
import { test, expect } from './coverage-fixtures.js'
|
||||
|
||||
test('operations bar shows managed model acquisition phase and bytes', async ({ page }) => {
|
||||
test('operations strip shows managed model acquisition phase and bytes', async ({ page }) => {
|
||||
await page.route('**/api/operations', (route) => route.fulfill({
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({
|
||||
@@ -14,7 +14,6 @@ test('operations bar shows managed model acquisition phase and bytes', async ({
|
||||
isDeletion: false,
|
||||
isBackend: false,
|
||||
isQueued: false,
|
||||
isCancelled: false,
|
||||
cancellable: true,
|
||||
phase: 'downloading',
|
||||
currentBytes: 1073741824,
|
||||
@@ -22,19 +21,12 @@ test('operations bar shows managed model acquisition phase and bytes', async ({
|
||||
}],
|
||||
}),
|
||||
}))
|
||||
let cancelledPath = ''
|
||||
await page.route('**/api/operations/artifact-job-123/cancel', (route) => {
|
||||
cancelledPath = new URL(route.request().url()).pathname
|
||||
return route.fulfill({ contentType: 'application/json', body: '{}' })
|
||||
})
|
||||
|
||||
await page.goto('/app/models')
|
||||
const operation = page.locator('.operation-item').filter({ hasText: 'qwen-asr' })
|
||||
await expect(operation).toContainText('Downloading model files')
|
||||
await expect(operation).toContainText('1 GB / 4 GB')
|
||||
await expect(operation.locator('.operation-progress')).toHaveText('45%')
|
||||
await expect(operation.locator('.operation-bar')).toHaveAttribute('style', /width: 45%/)
|
||||
|
||||
await operation.getByTitle('Cancel').click()
|
||||
expect(cancelledPath).toBe('/api/operations/artifact-job-123/cancel')
|
||||
const strip = page.locator('.operations-strip')
|
||||
await expect(strip.locator('.operations-strip__name')).toHaveText('qwen-asr')
|
||||
await expect(strip).toContainText('Downloading')
|
||||
await expect(strip).toContainText('1 GB / 4 GB')
|
||||
await expect(strip.locator('.operations-strip__pct')).toHaveText('45%')
|
||||
await expect(strip.locator('.operations-strip__fill')).toHaveAttribute('style', /width: 45%/)
|
||||
})
|
||||
|
||||
216
core/http/react-ui/e2e/operations-strip.spec.js
Normal file
216
core/http/react-ui/e2e/operations-strip.spec.js
Normal file
@@ -0,0 +1,216 @@
|
||||
import { test, expect } from './coverage-fixtures.js'
|
||||
|
||||
const op = (over = {}) => ({
|
||||
id: 'model-a',
|
||||
name: 'model-a',
|
||||
fullName: 'model-a',
|
||||
jobID: 'job-a',
|
||||
progress: 40,
|
||||
taskType: 'installation',
|
||||
isDeletion: false,
|
||||
isBackend: false,
|
||||
isQueued: false,
|
||||
cancellable: true,
|
||||
...over,
|
||||
})
|
||||
|
||||
const stubOperations = (page, operations) =>
|
||||
page.route('**/api/operations', (route) => route.fulfill({
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({ operations }),
|
||||
}))
|
||||
|
||||
test('renders exactly one row for four concurrent operations', async ({ page }) => {
|
||||
await stubOperations(page, [
|
||||
op({ id: 'model-a', name: 'model-a', jobID: 'job-a', progress: 10 }),
|
||||
op({ id: 'model-b', name: 'model-b', jobID: 'job-b', progress: 40 }),
|
||||
op({ id: 'model-c', name: 'model-c', jobID: 'job-c', progress: 70 }),
|
||||
op({ id: 'model-d', name: 'model-d', jobID: 'job-d', progress: 90 }),
|
||||
])
|
||||
|
||||
await page.goto('/app/models')
|
||||
|
||||
// One row, never four. The stacked bar is what this replaces.
|
||||
await expect(page.locator('.operations-strip')).toHaveCount(1)
|
||||
// The API sorts by progress ascending, so the least advanced op leads.
|
||||
await expect(page.locator('.operations-strip__name')).toHaveText('model-a')
|
||||
await expect(page.locator('.operations-strip__more')).toContainText('3')
|
||||
await expect(page.locator('.operations-strip__more')).toHaveAttribute('href', /\/app\/activity$/)
|
||||
})
|
||||
|
||||
test('a failure takes the strip over a running install', async ({ page }) => {
|
||||
await stubOperations(page, [
|
||||
op({ id: 'model-a', name: 'model-a', jobID: 'job-a', progress: 10 }),
|
||||
op({ id: 'sherpa-onnx', name: 'sherpa-onnx', jobID: 'job-f', isBackend: true, error: 'no space left on device' }),
|
||||
])
|
||||
|
||||
await page.goto('/app/models')
|
||||
|
||||
await expect(page.locator('.operations-strip__name')).toHaveText('sherpa-onnx')
|
||||
await expect(page.locator('.operations-strip')).toContainText('no space left on device')
|
||||
})
|
||||
|
||||
test('a hidden strip comes back when a different operation becomes primary', async ({ page }) => {
|
||||
// Hiding must not be able to silence a later failure, so the hidden state is
|
||||
// keyed by job rather than being a blanket mute.
|
||||
//
|
||||
// The swap is driven by the test rather than by a poll count: a count would
|
||||
// race the click, and a click landing on the failure would take the dismiss
|
||||
// path and fail this test for an unrelated reason.
|
||||
let swapped = false
|
||||
await page.route('**/api/operations', (route) => {
|
||||
const operations = swapped
|
||||
? [op({ id: 'sherpa-onnx', name: 'sherpa-onnx', jobID: 'job-f', error: 'no space left on device' })]
|
||||
: [op({ id: 'model-a', name: 'model-a', jobID: 'job-a' })]
|
||||
return route.fulfill({ contentType: 'application/json', body: JSON.stringify({ operations }) })
|
||||
})
|
||||
|
||||
await page.goto('/app/models')
|
||||
await expect(page.locator('.operations-strip__name')).toHaveText('model-a')
|
||||
|
||||
await page.locator('.operations-strip__hide').click()
|
||||
await expect(page.locator('.operations-strip')).toHaveCount(0)
|
||||
|
||||
// The poller swaps in a different job, which must re-render the strip.
|
||||
swapped = true
|
||||
await expect(page.locator('.operations-strip__name')).toHaveText('sherpa-onnx', { timeout: 10_000 })
|
||||
})
|
||||
|
||||
test('hiding a running operation does not silence that same job failing', async ({ page }) => {
|
||||
// A job keeps its jobID when it fails, so keying the hidden state on the job
|
||||
// alone would let a user mute the very failure they need to see.
|
||||
let failing = false
|
||||
await page.route('**/api/operations', (route) => {
|
||||
const operations = failing
|
||||
? [op({ error: 'no space left on device' })]
|
||||
: [op()]
|
||||
return route.fulfill({ contentType: 'application/json', body: JSON.stringify({ operations }) })
|
||||
})
|
||||
|
||||
await page.goto('/app/models')
|
||||
await expect(page.locator('.operations-strip__name')).toHaveText('model-a')
|
||||
|
||||
await page.locator('.operations-strip__hide').click()
|
||||
await expect(page.locator('.operations-strip')).toHaveCount(0)
|
||||
|
||||
failing = true
|
||||
await expect(page.locator('.operations-strip')).toContainText('no space left on device', { timeout: 10_000 })
|
||||
})
|
||||
|
||||
test('a long error message does not widen the page', async ({ page }) => {
|
||||
// An install error is arbitrarily long text, and the strip sits above every
|
||||
// page: if it cannot shrink, every page under it gets a horizontal scrollbar.
|
||||
await stubOperations(page, [op({ error: `disk write failed: ${'x'.repeat(180)}` })])
|
||||
|
||||
await page.setViewportSize({ width: 1280, height: 800 })
|
||||
await page.goto('/app/models')
|
||||
await expect(page.locator('.operations-strip')).toBeVisible()
|
||||
|
||||
const widths = await page.evaluate(() => ({
|
||||
scroll: document.documentElement.scrollWidth,
|
||||
client: document.documentElement.clientWidth,
|
||||
}))
|
||||
expect(widths.scroll).toBeLessThanOrEqual(widths.client)
|
||||
})
|
||||
|
||||
test('progress is exposed to assistive tech as a named progressbar', async ({ page }) => {
|
||||
// The percentage text is aria-hidden so the live region stops re-announcing
|
||||
// the strip once a second; the value has to reach assistive tech some other
|
||||
// way, and a progressbar is read on demand rather than announced.
|
||||
await stubOperations(page, [op({ progress: 45 })])
|
||||
|
||||
await page.goto('/app/models')
|
||||
const bar = page.locator('.operations-strip__track')
|
||||
await expect(bar).toHaveAttribute('role', 'progressbar')
|
||||
await expect(bar).toHaveAttribute('aria-valuenow', '45')
|
||||
await expect(bar).toHaveAttribute('aria-valuemin', '0')
|
||||
await expect(bar).toHaveAttribute('aria-valuemax', '100')
|
||||
await expect(bar).toHaveAttribute('aria-label', /model-a/)
|
||||
})
|
||||
|
||||
test('an operation waiting for the worker says queued, not installing', async ({ page }) => {
|
||||
// The real payload for an admitted-but-unstarted op: phase "queued", no
|
||||
// progress. It used to arrive with isQueued false, so the one state the
|
||||
// strip has a clock icon for never appeared and a queued install claimed to
|
||||
// be running.
|
||||
await stubOperations(page, [op({ isQueued: true, phase: 'queued', progress: 0 })])
|
||||
|
||||
await page.goto('/app/models')
|
||||
await expect(page.locator('.operations-strip')).toContainText('Queued')
|
||||
await expect(page.locator('.operations-strip')).not.toContainText('Installing')
|
||||
await expect(page.locator('.operations-strip__pct')).toHaveCount(0)
|
||||
})
|
||||
|
||||
test('cancelling the last operation does not announce it as installed', async ({ page }) => {
|
||||
// Cancelling deletes the operation server side, so the strip sees exactly
|
||||
// what it sees when an install finishes: the operation stops being listed.
|
||||
// The completion hold used to take that for success and put a green
|
||||
// "Installed model model-a" on screen for four seconds after the user
|
||||
// called it off.
|
||||
let cancelled = false
|
||||
await page.route('**/api/operations', (route) => route.fulfill({
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({ operations: cancelled ? [] : [op()] }),
|
||||
}))
|
||||
await page.route('**/api/operations/history', (route) => route.fulfill({
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({ operations: [] }),
|
||||
}))
|
||||
await page.route('**/api/operations/job-a/cancel', (route) => {
|
||||
cancelled = true
|
||||
return route.fulfill({ contentType: 'application/json', body: '{"success":true}' })
|
||||
})
|
||||
|
||||
// Cancel is on the card, never on the strip, so the page is where the user
|
||||
// does this. The strip sits above it the whole time.
|
||||
await page.goto('/app/activity')
|
||||
await expect(page.locator('.operations-strip')).toContainText('Installing model')
|
||||
|
||||
await page.locator('.operation-card__cancel').click()
|
||||
|
||||
// The poll that empties the page is the same render that would raise the
|
||||
// completion hold, so the strip has to be counted the instant the card
|
||||
// goes. A retrying assertion would simply wait the four-second hold out and
|
||||
// pass on the regression it is here to catch.
|
||||
await expect(page.locator('.operation-card')).toHaveCount(0)
|
||||
expect(await page.locator('.operations-strip').count()).toBe(0)
|
||||
|
||||
// And it must not turn up a moment later either.
|
||||
await page.waitForTimeout(1500)
|
||||
expect(await page.locator('.operations-strip').count()).toBe(0)
|
||||
})
|
||||
|
||||
test('a completed removal does not announce an install', async ({ page }) => {
|
||||
// The API drops an operation the instant it finishes, so the completion
|
||||
// phrase is rendered from the operation the strip was already holding.
|
||||
let removed = false
|
||||
await page.route('**/api/operations', (route) => {
|
||||
const operations = removed ? [] : [op({ isDeletion: true, progress: 0 })]
|
||||
return route.fulfill({ contentType: 'application/json', body: JSON.stringify({ operations }) })
|
||||
})
|
||||
|
||||
await page.goto('/app/models')
|
||||
await expect(page.locator('.operations-strip')).toContainText('Removing model')
|
||||
|
||||
removed = true
|
||||
await expect(page.locator('.operations-strip')).toContainText('Removed model', { timeout: 10_000 })
|
||||
await expect(page.locator('.operations-strip')).not.toContainText('Installed')
|
||||
})
|
||||
|
||||
test('the hide button hides the strip without cancelling', async ({ page }) => {
|
||||
await stubOperations(page, [op()])
|
||||
|
||||
let cancelCalled = false
|
||||
await page.route('**/api/operations/*/cancel', (route) => {
|
||||
cancelCalled = true
|
||||
return route.fulfill({ contentType: 'application/json', body: '{}' })
|
||||
})
|
||||
|
||||
await page.goto('/app/models')
|
||||
await expect(page.locator('.operations-strip')).toBeVisible()
|
||||
|
||||
await page.locator('.operations-strip__hide').click()
|
||||
|
||||
await expect(page.locator('.operations-strip')).toHaveCount(0)
|
||||
expect(cancelCalled).toBe(false)
|
||||
})
|
||||
@@ -19,6 +19,7 @@ const PAGES = [
|
||||
['/app/studio', 'Studio'],
|
||||
['/app/manage', 'Manage'],
|
||||
['/app/backends', 'Backends'],
|
||||
['/app/activity', 'Activity'],
|
||||
['/app/settings', 'Settings'],
|
||||
['/app/nodes', 'Nodes'],
|
||||
['/app/scheduling', 'Scheduling'],
|
||||
|
||||
73
core/http/react-ui/e2e/real-binary-activity.spec.js
Normal file
73
core/http/react-ui/e2e/real-binary-activity.spec.js
Normal file
@@ -0,0 +1,73 @@
|
||||
import { test, expect } from '@playwright/test'
|
||||
|
||||
// Runs against a REAL local-ai binary with NO route stubbing.
|
||||
//
|
||||
// Every other spec here stubs /api/operations. That is how a payload the server
|
||||
// could not actually emit (isDeletion:true on a live operation) stayed green
|
||||
// through an entire review while the UI rendered a removal as an install. These
|
||||
// assertions are only worth anything because the data came from the real handler.
|
||||
//
|
||||
// make build
|
||||
// ./local-ai run --address 127.0.0.1:8089 --models-path /tmp/lai-e2e
|
||||
// cd core/http/react-ui
|
||||
// LOCALAI_REAL_BINARY=1 PLAYWRIGHT_EXTERNAL_SERVER=1 PW_WORKERS=1 \
|
||||
// npx playwright test e2e/real-binary-activity.spec.js
|
||||
//
|
||||
// Skipped by default so CI, which runs the stub server, is unaffected.
|
||||
test.skip(!process.env.LOCALAI_REAL_BINARY, 'needs a real local-ai on 127.0.0.1:8089')
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
const MISSING = 'definitely-not-a-real-model-xyz'
|
||||
|
||||
test('a real failed install lands in Needs attention with the real error and a Retry', async ({ page, request }) => {
|
||||
// Self-contained: the gallery resolver rejects an unknown name, so this fails
|
||||
// fast without downloading anything.
|
||||
await request.post(`/api/models/install/${MISSING}`)
|
||||
|
||||
await page.goto('/app/activity')
|
||||
await expect(page.locator('.page-title')).toBeVisible()
|
||||
|
||||
const card = page.locator('.operation-card--error').filter({ hasText: MISSING })
|
||||
await expect(card).toBeVisible({ timeout: 20_000 })
|
||||
await expect(card).toContainText('no model found with name')
|
||||
// Retry is offered for a failed install; it is gated off for a failed removal.
|
||||
await expect(card.locator('.operation-card__retry')).toBeVisible()
|
||||
})
|
||||
|
||||
test('the strip shows the real failure on one line and does not widen the page', async ({ page }) => {
|
||||
await page.goto('/app/activity')
|
||||
|
||||
const strip = page.locator('.operations-strip')
|
||||
await expect(strip).toHaveCount(1)
|
||||
await expect(strip).toHaveClass(/operations-strip--error/)
|
||||
await expect(strip.locator('.operations-strip__name')).toHaveText(MISSING)
|
||||
|
||||
const box = await page.evaluate(() => ({
|
||||
scroll: document.documentElement.scrollWidth,
|
||||
client: document.documentElement.clientWidth,
|
||||
}))
|
||||
expect(box.scroll).toBeLessThanOrEqual(box.client)
|
||||
})
|
||||
|
||||
test('dismissing moves the failure into the record instead of destroying it', async ({ page }) => {
|
||||
await page.goto('/app/activity')
|
||||
|
||||
const card = page.locator('.operation-card--error').filter({ hasText: MISSING })
|
||||
await expect(card).toBeVisible()
|
||||
await card.locator('.operation-card__hide').click()
|
||||
|
||||
await expect(page.locator('.operation-card--error')).toHaveCount(0, { timeout: 15_000 })
|
||||
const row = page.locator('.activity-row').filter({ hasText: MISSING })
|
||||
await expect(row).toBeVisible({ timeout: 15_000 })
|
||||
await expect(row).toContainText('failed')
|
||||
})
|
||||
|
||||
test('Clear history empties the record against the real store', async ({ page }) => {
|
||||
await page.goto('/app/activity')
|
||||
|
||||
await expect(page.locator('.activity-row').first()).toBeVisible()
|
||||
await page.getByRole('button', { name: /clear history/i }).click()
|
||||
|
||||
await expect(page.locator('.activity-row')).toHaveCount(0, { timeout: 15_000 })
|
||||
await expect(page.locator('.activity-empty')).toBeVisible()
|
||||
})
|
||||
@@ -1,4 +1,84 @@
|
||||
{
|
||||
"activity": {
|
||||
"title": "Activity",
|
||||
"supporting": "Installs, downloads and removals on this instance.",
|
||||
"hide": "Hide",
|
||||
"moveToHistory": "Move to history",
|
||||
"moreCount": "{{count}} more",
|
||||
"waitingForInstaller": "waiting for the installer",
|
||||
"progressLabel": "Progress for {{name}}",
|
||||
"toNode": "to {{node}}",
|
||||
"nodesDone": "{{done}} of {{total}} nodes done",
|
||||
"timeLeft": "{{value}} left",
|
||||
"cancel": "Cancel",
|
||||
"cancelLabel": "Cancel {{name}}",
|
||||
"retry": "Retry",
|
||||
"retryLabel": "Retry {{name}}",
|
||||
"nodeCount": "{{count}} nodes",
|
||||
"showNodes": "Show {{count}} nodes",
|
||||
"hideNodes": "Hide per-node detail",
|
||||
"node": {
|
||||
"done": "Done",
|
||||
"failed": "Failed",
|
||||
"queued": "Queued",
|
||||
"workerBusy": "Worker busy",
|
||||
"downloading": "Downloading"
|
||||
},
|
||||
"kind": {
|
||||
"model": "model",
|
||||
"backend": "backend"
|
||||
},
|
||||
"verb": {
|
||||
"installing": "Installing {{kind}}",
|
||||
"installed": "Installed {{kind}}",
|
||||
"removed": "Removed {{kind}}",
|
||||
"staged": "Staged model",
|
||||
"failed": "Couldn't install {{kind}}",
|
||||
"queued": "Queued",
|
||||
"staging": "Staging model",
|
||||
"removing": "Removing {{kind}}",
|
||||
"failedRemoval": "Couldn't remove {{kind}}",
|
||||
"failedStaging": "Couldn't stage model"
|
||||
},
|
||||
"phase": {
|
||||
"resolving": "Resolving files",
|
||||
"downloading": "Downloading",
|
||||
"verifying": "Verifying",
|
||||
"committing": "Finalizing",
|
||||
"persisting": "Saving configuration"
|
||||
},
|
||||
"clearHistory": "Clear history",
|
||||
"inProgress": "In progress",
|
||||
"needsAttention": "Needs attention",
|
||||
"record": "Record",
|
||||
"historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
|
||||
"emptyTitle": "No operations since startup",
|
||||
"emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
|
||||
"browseModels": "Browse models",
|
||||
"viewInModels": "View in Models",
|
||||
"viewInBackends": "View in Backends",
|
||||
"rowInstalled": "installed in {{duration}}",
|
||||
"rowFailed": "failed: {{error}}",
|
||||
"rowCancelled": "cancelled",
|
||||
"rowRemoved": "removed",
|
||||
"retryFailed": "Retry failed: {{message}}",
|
||||
"filter": {
|
||||
"all": "All",
|
||||
"models": "Models",
|
||||
"backends": "Backends",
|
||||
"cluster": "Cluster"
|
||||
},
|
||||
"summaryRunning_one": "{{count}} operation running.",
|
||||
"summaryRunning_other": "{{count}} operations running.",
|
||||
"summaryFailed_one": "{{count}} operation needs attention.",
|
||||
"summaryFailed_other": "{{count}} operations need attention.",
|
||||
"summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
|
||||
"summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
|
||||
"summaryIdle": "Nothing running.",
|
||||
"emptyFiltered": "No operations match this filter.",
|
||||
"showAll": "Show all",
|
||||
"rowInstalledPlain": "installed"
|
||||
},
|
||||
"manage": {
|
||||
"title": "System",
|
||||
"subtitle": "Verwalten Sie installierte Modelle und Backends"
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
"cluster": "Cluster",
|
||||
"observability": "Observability",
|
||||
"access": "Access",
|
||||
"system": "System"
|
||||
"system": "System",
|
||||
"activity": "Activity"
|
||||
},
|
||||
"items": {
|
||||
"home": "Start",
|
||||
@@ -55,7 +56,8 @@
|
||||
"system": "System",
|
||||
"settings": "Einstellungen",
|
||||
"api": "API",
|
||||
"middleware": "Middleware"
|
||||
"middleware": "Middleware",
|
||||
"activity": "Aktivität"
|
||||
},
|
||||
"footer": {
|
||||
"github": "GitHub",
|
||||
|
||||
@@ -1,4 +1,84 @@
|
||||
{
|
||||
"activity": {
|
||||
"title": "Activity",
|
||||
"supporting": "Installs, downloads and removals on this instance.",
|
||||
"hide": "Hide",
|
||||
"moveToHistory": "Move to history",
|
||||
"moreCount": "{{count}} more",
|
||||
"waitingForInstaller": "waiting for the installer",
|
||||
"progressLabel": "Progress for {{name}}",
|
||||
"toNode": "to {{node}}",
|
||||
"nodesDone": "{{done}} of {{total}} nodes done",
|
||||
"timeLeft": "{{value}} left",
|
||||
"cancel": "Cancel",
|
||||
"cancelLabel": "Cancel {{name}}",
|
||||
"retry": "Retry",
|
||||
"retryLabel": "Retry {{name}}",
|
||||
"nodeCount": "{{count}} nodes",
|
||||
"showNodes": "Show {{count}} nodes",
|
||||
"hideNodes": "Hide per-node detail",
|
||||
"node": {
|
||||
"done": "Done",
|
||||
"failed": "Failed",
|
||||
"queued": "Queued",
|
||||
"workerBusy": "Worker busy",
|
||||
"downloading": "Downloading"
|
||||
},
|
||||
"kind": {
|
||||
"model": "model",
|
||||
"backend": "backend"
|
||||
},
|
||||
"verb": {
|
||||
"installing": "Installing {{kind}}",
|
||||
"installed": "Installed {{kind}}",
|
||||
"removed": "Removed {{kind}}",
|
||||
"staged": "Staged model",
|
||||
"failed": "Couldn't install {{kind}}",
|
||||
"queued": "Queued",
|
||||
"staging": "Staging model",
|
||||
"removing": "Removing {{kind}}",
|
||||
"failedRemoval": "Couldn't remove {{kind}}",
|
||||
"failedStaging": "Couldn't stage model"
|
||||
},
|
||||
"phase": {
|
||||
"resolving": "Resolving files",
|
||||
"downloading": "Downloading",
|
||||
"verifying": "Verifying",
|
||||
"committing": "Finalizing",
|
||||
"persisting": "Saving configuration"
|
||||
},
|
||||
"clearHistory": "Clear history",
|
||||
"inProgress": "In progress",
|
||||
"needsAttention": "Needs attention",
|
||||
"record": "Record",
|
||||
"historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
|
||||
"emptyTitle": "No operations since startup",
|
||||
"emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
|
||||
"browseModels": "Browse models",
|
||||
"viewInModels": "View in Models",
|
||||
"viewInBackends": "View in Backends",
|
||||
"rowInstalled": "installed in {{duration}}",
|
||||
"rowFailed": "failed: {{error}}",
|
||||
"rowCancelled": "cancelled",
|
||||
"rowRemoved": "removed",
|
||||
"retryFailed": "Retry failed: {{message}}",
|
||||
"filter": {
|
||||
"all": "All",
|
||||
"models": "Models",
|
||||
"backends": "Backends",
|
||||
"cluster": "Cluster"
|
||||
},
|
||||
"summaryRunning_one": "{{count}} operation running.",
|
||||
"summaryRunning_other": "{{count}} operations running.",
|
||||
"summaryFailed_one": "{{count}} operation needs attention.",
|
||||
"summaryFailed_other": "{{count}} operations need attention.",
|
||||
"summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
|
||||
"summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
|
||||
"summaryIdle": "Nothing running.",
|
||||
"emptyFiltered": "No operations match this filter.",
|
||||
"showAll": "Show all",
|
||||
"rowInstalledPlain": "installed"
|
||||
},
|
||||
"manage": {
|
||||
"title": "System",
|
||||
"subtitle": "Manage installed models and backends"
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
"cluster": "Cluster",
|
||||
"observability": "Observability",
|
||||
"access": "Access",
|
||||
"system": "System"
|
||||
"system": "System",
|
||||
"activity": "Activity"
|
||||
},
|
||||
"items": {
|
||||
"home": "Home",
|
||||
@@ -56,7 +57,8 @@
|
||||
"swarm": "Swarm",
|
||||
"system": "System",
|
||||
"settings": "Settings",
|
||||
"api": "API"
|
||||
"api": "API",
|
||||
"activity": "Activity"
|
||||
},
|
||||
"footer": {
|
||||
"github": "GitHub",
|
||||
|
||||
@@ -1,4 +1,84 @@
|
||||
{
|
||||
"activity": {
|
||||
"title": "Activity",
|
||||
"supporting": "Installs, downloads and removals on this instance.",
|
||||
"hide": "Hide",
|
||||
"moveToHistory": "Move to history",
|
||||
"moreCount": "{{count}} more",
|
||||
"waitingForInstaller": "waiting for the installer",
|
||||
"progressLabel": "Progress for {{name}}",
|
||||
"toNode": "to {{node}}",
|
||||
"nodesDone": "{{done}} of {{total}} nodes done",
|
||||
"timeLeft": "{{value}} left",
|
||||
"cancel": "Cancel",
|
||||
"cancelLabel": "Cancel {{name}}",
|
||||
"retry": "Retry",
|
||||
"retryLabel": "Retry {{name}}",
|
||||
"nodeCount": "{{count}} nodes",
|
||||
"showNodes": "Show {{count}} nodes",
|
||||
"hideNodes": "Hide per-node detail",
|
||||
"node": {
|
||||
"done": "Done",
|
||||
"failed": "Failed",
|
||||
"queued": "Queued",
|
||||
"workerBusy": "Worker busy",
|
||||
"downloading": "Downloading"
|
||||
},
|
||||
"kind": {
|
||||
"model": "model",
|
||||
"backend": "backend"
|
||||
},
|
||||
"verb": {
|
||||
"installing": "Installing {{kind}}",
|
||||
"installed": "Installed {{kind}}",
|
||||
"removed": "Removed {{kind}}",
|
||||
"staged": "Staged model",
|
||||
"failed": "Couldn't install {{kind}}",
|
||||
"queued": "Queued",
|
||||
"staging": "Staging model",
|
||||
"removing": "Removing {{kind}}",
|
||||
"failedRemoval": "Couldn't remove {{kind}}",
|
||||
"failedStaging": "Couldn't stage model"
|
||||
},
|
||||
"phase": {
|
||||
"resolving": "Resolving files",
|
||||
"downloading": "Downloading",
|
||||
"verifying": "Verifying",
|
||||
"committing": "Finalizing",
|
||||
"persisting": "Saving configuration"
|
||||
},
|
||||
"clearHistory": "Clear history",
|
||||
"inProgress": "In progress",
|
||||
"needsAttention": "Needs attention",
|
||||
"record": "Record",
|
||||
"historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
|
||||
"emptyTitle": "No operations since startup",
|
||||
"emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
|
||||
"browseModels": "Browse models",
|
||||
"viewInModels": "View in Models",
|
||||
"viewInBackends": "View in Backends",
|
||||
"rowInstalled": "installed in {{duration}}",
|
||||
"rowFailed": "failed: {{error}}",
|
||||
"rowCancelled": "cancelled",
|
||||
"rowRemoved": "removed",
|
||||
"retryFailed": "Retry failed: {{message}}",
|
||||
"filter": {
|
||||
"all": "All",
|
||||
"models": "Models",
|
||||
"backends": "Backends",
|
||||
"cluster": "Cluster"
|
||||
},
|
||||
"summaryRunning_one": "{{count}} operation running.",
|
||||
"summaryRunning_other": "{{count}} operations running.",
|
||||
"summaryFailed_one": "{{count}} operation needs attention.",
|
||||
"summaryFailed_other": "{{count}} operations need attention.",
|
||||
"summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
|
||||
"summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
|
||||
"summaryIdle": "Nothing running.",
|
||||
"emptyFiltered": "No operations match this filter.",
|
||||
"showAll": "Show all",
|
||||
"rowInstalledPlain": "installed"
|
||||
},
|
||||
"manage": {
|
||||
"title": "Sistema",
|
||||
"subtitle": "Administra modelos y backends instalados"
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
"cluster": "Cluster",
|
||||
"observability": "Observability",
|
||||
"access": "Access",
|
||||
"system": "System"
|
||||
"system": "System",
|
||||
"activity": "Activity"
|
||||
},
|
||||
"items": {
|
||||
"home": "Inicio",
|
||||
@@ -55,7 +56,8 @@
|
||||
"system": "Sistema",
|
||||
"settings": "Configuración",
|
||||
"api": "API",
|
||||
"middleware": "Middleware"
|
||||
"middleware": "Middleware",
|
||||
"activity": "Actividad"
|
||||
},
|
||||
"footer": {
|
||||
"github": "GitHub",
|
||||
|
||||
@@ -1,4 +1,84 @@
|
||||
{
|
||||
"activity": {
|
||||
"title": "Activity",
|
||||
"supporting": "Installs, downloads and removals on this instance.",
|
||||
"hide": "Hide",
|
||||
"moveToHistory": "Move to history",
|
||||
"moreCount": "{{count}} more",
|
||||
"waitingForInstaller": "waiting for the installer",
|
||||
"progressLabel": "Progress for {{name}}",
|
||||
"toNode": "to {{node}}",
|
||||
"nodesDone": "{{done}} of {{total}} nodes done",
|
||||
"timeLeft": "{{value}} left",
|
||||
"cancel": "Cancel",
|
||||
"cancelLabel": "Cancel {{name}}",
|
||||
"retry": "Retry",
|
||||
"retryLabel": "Retry {{name}}",
|
||||
"nodeCount": "{{count}} nodes",
|
||||
"showNodes": "Show {{count}} nodes",
|
||||
"hideNodes": "Hide per-node detail",
|
||||
"node": {
|
||||
"done": "Done",
|
||||
"failed": "Failed",
|
||||
"queued": "Queued",
|
||||
"workerBusy": "Worker busy",
|
||||
"downloading": "Downloading"
|
||||
},
|
||||
"kind": {
|
||||
"model": "model",
|
||||
"backend": "backend"
|
||||
},
|
||||
"verb": {
|
||||
"installing": "Installing {{kind}}",
|
||||
"installed": "Installed {{kind}}",
|
||||
"removed": "Removed {{kind}}",
|
||||
"staged": "Staged model",
|
||||
"failed": "Couldn't install {{kind}}",
|
||||
"queued": "Queued",
|
||||
"staging": "Staging model",
|
||||
"removing": "Removing {{kind}}",
|
||||
"failedRemoval": "Couldn't remove {{kind}}",
|
||||
"failedStaging": "Couldn't stage model"
|
||||
},
|
||||
"phase": {
|
||||
"resolving": "Resolving files",
|
||||
"downloading": "Downloading",
|
||||
"verifying": "Verifying",
|
||||
"committing": "Finalizing",
|
||||
"persisting": "Saving configuration"
|
||||
},
|
||||
"clearHistory": "Clear history",
|
||||
"inProgress": "In progress",
|
||||
"needsAttention": "Needs attention",
|
||||
"record": "Record",
|
||||
"historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
|
||||
"emptyTitle": "No operations since startup",
|
||||
"emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
|
||||
"browseModels": "Browse models",
|
||||
"viewInModels": "View in Models",
|
||||
"viewInBackends": "View in Backends",
|
||||
"rowInstalled": "installed in {{duration}}",
|
||||
"rowFailed": "failed: {{error}}",
|
||||
"rowCancelled": "cancelled",
|
||||
"rowRemoved": "removed",
|
||||
"retryFailed": "Retry failed: {{message}}",
|
||||
"filter": {
|
||||
"all": "All",
|
||||
"models": "Models",
|
||||
"backends": "Backends",
|
||||
"cluster": "Cluster"
|
||||
},
|
||||
"summaryRunning_one": "{{count}} operation running.",
|
||||
"summaryRunning_other": "{{count}} operations running.",
|
||||
"summaryFailed_one": "{{count}} operation needs attention.",
|
||||
"summaryFailed_other": "{{count}} operations need attention.",
|
||||
"summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
|
||||
"summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
|
||||
"summaryIdle": "Nothing running.",
|
||||
"emptyFiltered": "No operations match this filter.",
|
||||
"showAll": "Show all",
|
||||
"rowInstalledPlain": "installed"
|
||||
},
|
||||
"manage": {
|
||||
"title": "Sistem",
|
||||
"subtitle": "Kelola model dan backend yang terinstal"
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
"cluster": "Kluster",
|
||||
"observability": "Observabilitas",
|
||||
"access": "Akses",
|
||||
"system": "Sistem"
|
||||
"system": "Sistem",
|
||||
"activity": "Activity"
|
||||
},
|
||||
"items": {
|
||||
"home": "Beranda",
|
||||
@@ -55,7 +56,8 @@
|
||||
"swarm": "Swarm",
|
||||
"system": "Sistem",
|
||||
"settings": "Pengaturan",
|
||||
"api": "API"
|
||||
"api": "API",
|
||||
"activity": "Aktivitas"
|
||||
},
|
||||
"footer": {
|
||||
"github": "GitHub",
|
||||
|
||||
@@ -1,4 +1,84 @@
|
||||
{
|
||||
"activity": {
|
||||
"title": "Activity",
|
||||
"supporting": "Installs, downloads and removals on this instance.",
|
||||
"hide": "Hide",
|
||||
"moveToHistory": "Move to history",
|
||||
"moreCount": "{{count}} more",
|
||||
"waitingForInstaller": "waiting for the installer",
|
||||
"progressLabel": "Progress for {{name}}",
|
||||
"toNode": "to {{node}}",
|
||||
"nodesDone": "{{done}} of {{total}} nodes done",
|
||||
"timeLeft": "{{value}} left",
|
||||
"cancel": "Cancel",
|
||||
"cancelLabel": "Cancel {{name}}",
|
||||
"retry": "Retry",
|
||||
"retryLabel": "Retry {{name}}",
|
||||
"nodeCount": "{{count}} nodes",
|
||||
"showNodes": "Show {{count}} nodes",
|
||||
"hideNodes": "Hide per-node detail",
|
||||
"node": {
|
||||
"done": "Done",
|
||||
"failed": "Failed",
|
||||
"queued": "Queued",
|
||||
"workerBusy": "Worker busy",
|
||||
"downloading": "Downloading"
|
||||
},
|
||||
"kind": {
|
||||
"model": "model",
|
||||
"backend": "backend"
|
||||
},
|
||||
"verb": {
|
||||
"installing": "Installing {{kind}}",
|
||||
"installed": "Installed {{kind}}",
|
||||
"removed": "Removed {{kind}}",
|
||||
"staged": "Staged model",
|
||||
"failed": "Couldn't install {{kind}}",
|
||||
"queued": "Queued",
|
||||
"staging": "Staging model",
|
||||
"removing": "Removing {{kind}}",
|
||||
"failedRemoval": "Couldn't remove {{kind}}",
|
||||
"failedStaging": "Couldn't stage model"
|
||||
},
|
||||
"phase": {
|
||||
"resolving": "Resolving files",
|
||||
"downloading": "Downloading",
|
||||
"verifying": "Verifying",
|
||||
"committing": "Finalizing",
|
||||
"persisting": "Saving configuration"
|
||||
},
|
||||
"clearHistory": "Clear history",
|
||||
"inProgress": "In progress",
|
||||
"needsAttention": "Needs attention",
|
||||
"record": "Record",
|
||||
"historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
|
||||
"emptyTitle": "No operations since startup",
|
||||
"emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
|
||||
"browseModels": "Browse models",
|
||||
"viewInModels": "View in Models",
|
||||
"viewInBackends": "View in Backends",
|
||||
"rowInstalled": "installed in {{duration}}",
|
||||
"rowFailed": "failed: {{error}}",
|
||||
"rowCancelled": "cancelled",
|
||||
"rowRemoved": "removed",
|
||||
"retryFailed": "Retry failed: {{message}}",
|
||||
"filter": {
|
||||
"all": "All",
|
||||
"models": "Models",
|
||||
"backends": "Backends",
|
||||
"cluster": "Cluster"
|
||||
},
|
||||
"summaryRunning_one": "{{count}} operation running.",
|
||||
"summaryRunning_other": "{{count}} operations running.",
|
||||
"summaryFailed_one": "{{count}} operation needs attention.",
|
||||
"summaryFailed_other": "{{count}} operations need attention.",
|
||||
"summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
|
||||
"summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
|
||||
"summaryIdle": "Nothing running.",
|
||||
"emptyFiltered": "No operations match this filter.",
|
||||
"showAll": "Show all",
|
||||
"rowInstalledPlain": "installed"
|
||||
},
|
||||
"manage": {
|
||||
"title": "Sistema",
|
||||
"subtitle": "Gestisci modelli e backend installati"
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
"cluster": "Cluster",
|
||||
"observability": "Observability",
|
||||
"access": "Access",
|
||||
"system": "System"
|
||||
"system": "System",
|
||||
"activity": "Activity"
|
||||
},
|
||||
"items": {
|
||||
"home": "Home",
|
||||
@@ -55,7 +56,8 @@
|
||||
"system": "Sistema",
|
||||
"settings": "Impostazioni",
|
||||
"api": "API",
|
||||
"middleware": "Middleware"
|
||||
"middleware": "Middleware",
|
||||
"activity": "Attività"
|
||||
},
|
||||
"footer": {
|
||||
"github": "GitHub",
|
||||
|
||||
@@ -1,4 +1,84 @@
|
||||
{
|
||||
"activity": {
|
||||
"title": "Activity",
|
||||
"supporting": "Installs, downloads and removals on this instance.",
|
||||
"hide": "Hide",
|
||||
"moveToHistory": "Move to history",
|
||||
"moreCount": "{{count}} more",
|
||||
"waitingForInstaller": "waiting for the installer",
|
||||
"progressLabel": "Progress for {{name}}",
|
||||
"toNode": "to {{node}}",
|
||||
"nodesDone": "{{done}} of {{total}} nodes done",
|
||||
"timeLeft": "{{value}} left",
|
||||
"cancel": "Cancel",
|
||||
"cancelLabel": "Cancel {{name}}",
|
||||
"retry": "Retry",
|
||||
"retryLabel": "Retry {{name}}",
|
||||
"nodeCount": "{{count}} nodes",
|
||||
"showNodes": "Show {{count}} nodes",
|
||||
"hideNodes": "Hide per-node detail",
|
||||
"node": {
|
||||
"done": "Done",
|
||||
"failed": "Failed",
|
||||
"queued": "Queued",
|
||||
"workerBusy": "Worker busy",
|
||||
"downloading": "Downloading"
|
||||
},
|
||||
"kind": {
|
||||
"model": "model",
|
||||
"backend": "backend"
|
||||
},
|
||||
"verb": {
|
||||
"installing": "Installing {{kind}}",
|
||||
"installed": "Installed {{kind}}",
|
||||
"removed": "Removed {{kind}}",
|
||||
"staged": "Staged model",
|
||||
"failed": "Couldn't install {{kind}}",
|
||||
"queued": "Queued",
|
||||
"staging": "Staging model",
|
||||
"removing": "Removing {{kind}}",
|
||||
"failedRemoval": "Couldn't remove {{kind}}",
|
||||
"failedStaging": "Couldn't stage model"
|
||||
},
|
||||
"phase": {
|
||||
"resolving": "Resolving files",
|
||||
"downloading": "Downloading",
|
||||
"verifying": "Verifying",
|
||||
"committing": "Finalizing",
|
||||
"persisting": "Saving configuration"
|
||||
},
|
||||
"clearHistory": "Clear history",
|
||||
"inProgress": "In progress",
|
||||
"needsAttention": "Needs attention",
|
||||
"record": "Record",
|
||||
"historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
|
||||
"emptyTitle": "No operations since startup",
|
||||
"emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
|
||||
"browseModels": "Browse models",
|
||||
"viewInModels": "View in Models",
|
||||
"viewInBackends": "View in Backends",
|
||||
"rowInstalled": "installed in {{duration}}",
|
||||
"rowFailed": "failed: {{error}}",
|
||||
"rowCancelled": "cancelled",
|
||||
"rowRemoved": "removed",
|
||||
"retryFailed": "Retry failed: {{message}}",
|
||||
"filter": {
|
||||
"all": "All",
|
||||
"models": "Models",
|
||||
"backends": "Backends",
|
||||
"cluster": "Cluster"
|
||||
},
|
||||
"summaryRunning_one": "{{count}} operation running.",
|
||||
"summaryRunning_other": "{{count}} operations running.",
|
||||
"summaryFailed_one": "{{count}} operation needs attention.",
|
||||
"summaryFailed_other": "{{count}} operations need attention.",
|
||||
"summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
|
||||
"summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
|
||||
"summaryIdle": "Nothing running.",
|
||||
"emptyFiltered": "No operations match this filter.",
|
||||
"showAll": "Show all",
|
||||
"rowInstalledPlain": "installed"
|
||||
},
|
||||
"manage": {
|
||||
"title": "시스템",
|
||||
"subtitle": "설치된 모델과 백엔드를 관리합니다"
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
"cluster": "Cluster",
|
||||
"observability": "Observability",
|
||||
"access": "Access",
|
||||
"system": "System"
|
||||
"system": "System",
|
||||
"activity": "Activity"
|
||||
},
|
||||
"items": {
|
||||
"home": "홈",
|
||||
@@ -55,7 +56,8 @@
|
||||
"swarm": "Swarm",
|
||||
"system": "시스템",
|
||||
"settings": "설정",
|
||||
"api": "API"
|
||||
"api": "API",
|
||||
"activity": "활동"
|
||||
},
|
||||
"footer": {
|
||||
"github": "GitHub",
|
||||
|
||||
@@ -1,4 +1,84 @@
|
||||
{
|
||||
"activity": {
|
||||
"title": "Activity",
|
||||
"supporting": "Installs, downloads and removals on this instance.",
|
||||
"hide": "Hide",
|
||||
"moveToHistory": "Move to history",
|
||||
"moreCount": "{{count}} more",
|
||||
"waitingForInstaller": "waiting for the installer",
|
||||
"progressLabel": "Progress for {{name}}",
|
||||
"toNode": "to {{node}}",
|
||||
"nodesDone": "{{done}} of {{total}} nodes done",
|
||||
"timeLeft": "{{value}} left",
|
||||
"cancel": "Cancel",
|
||||
"cancelLabel": "Cancel {{name}}",
|
||||
"retry": "Retry",
|
||||
"retryLabel": "Retry {{name}}",
|
||||
"nodeCount": "{{count}} nodes",
|
||||
"showNodes": "Show {{count}} nodes",
|
||||
"hideNodes": "Hide per-node detail",
|
||||
"node": {
|
||||
"done": "Done",
|
||||
"failed": "Failed",
|
||||
"queued": "Queued",
|
||||
"workerBusy": "Worker busy",
|
||||
"downloading": "Downloading"
|
||||
},
|
||||
"kind": {
|
||||
"model": "model",
|
||||
"backend": "backend"
|
||||
},
|
||||
"verb": {
|
||||
"installing": "Installing {{kind}}",
|
||||
"installed": "Installed {{kind}}",
|
||||
"removed": "Removed {{kind}}",
|
||||
"staged": "Staged model",
|
||||
"failed": "Couldn't install {{kind}}",
|
||||
"queued": "Queued",
|
||||
"staging": "Staging model",
|
||||
"removing": "Removing {{kind}}",
|
||||
"failedRemoval": "Couldn't remove {{kind}}",
|
||||
"failedStaging": "Couldn't stage model"
|
||||
},
|
||||
"phase": {
|
||||
"resolving": "Resolving files",
|
||||
"downloading": "Downloading",
|
||||
"verifying": "Verifying",
|
||||
"committing": "Finalizing",
|
||||
"persisting": "Saving configuration"
|
||||
},
|
||||
"clearHistory": "Clear history",
|
||||
"inProgress": "In progress",
|
||||
"needsAttention": "Needs attention",
|
||||
"record": "Record",
|
||||
"historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
|
||||
"emptyTitle": "No operations since startup",
|
||||
"emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
|
||||
"browseModels": "Browse models",
|
||||
"viewInModels": "View in Models",
|
||||
"viewInBackends": "View in Backends",
|
||||
"rowInstalled": "installed in {{duration}}",
|
||||
"rowFailed": "failed: {{error}}",
|
||||
"rowCancelled": "cancelled",
|
||||
"rowRemoved": "removed",
|
||||
"retryFailed": "Retry failed: {{message}}",
|
||||
"filter": {
|
||||
"all": "All",
|
||||
"models": "Models",
|
||||
"backends": "Backends",
|
||||
"cluster": "Cluster"
|
||||
},
|
||||
"summaryRunning_one": "{{count}} operation running.",
|
||||
"summaryRunning_other": "{{count}} operations running.",
|
||||
"summaryFailed_one": "{{count}} operation needs attention.",
|
||||
"summaryFailed_other": "{{count}} operations need attention.",
|
||||
"summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
|
||||
"summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
|
||||
"summaryIdle": "Nothing running.",
|
||||
"emptyFiltered": "No operations match this filter.",
|
||||
"showAll": "Show all",
|
||||
"rowInstalledPlain": "installed"
|
||||
},
|
||||
"manage": {
|
||||
"title": "系统",
|
||||
"subtitle": "管理已安装的模型和后端"
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
"cluster": "Cluster",
|
||||
"observability": "Observability",
|
||||
"access": "Access",
|
||||
"system": "System"
|
||||
"system": "System",
|
||||
"activity": "Activity"
|
||||
},
|
||||
"items": {
|
||||
"home": "首页",
|
||||
@@ -55,7 +56,8 @@
|
||||
"system": "系统",
|
||||
"settings": "设置",
|
||||
"api": "API",
|
||||
"middleware": "Middleware"
|
||||
"middleware": "Middleware",
|
||||
"activity": "活动"
|
||||
},
|
||||
"footer": {
|
||||
"github": "GitHub",
|
||||
|
||||
@@ -13,6 +13,11 @@
|
||||
min-height: 100dvh;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
/* A flex item's automatic minimum is its content's minimum, so without this
|
||||
any single wide descendant (a long install error in the operations strip,
|
||||
a wide table on a phone) drags the whole column past the viewport and
|
||||
every page gets a horizontal scrollbar. */
|
||||
min-width: 0;
|
||||
transition: margin-left var(--duration-normal) var(--ease-default);
|
||||
}
|
||||
|
||||
@@ -663,41 +668,151 @@
|
||||
to { transform: rotate(360deg); }
|
||||
}
|
||||
|
||||
/* Operations bar */
|
||||
.operations-bar {
|
||||
background: var(--color-bg-secondary);
|
||||
border-bottom: 1px solid var(--color-border-subtle);
|
||||
padding: var(--spacing-xs) var(--spacing-md);
|
||||
}
|
||||
|
||||
.operation-text {
|
||||
font-family: var(--font-mono);
|
||||
}
|
||||
.operation-progress {
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
|
||||
.operation-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: var(--spacing-md);
|
||||
padding: var(--spacing-xs) 0;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.operation-info {
|
||||
/* Operations strip: always exactly one line. Anything that does not fit here
|
||||
belongs on /app/activity. */
|
||||
.operations-strip {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: var(--spacing-sm);
|
||||
flex: 2 1 0;
|
||||
min-height: 40px;
|
||||
padding: var(--spacing-xs) var(--spacing-md);
|
||||
background: var(--color-bg-secondary);
|
||||
border-bottom: 1px solid var(--color-border-subtle);
|
||||
border-left: 2px solid var(--color-primary);
|
||||
font-size: 0.8125rem;
|
||||
/* An install error is arbitrarily long text. This clips it; the shrinking
|
||||
is done by min-width on .main-content above and on __detail below. */
|
||||
overflow: hidden;
|
||||
}
|
||||
.operations-strip--error { border-left-color: var(--color-error); background: var(--color-error-light); }
|
||||
.operations-strip--queued { border-left-color: var(--color-text-disabled); }
|
||||
.operations-strip--staging { border-left-color: var(--color-info); }
|
||||
.operations-strip--removing { border-left-color: var(--color-warning); }
|
||||
.operations-strip--done { border-left-color: var(--color-success); }
|
||||
|
||||
.operations-strip__icon { flex: none; font-size: 0.8125rem; }
|
||||
.operations-strip--error .operations-strip__icon { color: var(--color-error); }
|
||||
.operations-strip--queued .operations-strip__icon { color: var(--color-text-muted); }
|
||||
.operations-strip--staging .operations-strip__icon { color: var(--color-info); }
|
||||
.operations-strip--removing .operations-strip__icon { color: var(--color-warning); }
|
||||
.operations-strip--done .operations-strip__icon { color: var(--color-success); }
|
||||
|
||||
.operations-strip__spinner {
|
||||
width: 13px;
|
||||
height: 13px;
|
||||
flex: none;
|
||||
border-radius: 50%;
|
||||
border: 2px solid var(--color-primary-light);
|
||||
border-top-color: var(--color-primary);
|
||||
animation: operationsStripSpin 0.9s linear infinite;
|
||||
}
|
||||
@keyframes operationsStripSpin { to { transform: rotate(360deg); } }
|
||||
|
||||
.operations-strip__verb { flex: none; color: var(--color-text-secondary); white-space: nowrap; }
|
||||
.operations-strip__name {
|
||||
font-family: var(--font-mono);
|
||||
font-weight: 500;
|
||||
white-space: nowrap;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
/* overflow: hidden zeroes the automatic minimum size, so without a floor the
|
||||
name loses the shrink contest to a long error and the identity of the
|
||||
thing that broke is the first casualty. */
|
||||
min-width: 12ch;
|
||||
}
|
||||
.operations-strip__sep { flex: none; color: var(--color-border-strong); }
|
||||
/* min-width and overflow are what let these shrink: a nowrap flex item's
|
||||
automatic minimum size is otherwise its full content width. */
|
||||
.operations-strip__detail,
|
||||
.operations-strip__bytes {
|
||||
flex: 0 1 auto;
|
||||
min-width: 0;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
color: var(--color-text-muted);
|
||||
font-size: 0.75rem;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.operations-strip__bytes { font-variant-numeric: tabular-nums; }
|
||||
.operations-strip__spacer { flex: 1 1 auto; min-width: var(--spacing-xs); }
|
||||
.operations-strip__pct {
|
||||
flex: none;
|
||||
font-family: var(--font-mono);
|
||||
font-variant-numeric: tabular-nums;
|
||||
font-size: 0.75rem;
|
||||
font-weight: 500;
|
||||
color: var(--color-primary);
|
||||
}
|
||||
.operations-strip__track {
|
||||
flex: 0 0 132px;
|
||||
height: 3px;
|
||||
border-radius: var(--radius-full);
|
||||
background: var(--color-surface-sunken);
|
||||
overflow: hidden;
|
||||
}
|
||||
.operations-strip__fill {
|
||||
display: block;
|
||||
height: 100%;
|
||||
border-radius: var(--radius-full);
|
||||
background: var(--color-primary);
|
||||
transition: width var(--duration-slow) var(--ease-spring);
|
||||
}
|
||||
.operations-strip--staging .operations-strip__fill { background: var(--color-info); }
|
||||
|
||||
.operations-strip__more {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
flex: none;
|
||||
min-height: 28px;
|
||||
padding: 0 var(--spacing-sm);
|
||||
border-radius: var(--radius-full);
|
||||
border: 1px solid var(--color-primary-border);
|
||||
background: var(--color-primary-light);
|
||||
color: var(--color-primary);
|
||||
font-size: 0.6875rem;
|
||||
font-weight: 600;
|
||||
white-space: nowrap;
|
||||
text-decoration: none;
|
||||
}
|
||||
.operations-strip__more:hover { background: var(--color-primary); color: var(--color-primary-text); }
|
||||
.operations-strip__more--neutral {
|
||||
background: transparent;
|
||||
border-color: var(--color-border-default);
|
||||
color: var(--color-text-secondary);
|
||||
}
|
||||
|
||||
.operation-info > .operation-text {
|
||||
flex: 1 1 auto;
|
||||
min-width: 0;
|
||||
.operations-strip__hide {
|
||||
flex: none;
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
border: 0;
|
||||
border-radius: var(--radius-sm);
|
||||
background: transparent;
|
||||
color: var(--color-text-muted);
|
||||
cursor: pointer;
|
||||
}
|
||||
.operations-strip__hide:hover { background: var(--color-bg-hover); color: var(--color-text-primary); }
|
||||
|
||||
/* Narrow screens drop the prose, never the name, the percentage or the
|
||||
counter. */
|
||||
@media (max-width: 640px) {
|
||||
.operations-strip__verb,
|
||||
.operations-strip__detail,
|
||||
.operations-strip__bytes,
|
||||
.operations-strip__sep,
|
||||
.operations-strip__track { display: none; }
|
||||
.operations-strip__name { max-width: 45vw; }
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.operations-strip__spinner { animation: none; }
|
||||
.operations-strip__fill { transition: none; }
|
||||
}
|
||||
|
||||
/* Row-level install indicator, used by the Models and Backends tables. */
|
||||
.operation-spinner {
|
||||
width: 16px;
|
||||
height: 16px;
|
||||
@@ -710,19 +825,6 @@
|
||||
display: inline-block;
|
||||
}
|
||||
|
||||
.operation-text {
|
||||
font-size: 0.8125rem;
|
||||
color: var(--color-text-secondary);
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
|
||||
.operation-progress {
|
||||
font-size: 0.75rem;
|
||||
color: var(--color-primary);
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.operation-bar-container {
|
||||
flex: 0 1 160px;
|
||||
min-width: 80px;
|
||||
@@ -760,38 +862,7 @@
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.operation-cancel {
|
||||
flex-shrink: 0;
|
||||
background: none;
|
||||
border: none;
|
||||
color: var(--color-text-muted);
|
||||
cursor: pointer;
|
||||
padding: 4px 6px;
|
||||
font-size: 0.875rem;
|
||||
}
|
||||
.operation-cancel:hover {
|
||||
color: var(--color-error);
|
||||
}
|
||||
|
||||
/* Operations bar: per-node breakdown (multi-worker installs) */
|
||||
.operation-expand {
|
||||
background: none;
|
||||
border: none;
|
||||
color: var(--color-text-muted);
|
||||
cursor: pointer;
|
||||
padding: 0 var(--spacing-xs);
|
||||
font-size: var(--text-xs);
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.25rem;
|
||||
}
|
||||
.operation-expand:hover {
|
||||
color: var(--color-text-primary);
|
||||
}
|
||||
.operation-expand-label {
|
||||
font-size: var(--text-xs);
|
||||
}
|
||||
|
||||
/* Per-node breakdown of a multi-worker install. */
|
||||
.operation-nodes-list {
|
||||
list-style: none;
|
||||
margin: var(--spacing-xs) 0 0;
|
||||
@@ -5563,20 +5634,6 @@ button.collapsible-header:focus-visible {
|
||||
border-right: 0;
|
||||
margin-inline: calc(-1 * var(--spacing-md));
|
||||
}
|
||||
|
||||
/* Operations toasts: scroll horizontally instead of wrapping */
|
||||
.operations-bar {
|
||||
overflow-x: auto;
|
||||
flex-wrap: nowrap;
|
||||
-webkit-overflow-scrolling: touch;
|
||||
}
|
||||
.operation-item { flex-shrink: 0; }
|
||||
.operation-text {
|
||||
max-width: 60vw;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
}
|
||||
|
||||
/* Reduced motion — disable non-essential transitions for users who
|
||||
@@ -5584,7 +5641,7 @@ button.collapsible-header:focus-visible {
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.sidebar,
|
||||
.page-transition,
|
||||
.operations-bar,
|
||||
.operations-strip,
|
||||
.page,
|
||||
.main-content {
|
||||
transition: none !important;
|
||||
@@ -9645,3 +9702,224 @@ button.collapsible-header:focus-visible {
|
||||
.variant-row__info,
|
||||
.variant-row__action { transition: none; }
|
||||
}
|
||||
|
||||
/* Live operation card on /app/activity. Status is carried by the icon and the
|
||||
tag rather than a coloured rail, so a page of cards does not read as stripes. */
|
||||
.operation-card {
|
||||
background: var(--color-bg-secondary);
|
||||
border: 1px solid var(--color-border-subtle);
|
||||
border-radius: var(--radius-md);
|
||||
overflow: hidden;
|
||||
}
|
||||
.operation-card--error { border-color: var(--color-error-border); background: var(--color-error-light); }
|
||||
.operation-card__main { display: flex; align-items: center; gap: var(--spacing-sm); padding: var(--spacing-sm); }
|
||||
.operation-card__icon { flex: none; }
|
||||
.operation-card__icon--error { color: var(--color-error); }
|
||||
.operation-card__icon--staging { color: var(--color-info); }
|
||||
.operation-card__icon--removing { color: var(--color-warning); }
|
||||
.operation-card__spinner {
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
flex: none;
|
||||
border-radius: 50%;
|
||||
border: 2px solid var(--color-primary-light);
|
||||
border-top-color: var(--color-primary);
|
||||
animation: operationsStripSpin 0.9s linear infinite;
|
||||
}
|
||||
.operation-card__body { flex: 1; min-width: 0; display: flex; flex-direction: column; gap: 4px; }
|
||||
.operation-card__title { display: flex; align-items: center; gap: var(--spacing-xs); flex-wrap: wrap; }
|
||||
.operation-card__name { font-family: var(--font-mono); font-weight: 500; }
|
||||
.operation-card__tag {
|
||||
font-size: 0.625rem;
|
||||
letter-spacing: 0.07em;
|
||||
text-transform: uppercase;
|
||||
font-weight: 600;
|
||||
padding: 1px var(--spacing-xs);
|
||||
border-radius: var(--radius-sm);
|
||||
border: 1px solid var(--color-border-default);
|
||||
color: var(--color-text-muted);
|
||||
}
|
||||
.operation-card__tag--model { color: var(--color-primary); border-color: var(--color-primary-border); background: var(--color-primary-light); }
|
||||
.operation-card__tag--backend { color: var(--color-info); border-color: var(--color-info-border); background: var(--color-info-light); }
|
||||
.operation-card__tag--cluster { color: var(--color-warning); border-color: var(--color-warning-border); background: var(--color-warning-light); }
|
||||
.operation-card__sub { display: flex; align-items: center; gap: var(--spacing-xs); flex-wrap: wrap; font-size: 0.75rem; color: var(--color-text-muted); }
|
||||
.operation-card__bytes { font-variant-numeric: tabular-nums; }
|
||||
/* The legacy installer message embeds an absolute file path, so it is both long
|
||||
and a single unbreakable token. Left unclamped it wraps to three lines and
|
||||
becomes the largest thing on the card, which is how it looked against a real
|
||||
download. One line, ellipsised, full text in the title. */
|
||||
.operation-card__message {
|
||||
min-width: 0;
|
||||
flex: 1 1 0;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.operation-card__verb { color: var(--color-text-secondary); }
|
||||
/* A Go error is arbitrarily long and often multi-line. Clamped so it cannot
|
||||
push the rest of the card off screen; the full text is in the title. */
|
||||
.operation-card__error {
|
||||
color: var(--color-error);
|
||||
font-family: var(--font-mono);
|
||||
/* A flex item defaults to min-width:auto, so one unbroken token (a URL
|
||||
inside a Go error) would refuse to shrink and push the sub row past the
|
||||
card. The vertical clamp below cannot help with that. */
|
||||
min-width: 0;
|
||||
display: -webkit-box;
|
||||
-webkit-box-orient: vertical;
|
||||
-webkit-line-clamp: 2;
|
||||
line-clamp: 2;
|
||||
overflow: hidden;
|
||||
}
|
||||
.operation-card__track { height: 4px; border-radius: var(--radius-full); background: var(--color-surface-sunken); overflow: hidden; }
|
||||
.operation-card__fill { display: block; height: 100%; border-radius: var(--radius-full); background: var(--color-primary); transition: width var(--duration-slow) var(--ease-spring); }
|
||||
.operation-card__actions { display: flex; align-items: center; gap: var(--spacing-xs); flex: none; }
|
||||
.operation-card__pct { font-family: var(--font-mono); font-variant-numeric: tabular-nums; font-weight: 600; color: var(--color-primary); }
|
||||
.operation-card__hide {
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
border: 0;
|
||||
border-radius: var(--radius-sm);
|
||||
background: transparent;
|
||||
color: var(--color-text-muted);
|
||||
cursor: pointer;
|
||||
}
|
||||
.operation-card__hide:hover { background: var(--color-bg-hover); color: var(--color-text-primary); }
|
||||
.operation-card__nodes-toggle {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: var(--spacing-xs);
|
||||
width: 100%;
|
||||
min-height: 28px;
|
||||
padding: var(--spacing-xs) var(--spacing-sm);
|
||||
border: 0;
|
||||
border-top: 1px solid var(--color-border-subtle);
|
||||
background: transparent;
|
||||
color: var(--color-text-muted);
|
||||
font-size: 0.6875rem;
|
||||
cursor: pointer;
|
||||
}
|
||||
.operation-card__nodes-toggle:hover { color: var(--color-text-secondary); background: var(--color-bg-hover); }
|
||||
|
||||
/* The node list carries its own inset here. It was written for the old bar,
|
||||
whose parent supplied one, and the card only pads its main row. */
|
||||
.operation-card .operation-nodes-list { padding: 0 var(--spacing-sm) var(--spacing-xs); }
|
||||
/* One rule between the card body and the node block, not two: when the
|
||||
disclosure is there, it is already the separator. */
|
||||
.operation-card__nodes-toggle + .operation-nodes-list { margin-top: 0; border-top: 0; }
|
||||
.operation-card .operation-node-error {
|
||||
max-width: 40ch;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.operation-card__spinner { animation: none; }
|
||||
.operation-card__fill { transition: none; }
|
||||
.operation-card .operation-node-bar { transition: none; }
|
||||
}
|
||||
|
||||
/* ── Activity page ────────────────────────────────────────────────────────── */
|
||||
.activity-page { display: flex; flex-direction: column; gap: var(--spacing-md); }
|
||||
.activity-filters { display: flex; gap: var(--spacing-xs); flex-wrap: wrap; }
|
||||
.activity-chip {
|
||||
min-height: 28px;
|
||||
padding: 0 var(--spacing-sm);
|
||||
border-radius: var(--radius-full);
|
||||
border: 1px solid var(--color-border-default);
|
||||
background: transparent;
|
||||
color: var(--color-text-muted);
|
||||
font-size: 0.75rem;
|
||||
cursor: pointer;
|
||||
}
|
||||
.activity-chip[aria-pressed="true"] {
|
||||
background: var(--color-primary-light);
|
||||
border-color: var(--color-primary-border);
|
||||
color: var(--color-primary);
|
||||
font-weight: 500;
|
||||
}
|
||||
.activity-section { display: flex; flex-direction: column; gap: var(--spacing-sm); }
|
||||
.activity-section__title {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
gap: var(--spacing-xs);
|
||||
margin: 0;
|
||||
font-size: 0.75rem;
|
||||
font-weight: 600;
|
||||
letter-spacing: 0.08em;
|
||||
text-transform: uppercase;
|
||||
color: var(--color-text-secondary);
|
||||
}
|
||||
.activity-section__count { font-family: var(--font-mono); color: var(--color-text-muted); font-weight: 400; }
|
||||
.activity-rows {
|
||||
background: var(--color-bg-secondary);
|
||||
border: 1px solid var(--color-border-subtle);
|
||||
border-radius: var(--radius-md);
|
||||
overflow: hidden;
|
||||
}
|
||||
.activity-row {
|
||||
display: grid;
|
||||
grid-template-columns: 16px 1fr auto auto;
|
||||
gap: var(--spacing-sm);
|
||||
align-items: center;
|
||||
min-height: 38px;
|
||||
padding: var(--spacing-xs) var(--spacing-sm);
|
||||
border-bottom: 1px solid var(--color-border-subtle);
|
||||
font-size: 0.8125rem;
|
||||
}
|
||||
.activity-row:last-child { border-bottom: 0; }
|
||||
.activity-row:hover { background: var(--color-bg-hover); }
|
||||
.activity-row__icon--completed { color: var(--color-success); }
|
||||
.activity-row__icon--failed { color: var(--color-error); }
|
||||
.activity-row__icon--cancelled { color: var(--color-warning); }
|
||||
.activity-row__name { font-family: var(--font-mono); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.activity-row__name small { font-family: var(--font-sans); color: var(--color-text-muted); margin-left: var(--spacing-xs); }
|
||||
.activity-row__when { color: var(--color-text-muted); font-size: 0.75rem; font-variant-numeric: tabular-nums; }
|
||||
.activity-row__action { color: var(--color-primary); font-size: 0.75rem; text-decoration: none; white-space: nowrap; }
|
||||
.activity-row__action:hover { text-decoration: underline; }
|
||||
.activity-note { font-size: 0.75rem; color: var(--color-text-muted); margin: 0; }
|
||||
.activity-empty {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
gap: var(--spacing-xs);
|
||||
padding: var(--spacing-xl) var(--spacing-md);
|
||||
text-align: center;
|
||||
}
|
||||
.activity-empty__icon { font-size: 1.5rem; color: var(--color-text-disabled); margin-bottom: var(--spacing-xs); }
|
||||
.activity-empty__title { margin: 0; font-weight: 500; color: var(--color-text-secondary); }
|
||||
.activity-empty__body { margin: 0; max-width: 46ch; font-size: 0.8125rem; color: var(--color-text-muted); }
|
||||
/* A chip matching nothing is a smaller event than an empty instance, and gets
|
||||
less of the page so the chip row above stays the obvious thing to change. */
|
||||
.activity-empty--filtered { padding: var(--spacing-lg) var(--spacing-md); }
|
||||
|
||||
.nav-badge {
|
||||
min-width: 18px;
|
||||
height: 18px;
|
||||
padding: 0 var(--spacing-xs);
|
||||
border-radius: var(--radius-full);
|
||||
background: var(--color-primary);
|
||||
color: var(--color-primary-text);
|
||||
font-family: var(--font-mono);
|
||||
font-size: 0.625rem;
|
||||
font-weight: 700;
|
||||
line-height: 18px;
|
||||
text-align: center;
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
.nav-badge--error { background: var(--color-error); color: var(--color-bg-secondary); }
|
||||
/* Collapsed, the label is gone and the item centres its icon: an inline badge
|
||||
would shove that icon off the rail's axis. Pinning it to the corner keeps the
|
||||
count visible without moving anything else. */
|
||||
.sidebar.collapsed .nav-badge {
|
||||
position: absolute;
|
||||
top: 2px;
|
||||
right: 4px;
|
||||
min-width: 16px;
|
||||
height: 16px;
|
||||
line-height: 16px;
|
||||
font-size: 0.5625rem;
|
||||
}
|
||||
|
||||
233
core/http/react-ui/src/components/OperationCard.jsx
Normal file
233
core/http/react-ui/src/components/OperationCard.jsx
Normal file
@@ -0,0 +1,233 @@
|
||||
import { useId, useState } from 'react'
|
||||
import { useTranslation } from 'react-i18next'
|
||||
import { formatBytes } from '../utils/format'
|
||||
|
||||
const phaseKeys = {
|
||||
resolving: 'activity.phase.resolving',
|
||||
downloading: 'activity.phase.downloading',
|
||||
verifying: 'activity.phase.verifying',
|
||||
committing: 'activity.phase.committing',
|
||||
persisting: 'activity.phase.persisting',
|
||||
}
|
||||
|
||||
const nodeStatusKeys = {
|
||||
success: 'activity.node.done',
|
||||
error: 'activity.node.failed',
|
||||
queued: 'activity.node.queued',
|
||||
running_on_worker: 'activity.node.workerBusy',
|
||||
downloading: 'activity.node.downloading',
|
||||
}
|
||||
|
||||
// etaSeconds is derived by OperationsContext from the byte delta between
|
||||
// polls. It is absent until two samples exist, and absent for every operation
|
||||
// when it is absent for any byte-tracked one.
|
||||
function formatEta(seconds) {
|
||||
if (!Number.isFinite(seconds) || seconds <= 0) return ''
|
||||
if (seconds < 60) return `${seconds}s`
|
||||
const minutes = Math.round(seconds / 60)
|
||||
if (minutes < 60) return `${minutes} min`
|
||||
return `${Math.floor(minutes / 60)}h ${minutes % 60}m`
|
||||
}
|
||||
|
||||
export default function OperationCard({ operation, onCancel, onDismiss, onRetry }) {
|
||||
const { t } = useTranslation('admin')
|
||||
const nodes = Array.isArray(operation.nodes) ? operation.nodes : []
|
||||
// Holds only what the user chose. The default has to stay a live
|
||||
// expression: an operation appears in /api/operations as soon as it is
|
||||
// admitted, but its nodes are filled in later, when the fan-out starts
|
||||
// reporting. State seeded at mount would latch on the empty list.
|
||||
const [nodesOpenOverride, setNodesOpenOverride] = useState(null)
|
||||
const listId = useId()
|
||||
|
||||
const failed = Boolean(operation.error)
|
||||
const name = operation.name || operation.id
|
||||
const kind = operation.isBackend ? t('activity.kind.backend') : t('activity.kind.model')
|
||||
|
||||
// Same chain as the one-line strip, so the two never describe one job
|
||||
// differently. Without it a removal and an install render identically: a
|
||||
// deletion has no phase, no bytes and no nodes to tell them apart.
|
||||
let icon
|
||||
let verb
|
||||
if (failed) {
|
||||
icon = <i className="fas fa-circle-exclamation operation-card__icon operation-card__icon--error" aria-hidden="true" />
|
||||
// The failure phrase has to name the work that actually failed. A removal
|
||||
// or a staging job reported as a failed install describes the opposite of
|
||||
// what happened, and would make the missing Retry button look like a bug.
|
||||
if (operation.isDeletion) verb = t('activity.verb.failedRemoval', { kind })
|
||||
else if (operation.taskType === 'staging') verb = t('activity.verb.failedStaging')
|
||||
else verb = t('activity.verb.failed', { kind })
|
||||
} else if (operation.isQueued) {
|
||||
icon = <i className="fas fa-clock operation-card__icon" aria-hidden="true" />
|
||||
verb = t('activity.verb.queued')
|
||||
} else if (operation.taskType === 'staging') {
|
||||
icon = <i className="fas fa-cloud-arrow-up operation-card__icon operation-card__icon--staging" aria-hidden="true" />
|
||||
verb = t('activity.verb.staging')
|
||||
} else if (operation.isDeletion) {
|
||||
icon = <i className="fas fa-trash operation-card__icon operation-card__icon--removing" aria-hidden="true" />
|
||||
verb = t('activity.verb.removing', { kind })
|
||||
} else {
|
||||
icon = <span className="operation-card__spinner" aria-hidden="true" />
|
||||
verb = t('activity.verb.installing', { kind })
|
||||
}
|
||||
|
||||
const byteLabel = Number.isFinite(operation.currentBytes) && Number.isFinite(operation.totalBytes) && operation.totalBytes > 0
|
||||
? `${formatBytes(operation.currentBytes)} / ${formatBytes(operation.totalBytes)}`
|
||||
: ''
|
||||
const phaseKey = phaseKeys[operation.phase]
|
||||
const etaLabel = formatEta(operation.etaSeconds)
|
||||
// Same call the strip makes, for the same reason: a failed operation
|
||||
// stopped where it broke and a queued one has not moved, so neither has a
|
||||
// bar worth drawing.
|
||||
const showProgress = !failed && !operation.isQueued && operation.progress > 0
|
||||
const canCancel = operation.cancellable && !failed
|
||||
// Retrying means reconstructing an install call out of the operation, which
|
||||
// is page knowledge. The card offers the button only when the page handed it
|
||||
// a handler, so the control can never be present with nothing behind it.
|
||||
const canRetry = failed && typeof onRetry === 'function'
|
||||
|
||||
// One node needs no disclosure: the single row is the whole story, and a
|
||||
// count-less string would render "Show 1 nodes".
|
||||
const showNodesToggle = nodes.length > 1
|
||||
const nodesOpen = nodesOpenOverride ?? (nodes.length > 0 && nodes.length <= 4)
|
||||
const showNodesList = nodes.length > 0 && (nodesOpen || !showNodesToggle)
|
||||
|
||||
return (
|
||||
<div className={`operation-card${failed ? ' operation-card--error' : ''}`}>
|
||||
<div className="operation-card__main">
|
||||
{icon}
|
||||
|
||||
<div className="operation-card__body">
|
||||
<div className="operation-card__title">
|
||||
<span className="operation-card__name">{name}</span>
|
||||
<span className={`operation-card__tag operation-card__tag--${operation.isBackend ? 'backend' : 'model'}`}>
|
||||
{kind}
|
||||
</span>
|
||||
{nodes.length > 1 && (
|
||||
<span className="operation-card__tag operation-card__tag--cluster">
|
||||
{t('activity.nodeCount', { count: nodes.length })}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="operation-card__sub">
|
||||
<span className="operation-card__verb">{verb}</span>
|
||||
{operation.nodeName && <span>{t('activity.toNode', { node: operation.nodeName })}</span>}
|
||||
{failed && <span className="operation-card__error" title={operation.error}>{operation.error}</span>}
|
||||
{!failed && phaseKey && <span>{t(phaseKey)}</span>}
|
||||
{/* Phases and byte counters exist only on the managed-artifact
|
||||
path, so a legacy files: gallery model and every backend
|
||||
install would otherwise say nothing beyond the verb. The
|
||||
server's own message is the only detail those jobs have. It is
|
||||
skipped while queued because there it is just "queued", which
|
||||
the line below already says in the user's language. */}
|
||||
{!failed && !phaseKey && !operation.isQueued && operation.message && (
|
||||
<span className="operation-card__message" title={operation.message}>{operation.message}</span>
|
||||
)}
|
||||
{!failed && operation.isQueued && <span>{t('activity.waitingForInstaller')}</span>}
|
||||
{!failed && byteLabel && <span className="operation-card__bytes">{byteLabel}</span>}
|
||||
{!failed && etaLabel && <span className="operation-card__bytes">{t('activity.timeLeft', { value: etaLabel })}</span>}
|
||||
</div>
|
||||
|
||||
{showProgress && (
|
||||
<div
|
||||
className="operation-card__track"
|
||||
role="progressbar"
|
||||
aria-valuenow={Math.round(operation.progress)}
|
||||
aria-valuemin={0}
|
||||
aria-valuemax={100}
|
||||
aria-label={t('activity.progressLabel', { name })}
|
||||
>
|
||||
<span className="operation-card__fill" style={{ width: `${operation.progress}%` }} />
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="operation-card__actions">
|
||||
{showProgress && <span className="operation-card__pct" aria-hidden="true">{Math.round(operation.progress)}%</span>}
|
||||
{canCancel && (
|
||||
// A page of cards would otherwise hand a screen reader a list of
|
||||
// identical "Cancel" buttons with nothing to tell them apart.
|
||||
<button
|
||||
type="button"
|
||||
className="btn btn-sm btn-danger operation-card__cancel"
|
||||
onClick={() => onCancel?.(operation.jobID)}
|
||||
aria-label={t('activity.cancelLabel', { name })}
|
||||
>
|
||||
{t('activity.cancel')}
|
||||
</button>
|
||||
)}
|
||||
{canRetry && (
|
||||
<button
|
||||
type="button"
|
||||
className="btn btn-sm btn-secondary operation-card__retry"
|
||||
onClick={() => onRetry(operation)}
|
||||
aria-label={t('activity.retryLabel', { name })}
|
||||
>
|
||||
{t('activity.retry')}
|
||||
</button>
|
||||
)}
|
||||
{failed && (
|
||||
<button
|
||||
type="button"
|
||||
className="operation-card__hide"
|
||||
onClick={() => onDismiss?.(operation.jobID)}
|
||||
title={t('activity.moveToHistory')}
|
||||
aria-label={t('activity.moveToHistory')}
|
||||
>
|
||||
<i className="fas fa-xmark" aria-hidden="true" />
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* The disclosure sits above what it discloses: a control that follows
|
||||
its own region reads backwards to anyone moving through the page. */}
|
||||
{showNodesToggle && (
|
||||
<button
|
||||
type="button"
|
||||
className="operation-card__nodes-toggle"
|
||||
aria-expanded={nodesOpen}
|
||||
aria-controls={listId}
|
||||
onClick={() => setNodesOpenOverride(!nodesOpen)}
|
||||
>
|
||||
<i className={`fas fa-chevron-${nodesOpen ? 'up' : 'down'}`} aria-hidden="true" />
|
||||
{nodesOpen ? t('activity.hideNodes') : t('activity.showNodes', { count: nodes.length })}
|
||||
</button>
|
||||
)}
|
||||
|
||||
{/* Hidden rather than unmounted while collapsed, so the toggle's
|
||||
aria-controls always points at something that exists. */}
|
||||
{nodes.length > 0 && (
|
||||
<ul className="operation-nodes-list" id={listId} hidden={!showNodesList}>
|
||||
{nodes.map((node) => (
|
||||
<li key={node.node_id} className={`operation-node operation-node-${node.status}`}>
|
||||
<span className={`operation-node-status operation-node-status-${node.status}`}>
|
||||
{/* An unmapped status is shown as it arrived: inventing
|
||||
"queued" for it would report a state the node is not in. */}
|
||||
{nodeStatusKeys[node.status] ? t(nodeStatusKeys[node.status]) : node.status}
|
||||
</span>
|
||||
<span className="operation-node-name">{node.node_name || node.node_id}</span>
|
||||
{node.file_name && (
|
||||
<span className="operation-node-file" title={node.file_name}>{node.file_name}</span>
|
||||
)}
|
||||
{(node.current || node.total) && (
|
||||
<span className="operation-node-bytes">{node.current || '?'} / {node.total || '?'}</span>
|
||||
)}
|
||||
{node.percentage > 0 && (
|
||||
<span className="operation-node-pct">{Math.round(node.percentage)}%</span>
|
||||
)}
|
||||
{node.error && (
|
||||
<span className="operation-node-error" title={node.error}>{node.error}</span>
|
||||
)}
|
||||
{node.percentage > 0 && node.percentage < 100 && (
|
||||
<div className="operation-node-bar-container">
|
||||
<div className="operation-node-bar" style={{ width: `${node.percentage}%` }} />
|
||||
</div>
|
||||
)}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -1,176 +1,200 @@
|
||||
import { useState } from 'react'
|
||||
import { useEffect, useRef, useState } from 'react'
|
||||
// eslint-plugin-react is not configured here, so eslint cannot see that a
|
||||
// JSX-only import is used.
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
import { Link } from 'react-router-dom'
|
||||
import { useTranslation } from 'react-i18next'
|
||||
import { useOperations } from '../hooks/useOperations'
|
||||
import { formatBytes } from '../utils/format'
|
||||
|
||||
const artifactPhaseLabels = {
|
||||
resolving: 'Resolving model files',
|
||||
downloading: 'Downloading model files',
|
||||
verifying: 'Verifying model files',
|
||||
committing: 'Finalizing model installation',
|
||||
persisting: 'Saving model configuration',
|
||||
const artifactPhaseKeys = {
|
||||
resolving: 'activity.phase.resolving',
|
||||
downloading: 'activity.phase.downloading',
|
||||
verifying: 'activity.phase.verifying',
|
||||
committing: 'activity.phase.committing',
|
||||
persisting: 'activity.phase.persisting',
|
||||
}
|
||||
|
||||
const nodeStatusLabels = {
|
||||
success: 'Done',
|
||||
error: 'Failed',
|
||||
queued: 'Queued',
|
||||
running_on_worker: 'Worker busy',
|
||||
downloading: 'Downloading',
|
||||
}
|
||||
// How long a finished operation stays on screen. The API drops an operation
|
||||
// the instant it succeeds, so without this a fast install is a flicker.
|
||||
const SUCCESS_HOLD_MS = 4000
|
||||
|
||||
const runningOnWorkerTooltip = 'NATS round-trip timed out, but the worker is still installing in the background. The reconciler will confirm completion.'
|
||||
// An unacknowledged failure outranks any progress; otherwise the API's own
|
||||
// sort (progress ascending) already puts the operation that gates the batch
|
||||
// first, and it is the most stable choice across polls.
|
||||
//
|
||||
// The strip is the only surface that picks one operation out of many. The
|
||||
// Activity page shows all of them, partitioned into failed and running, so it
|
||||
// has no primary to agree with.
|
||||
function primaryOperation(operations) {
|
||||
if (!operations || operations.length === 0) return null
|
||||
return operations.find((op) => op.error) || operations[0]
|
||||
}
|
||||
|
||||
export default function OperationsBar() {
|
||||
const { operations, cancelOperation, dismissFailedOp } = useOperations()
|
||||
const [expanded, setExpanded] = useState({})
|
||||
const { t } = useTranslation('admin')
|
||||
const { operations, dismissFailedOp, wasCancelled } = useOperations()
|
||||
// Which operation the user hid. Keyed by job so a different operation
|
||||
// becoming primary brings the strip back: hiding must never be able to
|
||||
// silence a later failure.
|
||||
const [hiddenJobID, setHiddenJobID] = useState(null)
|
||||
const [finished, setFinished] = useState(null)
|
||||
const previousRef = useRef(null)
|
||||
|
||||
if (operations.length === 0) return null
|
||||
const primary = primaryOperation(operations)
|
||||
|
||||
const toggle = (key) => setExpanded((m) => ({ ...m, [key]: !m[key] }))
|
||||
useEffect(() => {
|
||||
const previous = previousRef.current
|
||||
previousRef.current = primary
|
||||
|
||||
// The previous primary is gone from the live list and it was not failing:
|
||||
// it completed. Hold it on screen briefly, then drop it.
|
||||
//
|
||||
// Unless the user cancelled it. Cancelling deletes the operation server
|
||||
// side, so a cancel and a completion are the same event here: the
|
||||
// operation simply stops being listed. Nothing in the payload separates
|
||||
// them, which is why this asks the page whether it issued the cancel.
|
||||
// Without that, cancelling the last running install put a green
|
||||
// "Installed model X" on screen for four seconds.
|
||||
if (previous && !primary && !previous.error && !wasCancelled(previous.jobID)) {
|
||||
setFinished(previous)
|
||||
const timer = setTimeout(() => setFinished(null), SUCCESS_HOLD_MS)
|
||||
return () => clearTimeout(timer)
|
||||
}
|
||||
if (primary) setFinished(null)
|
||||
return undefined
|
||||
}, [primary, wasCancelled])
|
||||
|
||||
const shown = primary || finished
|
||||
if (!shown) return null
|
||||
// A job keeps its jobID when it turns into a failure, so hiding the running
|
||||
// operation would otherwise swallow that same job's error. Hiding is a way
|
||||
// to get on with your work, never a way to opt out of bad news.
|
||||
if (hiddenJobID === shown.jobID && !shown.error) return null
|
||||
|
||||
const extra = Math.max(0, operations.length - 1)
|
||||
const isFinished = !primary
|
||||
const phaseKey = artifactPhaseKeys[shown.phase]
|
||||
const byteLabel = Number.isFinite(shown.currentBytes) && Number.isFinite(shown.totalBytes) && shown.totalBytes > 0
|
||||
? `${formatBytes(shown.currentBytes)} / ${formatBytes(shown.totalBytes)}`
|
||||
: ''
|
||||
const kind = shown.isBackend ? t('activity.kind.backend') : t('activity.kind.model')
|
||||
|
||||
let modifier = ''
|
||||
let icon = null
|
||||
let verb = ''
|
||||
if (isFinished) {
|
||||
modifier = 'operations-strip--done'
|
||||
icon = <i className="fas fa-check operations-strip__icon" aria-hidden="true" />
|
||||
// The completion phrase has to match the work that just ended: a removal
|
||||
// that reports "Installed" reads as the opposite of what happened.
|
||||
if (shown.isDeletion) verb = t('activity.verb.removed', { kind })
|
||||
else if (shown.taskType === 'staging') verb = t('activity.verb.staged')
|
||||
else verb = t('activity.verb.installed', { kind })
|
||||
} else if (shown.error) {
|
||||
modifier = 'operations-strip--error'
|
||||
icon = <i className="fas fa-circle-exclamation operations-strip__icon" aria-hidden="true" />
|
||||
// Same split as the card, so the two surfaces never describe one failed
|
||||
// job differently: a removal reported as a failed install is the opposite
|
||||
// of what happened.
|
||||
if (shown.isDeletion) verb = t('activity.verb.failedRemoval', { kind })
|
||||
else if (shown.taskType === 'staging') verb = t('activity.verb.failedStaging')
|
||||
else verb = t('activity.verb.failed', { kind })
|
||||
} else if (shown.isQueued) {
|
||||
modifier = 'operations-strip--queued'
|
||||
icon = <i className="fas fa-clock operations-strip__icon" aria-hidden="true" />
|
||||
verb = t('activity.verb.queued')
|
||||
} else if (shown.taskType === 'staging') {
|
||||
modifier = 'operations-strip--staging'
|
||||
icon = <i className="fas fa-cloud-arrow-up operations-strip__icon" aria-hidden="true" />
|
||||
verb = t('activity.verb.staging')
|
||||
} else if (shown.isDeletion) {
|
||||
modifier = 'operations-strip--removing'
|
||||
icon = <i className="fas fa-trash operations-strip__icon" aria-hidden="true" />
|
||||
verb = t('activity.verb.removing', { kind })
|
||||
} else {
|
||||
icon = <span className="operations-strip__spinner" aria-hidden="true" />
|
||||
verb = t('activity.verb.installing', { kind })
|
||||
}
|
||||
|
||||
// A fanned-out backend install rolls its nodes up into one phrase. Without
|
||||
// this the strip would report one node's phase as if it were the whole job,
|
||||
// and the per-node list is what the Activity page is for.
|
||||
const nodes = Array.isArray(shown.nodes) ? shown.nodes : []
|
||||
const nodesDone = nodes.filter((node) => node.status === 'success').length
|
||||
const nodeRollup = nodes.length > 1
|
||||
? t('activity.nodesDone', { done: nodesDone, total: nodes.length })
|
||||
: ''
|
||||
|
||||
const detail = shown.error
|
||||
|| nodeRollup
|
||||
|| (shown.taskType === 'staging' && shown.nodeName ? t('activity.toNode', { node: shown.nodeName }) : '')
|
||||
|| (phaseKey ? t(phaseKey) : '')
|
||||
|| (shown.isQueued ? t('activity.waitingForInstaller') : '')
|
||||
|
||||
// A finished, failed or not-yet-started operation has no progress worth a
|
||||
// bar: the first is over, the second stopped where it broke and the third
|
||||
// has not moved.
|
||||
const showProgress = !isFinished && !shown.error && !shown.isQueued && shown.progress > 0
|
||||
|
||||
const onHide = () => {
|
||||
// A failure is dismissed server side, which moves it into the record.
|
||||
// Anything else is hidden locally: the work carries on and the sidebar
|
||||
// count still shows it.
|
||||
if (shown.error) {
|
||||
dismissFailedOp(shown.jobID)
|
||||
return
|
||||
}
|
||||
setHiddenJobID(shown.jobID)
|
||||
setFinished(null)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="operations-bar">
|
||||
{operations.map(op => {
|
||||
const key = op.jobID || op.id
|
||||
const nodes = Array.isArray(op.nodes) ? op.nodes : []
|
||||
const canExpand = nodes.length > 1
|
||||
const isOpen = !!expanded[key]
|
||||
const phaseLabel = artifactPhaseLabels[op.phase]
|
||||
const byteLabel = Number.isFinite(op.currentBytes) && Number.isFinite(op.totalBytes) && op.totalBytes > 0
|
||||
? `${formatBytes(op.currentBytes)} / ${formatBytes(op.totalBytes)}`
|
||||
: ''
|
||||
return (
|
||||
<div key={key} className="operation-item">
|
||||
<div className="operation-info">
|
||||
{op.error ? (
|
||||
<i className="fas fa-circle-exclamation" style={{ color: 'var(--color-error)', marginRight: 'var(--spacing-xs)' }} />
|
||||
) : op.isCancelled ? (
|
||||
<i className="fas fa-ban" style={{ color: 'var(--color-warning)', marginRight: 'var(--spacing-xs)' }} />
|
||||
) : op.isDeletion ? (
|
||||
<i className="fas fa-trash" style={{ color: 'var(--color-error)', marginRight: 'var(--spacing-xs)' }} />
|
||||
) : (
|
||||
<div className="operation-spinner" />
|
||||
)}
|
||||
<span className="operation-text">
|
||||
{op.error ? (
|
||||
<>
|
||||
Failed to install {op.isBackend ? 'backend' : 'model'}: {op.name || op.id}
|
||||
<span style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
|
||||
({op.error})
|
||||
</span>
|
||||
</>
|
||||
) : op.taskType === 'staging' ? (
|
||||
<>
|
||||
<i className="fas fa-cloud-arrow-up" style={{ marginRight: 'var(--spacing-xs)' }} />
|
||||
Staging model: {op.name}{op.nodeName ? ` → ${op.nodeName}` : ''}
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
{op.isDeletion ? 'Removing' : 'Installing'}{' '}
|
||||
{op.isBackend ? 'backend' : 'model'}: {op.name || op.id}
|
||||
</>
|
||||
)}
|
||||
</span>
|
||||
{!op.error && op.isQueued && (
|
||||
<span style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
|
||||
(Queued)
|
||||
</span>
|
||||
)}
|
||||
{!op.error && op.isCancelled && (
|
||||
<span style={{ fontSize: '0.75rem', color: 'var(--color-warning)', marginLeft: 'var(--spacing-xs)' }}>
|
||||
Cancelling...
|
||||
</span>
|
||||
)}
|
||||
{!op.error && phaseLabel && !op.isCancelled && (
|
||||
<span className="operation-phase" style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
|
||||
{phaseLabel}
|
||||
</span>
|
||||
)}
|
||||
{!op.error && byteLabel && !op.isCancelled && (
|
||||
<span className="operation-bytes" style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
|
||||
{byteLabel}
|
||||
</span>
|
||||
)}
|
||||
{!op.error && op.message && !phaseLabel && !op.isQueued && !op.isCancelled && (
|
||||
<span style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
|
||||
{op.message}
|
||||
</span>
|
||||
)}
|
||||
{!op.error && op.progress !== undefined && op.progress > 0 && (
|
||||
<span className="operation-progress">{Math.round(op.progress)}%</span>
|
||||
)}
|
||||
</div>
|
||||
{!op.error && op.progress !== undefined && op.progress > 0 && (
|
||||
<div className="operation-bar-container">
|
||||
<div className="operation-bar" style={{ width: `${op.progress}%` }} />
|
||||
</div>
|
||||
)}
|
||||
{op.error ? (
|
||||
<button
|
||||
className="operation-cancel"
|
||||
onClick={() => dismissFailedOp(op.id)}
|
||||
title="Dismiss"
|
||||
>
|
||||
<i className="fas fa-xmark" />
|
||||
</button>
|
||||
) : op.cancellable && !op.isCancelled ? (
|
||||
<button
|
||||
className="operation-cancel"
|
||||
onClick={() => cancelOperation(op.jobID)}
|
||||
title="Cancel"
|
||||
>
|
||||
<i className="fas fa-xmark" />
|
||||
</button>
|
||||
) : null}
|
||||
{canExpand && (
|
||||
<button
|
||||
type="button"
|
||||
className="operation-expand"
|
||||
onClick={() => toggle(key)}
|
||||
aria-expanded={isOpen}
|
||||
title={isOpen ? 'Hide per-node detail' : `Show ${nodes.length} nodes`}
|
||||
>
|
||||
<i className={`fas fa-chevron-${isOpen ? 'up' : 'down'}`} />
|
||||
<span className="operation-expand-label">{nodes.length} nodes</span>
|
||||
</button>
|
||||
)}
|
||||
{canExpand && isOpen && (
|
||||
<ul className="operation-nodes-list">
|
||||
{nodes.map((n) => (
|
||||
<li key={n.node_id} className={`operation-node operation-node-${n.status}`}>
|
||||
<span
|
||||
className={`operation-node-status operation-node-status-${n.status}`}
|
||||
title={n.status === 'running_on_worker' ? runningOnWorkerTooltip : undefined}
|
||||
>
|
||||
{nodeStatusLabels[n.status] || n.status}
|
||||
</span>
|
||||
<span className="operation-node-name">{n.node_name || n.node_id}</span>
|
||||
{n.file_name && <span className="operation-node-file">{n.file_name}</span>}
|
||||
{(n.current || n.total) && (
|
||||
<span className="operation-node-bytes">
|
||||
{n.current || '?'} / {n.total || '?'}
|
||||
</span>
|
||||
)}
|
||||
{n.percentage > 0 && (
|
||||
<span className="operation-node-pct">{Math.round(n.percentage)}%</span>
|
||||
)}
|
||||
{n.error && (
|
||||
<span className="operation-node-error" title={n.error}>
|
||||
{n.error.length > 80 ? n.error.slice(0, 80) + '...' : n.error}
|
||||
</span>
|
||||
)}
|
||||
{n.percentage > 0 && n.percentage < 100 && (
|
||||
<div className="operation-node-bar-container">
|
||||
<div className="operation-node-bar" style={{ width: `${n.percentage}%` }} />
|
||||
</div>
|
||||
)}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
// role="status" already implies a polite live region. It deliberately does
|
||||
// not cover the percentage, which changes every poll and would have a
|
||||
// screen reader re-reading the whole strip once a second.
|
||||
<div className={`operations-strip ${modifier}`.trim()} role="status">
|
||||
{icon}
|
||||
<span className="operations-strip__verb">{verb}</span>
|
||||
<span className="operations-strip__name">{shown.name || shown.id}</span>
|
||||
{detail && <span className="operations-strip__sep" aria-hidden="true">·</span>}
|
||||
{detail && <span className="operations-strip__detail">{detail}</span>}
|
||||
{byteLabel && !shown.error && <span className="operations-strip__bytes">{byteLabel}</span>}
|
||||
<span className="operations-strip__spacer" />
|
||||
{showProgress && (
|
||||
<>
|
||||
<span className="operations-strip__pct" aria-hidden="true">{Math.round(shown.progress)}%</span>
|
||||
{/* A progressbar carries the value without a live region's chatter:
|
||||
its updates are readable on demand rather than announced. */}
|
||||
<span
|
||||
className="operations-strip__track"
|
||||
role="progressbar"
|
||||
aria-valuenow={Math.round(shown.progress)}
|
||||
aria-valuemin={0}
|
||||
aria-valuemax={100}
|
||||
aria-label={t('activity.progressLabel', { name: shown.name || shown.id })}
|
||||
>
|
||||
<span className="operations-strip__fill" style={{ width: `${shown.progress}%` }} />
|
||||
</span>
|
||||
</>
|
||||
)}
|
||||
{extra > 0 && (
|
||||
<Link
|
||||
className={`operations-strip__more${shown.error ? ' operations-strip__more--neutral' : ''}`}
|
||||
to="/app/activity"
|
||||
>
|
||||
{t('activity.moreCount', { count: extra })}
|
||||
</Link>
|
||||
)}
|
||||
<button
|
||||
type="button"
|
||||
className="operations-strip__hide"
|
||||
onClick={onHide}
|
||||
title={shown.error ? t('activity.moveToHistory') : t('activity.hide')}
|
||||
aria-label={shown.error ? t('activity.moveToHistory') : t('activity.hide')}
|
||||
>
|
||||
<i className="fas fa-xmark" aria-hidden="true" />
|
||||
</button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import { useBranding } from '../contexts/BrandingContext'
|
||||
import { apiUrl } from '../utils/basePath'
|
||||
import { preloadRoute } from '../router'
|
||||
import { consoles, firstVisiblePath, consolePaths } from './console/consoleConfig'
|
||||
import { useOperations } from '../hooks/useOperations'
|
||||
|
||||
const COLLAPSED_KEY = 'localai_sidebar_collapsed'
|
||||
const SECTIONS_KEY = 'localai_sidebar_sections'
|
||||
@@ -83,6 +84,7 @@ export default function Sidebar({ isOpen, onClose }) {
|
||||
})
|
||||
const [openSections, setOpenSections] = useState(loadSectionState)
|
||||
const { isAdmin, authEnabled, user, logout, hasFeature } = useAuth()
|
||||
const { operations } = useOperations()
|
||||
const branding = useBranding()
|
||||
const navigate = useNavigate()
|
||||
const location = useLocation()
|
||||
@@ -160,6 +162,12 @@ export default function Sidebar({ isOpen, onClose }) {
|
||||
// Shared shape for the console gating helpers (consoleConfig.js).
|
||||
const auth = { isAdmin, authEnabled, hasFeature, features }
|
||||
|
||||
// One badge, on the always-visible sidebar entry. The console rail only
|
||||
// exists while the user is on an Operate route and can be collapsed, so
|
||||
// badging the rail item instead would let the count disappear entirely.
|
||||
const failedOps = operations.filter((op) => op.error).length
|
||||
const activeOps = operations.length
|
||||
|
||||
// Inline sections (Create) carry no gating; a plain filterItem pass suffices.
|
||||
const getVisibleSectionItems = (section) => section.items.filter(filterItem)
|
||||
|
||||
@@ -249,6 +257,11 @@ export default function Sidebar({ isOpen, onClose }) {
|
||||
>
|
||||
<i className={`${config.icon} nav-icon`} />
|
||||
<span className="nav-label">{label}</span>
|
||||
{config.groups.some(g => g.items.some(i => i.badge === 'operations')) && activeOps > 0 && (
|
||||
<span className={`nav-badge${failedOps > 0 ? ' nav-badge--error' : ''}`}>
|
||||
{failedOps > 0 ? failedOps : activeOps}
|
||||
</span>
|
||||
)}
|
||||
</NavLink>
|
||||
</div>
|
||||
)
|
||||
|
||||
@@ -56,6 +56,12 @@ export const operateConsole = {
|
||||
{ path: '/app/voice-library', icon: 'fas fa-wave-square', labelKey: 'items.voiceLibrary', adminOnly: true },
|
||||
],
|
||||
},
|
||||
{
|
||||
titleKey: 'operate.activity',
|
||||
items: [
|
||||
{ path: '/app/activity', icon: 'fas fa-download', labelKey: 'items.activity', adminOnly: true, badge: 'operations' },
|
||||
],
|
||||
},
|
||||
{
|
||||
titleKey: 'operate.cluster',
|
||||
items: [
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { createContext, useContext, useState, useEffect, useCallback, useRef } from 'react'
|
||||
import { createContext, useContext, useState, useEffect, useCallback, useMemo, useRef } from 'react'
|
||||
import { operationsApi } from '../utils/api'
|
||||
import { useAuth } from '../context/AuthContext'
|
||||
|
||||
@@ -12,6 +12,11 @@ function serializeOps(ops) {
|
||||
|
||||
const OperationsContext = createContext(null)
|
||||
|
||||
// How long a cancelled job is remembered. It only has to outlive the poll that
|
||||
// notices the operation left the list; a session that cancels all day must not
|
||||
// accumulate job IDs.
|
||||
const CANCELLED_MEMORY_MS = 60_000
|
||||
|
||||
// Single shared poller for /api/operations. Before this provider existed,
|
||||
// each useOperations() call ran its own setInterval; with OperationsBar
|
||||
// always mounted plus the per-page consumers (Models, Backends, Chat), the
|
||||
@@ -21,9 +26,43 @@ export function OperationsProvider({ children, pollInterval = 1000 }) {
|
||||
const [operations, setOperations] = useState([])
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [error, setError] = useState(null)
|
||||
const [history, setHistory] = useState([])
|
||||
const [historyLoading, setHistoryLoading] = useState(false)
|
||||
const { isAdmin } = useAuth()
|
||||
const intervalRef = useRef(null)
|
||||
const lastSerializedRef = useRef('[]')
|
||||
const liveIDsRef = useRef(new Set())
|
||||
// Jobs cancelled from this tab, by job ID. The cancel endpoint removes the
|
||||
// operation immediately, so on the next poll the only thing the UI can
|
||||
// observe is that the operation is gone, which is exactly what finishing
|
||||
// looks like. Nothing in the payload distinguishes them (a cancelled
|
||||
// operation is never listed), so the side that issued the cancel is the only
|
||||
// one that can remember it.
|
||||
const cancelledRef = useRef(new Map())
|
||||
|
||||
// History is fetched on demand, never on the poll interval: it only changes
|
||||
// when an operation finishes, and the Activity page is the only consumer.
|
||||
const fetchHistory = useCallback(async () => {
|
||||
if (!isAdmin) return
|
||||
setHistoryLoading(true)
|
||||
try {
|
||||
const data = await operationsApi.history()
|
||||
setHistory(data?.operations || [])
|
||||
} catch (err) {
|
||||
setError((prev) => (prev === err.message ? prev : err.message))
|
||||
} finally {
|
||||
setHistoryLoading(false)
|
||||
}
|
||||
}, [isAdmin])
|
||||
|
||||
const clearHistory = useCallback(async () => {
|
||||
try {
|
||||
await operationsApi.clearHistory()
|
||||
setHistory([])
|
||||
} catch (err) {
|
||||
setError(err.message)
|
||||
}
|
||||
}, [])
|
||||
|
||||
const fetchOperations = useCallback(async () => {
|
||||
if (!isAdmin) {
|
||||
@@ -40,13 +79,38 @@ export function OperationsProvider({ children, pollInterval = 1000 }) {
|
||||
setOperations(ops)
|
||||
}
|
||||
|
||||
// An operation leaving the live list is the one moment the record can
|
||||
// have changed. Refetching here keeps the page correct without polling
|
||||
// a second endpoint every second.
|
||||
//
|
||||
// Tracked by identity rather than by count: during a batch install one
|
||||
// operation finishing in the same second another starts leaves the
|
||||
// length unchanged, and a count comparison would miss the completion.
|
||||
const liveIDs = new Set(ops.map((op) => op.jobID || op.id))
|
||||
let departed = false
|
||||
for (const id of liveIDsRef.current) {
|
||||
if (!liveIDs.has(id)) {
|
||||
departed = true
|
||||
break
|
||||
}
|
||||
}
|
||||
liveIDsRef.current = liveIDs
|
||||
if (departed) {
|
||||
fetchHistory()
|
||||
}
|
||||
|
||||
const cutoff = Date.now() - CANCELLED_MEMORY_MS
|
||||
for (const [id, at] of cancelledRef.current) {
|
||||
if (at < cutoff) cancelledRef.current.delete(id)
|
||||
}
|
||||
|
||||
setError((prev) => (prev === null ? prev : null))
|
||||
} catch (err) {
|
||||
setError((prev) => (prev === err.message ? prev : err.message))
|
||||
} finally {
|
||||
setLoading((prev) => (prev ? false : prev))
|
||||
}
|
||||
}, [isAdmin])
|
||||
}, [isAdmin, fetchHistory])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isAdmin) return
|
||||
@@ -63,29 +127,106 @@ export function OperationsProvider({ children, pollInterval = 1000 }) {
|
||||
const cancelOperation = useCallback(async (jobID) => {
|
||||
try {
|
||||
await operationsApi.cancel(jobID)
|
||||
// Recorded before the refetch: that refetch is the one that sees the
|
||||
// operation gone, and a consumer reacting to the disappearance has to
|
||||
// find the cancel already remembered or it will call it a success.
|
||||
cancelledRef.current.set(jobID, Date.now())
|
||||
await fetchOperations()
|
||||
} catch (err) {
|
||||
setError(err.message)
|
||||
}
|
||||
}, [fetchOperations])
|
||||
|
||||
const dismissFailedOp = useCallback(async (opId) => {
|
||||
// Whether this tab cancelled the job. Read by the strip to tell "the last
|
||||
// operation finished" from "the user called it off": both look identical in
|
||||
// /api/operations, which lists neither.
|
||||
const wasCancelled = useCallback((jobID) => cancelledRef.current.has(jobID), [])
|
||||
|
||||
// Takes the jobID, never the display id. /api/operations strips the
|
||||
// "node:<nodeID>:" prefix before emitting, so a local install and a
|
||||
// node-scoped install of the same backend arrive as two distinct jobs
|
||||
// sharing one id: looking the job up by id could dismiss the wrong one,
|
||||
// leaving the failure the user acted on live and silently retiring another.
|
||||
const dismissFailedOp = useCallback(async (jobID) => {
|
||||
if (!jobID) return
|
||||
try {
|
||||
const op = operations.find((o) => o.id === opId)
|
||||
if (op?.jobID) {
|
||||
await operationsApi.dismiss(op.jobID)
|
||||
await fetchOperations()
|
||||
}
|
||||
await operationsApi.dismiss(jobID)
|
||||
await fetchOperations()
|
||||
} catch {
|
||||
// Ignore dismiss errors
|
||||
}
|
||||
}, [operations, fetchOperations])
|
||||
}, [fetchOperations])
|
||||
|
||||
// Time remaining is derived, not reported. We keep the previous
|
||||
// (bytes, timestamp) sample per job and estimate from the delta.
|
||||
//
|
||||
// All or nothing on purpose: an estimate needs two samples, and one card
|
||||
// showing "11 min left" while its neighbours show nothing reads as a
|
||||
// rendering bug rather than as missing data.
|
||||
const samplesRef = useRef(new Map())
|
||||
const operationsWithEta = useMemo(() => {
|
||||
const now = Date.now()
|
||||
const samples = samplesRef.current
|
||||
const seen = new Set()
|
||||
|
||||
const withEta = operations.map((op) => {
|
||||
const key = op.jobID || op.id
|
||||
seen.add(key)
|
||||
const current = op.currentBytes
|
||||
const total = op.totalBytes
|
||||
if (!Number.isFinite(current) || !Number.isFinite(total) || total <= 0) return op
|
||||
|
||||
const previous = samples.get(key)
|
||||
samples.set(key, { bytes: current, at: now })
|
||||
if (!previous || current <= previous.bytes) return op
|
||||
|
||||
const bytesPerMs = (current - previous.bytes) / Math.max(1, now - previous.at)
|
||||
if (bytesPerMs <= 0) return op
|
||||
return { ...op, etaSeconds: Math.round((total - current) / bytesPerMs / 1000) }
|
||||
})
|
||||
|
||||
// Drop samples for jobs that finished, so the map cannot grow forever.
|
||||
for (const key of samples.keys()) {
|
||||
if (!seen.has(key)) samples.delete(key)
|
||||
}
|
||||
|
||||
// All or nothing: if any operation still transferring has no estimate yet,
|
||||
// nobody shows one this tick.
|
||||
//
|
||||
// Only operations actually downloading get a vote. Every other phase
|
||||
// reports bytes but stops advancing them: verifying hashes a finished file
|
||||
// while the counter sits below the multi-file total, and committing sits
|
||||
// pinned at the total. Both can last minutes, and counting them would
|
||||
// blank every other operation's estimate for that whole window.
|
||||
//
|
||||
// The byte clauses are not redundant with the phase clause: a producer can
|
||||
// report downloading with bytes already at the total. The undefined-phase
|
||||
// arm keeps today's behaviour for producers that do not report a phase,
|
||||
// which in practice do not report totalBytes either.
|
||||
const tracked = withEta.filter(
|
||||
(op) =>
|
||||
Number.isFinite(op.totalBytes) &&
|
||||
op.totalBytes > 0 &&
|
||||
Number.isFinite(op.currentBytes) &&
|
||||
op.currentBytes < op.totalBytes &&
|
||||
(op.phase === undefined || op.phase === 'downloading')
|
||||
)
|
||||
if (tracked.length > 0 && tracked.some((op) => op.etaSeconds === undefined)) {
|
||||
return withEta.map(({ etaSeconds: _etaSeconds, ...op }) => op)
|
||||
}
|
||||
return withEta
|
||||
}, [operations])
|
||||
|
||||
const value = {
|
||||
operations,
|
||||
operations: operationsWithEta,
|
||||
loading,
|
||||
error,
|
||||
history,
|
||||
historyLoading,
|
||||
fetchHistory,
|
||||
clearHistory,
|
||||
cancelOperation,
|
||||
wasCancelled,
|
||||
dismissFailedOp,
|
||||
refetch: fetchOperations,
|
||||
}
|
||||
|
||||
269
core/http/react-ui/src/pages/Activity.jsx
Normal file
269
core/http/react-ui/src/pages/Activity.jsx
Normal file
@@ -0,0 +1,269 @@
|
||||
import { useCallback, useEffect, useMemo, useState } from 'react'
|
||||
import { useTranslation } from 'react-i18next'
|
||||
import { useOperations } from '../hooks/useOperations'
|
||||
import { modelsApi, backendsApi, nodesApi } from '../utils/api'
|
||||
// eslint-plugin-react is not configured here, so eslint cannot see that an
|
||||
// import used only inside JSX is used at all. Link, PageHeader and
|
||||
// OperationCard are each referenced from JSX only.
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
import { Link, useOutletContext } from 'react-router-dom'
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
import PageHeader from '../components/PageHeader'
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
import OperationCard from '../components/OperationCard'
|
||||
|
||||
const FILTERS = [
|
||||
{ id: 'all', labelKey: 'activity.filter.all' },
|
||||
{ id: 'models', labelKey: 'activity.filter.models' },
|
||||
{ id: 'backends', labelKey: 'activity.filter.backends' },
|
||||
{ id: 'cluster', labelKey: 'activity.filter.cluster' },
|
||||
]
|
||||
|
||||
function matchesFilter(entry, filter) {
|
||||
if (filter === 'all') return true
|
||||
if (filter === 'models') return !entry.isBackend && entry.taskType !== 'staging'
|
||||
if (filter === 'backends') return Boolean(entry.isBackend)
|
||||
// Cluster covers anything scoped to a node: staged files and node-scoped
|
||||
// backend installs.
|
||||
return entry.taskType === 'staging' || Boolean(entry.nodeID) || (Array.isArray(entry.nodes) && entry.nodes.length > 0)
|
||||
}
|
||||
|
||||
const outcomeIcon = {
|
||||
completed: 'fas fa-check',
|
||||
failed: 'fas fa-circle-exclamation',
|
||||
cancelled: 'fas fa-ban',
|
||||
}
|
||||
|
||||
function timeOfDay(iso) {
|
||||
const date = new Date(iso)
|
||||
if (Number.isNaN(date.getTime())) return ''
|
||||
return date.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })
|
||||
}
|
||||
|
||||
// Beyond this the elapsed time is not a duration, it is a broken start stamp.
|
||||
// A zero-value Go time reaching the page renders as a span of millennia, which
|
||||
// the row would state as fact; the page is the last place that can refuse to.
|
||||
const MAX_PLAUSIBLE_DURATION_SECONDS = 24 * 60 * 60
|
||||
|
||||
// Returns '' when the elapsed time cannot be trusted, which the caller renders
|
||||
// as a duration-less phrase rather than as "installed in " with nothing after
|
||||
// it. recordTerminal seeds StartedAt = FinishedAt and only overwrites it with a
|
||||
// real stamp, so a zero span is an ordinary arrival and gets a floor instead.
|
||||
function durationLabel(record) {
|
||||
const started = new Date(record.startedAt).getTime()
|
||||
const finished = new Date(record.finishedAt).getTime()
|
||||
if (Number.isNaN(started) || Number.isNaN(finished) || finished < started) return ''
|
||||
const seconds = Math.round((finished - started) / 1000)
|
||||
if (seconds > MAX_PLAUSIBLE_DURATION_SECONDS) return ''
|
||||
if (seconds < 1) return '< 1s'
|
||||
if (seconds < 60) return `${seconds}s`
|
||||
return `${Math.floor(seconds / 60)}m ${seconds % 60}s`
|
||||
}
|
||||
|
||||
// Cancellation is tested before the task type on purpose: a deletion cancelled
|
||||
// mid-flight must report the cancellation, not "removed". recordTerminal
|
||||
// produces exactly that pair, and the row's ban icon would otherwise sit beside
|
||||
// text claiming work that never happened.
|
||||
function recordSummary(record, t) {
|
||||
if (record.outcome === 'failed') return t('activity.rowFailed', { error: record.error })
|
||||
if (record.outcome === 'cancelled') return t('activity.rowCancelled')
|
||||
if (record.taskType === 'deletion') return t('activity.rowRemoved')
|
||||
const duration = durationLabel(record)
|
||||
return duration ? t('activity.rowInstalled', { duration }) : t('activity.rowInstalledPlain')
|
||||
}
|
||||
|
||||
// Retry only ever means "install this again". A failed deletion would need the
|
||||
// delete endpoint and a staging operation is driven by the router rather than
|
||||
// by a user action, so neither is retryable from here.
|
||||
function isRetryable(op) {
|
||||
return Boolean(op.error) && !op.isDeletion && op.taskType !== 'staging'
|
||||
}
|
||||
|
||||
export default function Activity() {
|
||||
const { t } = useTranslation('admin')
|
||||
const outlet = useOutletContext()
|
||||
const addToast = outlet?.addToast
|
||||
const { operations, history, fetchHistory, clearHistory, cancelOperation, dismissFailedOp } = useOperations()
|
||||
const [filter, setFilter] = useState('all')
|
||||
|
||||
useEffect(() => { fetchHistory() }, [fetchHistory])
|
||||
|
||||
const retryOperation = useCallback(async (op) => {
|
||||
// Dismiss before reinstalling, never after: the reinstall reuses the same
|
||||
// opcache key, and overwriting a failed entry in place skips recordTerminal
|
||||
// so the failure would never reach the record. Dismissing first is what
|
||||
// puts it there.
|
||||
//
|
||||
// By jobID, because the guarantee only holds while both calls address the
|
||||
// same job. Two ops can share an id (a local and a node-scoped install of
|
||||
// one backend), and dismissing by id could retire the other one instead,
|
||||
// leaving this failure to be overwritten in place by the reinstall below.
|
||||
await dismissFailedOp(op.jobID)
|
||||
// fullName is the gallery-qualified id the install endpoints expect;
|
||||
// `name` has the repo prefix stripped for display. Node-scoped ops already
|
||||
// had their prefix removed server side, so fullName is the bare slug there.
|
||||
const target = op.fullName || op.id
|
||||
try {
|
||||
if (op.nodeID) {
|
||||
await nodesApi.installBackend(op.nodeID, target)
|
||||
} else if (op.isBackend) {
|
||||
await backendsApi.install(target)
|
||||
} else {
|
||||
// The variant is not on the payload YET, so a pinned model retries as
|
||||
// an auto-select: someone who chose a specific quant, watched it fail
|
||||
// at 90% and pressed Retry gets a different build, with nothing on
|
||||
// screen saying so. Worth closing, and close to closed: ui_api.go
|
||||
// already reads ?variant= at enqueue and stores it on the ManagementOp,
|
||||
// so it only has to reach the /api/operations payload and this call.
|
||||
// Until then Retry stays, because nothing here distinguishes a pinned
|
||||
// install from an unpinned one and dropping it would cost every model
|
||||
// the button, including the common plain install that hit a network
|
||||
// error.
|
||||
await modelsApi.install(target)
|
||||
}
|
||||
} catch (err) {
|
||||
addToast?.(t('activity.retryFailed', { message: err.message }), 'error')
|
||||
}
|
||||
}, [dismissFailedOp, addToast, t])
|
||||
|
||||
const live = useMemo(
|
||||
() => operations.filter((op) => !op.error && matchesFilter(op, filter)),
|
||||
[operations, filter],
|
||||
)
|
||||
const failing = useMemo(
|
||||
() => operations.filter((op) => op.error && matchesFilter(op, filter)),
|
||||
[operations, filter],
|
||||
)
|
||||
const records = useMemo(
|
||||
() => history.filter((entry) => matchesFilter(entry, filter)),
|
||||
[history, filter],
|
||||
)
|
||||
|
||||
// The header describes the instance, not the current chip, which is what the
|
||||
// Clear-history button beside it already does. A filtered count here would
|
||||
// report "Nothing running" while two model installs were running just
|
||||
// offscreen; the filtered view explains itself through the sections and the
|
||||
// filtered empty state instead.
|
||||
//
|
||||
// "Nothing running" must also not be said while a failure is waiting for a
|
||||
// decision, so both counts get a clause. Each clause is dropped when its
|
||||
// count is zero rather than rendered as a literal 0: the ordinary happy path
|
||||
// would otherwise put "0 needs attention" under the page title on every
|
||||
// render, which reads as a report about failures rather than the absence of
|
||||
// one.
|
||||
const runningTotal = operations.filter((op) => !op.error).length
|
||||
const failingTotal = operations.length - runningTotal
|
||||
const summaryClauses = []
|
||||
if (runningTotal > 0) summaryClauses.push(t('activity.summaryRunning', { count: runningTotal }))
|
||||
if (failingTotal > 0) summaryClauses.push(t('activity.summaryFailed', { count: failingTotal }))
|
||||
let supporting
|
||||
if (summaryClauses.length > 0) supporting = summaryClauses.join(' ')
|
||||
else if (history.length > 0) supporting = t('activity.summaryQuiet', { count: history.length })
|
||||
// Saying "0 operations since startup" directly above "No operations since
|
||||
// startup" states the same nothing twice.
|
||||
else supporting = t('activity.summaryIdle')
|
||||
|
||||
return (
|
||||
<div className="page page--wide activity-page">
|
||||
<PageHeader
|
||||
title={t('activity.title')}
|
||||
supporting={supporting}
|
||||
actions={history.length > 0 ? (
|
||||
<button type="button" className="btn btn-secondary" onClick={clearHistory}>
|
||||
{t('activity.clearHistory')}
|
||||
</button>
|
||||
) : null}
|
||||
/>
|
||||
|
||||
<div className="activity-filters">
|
||||
{FILTERS.map((entry) => (
|
||||
<button
|
||||
key={entry.id}
|
||||
type="button"
|
||||
className="activity-chip"
|
||||
aria-pressed={filter === entry.id}
|
||||
onClick={() => setFilter(entry.id)}
|
||||
>
|
||||
{t(entry.labelKey)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{live.length > 0 && (
|
||||
<section className="activity-section">
|
||||
<h2 className="activity-section__title">
|
||||
{t('activity.inProgress')} <span className="activity-section__count">{live.length}</span>
|
||||
</h2>
|
||||
{live.map((op) => (
|
||||
<OperationCard key={op.jobID || op.id} operation={op} onCancel={cancelOperation} />
|
||||
))}
|
||||
</section>
|
||||
)}
|
||||
|
||||
{failing.length > 0 && (
|
||||
<section className="activity-section">
|
||||
<h2 className="activity-section__title">
|
||||
{t('activity.needsAttention')} <span className="activity-section__count">{failing.length}</span>
|
||||
</h2>
|
||||
{failing.map((op) => (
|
||||
<OperationCard
|
||||
key={op.jobID || op.id}
|
||||
operation={op}
|
||||
onDismiss={dismissFailedOp}
|
||||
onRetry={isRetryable(op) ? retryOperation : undefined}
|
||||
/>
|
||||
))}
|
||||
</section>
|
||||
)}
|
||||
|
||||
{records.length > 0 && (
|
||||
<section className="activity-section">
|
||||
<h2 className="activity-section__title">
|
||||
{t('activity.record')} <span className="activity-section__count">{records.length}</span>
|
||||
</h2>
|
||||
<div className="activity-rows">
|
||||
{records.map((record) => (
|
||||
<div key={record.jobID} className="activity-row">
|
||||
<i
|
||||
className={`${outcomeIcon[record.outcome] || 'fas fa-check'} activity-row__icon activity-row__icon--${record.outcome}`}
|
||||
aria-hidden="true"
|
||||
/>
|
||||
<span className="activity-row__name">
|
||||
{record.name}
|
||||
<small>{recordSummary(record, t)}</small>
|
||||
</span>
|
||||
<span className="activity-row__when">{timeOfDay(record.finishedAt)}</span>
|
||||
<Link className="activity-row__action" to={record.isBackend ? '/app/backends' : '/app/models'}>
|
||||
{record.isBackend ? t('activity.viewInBackends') : t('activity.viewInModels')}
|
||||
</Link>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<p className="activity-note">{t('activity.historyNote')}</p>
|
||||
</section>
|
||||
)}
|
||||
|
||||
{/* A chip that matches nothing is not an empty system. Telling someone
|
||||
with three model installs on record that nothing has ever run, while
|
||||
the line above them counts those same three, is simply false. */}
|
||||
{live.length === 0 && failing.length === 0 && records.length === 0 && (
|
||||
filter === 'all' ? (
|
||||
<div className="activity-empty">
|
||||
<i className="fas fa-download activity-empty__icon" aria-hidden="true" />
|
||||
<p className="activity-empty__title">{t('activity.emptyTitle')}</p>
|
||||
<p className="activity-empty__body">{t('activity.emptyBody')}</p>
|
||||
<Link className="btn btn-primary" to="/app/models">{t('activity.browseModels')}</Link>
|
||||
</div>
|
||||
) : (
|
||||
<div className="activity-empty activity-empty--filtered">
|
||||
<i className="fas fa-filter activity-empty__icon" aria-hidden="true" />
|
||||
<p className="activity-empty__title">{t('activity.emptyFiltered')}</p>
|
||||
<button type="button" className="btn btn-secondary" onClick={() => setFilter('all')}>
|
||||
{t('activity.showAll')}
|
||||
</button>
|
||||
</div>
|
||||
)
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -43,6 +43,11 @@ const Sound = page('sound', () => import('./pages/Sound'))
|
||||
const AudioTransform = page('transform', () => import('./pages/AudioTransform'))
|
||||
const Talk = page('talk', () => import('./pages/Talk'))
|
||||
const Backends = page('backends', () => import('./pages/Backends'))
|
||||
// Only referenced from JSX below, which eslint cannot see without
|
||||
// eslint-plugin-react. Suppressed here rather than left to widen the file's
|
||||
// warning count; the surrounding page consts predate the lint baseline.
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
const Activity = page('activity', () => import('./pages/Activity'))
|
||||
const Settings = page('settings', () => import('./pages/Settings'))
|
||||
const Traces = page('traces', () => import('./pages/Traces'))
|
||||
const P2P = page('p2p', () => import('./pages/P2P'))
|
||||
@@ -151,6 +156,7 @@ const appChildren = [
|
||||
element: <ConsoleLayout config={operateConsole} />,
|
||||
children: [
|
||||
{ path: 'backends', element: <Admin><Backends /></Admin> },
|
||||
{ path: 'activity', element: <Admin><Activity /></Admin> },
|
||||
{ path: 'voice-library', element: <Admin><VoiceLibrary /></Admin> },
|
||||
{ path: 'settings', element: <Admin><Settings /></Admin> },
|
||||
{ path: 'traces', element: <Admin><Traces /></Admin> },
|
||||
|
||||
2
core/http/react-ui/src/utils/api.js
vendored
2
core/http/react-ui/src/utils/api.js
vendored
@@ -174,6 +174,8 @@ export const operationsApi = {
|
||||
list: () => fetchJSON(API_CONFIG.endpoints.operations),
|
||||
cancel: (jobID) => postJSON(API_CONFIG.endpoints.cancelOperation(jobID), {}),
|
||||
dismiss: (jobID) => postJSON(API_CONFIG.endpoints.dismissOperation(jobID), {}),
|
||||
history: () => fetchJSON(API_CONFIG.endpoints.operationsHistory),
|
||||
clearHistory: () => fetchJSON(API_CONFIG.endpoints.operationsHistory, { method: 'DELETE' }),
|
||||
}
|
||||
|
||||
// Settings API
|
||||
|
||||
1
core/http/react-ui/src/utils/config.js
vendored
1
core/http/react-ui/src/utils/config.js
vendored
@@ -2,6 +2,7 @@ export const API_CONFIG = {
|
||||
endpoints: {
|
||||
// Operations
|
||||
operations: '/api/operations',
|
||||
operationsHistory: '/api/operations/history',
|
||||
cancelOperation: (jobID) => `/api/operations/${jobID}/cancel`,
|
||||
dismissOperation: (jobID) => `/api/operations/${jobID}/dismiss`,
|
||||
|
||||
|
||||
@@ -170,7 +170,6 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
progress := 0
|
||||
isDeletion := false
|
||||
isQueued := false
|
||||
isCancelled := false
|
||||
isCancellable := false
|
||||
message := ""
|
||||
phase := ""
|
||||
@@ -189,7 +188,13 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
|
||||
progress = int(status.Progress)
|
||||
isDeletion = status.Deletion
|
||||
isCancelled = status.Cancelled
|
||||
// Admission publishes a "queued" status before the op reaches
|
||||
// the worker, so a queued op HAS a status: the phase is the
|
||||
// only truthful signal. Reading "queued" off a missing status
|
||||
// instead (what this used to do) made the state unreachable,
|
||||
// and every queued operation rendered as if it were already
|
||||
// installing.
|
||||
isQueued = status.IsQueued()
|
||||
isCancellable = status.Cancellable
|
||||
message = status.Message
|
||||
phase = status.Phase
|
||||
@@ -198,11 +203,10 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
if isDeletion {
|
||||
taskType = "deletion"
|
||||
}
|
||||
if isCancelled {
|
||||
taskType = "cancelled"
|
||||
}
|
||||
} else {
|
||||
// Job is queued but hasn't started
|
||||
// No status at all: an op hydrated from the store or replicated
|
||||
// from a peer whose outcome this replica never held. It has not
|
||||
// been observed running, so it is reported as waiting.
|
||||
isQueued = true
|
||||
isCancellable = true
|
||||
message = "Operation queued"
|
||||
@@ -247,6 +251,11 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
}
|
||||
}
|
||||
|
||||
// No isCancelled field: a cancellation is terminal and removes the
|
||||
// operation from this list before the next poll (CancelOperation
|
||||
// marks the status Processed, GetStatus evicts it and the cancel
|
||||
// handler deletes it), so nothing here could ever report one. The
|
||||
// cancelled outcome is reported by /api/operations/history.
|
||||
opData := map[string]any{
|
||||
"id": galleryID,
|
||||
"name": displayName,
|
||||
@@ -257,7 +266,6 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
"isDeletion": isDeletion,
|
||||
"isBackend": isBackend,
|
||||
"isQueued": isQueued,
|
||||
"isCancelled": isCancelled,
|
||||
"cancellable": isCancellable,
|
||||
"message": message,
|
||||
}
|
||||
@@ -333,7 +341,6 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
"isDeletion": false,
|
||||
"isBackend": false,
|
||||
"isQueued": false,
|
||||
"isCancelled": false,
|
||||
"cancellable": false,
|
||||
"message": status.Message,
|
||||
"nodeName": status.NodeName,
|
||||
@@ -396,6 +403,28 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
})
|
||||
}, adminMiddleware)
|
||||
|
||||
// Finished operations. Separate from /api/operations on purpose: that
|
||||
// endpoint is polled once a second by every open tab, so the record does
|
||||
// not ride along with it.
|
||||
app.GET("/api/operations/history", func(c echo.Context) error {
|
||||
return c.JSON(200, map[string]any{
|
||||
"operations": opcache.History(),
|
||||
})
|
||||
}, adminMiddleware)
|
||||
|
||||
// Clear the record. Live operations and undismissed failures are untouched.
|
||||
app.DELETE("/api/operations/history", func(c echo.Context) error {
|
||||
if err := opcache.ClearHistory(); err != nil {
|
||||
xlog.Error("could not clear the operation record", "error", err)
|
||||
return c.JSON(http.StatusInternalServerError, map[string]any{
|
||||
"error": err.Error(),
|
||||
})
|
||||
}
|
||||
return c.JSON(200, map[string]any{
|
||||
"success": true,
|
||||
})
|
||||
}, adminMiddleware)
|
||||
|
||||
// Model Gallery APIs (admin only)
|
||||
app.GET("/api/models", func(c echo.Context) error {
|
||||
// Trimmed once, here, so "is the user searching?" has a single answer
|
||||
|
||||
141
core/http/routes/ui_api_operations_history_test.go
Normal file
141
core/http/routes/ui_api_operations_history_test.go
Normal file
@@ -0,0 +1,141 @@
|
||||
package routes_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
|
||||
"github.com/labstack/echo/v4"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/application"
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/http/routes"
|
||||
"github.com/mudler/LocalAI/core/services/distributed"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
"github.com/mudler/LocalAI/core/services/testutil"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
)
|
||||
|
||||
// The Activity page reads finished operations from /api/operations/history.
|
||||
// The live /api/operations payload must stay exactly as it was: it is polled
|
||||
// once a second by every open tab.
|
||||
var _ = Describe("/api/operations/history", func() {
|
||||
noopMw := func(next echo.HandlerFunc) echo.HandlerFunc { return next }
|
||||
|
||||
var (
|
||||
e *echo.Echo
|
||||
svc *galleryop.GalleryService
|
||||
opcache *galleryop.OpCache
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
tmpDir, err := os.MkdirTemp("", "ops-history-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() {
|
||||
Expect(os.RemoveAll(tmpDir)).To(Succeed())
|
||||
})
|
||||
|
||||
// /api/operations resolves a live op's name against the backend gallery,
|
||||
// which dereferences SystemState. A bare config would panic there.
|
||||
appCfg := &config.ApplicationConfig{
|
||||
SystemState: system.NewCapabilityState("default",
|
||||
system.WithBackendPath(tmpDir), system.WithBackendSystemPath(tmpDir)),
|
||||
}
|
||||
svc = galleryop.NewGalleryService(appCfg, nil)
|
||||
opcache = galleryop.NewOpCache(svc)
|
||||
e = echo.New()
|
||||
routes.RegisterUIAPIRoutes(e, nil, nil, appCfg, svc, opcache, &application.Application{}, noopMw)
|
||||
})
|
||||
|
||||
finish := func(key, jobID string) {
|
||||
opcache.Set(key, jobID)
|
||||
svc.UpdateStatus(jobID, &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
opcache.DeleteUUID(jobID)
|
||||
}
|
||||
|
||||
get := func(path string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(http.MethodGet, path, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
It("returns an empty list before anything has run", func() {
|
||||
rec := get("/api/operations/history")
|
||||
Expect(rec.Code).To(Equal(http.StatusOK))
|
||||
|
||||
var envelope struct {
|
||||
Operations []galleryop.OpRecord `json:"operations"`
|
||||
}
|
||||
Expect(json.Unmarshal(rec.Body.Bytes(), &envelope)).To(Succeed())
|
||||
Expect(envelope.Operations).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("returns finished operations newest first", func() {
|
||||
finish("model-one", "job-1")
|
||||
finish("model-two", "job-2")
|
||||
|
||||
rec := get("/api/operations/history")
|
||||
Expect(rec.Code).To(Equal(http.StatusOK))
|
||||
|
||||
var envelope struct {
|
||||
Operations []galleryop.OpRecord `json:"operations"`
|
||||
}
|
||||
Expect(json.Unmarshal(rec.Body.Bytes(), &envelope)).To(Succeed())
|
||||
Expect(envelope.Operations).To(HaveLen(2))
|
||||
Expect(envelope.Operations[0].Name).To(Equal("model-two"))
|
||||
Expect(envelope.Operations[0].Outcome).To(Equal("completed"))
|
||||
})
|
||||
|
||||
It("empties the record on DELETE", func() {
|
||||
finish("model-one", "job-1")
|
||||
|
||||
req := httptest.NewRequest(http.MethodDelete, "/api/operations/history", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
Expect(rec.Code).To(Equal(http.StatusOK))
|
||||
|
||||
Expect(opcache.History()).To(BeEmpty())
|
||||
})
|
||||
|
||||
// Answering 200 on a clear that did not happen makes the record vanish from
|
||||
// the page and come straight back on the next fetch, with nothing said
|
||||
// about why.
|
||||
It("reports a failed clear as a 500 rather than claiming success", func() {
|
||||
db := testutil.SetupTestDB()
|
||||
store, err := distributed.NewGalleryStore(db)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
opcache.SetGalleryStore(store)
|
||||
finish("model-one", "job-1")
|
||||
|
||||
// What a database outage looks like from the handler's side.
|
||||
Expect(db.Migrator().DropTable(&distributed.GalleryOperationRecord{})).To(Succeed())
|
||||
|
||||
req := httptest.NewRequest(http.MethodDelete, "/api/operations/history", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
Expect(rec.Code).To(Equal(http.StatusInternalServerError))
|
||||
|
||||
var body map[string]any
|
||||
Expect(json.Unmarshal(rec.Body.Bytes(), &body)).To(Succeed())
|
||||
Expect(body).To(HaveKey("error"))
|
||||
Expect(body).ToNot(HaveKey("success"))
|
||||
})
|
||||
|
||||
It("leaves the live operations payload unchanged", func() {
|
||||
// Regression guard: the 1s poll must not grow a history key.
|
||||
opcache.Set("model-live", "job-live")
|
||||
svc.UpdateStatus("job-live", &galleryop.OpStatus{Progress: 42, Cancellable: true})
|
||||
|
||||
rec := get("/api/operations")
|
||||
Expect(rec.Code).To(Equal(http.StatusOK))
|
||||
|
||||
var envelope map[string]any
|
||||
Expect(json.Unmarshal(rec.Body.Bytes(), &envelope)).To(Succeed())
|
||||
Expect(envelope).To(HaveLen(1))
|
||||
Expect(envelope).To(HaveKey("operations"))
|
||||
})
|
||||
})
|
||||
@@ -1,7 +1,9 @@
|
||||
package routes_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
|
||||
@@ -11,11 +13,33 @@ import (
|
||||
|
||||
"github.com/mudler/LocalAI/core/application"
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/gallery"
|
||||
"github.com/mudler/LocalAI/core/http/routes"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
)
|
||||
|
||||
// parkedModelManager parks inside the operation the worker is running so a
|
||||
// spec can read /api/operations while that operation is genuinely in flight.
|
||||
// Without it the terminal status lands before the request is served and the
|
||||
// running phase is unobservable.
|
||||
type parkedModelManager struct {
|
||||
entered chan string
|
||||
release chan struct{}
|
||||
}
|
||||
|
||||
func (m *parkedModelManager) InstallModel(_ context.Context, op *galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig], _ galleryop.ProgressCallback) error {
|
||||
m.entered <- op.GalleryElementName
|
||||
<-m.release
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *parkedModelManager) DeleteModel(name string) error {
|
||||
m.entered <- name
|
||||
<-m.release
|
||||
return nil
|
||||
}
|
||||
|
||||
// These specs guard the contract between the opcache (which stores
|
||||
// node-scoped backend installs under a "node:<nodeID>:<backend>" key) and the
|
||||
// /api/operations response surface the React UI polls. Without nodeID
|
||||
@@ -27,6 +51,29 @@ var _ = Describe("/api/operations with node-scoped backend ops", func() {
|
||||
// *DistributedServices, which is nil for a fresh Application{}.
|
||||
noopMw := func(next echo.HandlerFunc) echo.HandlerFunc { return next }
|
||||
|
||||
// operationByJobID serves GET /api/operations and returns the single
|
||||
// operation carrying jobID, or nil if the endpoint did not list it.
|
||||
operationByJobID := func(appCfg *config.ApplicationConfig, svc *galleryop.GalleryService, opcache *galleryop.OpCache, jobID string) map[string]any {
|
||||
GinkgoHelper()
|
||||
e := echo.New()
|
||||
routes.RegisterUIAPIRoutes(e, nil, nil, appCfg, svc, opcache, &application.Application{}, noopMw)
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/operations", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
Expect(rec.Code).To(Equal(http.StatusOK))
|
||||
|
||||
var envelope struct {
|
||||
Operations []map[string]any `json:"operations"`
|
||||
}
|
||||
Expect(json.Unmarshal(rec.Body.Bytes(), &envelope)).To(Succeed())
|
||||
for _, op := range envelope.Operations {
|
||||
if op["jobID"] == jobID {
|
||||
return op
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
It("emits nodeID and the un-prefixed backend name for keys built by NodeScopedKey", func() {
|
||||
appCfg := &config.ApplicationConfig{}
|
||||
galleryService := galleryop.NewGalleryService(appCfg, nil)
|
||||
@@ -154,6 +201,207 @@ var _ = Describe("/api/operations with node-scoped backend ops", func() {
|
||||
Expect(found["name"]).To(Equal("llama-cpp"))
|
||||
})
|
||||
|
||||
It("reports an admitted operation the worker has not started as queued", func() {
|
||||
state, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
appCfg := &config.ApplicationConfig{SystemState: state}
|
||||
galleryService := galleryop.NewGalleryService(appCfg, nil)
|
||||
opcache := galleryop.NewOpCache(galleryService)
|
||||
|
||||
// Nothing consumes ModelGalleryChannel here, which is exactly the state
|
||||
// of an op admitted while the serial worker is mid-install. The op is
|
||||
// admitted the way the install handlers admit it, so the spec fails if
|
||||
// the queued signal and the admission path ever disagree again.
|
||||
jobID := "job-queued-op"
|
||||
opcache.Set("localai@qwen3-4b", jobID)
|
||||
galleryService.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: jobID,
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
})
|
||||
Eventually(func() *galleryop.OpStatus {
|
||||
return galleryService.GetStatus(jobID)
|
||||
}, "2s", "10ms").ShouldNot(BeNil())
|
||||
|
||||
e := echo.New()
|
||||
routes.RegisterUIAPIRoutes(e, nil, nil, appCfg, galleryService, opcache, &application.Application{}, noopMw)
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/operations", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
Expect(rec.Code).To(Equal(http.StatusOK))
|
||||
|
||||
var envelope struct {
|
||||
Operations []map[string]any `json:"operations"`
|
||||
}
|
||||
Expect(json.Unmarshal(rec.Body.Bytes(), &envelope)).To(Succeed())
|
||||
var found map[string]any
|
||||
for _, op := range envelope.Operations {
|
||||
if op["jobID"] == jobID {
|
||||
found = op
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(found).ToNot(BeNil(), "an admitted op must be listed while it waits")
|
||||
Expect(found["isQueued"]).To(BeTrue(), "an op the worker has not started must report as queued")
|
||||
// Cancelling is what the queued state exists for: an op waiting behind
|
||||
// a long install is the one a user most wants to call off.
|
||||
Expect(found["cancellable"]).To(BeTrue())
|
||||
})
|
||||
|
||||
It("offers cancel on a queued removal", func() {
|
||||
state, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
appCfg := &config.ApplicationConfig{SystemState: state}
|
||||
galleryService := galleryop.NewGalleryService(appCfg, nil)
|
||||
opcache := galleryop.NewOpCache(galleryService)
|
||||
|
||||
// Nothing consumes the channel: the removal is admitted and waits, which
|
||||
// is the one window in a removal's life where calling it off both works
|
||||
// and leaves nothing behind.
|
||||
jobID := "job-queued-removal"
|
||||
opcache.Set("localai@qwen3-4b", jobID)
|
||||
galleryService.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: jobID,
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
Delete: true,
|
||||
})
|
||||
Eventually(func() *galleryop.OpStatus {
|
||||
return galleryService.GetStatus(jobID)
|
||||
}, "2s", "10ms").ShouldNot(BeNil())
|
||||
|
||||
found := operationByJobID(appCfg, galleryService, opcache, jobID)
|
||||
Expect(found).ToNot(BeNil(), "an admitted removal must be listed while it waits")
|
||||
Expect(found["isQueued"]).To(BeTrue())
|
||||
Expect(found["isDeletion"]).To(BeTrue())
|
||||
Expect(found["cancellable"]).To(BeTrue(),
|
||||
"hiding Cancel here strands the removal behind whatever is installing")
|
||||
})
|
||||
|
||||
It("does not offer cancel once a removal is running", func() {
|
||||
state, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
appCfg := &config.ApplicationConfig{SystemState: state}
|
||||
galleryService := galleryop.NewGalleryService(appCfg, nil)
|
||||
opcache := galleryop.NewOpCache(galleryService)
|
||||
|
||||
// A real worker running a real handler: the entry status is the thing
|
||||
// under test, so it must come from the handler and not from the spec.
|
||||
// The manager parks inside the removal to hold the running phase open.
|
||||
manager := &parkedModelManager{entered: make(chan string, 1), release: make(chan struct{})}
|
||||
galleryService.SetModelManager(manager)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
DeferCleanup(cancel)
|
||||
DeferCleanup(func() { close(manager.release) })
|
||||
Expect(galleryService.Start(ctx, nil, nil)).To(Succeed())
|
||||
|
||||
jobID := "job-running-removal"
|
||||
opcache.Set("localai@qwen3-4b", jobID)
|
||||
galleryService.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: jobID,
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
Delete: true,
|
||||
})
|
||||
Eventually(manager.entered, "5s").Should(Receive(Equal("localai@qwen3-4b")))
|
||||
|
||||
found := operationByJobID(appCfg, galleryService, opcache, jobID)
|
||||
Expect(found).ToNot(BeNil())
|
||||
Expect(found["isQueued"]).To(BeFalse())
|
||||
Expect(found["isDeletion"]).To(BeTrue())
|
||||
Expect(found["cancellable"]).To(BeFalse(),
|
||||
"a Cancel button on a running removal cannot be honoured: DeleteModel takes no context")
|
||||
})
|
||||
|
||||
It("does not emit isCancelled, which no live operation can ever be", func() {
|
||||
// CancelOperation marks the status processed and the cancel handler
|
||||
// deletes the op, so a cancelled operation is gone from this endpoint
|
||||
// by the next poll. A field that is structurally always false is a
|
||||
// state the UI would branch on and never reach.
|
||||
state, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
appCfg := &config.ApplicationConfig{SystemState: state}
|
||||
galleryService := galleryop.NewGalleryService(appCfg, nil)
|
||||
opcache := galleryop.NewOpCache(galleryService)
|
||||
jobID := "job-no-cancel-field"
|
||||
opcache.Set("qwen-asr", jobID)
|
||||
galleryService.UpdateStatus(jobID, &galleryop.OpStatus{Progress: 10, Cancellable: true})
|
||||
|
||||
e := echo.New()
|
||||
routes.RegisterUIAPIRoutes(e, nil, nil, appCfg, galleryService, opcache, &application.Application{}, noopMw)
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/operations", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
Expect(rec.Code).To(Equal(http.StatusOK))
|
||||
|
||||
var envelope struct {
|
||||
Operations []map[string]any `json:"operations"`
|
||||
}
|
||||
Expect(json.Unmarshal(rec.Body.Bytes(), &envelope)).To(Succeed())
|
||||
Expect(envelope.Operations).To(HaveLen(1))
|
||||
Expect(envelope.Operations[0]).ToNot(HaveKey("isCancelled"))
|
||||
})
|
||||
|
||||
It("reports a running removal as a deletion", func() {
|
||||
state, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
appCfg := &config.ApplicationConfig{SystemState: state}
|
||||
galleryService := galleryop.NewGalleryService(appCfg, nil)
|
||||
opcache := galleryop.NewOpCache(galleryService)
|
||||
|
||||
// Admitted the way the delete handlers admit it, so the spec fails if
|
||||
// admission and this endpoint ever disagree about what a removal is.
|
||||
jobID := "job-delete-running"
|
||||
opcache.Set("localai@qwen3-4b", jobID)
|
||||
galleryService.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: jobID,
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
Delete: true,
|
||||
})
|
||||
Eventually(func() *galleryop.OpStatus {
|
||||
return galleryService.GetStatus(jobID)
|
||||
}, "2s", "10ms").ShouldNot(BeNil())
|
||||
|
||||
// The worker's first write is a fresh OpStatus that says nothing about
|
||||
// the kind of job. Only the queued seed ever knew.
|
||||
galleryService.UpdateStatus(jobID, &galleryop.OpStatus{Message: "processing model: localai@qwen3-4b", Cancellable: true})
|
||||
|
||||
found := operationByJobID(appCfg, galleryService, opcache, jobID)
|
||||
Expect(found).ToNot(BeNil(), "a running removal must be listed")
|
||||
Expect(found["isQueued"]).To(BeFalse())
|
||||
Expect(found["isDeletion"]).To(BeTrue(), "a running removal must report as a removal, not an install")
|
||||
Expect(found["taskType"]).To(Equal("deletion"))
|
||||
})
|
||||
|
||||
It("still reports a failed removal as a deletion", func() {
|
||||
state, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
appCfg := &config.ApplicationConfig{SystemState: state}
|
||||
galleryService := galleryop.NewGalleryService(appCfg, nil)
|
||||
opcache := galleryop.NewOpCache(galleryService)
|
||||
|
||||
jobID := "job-delete-failed"
|
||||
opcache.Set("localai@qwen3-4b", jobID)
|
||||
galleryService.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: jobID,
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
Delete: true,
|
||||
})
|
||||
Eventually(func() *galleryop.OpStatus {
|
||||
return galleryService.GetStatus(jobID)
|
||||
}, "2s", "10ms").ShouldNot(BeNil())
|
||||
|
||||
galleryService.UpdateStatus(jobID, &galleryop.OpStatus{Message: "processing model: localai@qwen3-4b", Cancellable: true})
|
||||
galleryService.UpdateStatus(jobID, &galleryop.OpStatus{
|
||||
Error: errors.New("permission denied"), Processed: true, Message: "error: permission denied",
|
||||
})
|
||||
|
||||
found := operationByJobID(appCfg, galleryService, opcache, jobID)
|
||||
// This is the one that matters: the UI offers Retry on any failure that
|
||||
// is not a removal, and Retry installs. A failed removal that reports
|
||||
// isDeletion=false gets a Retry button that re-downloads the model.
|
||||
Expect(found).ToNot(BeNil(), "a failed removal stays listed until it is dismissed")
|
||||
Expect(found["isDeletion"]).To(BeTrue(), "a failed removal must not be offered a Retry that reinstalls")
|
||||
Expect(found["taskType"]).To(Equal("deletion"))
|
||||
})
|
||||
|
||||
It("surfaces managed model artifact phase and byte counters", func() {
|
||||
state, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
@@ -22,7 +22,7 @@ type GalleryOperationRecord struct {
|
||||
GalleryElementName string `gorm:"size:255" json:"gallery_element_name"`
|
||||
CacheKey string `gorm:"index;size:512" json:"cache_key,omitempty"` // OpCache key (galleryID or node:<id>:<backend>)
|
||||
IsBackendOp bool `json:"is_backend_op"` // true if installed via SetBackend
|
||||
OpType string `gorm:"size:32" json:"op_type"` // "model_install", "model_delete", "backend_install"
|
||||
OpType string `gorm:"size:32" json:"op_type"` // "model_install", "model_delete", "backend_install", "backend_delete"
|
||||
Status string `gorm:"size:32;default:pending" json:"status"` // pending, downloading, processing, completed, failed, cancelled
|
||||
Progress float64 `json:"progress"` // 0.0 to 1.0
|
||||
Phase string `gorm:"size:32" json:"phase,omitempty"`
|
||||
@@ -45,7 +45,24 @@ type GalleryOperationRecord struct {
|
||||
// "still active" means.
|
||||
var activeStatuses = []string{"pending", "downloading", "processing"}
|
||||
|
||||
func (GalleryOperationRecord) TableName() string { return "gallery_operations" }
|
||||
// terminalStatuses lists the gallery_operations.status values that represent a
|
||||
// finished operation. The Activity record and the retention reaper share this
|
||||
// set so the two never disagree about what "finished" means.
|
||||
var terminalStatuses = []string{"completed", "failed", "cancelled"}
|
||||
|
||||
// settledStatuses is the subset of terminalStatuses that may never be
|
||||
// rewritten. It deliberately excludes "failed", which the other two do not:
|
||||
// CleanStale writes a failure onto any operation that has sat in an active
|
||||
// status for 30 minutes, and the gallery worker consumes both channels
|
||||
// serially, so an operation queued behind a large download is reaped while it
|
||||
// is still going to run. That failure has to stay correctable by the real
|
||||
// outcome. A completion or a cancellation is what actually happened, and
|
||||
// nothing that arrives afterwards knows better.
|
||||
var settledStatuses = []string{"completed", "cancelled"}
|
||||
|
||||
const galleryOperationsTable = "gallery_operations"
|
||||
|
||||
func (GalleryOperationRecord) TableName() string { return galleryOperationsTable }
|
||||
|
||||
// GalleryStore manages gallery operation state in PostgreSQL.
|
||||
type GalleryStore struct {
|
||||
@@ -67,6 +84,14 @@ func NewGalleryStore(db *gorm.DB) (*GalleryStore, error) {
|
||||
// name, op type, status) rather than fail with a primary-key conflict.
|
||||
// CacheKey and IsBackendOp are intentionally not in DoUpdates so the
|
||||
// placeholder's values win.
|
||||
//
|
||||
// The status, cancellable and updated_at columns are frozen once the row has
|
||||
// settled. An admin can cancel an operation while it is still queued, and the
|
||||
// worker then dequeues it and calls Create with status "pending" — without the
|
||||
// freeze that reopens a cancelled operation, which reads as live forever. The
|
||||
// descriptive columns still update, so the row keeps gaining its name and
|
||||
// op_type, and a reaped-but-still-running operation is deliberately still
|
||||
// allowed to reopen (see settledStatuses).
|
||||
func (s *GalleryStore) Create(op *GalleryOperationRecord) error {
|
||||
if op.ID == "" {
|
||||
op.ID = uuid.New().String()
|
||||
@@ -75,13 +100,28 @@ func (s *GalleryStore) Create(op *GalleryOperationRecord) error {
|
||||
op.UpdatedAt = op.CreatedAt
|
||||
return s.db.Clauses(clause.OnConflict{
|
||||
Columns: []clause.Column{{Name: "id"}},
|
||||
DoUpdates: clause.AssignmentColumns([]string{
|
||||
"gallery_element_name", "op_type", "status",
|
||||
"frontend_id", "user_id", "cancellable", "updated_at",
|
||||
DoUpdates: clause.Assignments(map[string]any{
|
||||
"gallery_element_name": gorm.Expr("excluded.gallery_element_name"),
|
||||
"op_type": gorm.Expr("excluded.op_type"),
|
||||
"frontend_id": gorm.Expr("excluded.frontend_id"),
|
||||
"user_id": gorm.Expr("excluded.user_id"),
|
||||
"status": keepWhenSettled("status"),
|
||||
"cancellable": keepWhenSettled("cancellable"),
|
||||
"updated_at": keepWhenSettled("updated_at"),
|
||||
}),
|
||||
}).Create(op).Error
|
||||
}
|
||||
|
||||
// keepWhenSettled builds the upsert assignment for a column that must not be
|
||||
// rewritten once the operation has settled: it keeps the stored value for a
|
||||
// settled row and takes the incoming one otherwise.
|
||||
func keepWhenSettled(column string) clause.Expr {
|
||||
stored := galleryOperationsTable + "."
|
||||
return gorm.Expr(
|
||||
"CASE WHEN "+stored+"status IN ? THEN "+stored+column+" ELSE excluded."+column+" END",
|
||||
settledStatuses)
|
||||
}
|
||||
|
||||
// UpdateProgress updates progress for an operation. The cancellable flag is
|
||||
// persisted on every tick so a replica that restarts mid-install rehydrates the
|
||||
// op as still cancellable — otherwise the column keeps its Create-time zero
|
||||
@@ -112,16 +152,31 @@ func (s *GalleryStore) UpdateProgress(id string, progress float64, message, down
|
||||
// UpdateStatus updates the status of an operation. A terminal status is never
|
||||
// cancellable, so the flag is cleared here to keep the persisted row consistent
|
||||
// with what the UI should offer.
|
||||
//
|
||||
// A row that has already settled is left alone. An operation settles once, and
|
||||
// the paths that retire one are not mutually exclusive: cancelling writes
|
||||
// "cancelled" synchronously, and the handler goroutine then unwinds with the
|
||||
// context error, which without this guard would overwrite the row with
|
||||
// "failed: context canceled" and make the Activity page render a cancelled
|
||||
// install as a red failure offering Retry. The guard also keeps updated_at
|
||||
// pinned to when the operation actually finished, which is the key ListTerminal
|
||||
// orders the record by. A failure is not settled and stays correctable — see
|
||||
// settledStatuses for why.
|
||||
//
|
||||
// The error is written unconditionally so a corrected outcome drops the
|
||||
// previous attempt's reason: without that, an operation the reaper gave up on
|
||||
// and that then succeeded would be recorded as completed while still carrying
|
||||
// "stale operation reaped" as its error.
|
||||
func (s *GalleryStore) UpdateStatus(id, status, errMsg string) error {
|
||||
updates := map[string]any{
|
||||
"status": status,
|
||||
"cancellable": false,
|
||||
"updated_at": time.Now(),
|
||||
"error": errMsg,
|
||||
}
|
||||
if errMsg != "" {
|
||||
updates["error"] = errMsg
|
||||
}
|
||||
return s.db.Model(&GalleryOperationRecord{}).Where("id = ?", id).Updates(updates).Error
|
||||
return s.db.Model(&GalleryOperationRecord{}).
|
||||
Where("id = ? AND status NOT IN ?", id, settledStatuses).
|
||||
Updates(updates).Error
|
||||
}
|
||||
|
||||
// Get retrieves an operation by ID.
|
||||
@@ -237,7 +292,32 @@ func (s *GalleryStore) CleanStale(age time.Duration) (int64, error) {
|
||||
// CleanOld removes operations older than the given duration.
|
||||
func (s *GalleryStore) CleanOld(retention time.Duration) error {
|
||||
cutoff := time.Now().Add(-retention)
|
||||
return s.db.Where("created_at < ? AND status IN ?", cutoff,
|
||||
[]string{"completed", "failed", "cancelled"}).
|
||||
return s.db.Where("created_at < ? AND status IN ?", cutoff, terminalStatuses).
|
||||
Delete(&GalleryOperationRecord{}).Error
|
||||
}
|
||||
|
||||
// ListTerminal returns finished operations, newest-finished first. It backs the
|
||||
// Activity page's record: in distributed mode the per-replica in-memory ring
|
||||
// shows a different history depending on which replica served the request, and
|
||||
// a replica added by a scale-out has none at all.
|
||||
//
|
||||
// The order is by updated_at, which is when the operation reached its terminal
|
||||
// status, rather than created_at, which is when it was queued: the record
|
||||
// reports what finished and when.
|
||||
//
|
||||
// limit <= 0 returns every row.
|
||||
func (s *GalleryStore) ListTerminal(limit int) ([]GalleryOperationRecord, error) {
|
||||
var ops []GalleryOperationRecord
|
||||
q := s.db.Where("status IN ?", terminalStatuses).Order("updated_at DESC")
|
||||
if limit > 0 {
|
||||
q = q.Limit(limit)
|
||||
}
|
||||
return ops, q.Find(&ops).Error
|
||||
}
|
||||
|
||||
// ClearTerminal deletes every finished operation, cluster-wide. Operations
|
||||
// still in flight survive, so clearing the record cannot lose an install that
|
||||
// has not reported its outcome yet.
|
||||
func (s *GalleryStore) ClearTerminal() error {
|
||||
return s.db.Where("status IN ?", terminalStatuses).Delete(&GalleryOperationRecord{}).Error
|
||||
}
|
||||
|
||||
265
core/services/distributed/gallery_test.go
Normal file
265
core/services/distributed/gallery_test.go
Normal file
@@ -0,0 +1,265 @@
|
||||
package distributed_test
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/distributed"
|
||||
"github.com/mudler/LocalAI/core/services/testutil"
|
||||
)
|
||||
|
||||
var _ = Describe("GalleryStore terminal operations", func() {
|
||||
var store *distributed.GalleryStore
|
||||
|
||||
BeforeEach(func() {
|
||||
db := testutil.SetupTestDB()
|
||||
var err error
|
||||
store, err = distributed.NewGalleryStore(db)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
// finish drives a row through to a terminal status the same way the gallery
|
||||
// service does, so updated_at reflects when it actually finished.
|
||||
finish := func(id, status string) {
|
||||
Expect(store.UpdateStatus(id, status, "")).To(Succeed())
|
||||
// Postgres timestamps are microsecond-precision, but two updates issued
|
||||
// back to back can still land close enough to make the assertion on
|
||||
// ordering a coin toss.
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
|
||||
Describe("ListTerminal", func() {
|
||||
It("returns only finished operations", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "running", GalleryElementName: "busy", OpType: "model_install", Status: "downloading",
|
||||
})).To(Succeed())
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "queued", GalleryElementName: "waiting", OpType: "model_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "done", GalleryElementName: "installed", OpType: "model_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "broke", GalleryElementName: "failed-one", OpType: "backend_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "stopped", GalleryElementName: "cancelled-one", OpType: "model_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
|
||||
finish("done", "completed")
|
||||
finish("broke", "failed")
|
||||
finish("stopped", "cancelled")
|
||||
|
||||
ops, err := store.ListTerminal(0)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
ids := []string{}
|
||||
for _, op := range ops {
|
||||
ids = append(ids, op.ID)
|
||||
}
|
||||
Expect(ids).To(ConsistOf("done", "broke", "stopped"))
|
||||
})
|
||||
|
||||
It("orders by when the operation finished, newest first", func() {
|
||||
for _, id := range []string{"first", "second", "third"} {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: id, GalleryElementName: id, OpType: "model_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
}
|
||||
|
||||
// Finish them out of creation order: the record reports what
|
||||
// finished when, not what was queued when.
|
||||
finish("second", "completed")
|
||||
finish("first", "completed")
|
||||
finish("third", "completed")
|
||||
|
||||
ops, err := store.ListTerminal(0)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(ops).To(HaveLen(3))
|
||||
Expect(ops[0].ID).To(Equal("third"))
|
||||
Expect(ops[1].ID).To(Equal("first"))
|
||||
Expect(ops[2].ID).To(Equal("second"))
|
||||
})
|
||||
|
||||
It("caps the result at the limit, keeping the newest", func() {
|
||||
for _, id := range []string{"a", "b", "c"} {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: id, GalleryElementName: id, OpType: "model_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
finish(id, "completed")
|
||||
}
|
||||
|
||||
ops, err := store.ListTerminal(2)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(ops).To(HaveLen(2))
|
||||
Expect(ops[0].ID).To(Equal("c"))
|
||||
Expect(ops[1].ID).To(Equal("b"))
|
||||
})
|
||||
|
||||
It("returns every row when the limit is zero or negative", func() {
|
||||
for _, id := range []string{"a", "b", "c"} {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: id, GalleryElementName: id, OpType: "model_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
finish(id, "completed")
|
||||
}
|
||||
|
||||
ops, err := store.ListTerminal(-1)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(ops).To(HaveLen(3))
|
||||
})
|
||||
|
||||
It("carries the columns the Activity record is built from", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "op-1", GalleryElementName: "localai@qwen3-4b", OpType: "model_delete", Status: "pending",
|
||||
})).To(Succeed())
|
||||
Expect(store.UpsertCacheKey("op-1", "localai@qwen3-4b", false)).To(Succeed())
|
||||
Expect(store.UpdateStatus("op-1", "failed", "no space left on device")).To(Succeed())
|
||||
|
||||
ops, err := store.ListTerminal(0)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(ops).To(HaveLen(1))
|
||||
Expect(ops[0].CacheKey).To(Equal("localai@qwen3-4b"))
|
||||
Expect(ops[0].OpType).To(Equal("model_delete"))
|
||||
Expect(ops[0].Error).To(Equal("no space left on device"))
|
||||
Expect(ops[0].CreatedAt).ToNot(BeZero())
|
||||
Expect(ops[0].UpdatedAt).ToNot(BeZero())
|
||||
})
|
||||
})
|
||||
|
||||
// An operation settles exactly once. The paths that retire one are not
|
||||
// mutually exclusive, so the first settled outcome has to be the one that
|
||||
// sticks — but a failure is not a settled outcome, because the reaper
|
||||
// writes one onto an operation that is still going to finish.
|
||||
Describe("a settled outcome is final", func() {
|
||||
It("refuses to overwrite a cancelled operation with the handler's context error", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "op-1", GalleryElementName: "localai@qwen3-4b", OpType: "model_install", Status: "downloading",
|
||||
})).To(Succeed())
|
||||
|
||||
// The cancel handler persists the outcome synchronously...
|
||||
Expect(store.Cancel("op-1")).To(Succeed())
|
||||
// ...and the handler goroutine then unwinds with the context error,
|
||||
// which Start hands to updateError unconditionally.
|
||||
Expect(store.UpdateStatus("op-1", "failed", "context canceled")).To(Succeed())
|
||||
|
||||
op, err := store.Get("op-1")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(op.Status).To(Equal("cancelled"),
|
||||
"a cancelled install must not be rewritten as a failure")
|
||||
Expect(op.Error).To(BeEmpty(), "and must not carry a raw context error for the UI to render")
|
||||
})
|
||||
|
||||
It("keeps updated_at pinned to when the operation actually finished", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "op-1", GalleryElementName: "one", OpType: "model_install", Status: "downloading",
|
||||
})).To(Succeed())
|
||||
Expect(store.UpdateStatus("op-1", "completed", "")).To(Succeed())
|
||||
|
||||
finished, err := store.Get("op-1")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
Expect(store.UpdateStatus("op-1", "failed", "late write")).To(Succeed())
|
||||
|
||||
after, err := store.Get("op-1")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(after.UpdatedAt).To(BeTemporally("==", finished.UpdatedAt),
|
||||
"a late write must not reorder the record")
|
||||
})
|
||||
|
||||
It("does not let a worker dequeue reopen an operation cancelled while queued", func() {
|
||||
// Admission writes the placeholder row...
|
||||
Expect(store.UpsertCacheKey("op-1", "localai@qwen3-4b", false)).To(Succeed())
|
||||
// ...the admin cancels before the worker gets to it...
|
||||
Expect(store.Cancel("op-1")).To(Succeed())
|
||||
// ...and the worker then dequeues it and calls Create.
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "op-1", GalleryElementName: "localai@qwen3-4b", OpType: "model_install",
|
||||
Status: "pending", Cancellable: true,
|
||||
})).To(Succeed())
|
||||
|
||||
op, err := store.Get("op-1")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(op.Status).To(Equal("cancelled"),
|
||||
"reopening it would leave a cancelled operation reading as live forever")
|
||||
Expect(op.Cancellable).To(BeFalse())
|
||||
Expect(op.OpType).To(Equal("model_install"),
|
||||
"the descriptive columns must still be filled in")
|
||||
})
|
||||
|
||||
// The gallery worker is a single goroutine consuming both channels
|
||||
// serially, so an operation queued behind a large download sits in
|
||||
// "pending" with nothing bumping updated_at. After 30 minutes the stale
|
||||
// reaper gives up on it and writes "failed" — and then the worker gets
|
||||
// to it and it installs perfectly well. A failure has to stay
|
||||
// correctable or that row is wrong forever: a red card with Retry for a
|
||||
// model that is installed, invisible to ListActive, and no longer
|
||||
// deduped by FindDuplicate.
|
||||
It("lets a real completion correct a failure the stale reaper wrote", func() {
|
||||
Expect(store.UpsertCacheKey("op-1", "localai@qwen3-4b", false)).To(Succeed())
|
||||
|
||||
reaped, err := store.CleanStale(0)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(reaped).To(Equal(int64(1)), "a queued row is what the reaper acts on")
|
||||
|
||||
// The worker finally dequeues it and runs it to completion.
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "op-1", GalleryElementName: "localai@qwen3-4b",
|
||||
OpType: "model_install", Status: "pending", Cancellable: true,
|
||||
})).To(Succeed())
|
||||
Expect(store.UpdateStatus("op-1", "completed", "")).To(Succeed())
|
||||
|
||||
op, err := store.Get("op-1")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(op.Status).To(Equal("completed"))
|
||||
Expect(op.Error).To(BeEmpty(),
|
||||
"a corrected outcome must not keep the reaper's error as its reason")
|
||||
})
|
||||
|
||||
It("still applies a first terminal status to an active row", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "op-1", GalleryElementName: "one", OpType: "model_install", Status: "downloading",
|
||||
})).To(Succeed())
|
||||
Expect(store.UpdateStatus("op-1", "failed", "no space left on device")).To(Succeed())
|
||||
|
||||
op, err := store.Get("op-1")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(op.Status).To(Equal("failed"))
|
||||
Expect(op.Error).To(Equal("no space left on device"))
|
||||
Expect(op.Cancellable).To(BeFalse())
|
||||
})
|
||||
})
|
||||
|
||||
Describe("ClearTerminal", func() {
|
||||
It("removes finished operations and leaves in-flight ones alone", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "running", GalleryElementName: "busy", OpType: "model_install", Status: "downloading",
|
||||
})).To(Succeed())
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "queued", GalleryElementName: "waiting", OpType: "model_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "done", GalleryElementName: "installed", OpType: "model_install", Status: "pending",
|
||||
})).To(Succeed())
|
||||
finish("done", "completed")
|
||||
|
||||
Expect(store.ClearTerminal()).To(Succeed())
|
||||
|
||||
terminal, err := store.ListTerminal(0)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(terminal).To(BeEmpty())
|
||||
|
||||
active, err := store.ListActive()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
ids := []string{}
|
||||
for _, op := range active {
|
||||
ids = append(ids, op.ID)
|
||||
}
|
||||
Expect(ids).To(ConsistOf("running", "queued"),
|
||||
"clearing the record must never drop an operation still in flight")
|
||||
})
|
||||
})
|
||||
})
|
||||
149
core/services/galleryop/backend_delete_optype_test.go
Normal file
149
core/services/galleryop/backend_delete_optype_test.go
Normal file
@@ -0,0 +1,149 @@
|
||||
package galleryop_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/gallery"
|
||||
"github.com/mudler/LocalAI/core/services/distributed"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
"github.com/mudler/LocalAI/core/services/testutil"
|
||||
)
|
||||
|
||||
// blockingBackendManager holds an operation open so the test can read the row
|
||||
// as it stands while the operation is in flight. That is the only window the
|
||||
// cancellable column means anything: the terminal write clears it either way.
|
||||
type blockingBackendManager struct{ release chan struct{} }
|
||||
|
||||
func (b blockingBackendManager) InstallBackend(_ context.Context, _ *galleryop.ManagementOp[gallery.GalleryBackend, any], _ galleryop.ProgressCallback) error {
|
||||
<-b.release
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b blockingBackendManager) DeleteBackend(string) error {
|
||||
<-b.release
|
||||
return nil
|
||||
}
|
||||
|
||||
func (blockingBackendManager) ListBackends() (gallery.SystemBackends, error) {
|
||||
return gallery.SystemBackends{}, nil
|
||||
}
|
||||
|
||||
func (b blockingBackendManager) UpgradeBackend(_ context.Context, _ *galleryop.ManagementOp[gallery.GalleryBackend, any], _ galleryop.ProgressCallback) error {
|
||||
<-b.release
|
||||
return nil
|
||||
}
|
||||
|
||||
func (blockingBackendManager) CheckUpgrades(_ context.Context) (map[string]gallery.UpgradeInfo, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (blockingBackendManager) IsDistributed() bool { return false }
|
||||
|
||||
// What the backend channel persists about a removal. The row is all a replica
|
||||
// that restarts mid-operation has to go on, so a removal recorded as an install
|
||||
// is a removal that renders as "Installing backend X" and offers a Cancel
|
||||
// button that does not apply to it.
|
||||
var _ = Describe("backend removal persistence", func() {
|
||||
var (
|
||||
svc *galleryop.GalleryService
|
||||
store *distributed.GalleryStore
|
||||
release chan struct{}
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
db := testutil.SetupTestDB()
|
||||
var err error
|
||||
store, err = distributed.NewGalleryStore(db)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
release = make(chan struct{})
|
||||
DeferCleanup(func() { close(release) })
|
||||
|
||||
svc = galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
svc.SetBackendManager(blockingBackendManager{release: release})
|
||||
svc.SetGalleryStore(store)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
DeferCleanup(cancel)
|
||||
Expect(svc.Start(ctx, nil, nil)).To(Succeed())
|
||||
})
|
||||
|
||||
// inFlight waits for the row the consumer writes before it enters the
|
||||
// handler, and returns it.
|
||||
inFlight := func(id string) *distributed.GalleryOperationRecord {
|
||||
var rec *distributed.GalleryOperationRecord
|
||||
Eventually(func() error {
|
||||
op, err := store.Get(id)
|
||||
rec = op
|
||||
return err
|
||||
}, "5s", "20ms").Should(Succeed())
|
||||
return rec
|
||||
}
|
||||
|
||||
It("records a removal as a removal", func() {
|
||||
svc.EnqueueBackendOp(galleryop.ManagementOp[gallery.GalleryBackend, any]{
|
||||
ID: "job-del",
|
||||
GalleryElementName: "vllm",
|
||||
Delete: true,
|
||||
})
|
||||
|
||||
Expect(inFlight("job-del").OpType).To(Equal("backend_delete"))
|
||||
})
|
||||
|
||||
It("still records an install as an install", func() {
|
||||
svc.EnqueueBackendOp(galleryop.ManagementOp[gallery.GalleryBackend, any]{
|
||||
ID: "job-install",
|
||||
GalleryElementName: "vllm",
|
||||
})
|
||||
|
||||
Expect(inFlight("job-install").OpType).To(Equal("backend_install"))
|
||||
})
|
||||
|
||||
// Hydrate is the only reader that decides from op_type whether an operation
|
||||
// is a removal, and it is what a replica restarting mid-operation runs.
|
||||
It("hydrates a backend removal as a removal", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "job-hydrate", GalleryElementName: "vllm",
|
||||
OpType: "backend_delete", Status: "processing",
|
||||
})).To(Succeed())
|
||||
|
||||
fresh := galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
fresh.SetGalleryStore(store)
|
||||
Expect(fresh.Hydrate()).To(Succeed())
|
||||
|
||||
st := fresh.GetStatus("job-hydrate")
|
||||
Expect(st).ToNot(BeNil())
|
||||
Expect(st.Deletion).To(BeTrue(),
|
||||
"a hydrated backend removal reported as an install renders as 'Installing backend vllm'")
|
||||
})
|
||||
|
||||
It("still hydrates a model removal as a removal", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "job-hydrate", GalleryElementName: "localai@qwen3-4b",
|
||||
OpType: "model_delete", Status: "processing",
|
||||
})).To(Succeed())
|
||||
|
||||
fresh := galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
fresh.SetGalleryStore(store)
|
||||
Expect(fresh.Hydrate()).To(Succeed())
|
||||
|
||||
Expect(fresh.GetStatus("job-hydrate").Deletion).To(BeTrue())
|
||||
})
|
||||
|
||||
It("does not report an install as a removal", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "job-hydrate", GalleryElementName: "vllm",
|
||||
OpType: "backend_install", Status: "processing",
|
||||
})).To(Succeed())
|
||||
|
||||
fresh := galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
fresh.SetGalleryStore(store)
|
||||
Expect(fresh.Hydrate()).To(Succeed())
|
||||
|
||||
Expect(fresh.GetStatus("job-hydrate").Deletion).To(BeFalse())
|
||||
})
|
||||
})
|
||||
@@ -48,7 +48,13 @@ func (g *GalleryService) backendHandler(op *ManagementOp[gallery.GalleryBackend,
|
||||
}
|
||||
}
|
||||
|
||||
g.UpdateStatus(op.ID, &OpStatus{Message: fmt.Sprintf("processing backend: %s", op.GalleryElementName), Progress: 0, Cancellable: true})
|
||||
// Starting the operation NARROWS what can be cancelled, which is the reverse
|
||||
// of the usual shape: markQueued reports every queued op as cancellable
|
||||
// because abandoning it before the worker takes it leaves no trace. From
|
||||
// here on that only holds for an install or an upgrade, both of which take
|
||||
// ctx. DeleteBackend takes no context and cannot be interrupted, so a Cancel
|
||||
// button on a running removal is one the server cannot honour.
|
||||
g.UpdateStatus(op.ID, &OpStatus{Message: fmt.Sprintf("processing backend: %s", op.GalleryElementName), Progress: 0, Cancellable: !op.Delete})
|
||||
|
||||
// displayDownload displays the download progress
|
||||
progressCallback := func(fileName string, current string, total string, percentage float64) {
|
||||
|
||||
199
core/services/galleryop/cancellable_phase_test.go
Normal file
199
core/services/galleryop/cancellable_phase_test.go
Normal file
@@ -0,0 +1,199 @@
|
||||
package galleryop_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/gallery"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
)
|
||||
|
||||
// gatedModelManager records the element name of every operation the worker
|
||||
// actually starts and then parks inside the handler until the gate is released.
|
||||
// Parking is what makes the running phase observable at all: without it the
|
||||
// handler-entry status is overwritten by the terminal write before a spec can
|
||||
// read it.
|
||||
type gatedModelManager struct {
|
||||
mu sync.Mutex
|
||||
started []string
|
||||
gate chan struct{}
|
||||
}
|
||||
|
||||
func newGatedModelManager() *gatedModelManager {
|
||||
return &gatedModelManager{gate: make(chan struct{})}
|
||||
}
|
||||
|
||||
func (m *gatedModelManager) record(name string) {
|
||||
m.mu.Lock()
|
||||
m.started = append(m.started, name)
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func (m *gatedModelManager) Started() []string {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return append([]string(nil), m.started...)
|
||||
}
|
||||
|
||||
func (m *gatedModelManager) InstallModel(_ context.Context, op *galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig], _ galleryop.ProgressCallback) error {
|
||||
m.record(op.GalleryElementName)
|
||||
<-m.gate
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *gatedModelManager) DeleteModel(name string) error {
|
||||
m.record(name)
|
||||
<-m.gate
|
||||
return nil
|
||||
}
|
||||
|
||||
// An operation is cancellable for exactly as long as cancelling it can still
|
||||
// change the outcome, and that window is not the one the shape of the UI
|
||||
// suggests. A queued operation of ANY kind is cancellable: the delivery
|
||||
// goroutine is released by the operation context and abandons the op before the
|
||||
// worker ever sees it, so nothing has happened yet and nothing needs undoing. A
|
||||
// running removal is not: DeleteModel and DeleteBackend take no context, so once
|
||||
// the worker has entered the handler the files are going away regardless.
|
||||
// Reporting these backwards gave the admin a Cancel button on the removal that
|
||||
// could not honour it, and hid the button on the queued removal that could.
|
||||
var _ = Describe("operation cancellability by phase", func() {
|
||||
var svc *galleryop.GalleryService
|
||||
|
||||
BeforeEach(func() {
|
||||
svc = galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
})
|
||||
|
||||
// Nothing consumes the channel in this context, which is exactly the state
|
||||
// of an operation admitted while the serial worker is busy elsewhere.
|
||||
Context("while the operation is still queued", func() {
|
||||
It("reports a queued removal as cancellable", func() {
|
||||
svc.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: "job-queued-removal",
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
Delete: true,
|
||||
})
|
||||
|
||||
Eventually(func() *galleryop.OpStatus {
|
||||
return svc.GetStatus("job-queued-removal")
|
||||
}, "2s", "10ms").ShouldNot(BeNil())
|
||||
|
||||
st := svc.GetStatus("job-queued-removal")
|
||||
Expect(st.Deletion).To(BeTrue())
|
||||
Expect(st.Cancellable).To(BeTrue(),
|
||||
"a removal that has not reached the worker can still be called off with no trace")
|
||||
})
|
||||
|
||||
It("reports a queued install as cancellable", func() {
|
||||
svc.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: "job-queued-install",
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
})
|
||||
|
||||
Eventually(func() *galleryop.OpStatus {
|
||||
return svc.GetStatus("job-queued-install")
|
||||
}, "2s", "10ms").ShouldNot(BeNil())
|
||||
|
||||
Expect(svc.GetStatus("job-queued-install").Cancellable).To(BeTrue())
|
||||
})
|
||||
|
||||
It("reports a queued backend removal as cancellable", func() {
|
||||
svc.EnqueueBackendOp(galleryop.ManagementOp[gallery.GalleryBackend, any]{
|
||||
ID: "job-queued-backend-removal",
|
||||
GalleryElementName: "vllm",
|
||||
Delete: true,
|
||||
})
|
||||
|
||||
Eventually(func() *galleryop.OpStatus {
|
||||
return svc.GetStatus("job-queued-backend-removal")
|
||||
}, "2s", "10ms").ShouldNot(BeNil())
|
||||
|
||||
Expect(svc.GetStatus("job-queued-backend-removal").Cancellable).To(BeTrue())
|
||||
})
|
||||
})
|
||||
|
||||
Context("once the worker has started the operation", func() {
|
||||
var manager *gatedModelManager
|
||||
|
||||
BeforeEach(func() {
|
||||
manager = newGatedModelManager()
|
||||
DeferCleanup(func() { close(manager.gate) })
|
||||
|
||||
svc.SetModelManager(manager)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
DeferCleanup(cancel)
|
||||
Expect(svc.Start(ctx, nil, nil)).To(Succeed())
|
||||
})
|
||||
|
||||
It("reports a running removal as not cancellable", func() {
|
||||
svc.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: "job-running-removal",
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
Delete: true,
|
||||
})
|
||||
|
||||
Eventually(manager.Started, "5s", "20ms").Should(ContainElement("localai@qwen3-4b"))
|
||||
|
||||
st := svc.GetStatus("job-running-removal")
|
||||
Expect(st).ToNot(BeNil())
|
||||
Expect(st.Cancellable).To(BeFalse(),
|
||||
"DeleteModel takes no context, so a Cancel button here cannot stop anything")
|
||||
})
|
||||
|
||||
It("reports a running install as cancellable", func() {
|
||||
svc.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: "job-running-install",
|
||||
GalleryElementName: "localai@qwen3-4b",
|
||||
})
|
||||
|
||||
Eventually(manager.Started, "5s", "20ms").Should(ContainElement("localai@qwen3-4b"))
|
||||
|
||||
st := svc.GetStatus("job-running-install")
|
||||
Expect(st).ToNot(BeNil())
|
||||
Expect(st.Cancellable).To(BeTrue(),
|
||||
"an in-flight download is cancelled through the operation context")
|
||||
})
|
||||
|
||||
// The behaviour the whole asymmetry rests on: if cancelling a queued
|
||||
// removal did not actually stop it, reporting it as cancellable would be
|
||||
// the same lie in the other direction.
|
||||
It("keeps a cancelled queued removal from ever reaching the worker", func() {
|
||||
svc.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: "job-occupying-worker",
|
||||
GalleryElementName: "localai@long-install",
|
||||
})
|
||||
Eventually(manager.Started, "5s", "20ms").Should(ContainElement("localai@long-install"))
|
||||
|
||||
opCtx, cancelOp := context.WithCancel(context.Background())
|
||||
svc.EnqueueModelOp(galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{
|
||||
ID: "job-cancelled-while-queued",
|
||||
GalleryElementName: "localai@doomed-removal",
|
||||
Delete: true,
|
||||
Context: opCtx,
|
||||
CancelFunc: cancelOp,
|
||||
})
|
||||
Eventually(func() *galleryop.OpStatus {
|
||||
return svc.GetStatus("job-cancelled-while-queued")
|
||||
}, "2s", "10ms").ShouldNot(BeNil())
|
||||
|
||||
cancelOp()
|
||||
|
||||
Eventually(func() bool {
|
||||
st := svc.GetStatus("job-cancelled-while-queued")
|
||||
return st != nil && st.Processed
|
||||
}, "2s", "10ms").Should(BeTrue(), "abandonQueued must retire the op the worker never took")
|
||||
|
||||
// Free the worker: it loops back to an empty channel because the
|
||||
// delivery goroutine gave up on the send.
|
||||
close(manager.gate)
|
||||
manager.gate = make(chan struct{})
|
||||
|
||||
Consistently(manager.Started, "500ms", "20ms").ShouldNot(
|
||||
ContainElement("localai@doomed-removal"),
|
||||
"a removal cancelled while queued must never delete anything")
|
||||
})
|
||||
})
|
||||
})
|
||||
58
core/services/galleryop/deletion_flag_test.go
Normal file
58
core/services/galleryop/deletion_flag_test.go
Normal file
@@ -0,0 +1,58 @@
|
||||
package galleryop_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
)
|
||||
|
||||
// Whether a job is a removal or an install is decided once, at admission, and
|
||||
// never restated. markQueued is the only writer that sets Deletion; every
|
||||
// status the worker publishes afterwards leaves the field at its zero value.
|
||||
// Without carry-forward the flag survives only the queued window, which is the
|
||||
// one window no consumer reads it in.
|
||||
var _ = Describe("OpStatus.Deletion carry-forward", func() {
|
||||
var svc *galleryop.GalleryService
|
||||
|
||||
BeforeEach(func() {
|
||||
svc = galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
})
|
||||
|
||||
It("keeps the flag set once the worker starts the removal", func() {
|
||||
svc.UpdateStatus("op-del", &galleryop.OpStatus{
|
||||
Message: galleryop.PhaseQueued, Phase: galleryop.PhaseQueued,
|
||||
GalleryElementName: "localai@qwen3-4b", Deletion: true,
|
||||
})
|
||||
|
||||
// The worker's first write, verbatim from deleteModel/deleteBackend: a
|
||||
// fresh OpStatus that says nothing about what kind of job this is.
|
||||
svc.UpdateStatus("op-del", &galleryop.OpStatus{Message: "processing model: localai@qwen3-4b", Cancellable: true})
|
||||
|
||||
Expect(svc.GetStatus("op-del").Deletion).To(BeTrue(), "a running removal must still report as a removal")
|
||||
})
|
||||
|
||||
It("keeps the flag set when the removal fails", func() {
|
||||
svc.UpdateStatus("op-del", &galleryop.OpStatus{Deletion: true, Phase: galleryop.PhaseQueued})
|
||||
svc.UpdateStatus("op-del", &galleryop.OpStatus{Message: "processing model: localai@qwen3-4b", Cancellable: true})
|
||||
|
||||
// The failure path (updateError in Start) publishes only the error.
|
||||
svc.UpdateStatus("op-del", &galleryop.OpStatus{
|
||||
Error: errors.New("permission denied"), Processed: true, Message: "error: permission denied",
|
||||
})
|
||||
|
||||
st := svc.GetStatus("op-del")
|
||||
Expect(st.Deletion).To(BeTrue(), "a failed removal must not be mistaken for a failed install: Retry would re-download the model")
|
||||
Expect(st.Error).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("never turns an install into a deletion", func() {
|
||||
svc.UpdateStatus("op-install", &galleryop.OpStatus{Phase: galleryop.PhaseQueued})
|
||||
svc.UpdateStatus("op-install", &galleryop.OpStatus{Progress: 42, Cancellable: true})
|
||||
|
||||
Expect(svc.GetStatus("op-install").Deletion).To(BeFalse())
|
||||
})
|
||||
})
|
||||
@@ -8,9 +8,21 @@ import (
|
||||
"github.com/mudler/xlog"
|
||||
)
|
||||
|
||||
// queuedMessage is the status message an operation carries between admission
|
||||
// (the HTTP handler minting the job ID) and the moment the worker starts it.
|
||||
const queuedMessage = "queued"
|
||||
// PhaseQueued is the phase (and the status message) an operation carries
|
||||
// between admission (the HTTP handler minting the job ID) and the moment the
|
||||
// worker starts it.
|
||||
//
|
||||
// It is the ONLY signal for "this op has not started yet". A missing status is
|
||||
// not one: markQueued publishes a status at admission, so an op that is queued
|
||||
// behind a running install has a status for its whole queued life.
|
||||
const PhaseQueued = "queued"
|
||||
|
||||
// IsQueued reports whether the operation is still waiting for the gallery
|
||||
// worker to pick it up. Nil-safe so callers holding a status that may not
|
||||
// exist can ask without a guard of their own.
|
||||
func (o *OpStatus) IsQueued() bool {
|
||||
return o != nil && !o.Processed && o.Phase == PhaseQueued
|
||||
}
|
||||
|
||||
// EnqueueModelOp admits a model operation: it registers a queryable "queued"
|
||||
// status for op.ID and then hands the op to the gallery worker.
|
||||
@@ -65,16 +77,26 @@ func enqueueContext(ctx context.Context) context.Context {
|
||||
}
|
||||
|
||||
// markQueued publishes the pre-worker status for an admitted operation.
|
||||
//
|
||||
// Cancellable is unconditional here, removals included, which is the opposite
|
||||
// way round from the running phase (see the handler-entry writes in models.go
|
||||
// and backends.go). The asymmetry is real: cancelling a queued op releases the
|
||||
// delivery goroutine above and abandonQueued retires it, so the worker never
|
||||
// sees it and nothing is touched, so it is safe and effective for a removal as
|
||||
// it is for an install. It is once the worker starts a removal that cancelling
|
||||
// becomes impossible, because DeleteModel and DeleteBackend take no context.
|
||||
// Reporting a queued removal as uncancellable hid the Cancel button in the one
|
||||
// window where it would have worked.
|
||||
func (g *GalleryService) markQueued(id, elementName string, deletion bool) {
|
||||
if id == "" {
|
||||
return
|
||||
}
|
||||
g.UpdateStatus(id, &OpStatus{
|
||||
Message: queuedMessage,
|
||||
Phase: queuedMessage,
|
||||
Message: PhaseQueued,
|
||||
Phase: PhaseQueued,
|
||||
GalleryElementName: elementName,
|
||||
Deletion: deletion,
|
||||
Cancellable: !deletion,
|
||||
Cancellable: true,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
11
core/services/galleryop/export_test.go
Normal file
11
core/services/galleryop/export_test.go
Normal file
@@ -0,0 +1,11 @@
|
||||
package galleryop
|
||||
|
||||
// Test-only seams onto unexported behaviour. Compiled into the test binary
|
||||
// only, so nothing here reaches the shipped surface.
|
||||
|
||||
// ApplyEndForTest drives the NATS end path without standing up a broker.
|
||||
// applyEnd is unexported and only ever reached from a subscription callback,
|
||||
// which an external test package cannot trigger.
|
||||
func (m *OpCache) ApplyEndForTest(jobID string) {
|
||||
m.applyEnd(OpCacheEvent{JobID: jobID})
|
||||
}
|
||||
163
core/services/galleryop/history.go
Normal file
163
core/services/galleryop/history.go
Normal file
@@ -0,0 +1,163 @@
|
||||
package galleryop
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/distributed"
|
||||
"github.com/mudler/xlog"
|
||||
)
|
||||
|
||||
// DefaultHistorySize bounds the in-memory record of finished operations. The
|
||||
// Activity page is a "what just happened" view, not an audit log: 50 entries
|
||||
// covers a full onboarding session and costs a few kilobytes. Durable history
|
||||
// belongs in distributed.GalleryStore, which already persists terminal status.
|
||||
const DefaultHistorySize = 50
|
||||
|
||||
// OpRecord is a finished operation. Field names match the JSON the Activity
|
||||
// page consumes, so the handler can return the slice unwrapped.
|
||||
type OpRecord struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
JobID string `json:"jobID"`
|
||||
IsBackend bool `json:"isBackend"`
|
||||
NodeID string `json:"nodeID,omitempty"`
|
||||
TaskType string `json:"taskType"`
|
||||
Outcome string `json:"outcome"`
|
||||
Error string `json:"error,omitempty"`
|
||||
StartedAt time.Time `json:"startedAt"`
|
||||
FinishedAt time.Time `json:"finishedAt"`
|
||||
}
|
||||
|
||||
// Outcome values.
|
||||
const (
|
||||
OutcomeCompleted = "completed"
|
||||
OutcomeFailed = "failed"
|
||||
OutcomeCancelled = "cancelled"
|
||||
)
|
||||
|
||||
// IsDeleteOpType reports whether a persisted op_type describes a removal, and
|
||||
// IsBackendOpType whether it describes a backend rather than a model.
|
||||
//
|
||||
// Every reader that discriminates on op_type goes through these. Testing for
|
||||
// one specific value instead is how a backend removal came to be reported as an
|
||||
// installation: adding a fourth op_type has to be visible to every consumer at
|
||||
// once, not silently default to "install" in whichever one was missed.
|
||||
func IsDeleteOpType(opType string) bool { return strings.HasSuffix(opType, "_delete") }
|
||||
func IsBackendOpType(opType string) bool { return strings.HasPrefix(opType, "backend_") }
|
||||
|
||||
// recordFromStore maps a persisted gallery operation onto the record shape the
|
||||
// Activity page consumes, so the store-backed and ring-backed reads of the same
|
||||
// operation are indistinguishable to the page.
|
||||
func recordFromStore(op distributed.GalleryOperationRecord) OpRecord {
|
||||
// UpsertCacheKey lands when the request is admitted, which is after the row
|
||||
// exists: an operation that failed in between has a row and no cache key.
|
||||
// The gallery element name is what it was asked for by, and is the closest
|
||||
// stand-in the row carries.
|
||||
key := op.CacheKey
|
||||
if key == "" {
|
||||
key = op.GalleryElementName
|
||||
}
|
||||
name, nodeID := operationDisplayName(key)
|
||||
|
||||
rec := OpRecord{
|
||||
ID: key,
|
||||
Name: name,
|
||||
// IsBackendOp is only ever written by UpsertCacheKey, so the same rows
|
||||
// that need the name fallback above would report a backend operation as
|
||||
// a model one. op_type is written by a different path and answers the
|
||||
// same question, so either signal is enough.
|
||||
IsBackend: op.IsBackendOp || IsBackendOpType(op.OpType),
|
||||
JobID: op.ID,
|
||||
NodeID: nodeID,
|
||||
TaskType: "installation",
|
||||
Error: op.Error,
|
||||
StartedAt: op.CreatedAt,
|
||||
FinishedAt: op.UpdatedAt,
|
||||
}
|
||||
if IsDeleteOpType(op.OpType) {
|
||||
rec.TaskType = "deletion"
|
||||
}
|
||||
|
||||
// ListTerminal only returns these three statuses, and they are the same
|
||||
// three strings as the Outcome constants. Mapping them explicitly means a
|
||||
// status that is somehow neither gets logged rather than quietly filed as a
|
||||
// success.
|
||||
switch op.Status {
|
||||
case OutcomeFailed:
|
||||
rec.Outcome = OutcomeFailed
|
||||
case OutcomeCancelled:
|
||||
rec.Outcome = OutcomeCancelled
|
||||
case OutcomeCompleted:
|
||||
rec.Outcome = OutcomeCompleted
|
||||
default:
|
||||
xlog.Warn("unknown terminal gallery operation status; recording it as completed",
|
||||
"job_id", op.ID, "status", op.Status)
|
||||
rec.Outcome = OutcomeCompleted
|
||||
}
|
||||
return rec
|
||||
}
|
||||
|
||||
// opHistory is a bounded, deduped ring of finished operations, oldest first.
|
||||
// It knows nothing about jobs or statuses so it can be tested on its own.
|
||||
type opHistory struct {
|
||||
mu sync.Mutex
|
||||
records []OpRecord
|
||||
seen map[string]struct{}
|
||||
limit int
|
||||
}
|
||||
|
||||
func newOpHistory(limit int) *opHistory {
|
||||
return &opHistory{
|
||||
records: make([]OpRecord, 0, limit),
|
||||
seen: make(map[string]struct{}, limit),
|
||||
limit: limit,
|
||||
}
|
||||
}
|
||||
|
||||
// add appends rec unless its job ID was already recorded. Returns false when
|
||||
// the record was a duplicate. The originating replica both evicts locally and
|
||||
// receives its own NATS end broadcast, so without this every distributed
|
||||
// operation would be recorded twice.
|
||||
func (h *opHistory) add(rec OpRecord) bool {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
|
||||
if _, dup := h.seen[rec.JobID]; dup {
|
||||
return false
|
||||
}
|
||||
h.seen[rec.JobID] = struct{}{}
|
||||
h.records = append(h.records, rec)
|
||||
|
||||
// Drop the evicted record's job ID alongside it, so seen stays bounded by
|
||||
// limit rather than growing for the lifetime of the process. A duplicate
|
||||
// arriving 50 operations late would be re-added, which is both vanishingly
|
||||
// unlikely and harmless.
|
||||
for len(h.records) > h.limit {
|
||||
delete(h.seen, h.records[0].JobID)
|
||||
h.records = h.records[1:]
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// list returns a newest-first copy, so callers cannot mutate the ring and the
|
||||
// page does not have to sort.
|
||||
func (h *opHistory) list() []OpRecord {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
|
||||
out := make([]OpRecord, 0, len(h.records))
|
||||
for i := len(h.records) - 1; i >= 0; i-- {
|
||||
out = append(out, h.records[i])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *opHistory) clear() {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
|
||||
h.records = h.records[:0]
|
||||
h.seen = make(map[string]struct{}, h.limit)
|
||||
}
|
||||
177
core/services/galleryop/history_ring_test.go
Normal file
177
core/services/galleryop/history_ring_test.go
Normal file
@@ -0,0 +1,177 @@
|
||||
package galleryop
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
)
|
||||
|
||||
// The ring's dedupe and its bounded seen set are not reachable deterministically
|
||||
// through the exported API: a second DeleteUUID for the same job finds no cache
|
||||
// key and returns before add is ever called. They are driven directly here so
|
||||
// the guards are pinned rather than incidentally uncovered.
|
||||
var _ = Describe("opHistory", func() {
|
||||
It("refuses a job ID it already recorded", func() {
|
||||
h := newOpHistory(DefaultHistorySize)
|
||||
|
||||
Expect(h.add(OpRecord{JobID: "job-1", Name: "llama-cpp"})).To(BeTrue())
|
||||
Expect(h.add(OpRecord{JobID: "job-1", Name: "llama-cpp"})).To(BeFalse())
|
||||
Expect(h.list()).To(HaveLen(1))
|
||||
})
|
||||
|
||||
It("forgets a job ID once its record is evicted, so seen stays bounded", func() {
|
||||
h := newOpHistory(3)
|
||||
|
||||
for i := 1; i <= 4; i++ {
|
||||
Expect(h.add(OpRecord{JobID: fmt.Sprintf("job-%d", i)})).To(BeTrue())
|
||||
}
|
||||
Expect(h.list()).To(HaveLen(3))
|
||||
|
||||
// job-1 fell off the back, so its ID is free again. If seen kept every
|
||||
// ID ever added it would grow for the lifetime of the process.
|
||||
Expect(h.add(OpRecord{JobID: "job-1"})).To(BeTrue())
|
||||
Expect(h.list()).To(HaveLen(3))
|
||||
})
|
||||
|
||||
It("clears the seen set alongside the records", func() {
|
||||
h := newOpHistory(DefaultHistorySize)
|
||||
|
||||
Expect(h.add(OpRecord{JobID: "job-1"})).To(BeTrue())
|
||||
h.clear()
|
||||
|
||||
Expect(h.list()).To(BeEmpty())
|
||||
Expect(h.add(OpRecord{JobID: "job-1"})).To(BeTrue())
|
||||
})
|
||||
})
|
||||
|
||||
// Recording paths that only the in-package tests can reach, because they start
|
||||
// from an OpCache entry that Set/SetBackend never created.
|
||||
var _ = Describe("OpCache terminal recording (internal)", func() {
|
||||
var (
|
||||
svc *GalleryService
|
||||
cache *OpCache
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
svc = NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
cache = NewOpCache(svc)
|
||||
})
|
||||
|
||||
It("falls back to the finish time when the start was never stamped", func() {
|
||||
// hydrateFromStore and applyStart write status directly, so an op
|
||||
// recovered from PostgreSQL or replicated from a peer has no stamp. A
|
||||
// zero StartedAt would render as a two-millennia duration on the
|
||||
// Activity page, so the record reports a zero-length op instead.
|
||||
cache.applyStart(OpCacheEvent{CacheKey: "vllm", JobID: "job-hydrated", IsBackend: true})
|
||||
svc.UpdateStatus("job-hydrated", &OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
cache.DeleteUUID("job-hydrated")
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].StartedAt).NotTo(BeZero())
|
||||
Expect(history[0].StartedAt).To(Equal(history[0].FinishedAt))
|
||||
})
|
||||
|
||||
It("drops the previous job's start stamp when a key is reused", func() {
|
||||
// Reinstalling the same backend reuses the cache key with a fresh job
|
||||
// ID. Without the drop, every retry would orphan a timestamp.
|
||||
cache.Set("qwen3-8b", "job-old")
|
||||
cache.Set("qwen3-8b", "job-new")
|
||||
|
||||
Expect(cache.started.Exists("job-old")).To(BeFalse())
|
||||
Expect(cache.started.Exists("job-new")).To(BeTrue())
|
||||
})
|
||||
|
||||
It("drops the start stamp once the operation is recorded", func() {
|
||||
cache.SetBackend("piper", "job-done")
|
||||
svc.UpdateStatus("job-done", &OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
cache.DeleteUUID("job-done")
|
||||
|
||||
Expect(cache.started.Exists("job-done")).To(BeFalse())
|
||||
})
|
||||
|
||||
It("drops the previous job's start stamp when a peer reuses the cache key", func() {
|
||||
// A retry admitted on another replica reaches us as a start event for
|
||||
// the same key with a fresh job ID. The stamp the key used to point at
|
||||
// is now unreachable, so it has to go with it.
|
||||
cache.Set("qwen3-8b", "job-local")
|
||||
|
||||
cache.applyStart(OpCacheEvent{CacheKey: "qwen3-8b", JobID: "job-peer"})
|
||||
|
||||
Expect(cache.started.Exists("job-local")).To(BeFalse())
|
||||
})
|
||||
|
||||
It("drops the start stamp when the end event finds no cache key", func() {
|
||||
// The end broadcast can land before the local Set has written its status
|
||||
// key, and recordTerminal returns before its own stamp cleanup when no
|
||||
// key maps to the job. The operation is over cluster-wide either way, so
|
||||
// the stamp must not outlive it.
|
||||
cache.started.Set("job-raced", time.Now())
|
||||
|
||||
cache.ApplyEndForTest("job-raced")
|
||||
|
||||
Expect(cache.started.Exists("job-raced")).To(BeFalse())
|
||||
})
|
||||
|
||||
It("records nothing for an end event whose gallery status is gone", func() {
|
||||
// A replica that restarted mid-operation hydrates its cache keys from
|
||||
// PostgreSQL, but gallery statuses live in memory only and come back
|
||||
// empty. A missing status on this path means the outcome was lost, not
|
||||
// that the op never started, so reading it as cancelled would file a
|
||||
// successful install as a failure.
|
||||
cache.applyStart(OpCacheEvent{CacheKey: "vllm", JobID: "job-restarted", IsBackend: true})
|
||||
|
||||
cache.ApplyEndForTest("job-restarted")
|
||||
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("still records a locally dismissed op with no status as cancelled", func() {
|
||||
// The local reading of a missing status is unchanged: nothing ever set
|
||||
// one, so the op was queued, never started, then removed.
|
||||
cache.Set("qwen3-4b", "job-queued")
|
||||
|
||||
cache.DeleteUUID("job-queued")
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Outcome).To(Equal(OutcomeCancelled))
|
||||
})
|
||||
|
||||
It("records one entry when both terminal paths reach the ring", func() {
|
||||
// The external spec that fires DeleteUUID then the broadcast does not
|
||||
// reach the ring twice: the first call removes the status keys, so the
|
||||
// second returns before add. recordTerminal leaves the keys alone, so
|
||||
// calling it twice is what actually exercises the dedupe.
|
||||
cache.SetBackend("whisper-cpp", "job-twice")
|
||||
svc.UpdateStatus("job-twice", &OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
cache.recordTerminal("job-twice", terminalLocal)
|
||||
cache.recordTerminal("job-twice", terminalPeer)
|
||||
|
||||
Expect(cache.History()).To(HaveLen(1))
|
||||
})
|
||||
|
||||
It("keeps the finish-time fallback when the start stamp reads as zero", func() {
|
||||
// A concurrent recordTerminal for the same job deletes the stamp, and a
|
||||
// missing key reads back as the zero time. Letting that zero win over the
|
||||
// fallback serialises StartedAt as year one, which the Activity page
|
||||
// renders as a two-millennia run.
|
||||
cache.SetBackend("bark-cpp", "job-zero")
|
||||
cache.started.Set("job-zero", time.Time{})
|
||||
svc.UpdateStatus("job-zero", &OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
cache.DeleteUUID("job-zero")
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].StartedAt).NotTo(BeZero())
|
||||
Expect(history[0].StartedAt).To(Equal(history[0].FinishedAt))
|
||||
})
|
||||
})
|
||||
270
core/services/galleryop/history_store_test.go
Normal file
270
core/services/galleryop/history_store_test.go
Normal file
@@ -0,0 +1,270 @@
|
||||
package galleryop_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"gorm.io/gorm"
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/services/distributed"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
"github.com/mudler/LocalAI/core/services/testutil"
|
||||
)
|
||||
|
||||
// The record the Activity page reads. In standalone mode it is the in-memory
|
||||
// ring; in distributed mode it must come from PostgreSQL, or each replica
|
||||
// answers with its own private history and a replica added by a scale-out
|
||||
// answers with nothing.
|
||||
var _ = Describe("OpCache history source", func() {
|
||||
var (
|
||||
svc *galleryop.GalleryService
|
||||
cache *galleryop.OpCache
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
svc = galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
cache = galleryop.NewOpCache(svc)
|
||||
})
|
||||
|
||||
// recordLocally drives an operation through the local terminal path, which
|
||||
// is the only thing that puts an entry in the in-memory ring.
|
||||
recordLocally := func(key, jobID string) {
|
||||
cache.SetBackend(key, jobID)
|
||||
svc.UpdateStatus(jobID, &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
cache.DeleteUUID(jobID)
|
||||
}
|
||||
|
||||
Context("standalone, with no gallery store wired", func() {
|
||||
It("answers from the in-memory ring", func() {
|
||||
recordLocally("llama-cpp", "job-local")
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].JobID).To(Equal("job-local"))
|
||||
Expect(history[0].Name).To(Equal("llama-cpp"))
|
||||
})
|
||||
|
||||
It("clears the in-memory ring", func() {
|
||||
recordLocally("llama-cpp", "job-local")
|
||||
Expect(cache.ClearHistory()).To(Succeed())
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
Context("distributed, with a gallery store wired", func() {
|
||||
var (
|
||||
store *distributed.GalleryStore
|
||||
db *gorm.DB
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
db = testutil.SetupTestDB()
|
||||
var err error
|
||||
store, err = distributed.NewGalleryStore(db)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
cache.SetGalleryStore(store)
|
||||
})
|
||||
|
||||
// breakStore makes every subsequent store read and write fail, which is
|
||||
// what a database outage looks like from the OpCache's side.
|
||||
breakStore := func() {
|
||||
Expect(db.Migrator().DropTable(&distributed.GalleryOperationRecord{})).To(Succeed())
|
||||
}
|
||||
|
||||
// seed writes a finished operation the way a peer replica would have
|
||||
// left it: a row created for the job, then driven to a terminal status.
|
||||
seed := func(jobID, cacheKey, elementName, opType, status, errMsg string, isBackend bool) {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: jobID,
|
||||
GalleryElementName: elementName,
|
||||
OpType: opType,
|
||||
Status: "pending",
|
||||
})).To(Succeed())
|
||||
if cacheKey != "" {
|
||||
Expect(store.UpsertCacheKey(jobID, cacheKey, isBackend)).To(Succeed())
|
||||
}
|
||||
Expect(store.UpdateStatus(jobID, status, errMsg)).To(Succeed())
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
|
||||
It("answers from the store rather than the local ring", func() {
|
||||
// A local operation this replica happened to serve, plus one it
|
||||
// never saw. Only the store knows about both.
|
||||
recordLocally("llama-cpp", "job-local")
|
||||
seed("job-peer", "localai@qwen3-4b", "localai@qwen3-4b", "model_install", "completed", "", false)
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1),
|
||||
"the store is the whole record; the ring must not be merged in on top of it")
|
||||
Expect(history[0].JobID).To(Equal("job-peer"))
|
||||
Expect(history[0].ID).To(Equal("localai@qwen3-4b"))
|
||||
Expect(history[0].Name).To(Equal("qwen3-4b"), "the gallery prefix is not part of the name")
|
||||
Expect(history[0].Outcome).To(Equal(galleryop.OutcomeCompleted))
|
||||
Expect(history[0].TaskType).To(Equal("installation"))
|
||||
Expect(history[0].IsBackend).To(BeFalse())
|
||||
Expect(history[0].NodeID).To(BeEmpty())
|
||||
Expect(history[0].StartedAt).ToNot(BeZero())
|
||||
Expect(history[0].FinishedAt).ToNot(BeZero())
|
||||
})
|
||||
|
||||
It("reports a removal as a deletion", func() {
|
||||
seed("job-del", "localai@qwen3-4b", "localai@qwen3-4b", "model_delete", "completed", "", false)
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].TaskType).To(Equal("deletion"))
|
||||
})
|
||||
|
||||
It("reports a backend removal as a deletion too", func() {
|
||||
seed("job-bdel", "vllm", "vllm", "backend_delete", "completed", "", true)
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].TaskType).To(Equal("deletion"))
|
||||
Expect(history[0].IsBackend).To(BeTrue())
|
||||
})
|
||||
|
||||
It("splits a node-scoped key into the bare backend name and the node", func() {
|
||||
seed("job-node", galleryop.NodeScopedKey("worker-7", "llama-cpp"), "llama-cpp",
|
||||
"backend_install", "completed", "", true)
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Name).To(Equal("llama-cpp"))
|
||||
Expect(history[0].NodeID).To(Equal("worker-7"))
|
||||
Expect(history[0].IsBackend).To(BeTrue())
|
||||
})
|
||||
|
||||
It("falls back to the gallery element name when the cache key never landed", func() {
|
||||
// UpsertCacheKey runs on admission, after the row exists; a job that
|
||||
// failed in between has a row and no cache key.
|
||||
seed("job-nokey", "", "localai@qwen3-4b", "model_install", "failed", "disk full", false)
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].ID).To(Equal("localai@qwen3-4b"))
|
||||
Expect(history[0].Name).To(Equal("qwen3-4b"))
|
||||
Expect(history[0].Outcome).To(Equal(galleryop.OutcomeFailed))
|
||||
Expect(history[0].Error).To(Equal("disk full"))
|
||||
})
|
||||
|
||||
It("reads a backend operation off the op type when the cache key never landed", func() {
|
||||
// is_backend_op is only ever written by UpsertCacheKey, so the rows
|
||||
// that need the name fallback would report a backend operation as a
|
||||
// model one and file it under the wrong page filter.
|
||||
seed("job-nokey", "", "vllm", "backend_install", "failed", "oci pull failed", false)
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].IsBackend).To(BeTrue())
|
||||
})
|
||||
|
||||
It("carries a cancellation through as cancelled", func() {
|
||||
seed("job-cancel", "localai@qwen3-4b", "localai@qwen3-4b", "model_install", "cancelled", "", false)
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Outcome).To(Equal(galleryop.OutcomeCancelled))
|
||||
})
|
||||
|
||||
It("returns the record newest first", func() {
|
||||
seed("job-1", "localai@one", "localai@one", "model_install", "completed", "", false)
|
||||
seed("job-2", "localai@two", "localai@two", "model_install", "completed", "", false)
|
||||
seed("job-3", "localai@three", "localai@three", "model_install", "completed", "", false)
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(3))
|
||||
Expect(history[0].JobID).To(Equal("job-3"))
|
||||
Expect(history[1].JobID).To(Equal("job-2"))
|
||||
Expect(history[2].JobID).To(Equal("job-1"))
|
||||
})
|
||||
|
||||
It("omits operations still in flight", func() {
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "job-running", GalleryElementName: "busy", OpType: "model_install", Status: "downloading",
|
||||
})).To(Succeed())
|
||||
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
})
|
||||
|
||||
// The cancel handler persists "cancelled" synchronously, then the
|
||||
// handler goroutine unwinds with the context error and Start hands that
|
||||
// to updateError. If the second write wins, the page renders a
|
||||
// cancelled install as a red failure card offering Retry, with a raw
|
||||
// "context canceled" as the reason.
|
||||
It("records a cancelled install as cancelled, not as a context-cancelled failure", func() {
|
||||
svc.SetGalleryStore(store)
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "job-cancel", GalleryElementName: "localai@qwen3-4b",
|
||||
OpType: "model_install", Status: "downloading",
|
||||
})).To(Succeed())
|
||||
Expect(store.UpsertCacheKey("job-cancel", "localai@qwen3-4b", false)).To(Succeed())
|
||||
|
||||
Expect(store.Cancel("job-cancel")).To(Succeed())
|
||||
svc.UpdateStatus("job-cancel", &galleryop.OpStatus{
|
||||
Error: context.Canceled, Processed: true, Message: "error: context canceled",
|
||||
})
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Outcome).To(Equal(galleryop.OutcomeCancelled))
|
||||
Expect(history[0].Error).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("falls back to the local ring when the store read fails", func() {
|
||||
recordLocally("llama-cpp", "job-local")
|
||||
seed("job-peer", "localai@qwen3-4b", "localai@qwen3-4b", "model_install", "completed", "", false)
|
||||
|
||||
// A database blip must not blank the page: the ring holds a subset
|
||||
// of the same record, which beats returning nothing.
|
||||
breakStore()
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].JobID).To(Equal("job-local"))
|
||||
})
|
||||
|
||||
It("clears the record for the whole cluster, leaving live operations", func() {
|
||||
recordLocally("llama-cpp", "job-local")
|
||||
seed("job-peer", "localai@qwen3-4b", "localai@qwen3-4b", "model_install", "completed", "", false)
|
||||
Expect(store.Create(&distributed.GalleryOperationRecord{
|
||||
ID: "job-running", GalleryElementName: "busy", OpType: "model_install", Status: "downloading",
|
||||
})).To(Succeed())
|
||||
|
||||
Expect(cache.ClearHistory()).To(Succeed())
|
||||
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
|
||||
active, err := store.ListActive()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
ids := []string{}
|
||||
for _, op := range active {
|
||||
ids = append(ids, op.ID)
|
||||
}
|
||||
Expect(ids).To(ContainElement("job-running"))
|
||||
})
|
||||
|
||||
It("empties the local ring too, so a store failure cannot resurrect the record", func() {
|
||||
recordLocally("llama-cpp", "job-local")
|
||||
Expect(cache.ClearHistory()).To(Succeed())
|
||||
breakStore()
|
||||
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
})
|
||||
|
||||
// Reporting success on a clear that did not happen makes the record
|
||||
// vanish from the page and come straight back on the next fetch, with
|
||||
// nothing said about why.
|
||||
It("reports a failed clear instead of claiming success", func() {
|
||||
recordLocally("llama-cpp", "job-local")
|
||||
breakStore()
|
||||
|
||||
Expect(cache.ClearHistory()).ToNot(Succeed())
|
||||
Expect(cache.History()).To(HaveLen(1),
|
||||
"a failed clear must leave the fallback record intact rather than fake an empty one")
|
||||
})
|
||||
})
|
||||
})
|
||||
71
core/services/galleryop/history_test.go
Normal file
71
core/services/galleryop/history_test.go
Normal file
@@ -0,0 +1,71 @@
|
||||
package galleryop_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
)
|
||||
|
||||
// The history ring is the record behind the Activity page. It is bounded so a
|
||||
// long-running instance cannot grow it without limit, deduped by job ID so the
|
||||
// local eviction and the NATS end event for the same job record once, and
|
||||
// newest-first so the page renders without sorting.
|
||||
var _ = Describe("OpCache history ring", func() {
|
||||
var (
|
||||
svc *galleryop.GalleryService
|
||||
cache *galleryop.OpCache
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
svc = galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
cache = galleryop.NewOpCache(svc)
|
||||
})
|
||||
|
||||
It("starts empty", func() {
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("returns records newest first", func() {
|
||||
for i := 1; i <= 3; i++ {
|
||||
key := fmt.Sprintf("model-%d", i)
|
||||
job := fmt.Sprintf("job-%d", i)
|
||||
cache.Set(key, job)
|
||||
svc.UpdateStatus(job, &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
cache.DeleteUUID(job)
|
||||
}
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(3))
|
||||
Expect(history[0].Name).To(Equal("model-3"))
|
||||
Expect(history[2].Name).To(Equal("model-1"))
|
||||
})
|
||||
|
||||
It("caps at DefaultHistorySize and evicts the oldest first", func() {
|
||||
total := galleryop.DefaultHistorySize + 5
|
||||
for i := 1; i <= total; i++ {
|
||||
key := fmt.Sprintf("model-%d", i)
|
||||
job := fmt.Sprintf("job-%d", i)
|
||||
cache.Set(key, job)
|
||||
svc.UpdateStatus(job, &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
cache.DeleteUUID(job)
|
||||
}
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(galleryop.DefaultHistorySize))
|
||||
Expect(history[0].Name).To(Equal(fmt.Sprintf("model-%d", total)))
|
||||
// model-1..model-5 fell off the back.
|
||||
Expect(history[len(history)-1].Name).To(Equal("model-6"))
|
||||
})
|
||||
|
||||
It("clears on request", func() {
|
||||
cache.Set("model-a", "job-a")
|
||||
svc.UpdateStatus("job-a", &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
cache.DeleteUUID("job-a")
|
||||
Expect(cache.History()).To(HaveLen(1))
|
||||
|
||||
Expect(cache.ClearHistory()).To(Succeed())
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
@@ -53,7 +53,13 @@ func (g *GalleryService) modelHandler(op *ManagementOp[gallery.GalleryModel, gal
|
||||
}
|
||||
}
|
||||
|
||||
g.UpdateStatus(op.ID, &OpStatus{Message: fmt.Sprintf("processing model: %s", op.GalleryElementName), Progress: 0, Cancellable: true})
|
||||
// Starting the operation NARROWS what can be cancelled, which is the reverse
|
||||
// of the usual shape: markQueued reports every queued op as cancellable
|
||||
// because abandoning it before the worker takes it leaves no trace. From
|
||||
// here on that only holds for an install, whose download watches
|
||||
// operationCtx. DeleteModel takes no context and cannot be interrupted, so a
|
||||
// Cancel button on a running removal is one the server cannot honour.
|
||||
g.UpdateStatus(op.ID, &OpStatus{Message: fmt.Sprintf("processing model: %s", op.GalleryElementName), Progress: 0, Cancellable: !op.Delete})
|
||||
|
||||
bridge := newArtifactProgressBridge(func(status *OpStatus) {
|
||||
status.GalleryElementName = op.GalleryElementName
|
||||
|
||||
176
core/services/galleryop/opcache_history_test.go
Normal file
176
core/services/galleryop/opcache_history_test.go
Normal file
@@ -0,0 +1,176 @@
|
||||
package galleryop_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
)
|
||||
|
||||
// What lands in history, for each way an operation can leave the cache.
|
||||
var _ = Describe("OpCache terminal recording", func() {
|
||||
var (
|
||||
svc *galleryop.GalleryService
|
||||
cache *galleryop.OpCache
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
svc = galleryop.NewGalleryService(&config.ApplicationConfig{}, nil)
|
||||
cache = galleryop.NewOpCache(svc)
|
||||
})
|
||||
|
||||
It("records a completed install evicted by GetStatus", func() {
|
||||
cache.SetBackend("llama-cpp", "job-1")
|
||||
svc.UpdateStatus("job-1", &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
// GetStatus is what the UI polls; it self-evicts terminal ops.
|
||||
cache.GetStatus()
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Outcome).To(Equal("completed"))
|
||||
Expect(history[0].Name).To(Equal("llama-cpp"))
|
||||
Expect(history[0].JobID).To(Equal("job-1"))
|
||||
Expect(history[0].IsBackend).To(BeTrue())
|
||||
Expect(history[0].TaskType).To(Equal("installation"))
|
||||
Expect(history[0].Error).To(BeEmpty())
|
||||
Expect(history[0].StartedAt).NotTo(BeZero())
|
||||
Expect(history[0].FinishedAt).NotTo(BeZero())
|
||||
})
|
||||
|
||||
It("records a failed install only once it is dismissed", func() {
|
||||
cache.SetBackend("sherpa-onnx", "job-2")
|
||||
svc.UpdateStatus("job-2", &galleryop.OpStatus{Processed: true, Error: errors.New("no space left on device")})
|
||||
|
||||
// A failure is deliberately never auto-evicted: it stays live so the
|
||||
// UI can surface it under "Needs attention".
|
||||
cache.GetStatus()
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
Expect(cache.Exists("sherpa-onnx")).To(BeTrue())
|
||||
|
||||
// Dismissing is what moves it into the record.
|
||||
cache.DeleteUUID("job-2")
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Outcome).To(Equal("failed"))
|
||||
Expect(history[0].Error).To(Equal("no space left on device"))
|
||||
})
|
||||
|
||||
It("records an op removed before it was processed as cancelled, not completed", func() {
|
||||
// An operation that left the cache while it was still unprocessed never
|
||||
// got to finish, so it is not a success. This is what POST
|
||||
// /api/operations/:jobID/dismiss produces when it fires on an op that is
|
||||
// still in flight. Without the !Processed clause it would be recorded as
|
||||
// a successful install.
|
||||
cache.Set("deepseek-r1-70b", "job-3")
|
||||
svc.UpdateStatus("job-3", &galleryop.OpStatus{Progress: 8, Cancellable: true})
|
||||
|
||||
cache.DeleteUUID("job-3")
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Outcome).To(Equal("cancelled"))
|
||||
})
|
||||
|
||||
It("records an errored op as failed even when it never reached Processed", func() {
|
||||
// Pins the order of the outcome arms: an error outweighs the
|
||||
// "left while unprocessed" reading, so this is a failure to surface,
|
||||
// not a cancellation to ignore.
|
||||
cache.Set("stable-diffusion-3.5", "job-8")
|
||||
svc.UpdateStatus("job-8", &galleryop.OpStatus{Error: errors.New("download interrupted")})
|
||||
|
||||
cache.DeleteUUID("job-8")
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Outcome).To(Equal("failed"))
|
||||
Expect(history[0].Error).To(Equal("download interrupted"))
|
||||
})
|
||||
|
||||
It("records an explicitly cancelled op as cancelled", func() {
|
||||
cache.Set("flux.1-dev", "job-4")
|
||||
svc.UpdateStatus("job-4", &galleryop.OpStatus{Processed: true, Cancelled: true, Message: "cancelled"})
|
||||
|
||||
cache.GetStatus()
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Outcome).To(Equal("cancelled"))
|
||||
})
|
||||
|
||||
It("records a deletion with taskType deletion", func() {
|
||||
cache.Set("llama-3.2-1b-instruct", "job-5")
|
||||
svc.UpdateStatus("job-5", &galleryop.OpStatus{Processed: true, Progress: 100, Deletion: true, Message: "completed"})
|
||||
|
||||
cache.GetStatus()
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].TaskType).To(Equal("deletion"))
|
||||
Expect(history[0].Outcome).To(Equal("completed"))
|
||||
})
|
||||
|
||||
It("strips the gallery prefix from the display name", func() {
|
||||
cache.Set("localai@qwen3-8b-instruct", "job-6")
|
||||
svc.UpdateStatus("job-6", &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
cache.GetStatus()
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].ID).To(Equal("localai@qwen3-8b-instruct"))
|
||||
Expect(history[0].Name).To(Equal("qwen3-8b-instruct"))
|
||||
})
|
||||
|
||||
It("parses a node-scoped key into a backend name plus a node ID", func() {
|
||||
cache.SetBackend("node:worker-03:vllm", "job-7")
|
||||
svc.UpdateStatus("job-7", &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
cache.GetStatus()
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Name).To(Equal("vllm"))
|
||||
Expect(history[0].NodeID).To(Equal("worker-03"))
|
||||
Expect(history[0].IsBackend).To(BeTrue())
|
||||
})
|
||||
|
||||
It("ignores a job ID that was never in the cache", func() {
|
||||
cache.DeleteUUID("job-never-existed")
|
||||
Expect(cache.History()).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("records an operation that ended on a peer replica", func() {
|
||||
// applyEnd is the NATS SubjectGalleryOpEnd handler: a peer replica
|
||||
// finished the install and broadcast it. This replica must record it,
|
||||
// or history would be empty on every node except the one that ran the job.
|
||||
cache.SetBackend("vllm", "job-remote")
|
||||
svc.UpdateStatus("job-remote", &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
cache.ApplyEndForTest("job-remote")
|
||||
|
||||
history := cache.History()
|
||||
Expect(history).To(HaveLen(1))
|
||||
Expect(history[0].Name).To(Equal("vllm"))
|
||||
Expect(history[0].Outcome).To(Equal("completed"))
|
||||
})
|
||||
|
||||
It("records once when the local delete and the broadcast both land", func() {
|
||||
// The originating replica evicts locally AND receives its own broadcast.
|
||||
// This covers the sequence end to end: one record for one operation. It
|
||||
// is not what pins the ring's dedupe, because the local delete removes
|
||||
// the status keys and the broadcast then returns before reaching it;
|
||||
// the in-package spec that calls recordTerminal twice does that.
|
||||
cache.SetBackend("whisper-cpp", "job-both")
|
||||
svc.UpdateStatus("job-both", &galleryop.OpStatus{Processed: true, Progress: 100, Message: "completed"})
|
||||
|
||||
cache.DeleteUUID("job-both")
|
||||
cache.ApplyEndForTest("job-both")
|
||||
|
||||
Expect(cache.History()).To(HaveLen(1))
|
||||
})
|
||||
})
|
||||
@@ -4,8 +4,10 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/services/distributed"
|
||||
@@ -226,6 +228,11 @@ type OpCache struct {
|
||||
backendOps *xsync.SyncedMap[string, bool] // Tracks which operations are backend operations
|
||||
galleryService *GalleryService
|
||||
|
||||
// Finished operations, for GET /api/operations/history. started stamps the
|
||||
// start time when an op enters the cache so the record can report duration.
|
||||
history *opHistory
|
||||
started *xsync.SyncedMap[string, time.Time]
|
||||
|
||||
// Distributed sync (nil when standalone).
|
||||
mu sync.RWMutex
|
||||
nats messaging.MessagingClient
|
||||
@@ -238,6 +245,8 @@ func NewOpCache(galleryService *GalleryService) *OpCache {
|
||||
status: xsync.NewSyncedMap[string, string](),
|
||||
backendOps: xsync.NewSyncedMap[string, bool](),
|
||||
galleryService: galleryService,
|
||||
history: newOpHistory(DefaultHistorySize),
|
||||
started: xsync.NewSyncedMap[string, time.Time](),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -343,6 +352,11 @@ func (m *OpCache) applyStart(evt OpCacheEvent) {
|
||||
if evt.CacheKey == "" || evt.JobID == "" {
|
||||
return
|
||||
}
|
||||
// A retry admitted on a peer arrives as a start event reusing the key, which
|
||||
// strands the stamp of whichever job the key pointed at here. Only the drop
|
||||
// half of stampStart runs: a replicated op deliberately carries no stamp and
|
||||
// reports as zero-length through recordTerminal's finish-time fallback.
|
||||
m.dropReplacedStamp(evt.CacheKey, evt.JobID)
|
||||
m.status.Set(evt.CacheKey, evt.JobID)
|
||||
if evt.IsBackend {
|
||||
m.backendOps.Set(evt.CacheKey, true)
|
||||
@@ -354,26 +368,54 @@ func (m *OpCache) applyEnd(evt OpCacheEvent) {
|
||||
if evt.JobID == "" {
|
||||
return
|
||||
}
|
||||
// Record before the keys go. The history ring dedupes by job ID, so the
|
||||
// originating replica recording locally and then receiving its own
|
||||
// broadcast still produces one entry.
|
||||
m.recordTerminal(evt.JobID, terminalPeer)
|
||||
for _, k := range m.status.Keys() {
|
||||
if m.status.Get(k) == evt.JobID {
|
||||
m.status.Delete(k)
|
||||
m.backendOps.Delete(k)
|
||||
}
|
||||
}
|
||||
// recordTerminal only drops the stamp on the path where it found a cache
|
||||
// key; an end event that overtakes the local Set finds none. The operation
|
||||
// is over cluster-wide either way, so the stamp goes unconditionally.
|
||||
m.started.Delete(evt.JobID)
|
||||
}
|
||||
|
||||
func (m *OpCache) Set(key string, value string) {
|
||||
m.stampStart(key, value)
|
||||
m.status.Set(key, value)
|
||||
m.persistAndBroadcastStart(key, value, false)
|
||||
}
|
||||
|
||||
// SetBackend sets a key-value pair and marks it as a backend operation
|
||||
func (m *OpCache) SetBackend(key string, value string) {
|
||||
m.stampStart(key, value)
|
||||
m.status.Set(key, value)
|
||||
m.backendOps.Set(key, true)
|
||||
m.persistAndBroadcastStart(key, value, true)
|
||||
}
|
||||
|
||||
// stampStart records when jobID started, so the history record can report a
|
||||
// duration.
|
||||
func (m *OpCache) stampStart(key, jobID string) {
|
||||
m.dropReplacedStamp(key, jobID)
|
||||
m.started.Set(jobID, time.Now())
|
||||
}
|
||||
|
||||
// dropReplacedStamp forgets the start time of the job a cache key is about to
|
||||
// stop pointing at. Retrying an operation reuses the key with a fresh job ID,
|
||||
// so without this the map would grow for the lifetime of the process: the
|
||||
// callers in ui_api.go are not uniformly guarded by Exists. Must run before
|
||||
// status.Set, which is what the previous job ID is read from.
|
||||
func (m *OpCache) dropReplacedStamp(key, jobID string) {
|
||||
if prev := m.status.Get(key); prev != "" && prev != jobID {
|
||||
m.started.Delete(prev)
|
||||
}
|
||||
}
|
||||
|
||||
func (m *OpCache) persistAndBroadcastStart(key, value string, isBackend bool) {
|
||||
m.mu.RLock()
|
||||
store := m.store
|
||||
@@ -405,7 +447,173 @@ func (m *OpCache) Get(key string) string {
|
||||
return m.status.Get(key)
|
||||
}
|
||||
|
||||
// terminalSource is the path that retired an operation. It exists for one
|
||||
// decision: what a missing gallery status means. Locally it means the op was
|
||||
// queued and removed before anything ran; on the peer path it means this
|
||||
// replica never held the outcome, which is a different thing wearing the same
|
||||
// signal.
|
||||
type terminalSource int
|
||||
|
||||
const (
|
||||
terminalLocal terminalSource = iota
|
||||
terminalPeer
|
||||
)
|
||||
|
||||
// recordTerminal appends a finished operation to the history ring. It must run
|
||||
// BEFORE the cache entry is deleted: the gallery key is the only place the
|
||||
// display name, the backend flag and the node scoping live.
|
||||
//
|
||||
// Safe to call for an unknown job ID (no key, no record) and safe to call
|
||||
// twice for the same job (the ring dedupes), which is what makes it usable
|
||||
// from both the local delete path and the NATS end event.
|
||||
func (m *OpCache) recordTerminal(jobID string, src terminalSource) {
|
||||
if jobID == "" {
|
||||
return
|
||||
}
|
||||
|
||||
key := ""
|
||||
for _, k := range m.status.Keys() {
|
||||
if m.status.Get(k) == jobID {
|
||||
key = k
|
||||
break
|
||||
}
|
||||
}
|
||||
if key == "" {
|
||||
return
|
||||
}
|
||||
|
||||
// A replica that restarted mid-operation hydrates its cache keys from
|
||||
// PostgreSQL, but gallery statuses are in-memory only and come back empty.
|
||||
// The end broadcast then arrives with nothing to read the outcome from, and
|
||||
// guessing would file a successful install as cancelled. Record nothing:
|
||||
// absence beats wrong data in a "what just happened" view, and it is exactly
|
||||
// what this replica produced before the end event started recording.
|
||||
status := m.galleryService.GetStatus(jobID)
|
||||
if status == nil && src == terminalPeer {
|
||||
return
|
||||
}
|
||||
|
||||
rec := OpRecord{
|
||||
ID: key,
|
||||
JobID: jobID,
|
||||
IsBackend: m.backendOps.Get(key),
|
||||
TaskType: "installation",
|
||||
Outcome: OutcomeCompleted,
|
||||
FinishedAt: time.Now(),
|
||||
}
|
||||
// hydrateFromStore and applyStart populate status without a stamp, so an op
|
||||
// recovered from PostgreSQL or replicated from a peer has no start time.
|
||||
// Reporting the finish time makes such a record a zero-length operation
|
||||
// rather than one that appears to have run since year one.
|
||||
//
|
||||
// Read the stamp once and reject a zero value rather than testing Exists and
|
||||
// then reading: a concurrent recordTerminal for the same job can delete the
|
||||
// stamp between the two, and the read that follows returns the zero time,
|
||||
// which would overwrite the fallback with year one.
|
||||
rec.StartedAt = rec.FinishedAt
|
||||
if started := m.started.Get(jobID); !started.IsZero() {
|
||||
rec.StartedAt = started
|
||||
}
|
||||
|
||||
rec.Name, rec.NodeID = operationDisplayName(key)
|
||||
|
||||
// Outcome order matters: an error outweighs everything else, because an op
|
||||
// can carry both an error and an unfinished status and the failure is what
|
||||
// the user needs to see.
|
||||
//
|
||||
// An operation that left the cache while it was still unprocessed never got
|
||||
// to finish, so it is not a success. That is what the dismiss endpoint
|
||||
// produces when it fires on an op that is still in flight, and what a status
|
||||
// that exists but never started looks like. The cancel endpoint is already
|
||||
// covered by status.Cancelled, which CancelOperation sets synchronously
|
||||
// before the handler removes the entry.
|
||||
if status != nil {
|
||||
if status.Deletion {
|
||||
rec.TaskType = "deletion"
|
||||
}
|
||||
switch {
|
||||
case status.Error != nil:
|
||||
rec.Outcome = OutcomeFailed
|
||||
rec.Error = status.Error.Error()
|
||||
case status.Cancelled || !status.Processed:
|
||||
rec.Outcome = OutcomeCancelled
|
||||
}
|
||||
} else {
|
||||
// Queued but never started, then removed.
|
||||
rec.Outcome = OutcomeCancelled
|
||||
}
|
||||
|
||||
m.history.add(rec)
|
||||
m.started.Delete(jobID)
|
||||
}
|
||||
|
||||
// History returns finished operations, newest first.
|
||||
//
|
||||
// With a gallery store wired the record comes from PostgreSQL, so every replica
|
||||
// answers with the same history: the in-memory ring only holds what this
|
||||
// replica happened to serve, which makes the page's contents depend on which
|
||||
// replica the poll was routed to and leaves a replica added by a scale-out or a
|
||||
// rolling deploy blank forever.
|
||||
func (m *OpCache) History() []OpRecord {
|
||||
m.mu.RLock()
|
||||
store := m.store
|
||||
m.mu.RUnlock()
|
||||
|
||||
if store == nil {
|
||||
return m.history.list()
|
||||
}
|
||||
|
||||
ops, err := store.ListTerminal(DefaultHistorySize)
|
||||
if err != nil {
|
||||
// A transient database failure must not blank the page. The ring holds
|
||||
// a subset of the same record, so it is a strictly better answer than
|
||||
// nothing.
|
||||
xlog.Warn("OpCache failed to read the operation record; falling back to the local ring", "error", err)
|
||||
return m.history.list()
|
||||
}
|
||||
|
||||
records := make([]OpRecord, 0, len(ops))
|
||||
for _, op := range ops {
|
||||
records = append(records, recordFromStore(op))
|
||||
}
|
||||
return records
|
||||
}
|
||||
|
||||
// ClearHistory empties the record. Live operations are untouched.
|
||||
//
|
||||
// With a gallery store wired this clears the record for the whole cluster,
|
||||
// which is the only way "Clear history" can mean anything: clearing one
|
||||
// replica's ring leaves the record to reappear on the next poll routed
|
||||
// elsewhere.
|
||||
//
|
||||
// The error is returned rather than logged because the caller is an HTTP
|
||||
// handler and the rows are what the next read returns: reporting success on a
|
||||
// failed delete makes the record vanish from the page and come straight back on
|
||||
// the next fetch, with nothing said about why.
|
||||
func (m *OpCache) ClearHistory() error {
|
||||
m.mu.RLock()
|
||||
store := m.store
|
||||
m.mu.RUnlock()
|
||||
|
||||
if store == nil {
|
||||
m.history.clear()
|
||||
return nil
|
||||
}
|
||||
|
||||
// Store first: the ring is only ever a fallback for a failed read, so
|
||||
// emptying it before the rows are confirmed gone would report an empty
|
||||
// record while the real one is still there.
|
||||
if err := store.ClearTerminal(); err != nil {
|
||||
return fmt.Errorf("clearing the persisted operation record: %w", err)
|
||||
}
|
||||
m.history.clear()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *OpCache) DeleteUUID(uuid string) {
|
||||
// Before the keys go: they carry the name and the backend flag.
|
||||
m.recordTerminal(uuid, terminalLocal)
|
||||
|
||||
deleted := false
|
||||
for _, k := range m.status.Keys() {
|
||||
if m.status.Get(k) == uuid {
|
||||
@@ -488,6 +696,25 @@ func (m *OpCache) GetStatus() (map[string]string, map[string]string) {
|
||||
return processingModelsData, taskTypes
|
||||
}
|
||||
|
||||
// operationDisplayName reduces an operation key to what the record shows: the
|
||||
// node prefix ("node:<nodeID>:") detached into a node ID the page reports
|
||||
// separately, and the gallery prefix ("<gallery>@") dropped.
|
||||
//
|
||||
// The in-memory ring and the store-backed record both go through this, so the
|
||||
// same operation cannot end up named two different ways depending on which
|
||||
// source answered.
|
||||
func operationDisplayName(key string) (name, nodeID string) {
|
||||
name = key
|
||||
if id, backend, ok := ParseNodeScopedKey(key); ok {
|
||||
nodeID = id
|
||||
name = backend
|
||||
}
|
||||
if _, after, found := strings.Cut(name, "@"); found {
|
||||
name = after
|
||||
}
|
||||
return name, nodeID
|
||||
}
|
||||
|
||||
// NodeScopedKeyPrefix is the opcache key prefix used by InstallBackendOnNodeEndpoint
|
||||
// so per-node installs do not collide on the bare backend name. Format:
|
||||
// "node:<nodeID>:<backend>". Read by /api/operations to extract nodeID for the UI.
|
||||
|
||||
@@ -149,9 +149,18 @@ func (g *GalleryService) UpdateStatus(s string, op *OpStatus) {
|
||||
// another. If the caller explicitly populates Nodes on the incoming op,
|
||||
// that wins; an empty Nodes slice on the incoming op is treated as "no
|
||||
// new per-node data" and the previous Nodes are carried forward.
|
||||
if op != nil && len(op.Nodes) == 0 {
|
||||
if prev := g.statuses[s]; prev != nil && len(prev.Nodes) > 0 {
|
||||
op.Nodes = prev.Nodes
|
||||
if op != nil {
|
||||
if prev := g.statuses[s]; prev != nil {
|
||||
if len(op.Nodes) == 0 && len(prev.Nodes) > 0 {
|
||||
op.Nodes = prev.Nodes
|
||||
}
|
||||
// A job is a delete or an install for its whole life. markQueued is
|
||||
// the only writer that knows which; every later status omits the
|
||||
// flag, so an unset value means "no new information", not "this is
|
||||
// an install".
|
||||
if !op.Deletion {
|
||||
op.Deletion = prev.Deletion
|
||||
}
|
||||
}
|
||||
}
|
||||
g.statuses[s] = op
|
||||
@@ -552,12 +561,20 @@ func (g *GalleryService) Start(c context.Context, cl *config.ModelConfigLoader,
|
||||
}
|
||||
// Create DB record for distributed tracking
|
||||
if g.galleryStore != nil {
|
||||
opType := "backend_install"
|
||||
if op.Delete {
|
||||
opType = "backend_delete"
|
||||
}
|
||||
if err := g.galleryStore.Create(&distributed.GalleryOperationRecord{
|
||||
ID: op.ID,
|
||||
GalleryElementName: op.GalleryElementName,
|
||||
OpType: "backend_install",
|
||||
OpType: opType,
|
||||
Status: "pending",
|
||||
Cancellable: true,
|
||||
// Create runs at dequeue, so this is the running-phase
|
||||
// value: a running delete is not cancellable, an install
|
||||
// is, matching the model channel. The queued phase before
|
||||
// this point is cancellable either way (see markQueued).
|
||||
Cancellable: !op.Delete,
|
||||
}); err != nil {
|
||||
// Not fatal: the install still runs and the in-memory
|
||||
// status still updates. Logged because without the row
|
||||
@@ -596,7 +613,10 @@ func (g *GalleryService) Start(c context.Context, cl *config.ModelConfigLoader,
|
||||
GalleryElementName: op.GalleryElementName,
|
||||
OpType: opType,
|
||||
Status: "pending",
|
||||
// A delete is not cancellable; an install is.
|
||||
// Create runs at dequeue, so this is the running-phase
|
||||
// value: a running delete is not cancellable, an install
|
||||
// is. The queued phase before this point is cancellable
|
||||
// either way (see markQueued).
|
||||
Cancellable: !op.Delete,
|
||||
}); err != nil {
|
||||
xlog.Warn("Failed to create gallery operation record", "op_id", op.ID, "error", err)
|
||||
@@ -750,7 +770,7 @@ func (g *GalleryService) Hydrate() error {
|
||||
DownloadedFileSize: op.DownloadedFileSize,
|
||||
GalleryElementName: op.GalleryElementName,
|
||||
Cancellable: op.Cancellable,
|
||||
Deletion: op.OpType == "model_delete",
|
||||
Deletion: IsDeleteOpType(op.OpType),
|
||||
}
|
||||
if op.Error != "" {
|
||||
st.Error = errors.New(op.Error)
|
||||
|
||||
@@ -157,7 +157,8 @@ When authentication is enabled, the following endpoints require admin role:
|
||||
**Model & Backend Management:**
|
||||
- `GET /api/models`, `POST /api/models/install/*`, `POST /api/models/delete/*`
|
||||
- `GET /api/backends`, `POST /api/backends/install/*`, `POST /api/backends/delete/*`
|
||||
- `GET /api/operations`, `POST /api/operations/*/cancel`
|
||||
- `GET /api/operations`, `POST /api/operations/*/cancel`, `POST /api/operations/*/dismiss`
|
||||
- `GET /api/operations/history`, `DELETE /api/operations/history`
|
||||
- `GET /models/available`, `GET /models/galleries`, `GET /models/jobs/*`
|
||||
- `GET /backends`, `GET /backends/available`, `GET /backends/galleries`
|
||||
|
||||
|
||||
@@ -25,6 +25,11 @@ The LocalAI web interface provides an intuitive way to manage your backends:
|
||||
5. Install or delete backends with a single click
|
||||
6. Monitor installation progress in real-time
|
||||
|
||||
Installs run in the background. The strip at the top of the app follows the
|
||||
current one, and **Operate → Activity** lists everything in flight, what needs
|
||||
attention, and what has finished, and is where a running install is cancelled
|
||||
or a failed one retried. See [Activity]({{% relref "operations/activity" %}}).
|
||||
|
||||
Each backend card displays:
|
||||
- Backend name and description
|
||||
- Type of models it supports
|
||||
|
||||
@@ -202,13 +202,16 @@ The worker HTTP file transfer server is authenticated by `LOCALAI_REGISTRATION_T
|
||||
|
||||
### Watching Backend Installs
|
||||
|
||||
While a worker downloads a backend, the admin **Operations Bar** at the top
|
||||
of the UI shows real-time progress: current file, downloaded/total bytes,
|
||||
and percentage. This works the same as single-node mode.
|
||||
While a worker downloads a backend, the admin operations strip at the top
|
||||
of the UI shows real-time progress: a percentage, and, when the install
|
||||
targets several workers, a roll-up of how far the fan-out has got,
|
||||
`2 of 5 nodes done`. Per-file byte counts are not on the strip; they are in
|
||||
the per-node detail below.
|
||||
|
||||
When an install targets more than one worker, an **N nodes** chevron
|
||||
appears on the operation row. Click it to expand a per-node breakdown,
|
||||
with one row per worker showing:
|
||||
The per-node detail is on the **Operate → Activity** page
|
||||
([Activity]({{% relref "operations/activity" %}})). When an install targets
|
||||
more than one worker, an **N nodes** tag appears on the operation card, with
|
||||
one row per worker showing:
|
||||
|
||||
- A status pill: **Queued** (gray), **Downloading** (blue), **Worker busy**
|
||||
(yellow), **Done** (green), or **Failed** (red).
|
||||
@@ -225,6 +228,12 @@ If a worker is running an older LocalAI release that does not report
|
||||
progress, its row in the breakdown will still show terminal status
|
||||
(queued / done / failed / worker busy) but no per-file progress.
|
||||
|
||||
The **Record** on that page - what model and backend installs and removals have
|
||||
finished - is read from PostgreSQL rather than from the replica's memory. Every
|
||||
replica reports the same record, it survives restarts, a replica added by a
|
||||
scale-out or a rolling deploy reports it in full, and **Clear history** clears
|
||||
it for every replica.
|
||||
|
||||
## Worker Configuration
|
||||
|
||||
Workers are started with the `worker` subcommand. Each worker is generic - it doesn't need a backend type at startup:
|
||||
|
||||
@@ -304,9 +304,10 @@ installation. Their operation progresses through these phases:
|
||||
resolving -> downloading -> verifying -> committing -> persisting
|
||||
```
|
||||
|
||||
The admin Operations Bar and `GET /api/operations` expose `currentBytes` and
|
||||
`totalBytes` as raw transport bytes. Cancelling an active download leaves its
|
||||
partial files in place so a retry can resume. A verification failure never
|
||||
The admin operations strip, the [Activity]({{% relref "operations/activity" %}})
|
||||
page and `GET /api/operations` expose `currentBytes` and `totalBytes` as raw
|
||||
transport bytes. Cancelling an active download leaves its partial files in
|
||||
place so a retry can resume. A verification failure never
|
||||
exposes a completed snapshot, while a retry or another installation reuses an
|
||||
already verified content-addressed snapshot.
|
||||
|
||||
|
||||
@@ -25,7 +25,9 @@ The Model Gallery is the simplest way to install models. It provides pre-configu
|
||||
2. Navigate to the "Models" tab
|
||||
3. Browse available models
|
||||
4. Click "Install" on any model you want
|
||||
5. Wait for installation to complete
|
||||
5. Wait for installation to complete. Progress appears in the strip at the top
|
||||
of the app, and **Operate → Activity** shows every install in flight, plus
|
||||
what failed and what finished (see [Activity]({{% relref "operations/activity" %}}))
|
||||
|
||||
For more details, refer to the [Gallery Documentation]({{% relref "features/model-gallery" %}}).
|
||||
|
||||
|
||||
@@ -14,3 +14,4 @@ This section collects the operator-facing concerns for running LocalAI in produc
|
||||
- [Cloud passthrough proxy]({{% relref "operations/cloud-proxy" %}}) - forward requests to OpenAI, Anthropic, or any compatible provider.
|
||||
- [MITM proxy for Claude Code / Codex CLI]({{% relref "operations/mitm-proxy" %}}) - redact PII from cloud-AI traffic without LocalAI holding API keys.
|
||||
- [Backend Monitor]({{% relref "operations/backend-monitor" %}}) - monitor, pre-load, and shut down running model backends.
|
||||
- [Activity]({{% relref "operations/activity" %}}) - watch, cancel, and retry model and backend installs from the WebUI.
|
||||
|
||||
215
docs/content/operations/activity.md
Normal file
215
docs/content/operations/activity.md
Normal file
@@ -0,0 +1,215 @@
|
||||
+++
|
||||
disableToc = false
|
||||
title = "Activity"
|
||||
weight = 21
|
||||
url = "/features/activity/"
|
||||
description = "Watch, cancel and retry model and backend installs from the WebUI"
|
||||
+++
|
||||
|
||||
Model installs, backend installs, removals and cluster staging all run in the
|
||||
background. Two surfaces in the WebUI report them: a one-line strip at the top
|
||||
of the app, and the **Activity** page, which holds the full picture.
|
||||
|
||||
Both are admin-only.
|
||||
|
||||
## The operations strip
|
||||
|
||||
While there is anything to report, a single line appears at the top of the app.
|
||||
It shows one operation at a time:
|
||||
|
||||
- a failure, if there is one, because an error waiting for a decision outranks
|
||||
any progress;
|
||||
- otherwise the least advanced running operation, which is the one everything
|
||||
else is waiting behind.
|
||||
|
||||
Every operation shows what is being done and to what, for example
|
||||
`Installing model qwen3-4b`. Everything else depends on what the operation can
|
||||
report. A percentage and its bar appear only once the operation is running and
|
||||
has reported progress, so a queued operation, a failed one and a removal show
|
||||
none. Artifact-backed gallery models report two things more: the phase
|
||||
(`Resolving files`, `Downloading`, `Verifying`, `Finalizing`, `Saving
|
||||
configuration`) and downloaded/total bytes. A backend installing on several
|
||||
workers is rolled up into one phrase, `2 of 5 nodes done`; the per-node
|
||||
breakdown lives on the Activity page.
|
||||
|
||||
When more than one operation is in flight, a **N more** counter on the right
|
||||
links to the Activity page.
|
||||
|
||||
When the last operation finishes, it stays on the strip for about four seconds
|
||||
and then goes, so a fast install is not a flicker. While others are still
|
||||
running, the strip moves straight on to the next one. An operation you cancel
|
||||
is not held: it goes as soon as it stops.
|
||||
|
||||
The **X** on the right hides the strip. It does not cancel anything: the work
|
||||
carries on, the sidebar still counts it, and the Activity page still lists it.
|
||||
Hiding applies to that one operation, so a later failure brings the strip back.
|
||||
On a failure the same button acknowledges the error and moves it into the
|
||||
record instead.
|
||||
|
||||
## The Activity page
|
||||
|
||||
**Operate → Activity** (`/app/activity`). The page has up to three sections,
|
||||
each shown only when it has something in it.
|
||||
|
||||
### In progress
|
||||
|
||||
One card per running or queued operation, each naming what is being done and to
|
||||
what. A percentage and progress bar appear only once the operation is running
|
||||
and has reported progress, so a queued operation and a removal show none.
|
||||
Artifact-backed gallery models also report the phase, downloaded and total
|
||||
bytes, and an estimate of the time left once enough of the transfer has been
|
||||
observed to work one out. Every backend install, and every gallery model that
|
||||
lists its files directly, reports no phase and no byte counters; those cards
|
||||
show the installer's own status line instead, which names the file being
|
||||
fetched and its size.
|
||||
|
||||
An install that involves workers shows a per-node list, with one row per
|
||||
worker:
|
||||
|
||||
- a status pill: **Queued**, **Downloading**, **Worker busy**, **Done** or
|
||||
**Failed**;
|
||||
- the file being transferred, with current/total bytes and a percentage;
|
||||
- any error the worker returned.
|
||||
|
||||
An install that fans out to more than one worker also carries an **N nodes**
|
||||
tag and a toggle over the list. Lists of up to four nodes start expanded, longer
|
||||
ones start collapsed behind **Show N nodes**, and the toggle reads **Hide
|
||||
per-node detail** while the list is open.
|
||||
|
||||
**Worker busy** means the worker took longer than `--backend-install-timeout`
|
||||
to acknowledge but is most likely still working. It clears on its own when the
|
||||
worker finishes.
|
||||
|
||||
### Needs attention
|
||||
|
||||
Model and backend operations that failed and have not been acknowledged yet,
|
||||
each card carrying the error returned by the installer. Cluster staging never
|
||||
appears here: a staging job reports no error to the page, so a staging failure
|
||||
has to be read from the logs.
|
||||
|
||||
### Record
|
||||
|
||||
What finished, newest first, one row each: the name, what happened
|
||||
(`installed in 1m 12s`, `removed`, `cancelled`, or `failed:` with the error),
|
||||
the time of day it finished, and a link into Models or Backends. Model and
|
||||
backend installs and removals are recorded; cluster staging is not, so a
|
||||
staging run leaves nothing behind here once it finishes.
|
||||
|
||||
### Filters
|
||||
|
||||
Four chips above the sections filter the whole page: **All**, **Models**,
|
||||
**Backends** and **Cluster**. In the live sections, Cluster covers staged model
|
||||
files and any install that involves workers, whether it targets one node or
|
||||
fans out across several. In the Record it covers only the node-targeted case: a
|
||||
finished fan-out install is filed under Models or Backends, not under Cluster.
|
||||
|
||||
## Cancelling, retrying and dismissing
|
||||
|
||||
These are on the operation cards. **Cancel** and **Retry** are labelled
|
||||
buttons; dismissing is the **X** at the end of a failed card. The strip has no
|
||||
cancel button; the page is the only place work is stopped or restarted.
|
||||
|
||||
- **Cancel** is offered while an operation is queued, whatever it is, and while
|
||||
an install is running. It is not offered once a removal has started: a
|
||||
removal in progress cannot be interrupted, so the window to call one off is
|
||||
the queue, before a worker picks it up. Cancelling there stops it before
|
||||
anything is touched. For artifact-backed gallery models, cancelling an active
|
||||
download leaves its partial files in place so a later install resumes rather
|
||||
than starting over. A cancelled operation leaves the live sections
|
||||
immediately and is not held on the strip the way a completed one is; it
|
||||
appears in the record as `cancelled`.
|
||||
- **Retry** is offered on a failed model or backend install. It acknowledges
|
||||
the failure, which moves it into the record, and installs the same target
|
||||
again. It is not offered on a failed removal, which is not restarted by
|
||||
reinstalling.
|
||||
- **Dismiss**, the **X** on a failed card, acknowledges the failure without
|
||||
retrying. The operation moves into the record with a `failed` outcome; it is
|
||||
not deleted. This is why the same failure can be found either under **Needs
|
||||
attention** or in the **Record**, depending on whether it has been
|
||||
acknowledged.
|
||||
|
||||
{{% notice note %}}
|
||||
Retrying a model that was installed with an explicit `variant` reinstalls it
|
||||
with automatic variant selection, because the variant is not carried on the
|
||||
operation. If you pinned a build, reinstall it from the Models page or the API
|
||||
with the `variant` you want rather than using **Retry**. See
|
||||
[Model Gallery]({{% relref "features/model-gallery" %}}) for variants.
|
||||
{{% /notice %}}
|
||||
|
||||
## History
|
||||
|
||||
The record holds the last 50 finished operations. Where it is kept depends on
|
||||
how LocalAI is running:
|
||||
|
||||
- **Standalone**: in memory, so it is empty again after a restart. For durable
|
||||
per-model install state, use the model and backend listings rather than this
|
||||
page.
|
||||
- **[Distributed mode]({{% relref "features/distributed-mode" %}})**: in
|
||||
PostgreSQL, alongside the operations themselves. It is the same record on
|
||||
every replica, it survives restarts, and a replica added by a scale-out or a
|
||||
rolling deploy reports it in full rather than starting empty.
|
||||
|
||||
**Clear history**, beside the page title whenever the record has anything in
|
||||
it, empties it. Running operations and unacknowledged failures are untouched.
|
||||
In distributed mode it clears the record for every replica, not just the one
|
||||
serving the page.
|
||||
|
||||
## The sidebar count
|
||||
|
||||
The **Operate** entry in the sidebar carries a badge whenever something is in
|
||||
flight, showing how many operations are running or queued. If any failure is
|
||||
waiting to be acknowledged, the badge turns red and counts the failures
|
||||
instead, so the number always reports the thing that most wants your attention.
|
||||
|
||||
## API
|
||||
|
||||
Both surfaces are backed by these endpoints. All of them are admin-only when
|
||||
[authentication]({{% relref "features/authentication" %}}) is enabled.
|
||||
|
||||
| Method | Path | Description |
|
||||
| -------- | -------------------------------- | ------------------------------------------------------------------------ |
|
||||
| `GET` | `/api/operations` | Running, queued and failed operations, least advanced first. |
|
||||
| `POST` | `/api/operations/{jobID}/cancel` | Cancel a queued operation, or a running install. Not a running removal. |
|
||||
| `POST` | `/api/operations/{jobID}/dismiss`| Acknowledge a failed operation and move it into the record. |
|
||||
| `GET` | `/api/operations/history` | List finished operations, newest first. |
|
||||
| `DELETE` | `/api/operations/history` | Clear the record. Live operations are untouched. |
|
||||
|
||||
Both `GET` endpoints wrap their list in an `operations` key rather than
|
||||
returning a bare array:
|
||||
|
||||
```bash
|
||||
# What has finished
|
||||
curl http://localhost:8080/api/operations/history \
|
||||
-H "Authorization: Bearer <admin-key>"
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"operations": [
|
||||
{
|
||||
"id": "localai@qwen3-4b",
|
||||
"name": "qwen3-4b",
|
||||
"jobID": "6f1b0c2e-...",
|
||||
"isBackend": false,
|
||||
"taskType": "installation",
|
||||
"outcome": "completed",
|
||||
"startedAt": "2026-07-27T10:02:11.482913574Z",
|
||||
"finishedAt": "2026-07-27T10:03:23.117402881Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`nodeID` is present when the operation was scoped to a worker, and `error`
|
||||
when the `outcome` is `failed`. The `outcome` is one of `completed`, `failed`
|
||||
or `cancelled`.
|
||||
|
||||
```bash
|
||||
# Forget the record
|
||||
curl -X DELETE http://localhost:8080/api/operations/history \
|
||||
-H "Authorization: Bearer <admin-key>"
|
||||
```
|
||||
|
||||
The `DELETE` answers `500` with an `error` if the record could not be cleared,
|
||||
which in distributed mode means the rows are still there. It never reports
|
||||
success on a record it did not clear.
|
||||
Reference in New Issue
Block a user