mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-04 12:22:22 -04:00
* feat(ui): give the Operate overview real numbers and traces a latency shape First two items from a component-by-component comparison against the mocks. The pattern that audit found: everything newly built matched, everything pre-existing got the palette but not the layout, and an "absent rather than empty" rule hid most of the overview exactly when someone was looking at an idle installation. **The headline grid is always rendered**, including at zero, with a fourth cell for host memory. Hiding it removed the page's structure precisely when it was most likely to be read, and "0 failed" is information — an absent panel is not. The quiet case is now said in a line underneath instead of by showing nothing. **The sections state counts** rather than listing their destinations: backends, models, updates and running operations instead of the words "Usage and traces". That needed installed backend and model counts in the summary context, which are two more cheap reads on the poll that was already running. **Traces rows carry latency as a bar as well as a figure**, scaled against the slowest request currently in view and turning amber past two seconds. The table had no latency column at all — the number was buried in the expanded detail, so the shape of the tail was invisible while scanning. Scaling against the view rather than an absolute ceiling is deliberate: what matters when reading a page of traces is which of these are the outliers, and an absolute scale flattens every row on a fast installation into nothing. Full e2e suite: 409 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): name the engine on Home's resident models, and add jump-back-in Third item from the mock comparison. The mock showed each resident model with the engine serving it. /system carried only the id, so the audit recorded this as blocked on a server field — but the config loader is already in scope where that response is built, so it is one lookup. SysInfoModel gains an optional `backend`, resolved from the model's config and omitted rather than guessed when there is none (a loose file, or a config since removed). Home renders the column blank in that case; the test pins both halves of that. Memory per model stays out. It is not one lookup — it would mean asking each backend process — and inventing a number beside a real one is worse than leaving the column off. "Jump back in" is the block the mock had and Home did not. The quick-links row above it is a set of first-run actions; these are the three places someone returns to, each stated with what it currently holds rather than as a bare label. Go: routes suite passes. Full e2e suite: 412 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): rank the recommended models as lanes instead of equal cards The hardware recommendations were a grid of equally-weighted cards. The list is already sorted by fit, and a grid throws that order away: three cards side by side say "pick one", when the page has actually formed an opinion about which one. They are lanes now, read top to bottom in fit order, with the leader carrying the single amber "Best fit" label and the rest marked "Also fits". One opinion per page — the alternatives are alternatives, not runners-up each worth their own colour, which is how a strip of coloured badges ends up meaning nothing. Below 720px the size and VRAM columns drop and the lane keeps the name and the install action, which are the two things a narrow screen needs. The existing panel spec moves off .rec-models-item onto .lane rather than being deleted; dismissal, collapse, keyboard operation and install all still pass unchanged, and there is a new assertion that exactly one row is called out. Full e2e suite: 413 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * fix(ui): drop capsule chips app-wide, and un-break the empty voice library **Pills are gone.** A capsule radius reads as a tag floating on the surface, which fights a system whose structure is hairlines and square corners — and with chips on Discover, Host, Activity and the biometrics pages, "some pages have pills" was the real inconsistency rather than any one page. Sixteen selectors move to the small radius: filter buttons, tab pills, activity and biometrics chips, file and count badges, the jump-to-latest control, the nav badge. Round *buttons* keep their circle — .lightbox__nav and .home-send-btn are circles, not capsules — as do every progress track, status dot and avatar, which are round because they are round, not because they are tags. **The empty voice library was unusable.** `.voice-library-empty` sets min-height: 430px, border: 0 and background: transparent — a description of the empty PANEL — and it had been attached to the action instead. The create button was therefore a 430px transparent box that pushed itself out of the panel and could not be seen. Moved onto the container it describes, which now centres its action rather than letting it fall off the bottom. Same class-mangling shape as the Agents header fixed earlier. Full e2e suite: 416 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): put Host's headline figures on the shared hairline strip Host had shadowed, clickable StatCards above a page that already has a rail, a pane and a tab bar — a second dashboard language on one screen, and a different one again from the figures inside its own detail pane. The Operate overview's figure grid is generalised into a shared `.stat-strip` and Host adopts it, so the two pages read as one system: same cell, same figure scale, same tone vocabulary, and the same hairline grid the split-view StatGrid already uses. The cells stay clickable and still route into the tab and filter they describe, because a count is worth more when it is also the way to the thing counted. Tone is spent only where the number means something — running and updates when non-zero — since a strip where every cell is coloured has no emphasis left. Two bugs made on the way, both now covered: - The first version put `<button>` elements inside a `<dl>` with `<dt>`/`<dd>` inside the buttons. Neither is valid, the browser re-parents both, and the cells collapsed. These cells are a set of controls, so a plain container of buttons is also the honest markup. - Even correct, the strip rendered 2px tall: `.page--app` is a flex column whose split view takes flex:1, so a child with no intrinsic minimum is shrunk away. The old cards survived only because `.stat-card` carried min-height:96px. The strip now declines to shrink, with a test pinning it. The stat-card specs are retargeted rather than deleted: they were written to guard a class collision on a page that no longer uses cards, so they now guard the strip's labels and its height. Full e2e suite: 417 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): make Backends notices an edge rather than a filled card The install and upgrade banners were tinted cards with a full border. A filled panel makes every notice shout at the weight of an error, which is how notices stop being read — and Backends shows one on most visits, so it was shouting routinely. They are now a hairline with a coloured left edge, the same treatment the Operate overview gives rows that want a decision, so "this needs you" looks the same wherever it appears. Counts in the notice take the monospace tabular figures the rest of the console uses. Also drops the last inline style on the page, and refreshes the inline-style baseline, which has read 624 against a real count since #11288 landed. The gate exits 0 either way, so nothing was failing — but a baseline 86 above the truth would have let that many inline styles back in unnoticed. Now at 538, which tightens the ratchet rather than loosening it. The spec creates the upgrade it asserts on rather than skipping when the mock has no notice: a test that skips is a test that proves nothing. Full e2e suite: 418 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): finish the mock parity list, and stop hiding the recommendations The last two items from the audit, plus a correction. **Discover's use-case shelf is lanes.** These are a list of ways in, read in order; a grid of equal cards asks the reader to compare them, which is not the choice on offer. **The request panel reaches every generator.** Video, 3D, Sound and Audio FX join Images and Speech, so each one teaches its own endpoint rather than two of six doing it. Audio FX records the fields that shape the request rather than the bytes, since its payload is multipart. **Recommendations no longer collapse themselves.** They were folded away by default once anything was installed. That is the page's one opinion about this host, and an opinion hidden by default is one the reader never gets. Someone who disagrees can still collapse it and that choice is remembered — the difference is that we no longer make it for them. Three specs asserted the old default and now assert the new one. The use-case heading also sat a line's width from the text it introduces, so the two read as one paragraph. It has air under it now, and the shelf is separated from the recommendations above it. Two tests removed rather than kept: a generator loop whose only real assertion was `expect(endpoint.length).toBeGreaterThan(0)`, and an earlier card-gap guard that could only skip. A test that cannot fail is worse than no test, because it reads as coverage. Full e2e suite: 418 passed, 4 skipped. Inline styles at baseline. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * fix(ui): make the Host figures legible and give the strip its spacing back Three defects introduced by the Host redesign, all found by looking at the running app rather than by the suite. **The figures were invisible.** "Running now" and "Updates available" rendered pure black on the dark ground. Two causes compounding: the `--muted` tone alias never landed, because the source rule has extra spaces before its brace and the exact-match edit missed it silently; and a `<button>` does not inherit colour, so with no tone rule the value fell back to the user agent's `buttontext`. Both fixed, and a test now fails on any figure computing to pure black. **The strip sat flush against the resources panel.** `.stat-strip` declares `margin: 0 0 ...` and is declared later in the file than `.manage-summary`, so the shorthand quietly won and the top margin became zero. Raised to `.stat-strip.manage-summary` so it beats the shorthand on specificity rather than on declaration order, which is the kind of thing that breaks again the next time a rule moves. **Discover's use-case heading had a doubled gap.** `.zero-pane` is a flex column that already separates its children; adding a margin on top of the gap stacked the two. The margin is gone and the heading keeps only its own breathing room. Full e2e suite: 420 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): make Studio's tabs path segments rather than a query parameter `/app/studio?tab=images` reads like a filter applied to a page. It is navigation: a different generator, with its own state and its own deep link. It is now `/app/studio/images`, with the overview at `/app/studio`. Legacy `?tab=` links are redirected once to the path form, replacing the history entry so Back does not bounce between two spellings of the same place. Bookmarks and older links keep working and land on the canonical URL rather than a second version of it, which is the part worth having a test for. The nine `?tab=` references were all in specs, none in docs, so the migration is contained. They move to paths, and a new spec pins the redirect. Full e2e suite: 421 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): make the hardware recommendation a section, not a dismissable card It was a bordered card with a collapse control and a close button, sitting inside a pane that is otherwise hairline sections. Two problems: it read as something bolted onto the page rather than part of it, and treating it as an interruption to be shut is the wrong frame for the one thing the page has to say about the machine it is running on. It is now a plain section with the same heading treatment as the shelves below it. The collapse state, the dismissal, their storage keys and the legacy key read for backwards compatibility all go with it, along with the installedCount prop that existed only to pick a default collapse. Five specs described behaviour that no longer exists and are removed rather than adjusted — collapsing, dismissing, persistence of both, and the toggle's keyboard handling. One new spec asserts the replacement contract: no control with aria-expanded, no dismiss, and no card border. Full e2e suite: 416 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * fix(ui): restore every stripped icon and every default-chrome control You reported two broken icons. They were not two: an earlier automated edit had stripped the `fa-*` class from twenty `<i>` elements across eleven pages, and an `<i>` with no icon class renders nothing at all. Settings' save button, the voice-profile back link, and eighteen others — agent row actions, task and job buttons, import and create actions — were all drawing empty space. Each is restored from its own context rather than a blanket icon: the agent row gets pause/play, pen, comments, file-export and trash; the fine-tune toggle swaps plus for xmark as it opens; the P2P documentation link gets the external-link glyph. The same edit left controls without their classes. Fine-tune's "Import config" was rendering in the browser's own chrome, and `.p2p-cmd__copy` set a border but no background, so it fell back to `buttonface` — a pale grey chip on a dark command block. FineTune's "New job" also had its icon classes folded into the button's className, the same mangling already fixed on the Agents header. Rather than fix the reported two and wait for the next report, this adds a standing audit: twenty-five routes are walked and the test fails on any visible control rendering with user-agent chrome, or any `<i>` without an `fa-*` class. It found the three remaining cases after the first sweep, and it is the reason the next one cannot ship quietly. Full e2e suite: 417 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
624 lines
27 KiB
Go
624 lines
27 KiB
Go
package schema
|
||
|
||
import (
|
||
"encoding/json"
|
||
"time"
|
||
|
||
gopsutil "github.com/shirou/gopsutil/v3/process"
|
||
)
|
||
|
||
type BackendMonitorRequest struct {
|
||
BasicModelRequest
|
||
}
|
||
|
||
// ModelLoadRequest asks LocalAI to pre-load a model into memory by name, so the
|
||
// first request that uses it pays no cold-start load cost. For a realtime
|
||
// pipeline model, every configured sub-model (VAD, transcription, LLM, TTS,
|
||
// sound_detection, voice_recognition) is loaded instead of the pipeline stub.
|
||
// It is the inverse of the /backend/shutdown request.
|
||
type ModelLoadRequest struct {
|
||
BasicModelRequest
|
||
}
|
||
|
||
// ModelLoadResponse reports the outcome of a /backend/load call.
|
||
type ModelLoadResponse struct {
|
||
// Loaded lists the model names actually resident in memory after the call.
|
||
// For a pipeline model these are its sub-models, not the pipeline name.
|
||
Loaded []string `json:"loaded"`
|
||
// Message is a short human-readable status ("model loaded", or an error).
|
||
Message string `json:"message"`
|
||
}
|
||
|
||
type TokenMetricsRequest struct {
|
||
BasicModelRequest
|
||
}
|
||
|
||
type BackendMonitorResponse struct {
|
||
MemoryInfo *gopsutil.MemoryInfoStat
|
||
MemoryPercent float32
|
||
CPUPercent float64
|
||
}
|
||
|
||
type GalleryResponse struct {
|
||
ID string `json:"uuid"`
|
||
StatusURL string `json:"status"`
|
||
|
||
EstimatedVRAMBytes uint64 `json:"estimated_vram_bytes,omitempty"`
|
||
EstimatedVRAMDisplay string `json:"estimated_vram_display,omitempty"`
|
||
EstimatedSizeBytes uint64 `json:"estimated_size_bytes,omitempty"`
|
||
EstimatedSizeDisplay string `json:"estimated_size_display,omitempty"`
|
||
}
|
||
|
||
type VideoRequest struct {
|
||
BasicModelRequest
|
||
Prompt string `json:"prompt" yaml:"prompt"` // text description of the video to generate
|
||
NegativePrompt string `json:"negative_prompt" yaml:"negative_prompt"` // things to avoid in the output
|
||
StartImage string `json:"start_image" yaml:"start_image"` // URL or base64 of the first frame
|
||
EndImage string `json:"end_image" yaml:"end_image"` // URL or base64 of the last frame
|
||
Audio string `json:"audio,omitempty" yaml:"audio,omitempty"` // URL or base64 audio for audio-conditioned generation
|
||
Width int32 `json:"width" yaml:"width"` // output width in pixels
|
||
Height int32 `json:"height" yaml:"height"` // output height in pixels
|
||
NumFrames int32 `json:"num_frames" yaml:"num_frames"` // total number of frames to generate
|
||
FPS int32 `json:"fps" yaml:"fps"` // frames per second
|
||
Seconds string `json:"seconds,omitempty" yaml:"seconds,omitempty"` // duration in seconds (alternative to num_frames)
|
||
Size string `json:"size,omitempty" yaml:"size,omitempty"` // WxH shorthand (e.g. "512x512")
|
||
InputReference string `json:"input_reference,omitempty" yaml:"input_reference,omitempty"` // reference image or video URL
|
||
Seed int32 `json:"seed" yaml:"seed"` // random seed for reproducibility
|
||
CFGScale float32 `json:"cfg_scale" yaml:"cfg_scale"` // classifier-free guidance scale
|
||
Step int32 `json:"step" yaml:"step"` // number of diffusion steps
|
||
ResponseFormat string `json:"response_format" yaml:"response_format"` // output format (url or b64_json)
|
||
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"` // backend-specific generation parameters
|
||
}
|
||
|
||
// @Description 3D asset generation request body. Generation is image-conditioned
|
||
// (TRELLIS.2 has no text-prompt path); the response is a binary glTF (.glb).
|
||
type Model3DRequest struct {
|
||
BasicModelRequest
|
||
Image string `json:"image" yaml:"image"` // conditioning image: URL, base64, or data URI (required)
|
||
Seed int32 `json:"seed,omitempty" yaml:"seed,omitempty"` // random seed; <=0 picks a random seed
|
||
Step int32 `json:"step,omitempty" yaml:"step,omitempty"` // flow sampling steps (backend default 12)
|
||
CFGScale float32 `json:"cfg_scale,omitempty" yaml:"cfg_scale,omitempty"` // classifier-free guidance scale (backend default 7.5)
|
||
TextureSteps int32 `json:"texture_steps,omitempty" yaml:"texture_steps,omitempty"` // texture flow sampling steps (backend default 12)
|
||
Quality string `json:"quality,omitempty" yaml:"quality,omitempty"` // mesh pipeline: auto|coarse|512|1024
|
||
Background string `json:"background,omitempty" yaml:"background,omitempty"` // background handling: auto|keep|black|white
|
||
ResponseFormat string `json:"response_format,omitempty" yaml:"response_format,omitempty"` // output format (url or b64_json)
|
||
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"` // backend-specific generation parameters
|
||
}
|
||
|
||
// @Description Print-remesh an existing trellis2.cpp GLB. The multipart mesh
|
||
// is wrapped into a watertight manifold; detail is a percentage of the source
|
||
// bounding-box diagonal and the enclosing offset is derived automatically.
|
||
type Model3DRemeshRequest struct {
|
||
BasicModelRequest
|
||
Detail float32 `json:"detail,omitempty" yaml:"detail,omitempty" form:"detail"` // detail size in percent (0.35–2.5; default 0.5)
|
||
}
|
||
|
||
// @Description TTS request body
|
||
type TTSRequest struct {
|
||
BasicModelRequest
|
||
Input string `json:"input" yaml:"input"` // text input
|
||
Voice string `json:"voice" yaml:"voice"` // voice audio file or speaker id
|
||
Backend string `json:"backend" yaml:"backend"` // backend engine override
|
||
Language string `json:"language,omitempty" yaml:"language,omitempty"` // (optional) language to use with TTS model
|
||
Format string `json:"response_format,omitempty" yaml:"response_format,omitempty"` // (optional) output format
|
||
Stream bool `json:"stream,omitempty" yaml:"stream,omitempty"` // (optional) enable streaming TTS
|
||
SampleRate int `json:"sample_rate,omitempty" yaml:"sample_rate,omitempty"` // (optional) desired output sample rate
|
||
// Speed is the OpenAI `speed` field (0.25-4.0). It is a pointer so an
|
||
// explicit `"speed": 0` (invalid, rejected with 400) is distinguishable
|
||
// from an omitted field (left at the backend default). It is normalised
|
||
// into Params["speed"] so it reaches the backend over the same channel as
|
||
// the other per-request generation parameters.
|
||
Speed *float32 `json:"speed,omitempty" yaml:"speed,omitempty"`
|
||
// Instructions is a free-form, per-request style/voice description. It maps to
|
||
// the OpenAI `instructions` field and is forwarded to the backend so expressive
|
||
// TTS models (e.g. Qwen3-TTS CustomVoice/VoiceDesign) can vary tone or designed
|
||
// voice per request instead of only via the static YAML option.
|
||
Instructions string `json:"instructions,omitempty" yaml:"instructions,omitempty"`
|
||
// Params carries optional, backend-specific per-request generation parameters
|
||
// (LocalAI extension, e.g. Chatterbox exaggeration/cfg_weight/temperature).
|
||
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"`
|
||
}
|
||
|
||
// @Description VAD request body
|
||
type VADRequest struct {
|
||
BasicModelRequest
|
||
Audio []float32 `json:"audio" yaml:"audio"` // raw audio samples as float32 PCM
|
||
}
|
||
|
||
type VADSegment struct {
|
||
Start float32 `json:"start" yaml:"start"`
|
||
End float32 `json:"end" yaml:"end"`
|
||
}
|
||
|
||
type VADResponse struct {
|
||
Segments []VADSegment `json:"segments" yaml:"segments"`
|
||
}
|
||
|
||
type StoreCommon struct {
|
||
Backend string `json:"backend,omitempty" yaml:"backend,omitempty"`
|
||
}
|
||
type StoresSet struct {
|
||
Store string `json:"store,omitempty" yaml:"store,omitempty"`
|
||
|
||
Keys [][]float32 `json:"keys" yaml:"keys"`
|
||
Values []string `json:"values" yaml:"values"`
|
||
StoreCommon
|
||
}
|
||
|
||
type StoresDelete struct {
|
||
Store string `json:"store,omitempty" yaml:"store,omitempty"`
|
||
|
||
Keys [][]float32 `json:"keys"`
|
||
StoreCommon
|
||
}
|
||
|
||
type StoresGet struct {
|
||
Store string `json:"store,omitempty" yaml:"store,omitempty"`
|
||
|
||
Keys [][]float32 `json:"keys" yaml:"keys"`
|
||
StoreCommon
|
||
}
|
||
|
||
type StoresGetResponse struct {
|
||
Keys [][]float32 `json:"keys" yaml:"keys"`
|
||
Values []string `json:"values" yaml:"values"`
|
||
}
|
||
|
||
type StoresFind struct {
|
||
Store string `json:"store,omitempty" yaml:"store,omitempty"`
|
||
|
||
Key []float32 `json:"key" yaml:"key"`
|
||
Topk int `json:"topk" yaml:"topk"`
|
||
StoreCommon
|
||
}
|
||
|
||
type StoresFindResponse struct {
|
||
Keys [][]float32 `json:"keys" yaml:"keys"`
|
||
Values []string `json:"values" yaml:"values"`
|
||
Similarities []float32 `json:"similarities" yaml:"similarities"`
|
||
}
|
||
|
||
type NodeData struct {
|
||
Name string
|
||
ID string
|
||
TunnelAddress string
|
||
ServiceID string
|
||
LastSeen time.Time
|
||
}
|
||
|
||
func (d NodeData) IsOnline() bool {
|
||
now := time.Now()
|
||
// if the node was seen in the last 40 seconds, it's online
|
||
return now.Sub(d.LastSeen) < 40*time.Second
|
||
}
|
||
|
||
type P2PNodesResponse struct {
|
||
LlamaCPPNodes []NodeData `json:"llama_cpp_nodes" yaml:"llama_cpp_nodes"`
|
||
FederatedNodes []NodeData `json:"federated_nodes" yaml:"federated_nodes"`
|
||
MLXNodes []NodeData `json:"mlx_nodes" yaml:"mlx_nodes"`
|
||
}
|
||
|
||
type SysInfoModel struct {
|
||
ID string `json:"id"`
|
||
// Backend is the engine serving this model. The loader knows only the ID,
|
||
// so it is resolved from the model's config; empty when the model was
|
||
// loaded without one (a loose file, or a config since removed).
|
||
Backend string `json:"backend,omitempty"`
|
||
}
|
||
|
||
type SystemInformationResponse struct {
|
||
Backends []string `json:"backends"` // available backend engines
|
||
Models []SysInfoModel `json:"loaded_models"` // currently loaded models
|
||
}
|
||
|
||
type DetectionRequest struct {
|
||
BasicModelRequest
|
||
Image string `json:"image"` // URL or base64-encoded image to analyze
|
||
Prompt string `json:"prompt,omitempty"` // Text prompt (for SAM 3 PCS mode)
|
||
Points []float32 `json:"points,omitempty"` // Point coordinates as [x,y,label,...] triples (label: 1=pos, 0=neg)
|
||
Boxes []float32 `json:"boxes,omitempty"` // Box coordinates as [x1,y1,x2,y2,...] quads
|
||
Threshold float32 `json:"threshold,omitempty"` // Detection confidence threshold
|
||
}
|
||
|
||
type DetectionResponse struct {
|
||
Detections []Detection `json:"detections"`
|
||
}
|
||
|
||
type Detection struct {
|
||
X float32 `json:"x"`
|
||
Y float32 `json:"y"`
|
||
Width float32 `json:"width"`
|
||
Height float32 `json:"height"`
|
||
ClassName string `json:"class_name"`
|
||
Confidence float32 `json:"confidence,omitempty"`
|
||
Mask string `json:"mask,omitempty"` // base64-encoded PNG segmentation mask
|
||
}
|
||
|
||
// DepthRequest is the request body for the /v1/depth endpoint. It exposes the
|
||
// full Depth Anything 3 output surface; the include_* flags and exports let a
|
||
// caller ask for less work (e.g. depth only, or depth+pose without the point
|
||
// cloud).
|
||
type DepthRequest struct {
|
||
BasicModelRequest
|
||
Image string `json:"image"` // URL or base64-encoded image to analyze
|
||
Dst string `json:"dst,omitempty"` // optional output directory for exports (glb/colmap)
|
||
IncludeDepth bool `json:"include_depth,omitempty"` // return the per-pixel depth map
|
||
IncludeConfidence bool `json:"include_confidence,omitempty"` // return the per-pixel confidence map (DualDPT)
|
||
IncludePose bool `json:"include_pose,omitempty"` // return camera extrinsics/intrinsics (DualDPT)
|
||
IncludeSky bool `json:"include_sky,omitempty"` // return the per-pixel sky map (mono models)
|
||
IncludePoints bool `json:"include_points,omitempty"` // back-project to a 3D point cloud (DualDPT)
|
||
PointsConfThresh float32 `json:"points_conf_thresh,omitempty"` // keep points with confidence >= this threshold
|
||
Exports []string `json:"exports,omitempty"` // requested exports: "glb", "colmap"
|
||
}
|
||
|
||
// DepthResponse is the JSON response for the /v1/depth endpoint, mirroring the
|
||
// DepthResponse proto.
|
||
type DepthResponse struct {
|
||
Width int32 `json:"width"`
|
||
Height int32 `json:"height"`
|
||
Depth []float32 `json:"depth,omitempty"` // width*height row-major metric depth
|
||
Confidence []float32 `json:"confidence,omitempty"` // width*height row-major confidence (DualDPT)
|
||
Sky []float32 `json:"sky,omitempty"` // width*height row-major sky map (mono)
|
||
Extrinsics []float32 `json:"extrinsics,omitempty"` // 12 floats, 3x4 row-major (world-to-camera)
|
||
Intrinsics []float32 `json:"intrinsics,omitempty"` // 9 floats, 3x3 row-major
|
||
NumPoints int32 `json:"num_points,omitempty"` // number of 3D points
|
||
Points []float32 `json:"points,omitempty"` // num_points*3 xyz, world space
|
||
PointColors string `json:"point_colors,omitempty"` // base64-encoded num_points*3 uint8 rgb
|
||
ExportPaths []string `json:"export_paths,omitempty"` // paths written for the requested exports
|
||
IsMetric bool `json:"is_metric"` // depth is in metric units
|
||
}
|
||
|
||
// ─── Face recognition ──────────────────────────────────────────────
|
||
//
|
||
// FacialArea describes a bounding box for a detected face.
|
||
type FacialArea struct {
|
||
X float32 `json:"x"`
|
||
Y float32 `json:"y"`
|
||
W float32 `json:"w"`
|
||
H float32 `json:"h"`
|
||
}
|
||
|
||
// FaceVerifyRequest compares two images to decide whether they depict
|
||
// the same person. Img1 and Img2 accept URL, base64, or data-URI.
|
||
type FaceVerifyRequest struct {
|
||
BasicModelRequest
|
||
Img1 string `json:"img1"`
|
||
Img2 string `json:"img2"`
|
||
Threshold float32 `json:"threshold,omitempty"`
|
||
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
|
||
}
|
||
|
||
type FaceVerifyResponse struct {
|
||
Verified bool `json:"verified"`
|
||
Distance float32 `json:"distance"`
|
||
Threshold float32 `json:"threshold"`
|
||
Confidence float32 `json:"confidence"`
|
||
Model string `json:"model"`
|
||
Img1Area FacialArea `json:"img1_area"`
|
||
Img2Area FacialArea `json:"img2_area"`
|
||
ProcessingTimeMs float32 `json:"processing_time_ms,omitempty"`
|
||
// Liveness fields are only populated when the request set
|
||
// anti_spoofing=true. Pointers keep them fully absent from the
|
||
// JSON response otherwise, so callers can tell "not checked"
|
||
// apart from "checked and fake" (which would collapse to zero
|
||
// values with plain bool+omitempty).
|
||
Img1IsReal *bool `json:"img1_is_real,omitempty"`
|
||
Img1AntispoofScore *float32 `json:"img1_antispoof_score,omitempty"`
|
||
Img2IsReal *bool `json:"img2_is_real,omitempty"`
|
||
Img2AntispoofScore *float32 `json:"img2_antispoof_score,omitempty"`
|
||
}
|
||
|
||
// FaceAnalyzeRequest asks the backend for demographic attributes on
|
||
// every face detected in Img.
|
||
type FaceAnalyzeRequest struct {
|
||
BasicModelRequest
|
||
Img string `json:"img"`
|
||
Actions []string `json:"actions,omitempty"` // subset of {"age","gender","emotion","race"}
|
||
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
|
||
}
|
||
|
||
type FaceAnalyzeResponse struct {
|
||
Faces []FaceAnalysis `json:"faces"`
|
||
}
|
||
|
||
type FaceAnalysis struct {
|
||
Region FacialArea `json:"region"`
|
||
FaceConfidence float32 `json:"face_confidence"`
|
||
Age float32 `json:"age,omitempty"`
|
||
DominantGender string `json:"dominant_gender,omitempty"`
|
||
Gender map[string]float32 `json:"gender,omitempty"`
|
||
DominantEmotion string `json:"dominant_emotion,omitempty"`
|
||
Emotion map[string]float32 `json:"emotion,omitempty"`
|
||
DominantRace string `json:"dominant_race,omitempty"`
|
||
Race map[string]float32 `json:"race,omitempty"`
|
||
// Liveness fields — see FaceVerifyResponse for why these are pointers.
|
||
IsReal *bool `json:"is_real,omitempty"`
|
||
AntispoofScore *float32 `json:"antispoof_score,omitempty"`
|
||
}
|
||
|
||
// FaceEmbedRequest extracts a face embedding from an image. Distinct
|
||
// from /v1/embeddings (which is OpenAI-compatible and text-only); this
|
||
// endpoint accepts URL / base64 / data-URI image inputs.
|
||
type FaceEmbedRequest struct {
|
||
BasicModelRequest
|
||
Img string `json:"img"`
|
||
}
|
||
|
||
type FaceEmbedResponse struct {
|
||
Embedding []float32 `json:"embedding"`
|
||
Dim int `json:"dim"`
|
||
Model string `json:"model,omitempty"`
|
||
}
|
||
|
||
// FaceRegisterRequest enrolls a face into the 1:N recognition store.
|
||
type FaceRegisterRequest struct {
|
||
BasicModelRequest
|
||
Img string `json:"img"`
|
||
Name string `json:"name"`
|
||
Labels map[string]string `json:"labels,omitempty"`
|
||
Store string `json:"store,omitempty"` // vector store model; empty = local-store default
|
||
}
|
||
|
||
type FaceRegisterResponse struct {
|
||
ID string `json:"id"`
|
||
Name string `json:"name"`
|
||
RegisteredAt time.Time `json:"registered_at"`
|
||
}
|
||
|
||
// FaceIdentifyRequest runs 1:N recognition: embed the probe and
|
||
// return the top-K nearest registered faces.
|
||
type FaceIdentifyRequest struct {
|
||
BasicModelRequest
|
||
Img string `json:"img"`
|
||
TopK int `json:"top_k,omitempty"`
|
||
Threshold float32 `json:"threshold,omitempty"` // optional cutoff on distance
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
type FaceIdentifyResponse struct {
|
||
Matches []FaceIdentifyMatch `json:"matches"`
|
||
}
|
||
|
||
type FaceIdentifyMatch struct {
|
||
ID string `json:"id"`
|
||
Name string `json:"name"`
|
||
Labels map[string]string `json:"labels,omitempty"`
|
||
Distance float32 `json:"distance"`
|
||
Confidence float32 `json:"confidence"`
|
||
Match bool `json:"match"` // true when distance <= threshold
|
||
}
|
||
|
||
// FaceForgetRequest removes a previously-registered face by ID.
|
||
type FaceForgetRequest struct {
|
||
BasicModelRequest
|
||
ID string `json:"id"`
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
// ─── Voice (speaker) recognition ───────────────────────────────────
|
||
//
|
||
// VoiceVerifyRequest compares two audio clips and reports whether they
|
||
// were spoken by the same speaker. Audio1/Audio2 accept URL, base64,
|
||
// or data-URI (the HTTP layer materialises the bytes to a temp file
|
||
// before calling the gRPC backend).
|
||
type VoiceVerifyRequest struct {
|
||
BasicModelRequest
|
||
Audio1 string `json:"audio1"`
|
||
Audio2 string `json:"audio2"`
|
||
Threshold float32 `json:"threshold,omitempty"`
|
||
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
|
||
}
|
||
|
||
type VoiceVerifyResponse struct {
|
||
Verified bool `json:"verified"`
|
||
Distance float32 `json:"distance"`
|
||
Threshold float32 `json:"threshold"`
|
||
Confidence float32 `json:"confidence"`
|
||
Model string `json:"model"`
|
||
ProcessingTimeMs float32 `json:"processing_time_ms,omitempty"`
|
||
}
|
||
|
||
// VoiceAnalyzeRequest asks the backend for demographic attributes
|
||
// (age, gender, emotion) inferred from the audio clip.
|
||
type VoiceAnalyzeRequest struct {
|
||
BasicModelRequest
|
||
Audio string `json:"audio"`
|
||
Actions []string `json:"actions,omitempty"` // subset of {"age","gender","emotion"}
|
||
}
|
||
|
||
type VoiceAnalyzeResponse struct {
|
||
Segments []VoiceAnalysis `json:"segments"`
|
||
}
|
||
|
||
type VoiceAnalysis struct {
|
||
Start float32 `json:"start"`
|
||
End float32 `json:"end"`
|
||
Age float32 `json:"age,omitempty"`
|
||
DominantGender string `json:"dominant_gender,omitempty"`
|
||
Gender map[string]float32 `json:"gender,omitempty"`
|
||
DominantEmotion string `json:"dominant_emotion,omitempty"`
|
||
Emotion map[string]float32 `json:"emotion,omitempty"`
|
||
}
|
||
|
||
// VoiceEmbedRequest extracts a speaker embedding from an audio clip.
|
||
// Distinct from /v1/embeddings (OpenAI-compatible, text-only) — this
|
||
// endpoint accepts URL / base64 / data-URI audio inputs.
|
||
type VoiceEmbedRequest struct {
|
||
BasicModelRequest
|
||
Audio string `json:"audio"`
|
||
}
|
||
|
||
type VoiceEmbedResponse struct {
|
||
Embedding []float32 `json:"embedding"`
|
||
Dim int `json:"dim"`
|
||
Model string `json:"model,omitempty"`
|
||
}
|
||
|
||
// VoiceRegisterRequest enrolls a speaker into the 1:N identification store.
|
||
type VoiceRegisterRequest struct {
|
||
BasicModelRequest
|
||
Audio string `json:"audio"`
|
||
Name string `json:"name"`
|
||
Labels map[string]string `json:"labels,omitempty"`
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
type VoiceRegisterResponse struct {
|
||
ID string `json:"id"`
|
||
Name string `json:"name"`
|
||
RegisteredAt time.Time `json:"registered_at"`
|
||
}
|
||
|
||
// VoiceIdentifyRequest runs 1:N recognition: embed the probe and
|
||
// return the top-K nearest registered speakers.
|
||
type VoiceIdentifyRequest struct {
|
||
BasicModelRequest
|
||
Audio string `json:"audio"`
|
||
TopK int `json:"top_k,omitempty"`
|
||
Threshold float32 `json:"threshold,omitempty"`
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
type VoiceIdentifyResponse struct {
|
||
Matches []VoiceIdentifyMatch `json:"matches"`
|
||
}
|
||
|
||
type VoiceIdentifyMatch struct {
|
||
ID string `json:"id"`
|
||
Name string `json:"name"`
|
||
Labels map[string]string `json:"labels,omitempty"`
|
||
Distance float32 `json:"distance"`
|
||
Confidence float32 `json:"confidence"`
|
||
Match bool `json:"match"`
|
||
}
|
||
|
||
// VoiceForgetRequest removes a previously-registered speaker by ID.
|
||
type VoiceForgetRequest struct {
|
||
BasicModelRequest
|
||
ID string `json:"id"`
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
type ImportModelRequest struct {
|
||
URI string `json:"uri"`
|
||
Preferences json.RawMessage `json:"preferences,omitempty"`
|
||
}
|
||
|
||
// KnownBackend describes a backend that the importer knows about.
|
||
// Used by GET /backends/known to populate the import form dropdown.
|
||
type KnownBackend struct {
|
||
Name string `json:"name"`
|
||
Modality string `json:"modality"`
|
||
AutoDetect bool `json:"auto_detect"`
|
||
Description string `json:"description,omitempty"`
|
||
// Installed is true when the backend is currently present on disk — i.e. it
|
||
// appears in gallery.ListSystemBackends(systemState). Importer-registered or
|
||
// curated pref-only backends default to false unless they also show up on
|
||
// disk. The import form uses this to warn users that submitting an import
|
||
// may trigger an automatic backend download.
|
||
Installed bool `json:"installed"`
|
||
}
|
||
|
||
// SettingsResponse is the response type for settings API operations
|
||
type SettingsResponse struct {
|
||
Success bool `json:"success"`
|
||
Error string `json:"error,omitempty"`
|
||
Message string `json:"message,omitempty"`
|
||
}
|
||
|
||
// RouterDecideRequest is the input for POST /api/router/decide — the
|
||
// programmatic decision-oracle endpoint. Given the name of a router
|
||
// model (a ModelConfig that carries a `router:` block) and a prompt,
|
||
// the endpoint returns the classifier's label set plus the candidate
|
||
// model the in-band RouteModel middleware would have chosen. The
|
||
// endpoint does NOT rewrite any request, forward to a backend, or
|
||
// record a row in the decision store — it is a pure decision oracle
|
||
// for external routers that want LocalAI's classifier opinion without
|
||
// committing LocalAI to handle the request.
|
||
type RouterDecideRequest struct {
|
||
// Router is the name of the router model (a ModelConfig with a
|
||
// `router:` block). Required.
|
||
Router string `json:"router"`
|
||
// Input is the user-visible prompt text to classify. Required.
|
||
// Schema-shape extraction (chat-message concatenation, etc.) is
|
||
// the caller's responsibility — matches the Probe contract used
|
||
// by the in-band middleware.
|
||
Input string `json:"input"`
|
||
}
|
||
|
||
// RouterDecideResponse carries the classifier's decision plus the
|
||
// resolved candidate. Mirrors router.Decision with the addition of
|
||
// Candidate/Fallback so the caller learns which downstream model
|
||
// would have served the request without re-implementing the
|
||
// label-set → candidate match locally.
|
||
type RouterDecideResponse struct {
|
||
// Router echoes the requested router model.
|
||
Router string `json:"router"`
|
||
// Classifier is the classifier name that produced the decision
|
||
// (e.g. "score").
|
||
Classifier string `json:"classifier"`
|
||
// Labels is the set of active policy labels.
|
||
Labels []string `json:"labels"`
|
||
// Candidate is the model that would be routed to. Empty when no
|
||
// candidate covers Labels AND no fallback is configured.
|
||
Candidate string `json:"candidate,omitempty"`
|
||
// Fallback is true when Candidate is the router's configured
|
||
// fallback because no candidate covered Labels. Lets callers
|
||
// distinguish "matched" from "fell back" without comparing names.
|
||
Fallback bool `json:"fallback,omitempty"`
|
||
// Score is the top label's softmax probability (the
|
||
// classifier-side confidence signal).
|
||
Score float64 `json:"score"`
|
||
// LatencyMs is the classifier's wall-clock cost.
|
||
LatencyMs int64 `json:"latency_ms"`
|
||
// Cached is true when the decision came from the L2 embedding
|
||
// cache rather than a fresh classifier run.
|
||
Cached bool `json:"cached,omitempty"`
|
||
// CacheSimilarity carries the cosine similarity of the cache hit
|
||
// (0 when not cached).
|
||
CacheSimilarity float64 `json:"cache_similarity,omitempty"`
|
||
}
|
||
|
||
// PIIDecideRequest is the input for POST /api/pii/decide — the
|
||
// programmatic PII-decision oracle. External routers call it before
|
||
// dispatching a request to learn whether the content carries PII and
|
||
// what action the configured pattern set would take. The endpoint
|
||
// inspects the text and returns findings + a suggested action; it
|
||
// does NOT mutate the input, record an audit event, or rewrite any
|
||
// downstream request. The caller composes the decision with its own
|
||
// policy (mask, block, or allow).
|
||
type PIIDecideRequest struct {
|
||
// Text is the user-visible content to inspect. Required.
|
||
Text string `json:"text"`
|
||
}
|
||
|
||
// PIIDecideResponse carries the redactor's findings.
|
||
// SuggestedAction is derived from the action ordering used by the
|
||
// internal redactor (block > mask > allow) so callers don't need to
|
||
// replicate that logic.
|
||
type PIIDecideResponse struct {
|
||
// Findings is one entry per matched span — pattern id, byte
|
||
// range, and audit-safe hash prefix (never the matched value).
|
||
Findings []PIIFinding `json:"findings"`
|
||
// SuggestedAction is the strongest action across all findings:
|
||
// "block", "mask", or "allow" (no findings, or all findings
|
||
// resolved to the allow action).
|
||
SuggestedAction string `json:"suggested_action"`
|
||
// RedactedPreview is the input with mask-action spans replaced
|
||
// by their placeholders. Identical to Text when no findings or
|
||
// when the strongest action is block/allow (which don't rewrite
|
||
// content).
|
||
RedactedPreview string `json:"redacted_preview"`
|
||
}
|
||
|
||
// PIIFinding mirrors pii.Span on the wire. Pattern is the pattern id
|
||
// that matched (e.g. "email"). HashPrefix is the first 8 chars of
|
||
// sha256(matched value) — lets admins correlate recurring leaks
|
||
// without recovering the value itself.
|
||
type PIIFinding struct {
|
||
Start int `json:"start"`
|
||
End int `json:"end"`
|
||
Pattern string `json:"pattern"`
|
||
HashPrefix string `json:"hash_prefix"`
|
||
}
|