Files
LocalAI/core/schema/localai.go
mudler's LocalAI [bot] 5cb0c1a872 feat(ui): close the gap between the shipped UI and the design mocks (#11307)
* feat(ui): give the Operate overview real numbers and traces a latency shape

First two items from a component-by-component comparison against the mocks.
The pattern that audit found: everything newly built matched, everything
pre-existing got the palette but not the layout, and an "absent rather than
empty" rule hid most of the overview exactly when someone was looking at an
idle installation.

**The headline grid is always rendered**, including at zero, with a fourth cell
for host memory. Hiding it removed the page's structure precisely when it was
most likely to be read, and "0 failed" is information — an absent panel is not.
The quiet case is now said in a line underneath instead of by showing nothing.

**The sections state counts** rather than listing their destinations: backends,
models, updates and running operations instead of the words "Usage and traces".
That needed installed backend and model counts in the summary context, which
are two more cheap reads on the poll that was already running.

**Traces rows carry latency as a bar as well as a figure**, scaled against the
slowest request currently in view and turning amber past two seconds. The table
had no latency column at all — the number was buried in the expanded detail, so
the shape of the tail was invisible while scanning. Scaling against the view
rather than an absolute ceiling is deliberate: what matters when reading a page
of traces is which of these are the outliers, and an absolute scale flattens
every row on a fast installation into nothing.

Full e2e suite: 409 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* feat(ui): name the engine on Home's resident models, and add jump-back-in

Third item from the mock comparison.

The mock showed each resident model with the engine serving it. /system carried
only the id, so the audit recorded this as blocked on a server field — but the
config loader is already in scope where that response is built, so it is one
lookup. SysInfoModel gains an optional `backend`, resolved from the model's
config and omitted rather than guessed when there is none (a loose file, or a
config since removed). Home renders the column blank in that case; the test
pins both halves of that.

Memory per model stays out. It is not one lookup — it would mean asking each
backend process — and inventing a number beside a real one is worse than
leaving the column off.

"Jump back in" is the block the mock had and Home did not. The quick-links row
above it is a set of first-run actions; these are the three places someone
returns to, each stated with what it currently holds rather than as a bare
label.

Go: routes suite passes. Full e2e suite: 412 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* feat(ui): rank the recommended models as lanes instead of equal cards

The hardware recommendations were a grid of equally-weighted cards. The list is
already sorted by fit, and a grid throws that order away: three cards side by
side say "pick one", when the page has actually formed an opinion about which
one.

They are lanes now, read top to bottom in fit order, with the leader carrying
the single amber "Best fit" label and the rest marked "Also fits". One opinion
per page — the alternatives are alternatives, not runners-up each worth their
own colour, which is how a strip of coloured badges ends up meaning nothing.

Below 720px the size and VRAM columns drop and the lane keeps the name and the
install action, which are the two things a narrow screen needs.

The existing panel spec moves off .rec-models-item onto .lane rather than being
deleted; dismissal, collapse, keyboard operation and install all still pass
unchanged, and there is a new assertion that exactly one row is called out.

Full e2e suite: 413 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* fix(ui): drop capsule chips app-wide, and un-break the empty voice library

**Pills are gone.** A capsule radius reads as a tag floating on the surface,
which fights a system whose structure is hairlines and square corners — and
with chips on Discover, Host, Activity and the biometrics pages, "some pages
have pills" was the real inconsistency rather than any one page.

Sixteen selectors move to the small radius: filter buttons, tab pills, activity
and biometrics chips, file and count badges, the jump-to-latest control, the
nav badge. Round *buttons* keep their circle — .lightbox__nav and
.home-send-btn are circles, not capsules — as do every progress track, status
dot and avatar, which are round because they are round, not because they are
tags.

**The empty voice library was unusable.** `.voice-library-empty` sets
min-height: 430px, border: 0 and background: transparent — a description of the
empty PANEL — and it had been attached to the action instead. The create button
was therefore a 430px transparent box that pushed itself out of the panel and
could not be seen. Moved onto the container it describes, which now centres its
action rather than letting it fall off the bottom. Same class-mangling shape as
the Agents header fixed earlier.

Full e2e suite: 416 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* feat(ui): put Host's headline figures on the shared hairline strip

Host had shadowed, clickable StatCards above a page that already has a rail, a
pane and a tab bar — a second dashboard language on one screen, and a different
one again from the figures inside its own detail pane.

The Operate overview's figure grid is generalised into a shared `.stat-strip`
and Host adopts it, so the two pages read as one system: same cell, same figure
scale, same tone vocabulary, and the same hairline grid the split-view StatGrid
already uses. The cells stay clickable and still route into the tab and filter
they describe, because a count is worth more when it is also the way to the
thing counted.

Tone is spent only where the number means something — running and updates when
non-zero — since a strip where every cell is coloured has no emphasis left.

Two bugs made on the way, both now covered:

- The first version put `<button>` elements inside a `<dl>` with `<dt>`/`<dd>`
  inside the buttons. Neither is valid, the browser re-parents both, and the
  cells collapsed. These cells are a set of controls, so a plain container of
  buttons is also the honest markup.
- Even correct, the strip rendered 2px tall: `.page--app` is a flex column
  whose split view takes flex:1, so a child with no intrinsic minimum is shrunk
  away. The old cards survived only because `.stat-card` carried
  min-height:96px. The strip now declines to shrink, with a test pinning it.

The stat-card specs are retargeted rather than deleted: they were written to
guard a class collision on a page that no longer uses cards, so they now guard
the strip's labels and its height.

Full e2e suite: 417 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* feat(ui): make Backends notices an edge rather than a filled card

The install and upgrade banners were tinted cards with a full border. A filled
panel makes every notice shout at the weight of an error, which is how notices
stop being read — and Backends shows one on most visits, so it was shouting
routinely.

They are now a hairline with a coloured left edge, the same treatment the
Operate overview gives rows that want a decision, so "this needs you" looks the
same wherever it appears. Counts in the notice take the monospace tabular
figures the rest of the console uses.

Also drops the last inline style on the page, and refreshes the inline-style
baseline, which has read 624 against a real count since #11288 landed. The gate
exits 0 either way, so nothing was failing — but a baseline 86 above the truth
would have let that many inline styles back in unnoticed. Now at 538, which
tightens the ratchet rather than loosening it.

The spec creates the upgrade it asserts on rather than skipping when the mock
has no notice: a test that skips is a test that proves nothing.

Full e2e suite: 418 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* feat(ui): finish the mock parity list, and stop hiding the recommendations

The last two items from the audit, plus a correction.

**Discover's use-case shelf is lanes.** These are a list of ways in, read in
order; a grid of equal cards asks the reader to compare them, which is not the
choice on offer.

**The request panel reaches every generator.** Video, 3D, Sound and Audio FX
join Images and Speech, so each one teaches its own endpoint rather than two of
six doing it. Audio FX records the fields that shape the request rather than
the bytes, since its payload is multipart.

**Recommendations no longer collapse themselves.** They were folded away by
default once anything was installed. That is the page's one opinion about this
host, and an opinion hidden by default is one the reader never gets. Someone
who disagrees can still collapse it and that choice is remembered — the
difference is that we no longer make it for them. Three specs asserted the old
default and now assert the new one.

The use-case heading also sat a line's width from the text it introduces, so
the two read as one paragraph. It has air under it now, and the shelf is
separated from the recommendations above it.

Two tests removed rather than kept: a generator loop whose only real assertion
was `expect(endpoint.length).toBeGreaterThan(0)`, and an earlier card-gap guard
that could only skip. A test that cannot fail is worse than no test, because it
reads as coverage.

Full e2e suite: 418 passed, 4 skipped. Inline styles at baseline.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* fix(ui): make the Host figures legible and give the strip its spacing back

Three defects introduced by the Host redesign, all found by looking at the
running app rather than by the suite.

**The figures were invisible.** "Running now" and "Updates available" rendered
pure black on the dark ground. Two causes compounding: the `--muted` tone alias
never landed, because the source rule has extra spaces before its brace and the
exact-match edit missed it silently; and a `<button>` does not inherit colour,
so with no tone rule the value fell back to the user agent's `buttontext`.
Both fixed, and a test now fails on any figure computing to pure black.

**The strip sat flush against the resources panel.** `.stat-strip` declares
`margin: 0 0 ...` and is declared later in the file than `.manage-summary`, so
the shorthand quietly won and the top margin became zero. Raised to
`.stat-strip.manage-summary` so it beats the shorthand on specificity rather
than on declaration order, which is the kind of thing that breaks again the
next time a rule moves.

**Discover's use-case heading had a doubled gap.** `.zero-pane` is a flex
column that already separates its children; adding a margin on top of the gap
stacked the two. The margin is gone and the heading keeps only its own breathing
room.

Full e2e suite: 420 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* feat(ui): make Studio's tabs path segments rather than a query parameter

`/app/studio?tab=images` reads like a filter applied to a page. It is
navigation: a different generator, with its own state and its own deep link. It
is now `/app/studio/images`, with the overview at `/app/studio`.

Legacy `?tab=` links are redirected once to the path form, replacing the
history entry so Back does not bounce between two spellings of the same place.
Bookmarks and older links keep working and land on the canonical URL rather
than a second version of it, which is the part worth having a test for.

The nine `?tab=` references were all in specs, none in docs, so the migration
is contained. They move to paths, and a new spec pins the redirect.

Full e2e suite: 421 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* feat(ui): make the hardware recommendation a section, not a dismissable card

It was a bordered card with a collapse control and a close button, sitting
inside a pane that is otherwise hairline sections. Two problems: it read as
something bolted onto the page rather than part of it, and treating it as an
interruption to be shut is the wrong frame for the one thing the page has to
say about the machine it is running on.

It is now a plain section with the same heading treatment as the shelves below
it. The collapse state, the dismissal, their storage keys and the legacy key
read for backwards compatibility all go with it, along with the installedCount
prop that existed only to pick a default collapse.

Five specs described behaviour that no longer exists and are removed rather
than adjusted — collapsing, dismissing, persistence of both, and the toggle's
keyboard handling. One new spec asserts the replacement contract: no control
with aria-expanded, no dismiss, and no card border.

Full e2e suite: 416 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

* fix(ui): restore every stripped icon and every default-chrome control

You reported two broken icons. They were not two: an earlier automated edit had
stripped the `fa-*` class from twenty `<i>` elements across eleven pages, and an
`<i>` with no icon class renders nothing at all. Settings' save button, the
voice-profile back link, and eighteen others — agent row actions, task and job
buttons, import and create actions — were all drawing empty space.

Each is restored from its own context rather than a blanket icon: the agent row
gets pause/play, pen, comments, file-export and trash; the fine-tune toggle
swaps plus for xmark as it opens; the P2P documentation link gets the
external-link glyph.

The same edit left controls without their classes. Fine-tune's "Import config"
was rendering in the browser's own chrome, and `.p2p-cmd__copy` set a border
but no background, so it fell back to `buttonface` — a pale grey chip on a dark
command block. FineTune's "New job" also had its icon classes folded into the
button's className, the same mangling already fixed on the Agents header.

Rather than fix the reported two and wait for the next report, this adds a
standing audit: twenty-five routes are walked and the test fails on any visible
control rendering with user-agent chrome, or any `<i>` without an `fa-*` class.
It found the three remaining cases after the first sweep, and it is the reason
the next one cannot ship quietly.

Full e2e suite: 417 passed, 4 skipped.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-08-03 12:16:09 +02:00

624 lines
27 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package schema
import (
"encoding/json"
"time"
gopsutil "github.com/shirou/gopsutil/v3/process"
)
type BackendMonitorRequest struct {
BasicModelRequest
}
// ModelLoadRequest asks LocalAI to pre-load a model into memory by name, so the
// first request that uses it pays no cold-start load cost. For a realtime
// pipeline model, every configured sub-model (VAD, transcription, LLM, TTS,
// sound_detection, voice_recognition) is loaded instead of the pipeline stub.
// It is the inverse of the /backend/shutdown request.
type ModelLoadRequest struct {
BasicModelRequest
}
// ModelLoadResponse reports the outcome of a /backend/load call.
type ModelLoadResponse struct {
// Loaded lists the model names actually resident in memory after the call.
// For a pipeline model these are its sub-models, not the pipeline name.
Loaded []string `json:"loaded"`
// Message is a short human-readable status ("model loaded", or an error).
Message string `json:"message"`
}
type TokenMetricsRequest struct {
BasicModelRequest
}
type BackendMonitorResponse struct {
MemoryInfo *gopsutil.MemoryInfoStat
MemoryPercent float32
CPUPercent float64
}
type GalleryResponse struct {
ID string `json:"uuid"`
StatusURL string `json:"status"`
EstimatedVRAMBytes uint64 `json:"estimated_vram_bytes,omitempty"`
EstimatedVRAMDisplay string `json:"estimated_vram_display,omitempty"`
EstimatedSizeBytes uint64 `json:"estimated_size_bytes,omitempty"`
EstimatedSizeDisplay string `json:"estimated_size_display,omitempty"`
}
type VideoRequest struct {
BasicModelRequest
Prompt string `json:"prompt" yaml:"prompt"` // text description of the video to generate
NegativePrompt string `json:"negative_prompt" yaml:"negative_prompt"` // things to avoid in the output
StartImage string `json:"start_image" yaml:"start_image"` // URL or base64 of the first frame
EndImage string `json:"end_image" yaml:"end_image"` // URL or base64 of the last frame
Audio string `json:"audio,omitempty" yaml:"audio,omitempty"` // URL or base64 audio for audio-conditioned generation
Width int32 `json:"width" yaml:"width"` // output width in pixels
Height int32 `json:"height" yaml:"height"` // output height in pixels
NumFrames int32 `json:"num_frames" yaml:"num_frames"` // total number of frames to generate
FPS int32 `json:"fps" yaml:"fps"` // frames per second
Seconds string `json:"seconds,omitempty" yaml:"seconds,omitempty"` // duration in seconds (alternative to num_frames)
Size string `json:"size,omitempty" yaml:"size,omitempty"` // WxH shorthand (e.g. "512x512")
InputReference string `json:"input_reference,omitempty" yaml:"input_reference,omitempty"` // reference image or video URL
Seed int32 `json:"seed" yaml:"seed"` // random seed for reproducibility
CFGScale float32 `json:"cfg_scale" yaml:"cfg_scale"` // classifier-free guidance scale
Step int32 `json:"step" yaml:"step"` // number of diffusion steps
ResponseFormat string `json:"response_format" yaml:"response_format"` // output format (url or b64_json)
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"` // backend-specific generation parameters
}
// @Description 3D asset generation request body. Generation is image-conditioned
// (TRELLIS.2 has no text-prompt path); the response is a binary glTF (.glb).
type Model3DRequest struct {
BasicModelRequest
Image string `json:"image" yaml:"image"` // conditioning image: URL, base64, or data URI (required)
Seed int32 `json:"seed,omitempty" yaml:"seed,omitempty"` // random seed; <=0 picks a random seed
Step int32 `json:"step,omitempty" yaml:"step,omitempty"` // flow sampling steps (backend default 12)
CFGScale float32 `json:"cfg_scale,omitempty" yaml:"cfg_scale,omitempty"` // classifier-free guidance scale (backend default 7.5)
TextureSteps int32 `json:"texture_steps,omitempty" yaml:"texture_steps,omitempty"` // texture flow sampling steps (backend default 12)
Quality string `json:"quality,omitempty" yaml:"quality,omitempty"` // mesh pipeline: auto|coarse|512|1024
Background string `json:"background,omitempty" yaml:"background,omitempty"` // background handling: auto|keep|black|white
ResponseFormat string `json:"response_format,omitempty" yaml:"response_format,omitempty"` // output format (url or b64_json)
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"` // backend-specific generation parameters
}
// @Description Print-remesh an existing trellis2.cpp GLB. The multipart mesh
// is wrapped into a watertight manifold; detail is a percentage of the source
// bounding-box diagonal and the enclosing offset is derived automatically.
type Model3DRemeshRequest struct {
BasicModelRequest
Detail float32 `json:"detail,omitempty" yaml:"detail,omitempty" form:"detail"` // detail size in percent (0.352.5; default 0.5)
}
// @Description TTS request body
type TTSRequest struct {
BasicModelRequest
Input string `json:"input" yaml:"input"` // text input
Voice string `json:"voice" yaml:"voice"` // voice audio file or speaker id
Backend string `json:"backend" yaml:"backend"` // backend engine override
Language string `json:"language,omitempty" yaml:"language,omitempty"` // (optional) language to use with TTS model
Format string `json:"response_format,omitempty" yaml:"response_format,omitempty"` // (optional) output format
Stream bool `json:"stream,omitempty" yaml:"stream,omitempty"` // (optional) enable streaming TTS
SampleRate int `json:"sample_rate,omitempty" yaml:"sample_rate,omitempty"` // (optional) desired output sample rate
// Speed is the OpenAI `speed` field (0.25-4.0). It is a pointer so an
// explicit `"speed": 0` (invalid, rejected with 400) is distinguishable
// from an omitted field (left at the backend default). It is normalised
// into Params["speed"] so it reaches the backend over the same channel as
// the other per-request generation parameters.
Speed *float32 `json:"speed,omitempty" yaml:"speed,omitempty"`
// Instructions is a free-form, per-request style/voice description. It maps to
// the OpenAI `instructions` field and is forwarded to the backend so expressive
// TTS models (e.g. Qwen3-TTS CustomVoice/VoiceDesign) can vary tone or designed
// voice per request instead of only via the static YAML option.
Instructions string `json:"instructions,omitempty" yaml:"instructions,omitempty"`
// Params carries optional, backend-specific per-request generation parameters
// (LocalAI extension, e.g. Chatterbox exaggeration/cfg_weight/temperature).
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"`
}
// @Description VAD request body
type VADRequest struct {
BasicModelRequest
Audio []float32 `json:"audio" yaml:"audio"` // raw audio samples as float32 PCM
}
type VADSegment struct {
Start float32 `json:"start" yaml:"start"`
End float32 `json:"end" yaml:"end"`
}
type VADResponse struct {
Segments []VADSegment `json:"segments" yaml:"segments"`
}
type StoreCommon struct {
Backend string `json:"backend,omitempty" yaml:"backend,omitempty"`
}
type StoresSet struct {
Store string `json:"store,omitempty" yaml:"store,omitempty"`
Keys [][]float32 `json:"keys" yaml:"keys"`
Values []string `json:"values" yaml:"values"`
StoreCommon
}
type StoresDelete struct {
Store string `json:"store,omitempty" yaml:"store,omitempty"`
Keys [][]float32 `json:"keys"`
StoreCommon
}
type StoresGet struct {
Store string `json:"store,omitempty" yaml:"store,omitempty"`
Keys [][]float32 `json:"keys" yaml:"keys"`
StoreCommon
}
type StoresGetResponse struct {
Keys [][]float32 `json:"keys" yaml:"keys"`
Values []string `json:"values" yaml:"values"`
}
type StoresFind struct {
Store string `json:"store,omitempty" yaml:"store,omitempty"`
Key []float32 `json:"key" yaml:"key"`
Topk int `json:"topk" yaml:"topk"`
StoreCommon
}
type StoresFindResponse struct {
Keys [][]float32 `json:"keys" yaml:"keys"`
Values []string `json:"values" yaml:"values"`
Similarities []float32 `json:"similarities" yaml:"similarities"`
}
type NodeData struct {
Name string
ID string
TunnelAddress string
ServiceID string
LastSeen time.Time
}
func (d NodeData) IsOnline() bool {
now := time.Now()
// if the node was seen in the last 40 seconds, it's online
return now.Sub(d.LastSeen) < 40*time.Second
}
type P2PNodesResponse struct {
LlamaCPPNodes []NodeData `json:"llama_cpp_nodes" yaml:"llama_cpp_nodes"`
FederatedNodes []NodeData `json:"federated_nodes" yaml:"federated_nodes"`
MLXNodes []NodeData `json:"mlx_nodes" yaml:"mlx_nodes"`
}
type SysInfoModel struct {
ID string `json:"id"`
// Backend is the engine serving this model. The loader knows only the ID,
// so it is resolved from the model's config; empty when the model was
// loaded without one (a loose file, or a config since removed).
Backend string `json:"backend,omitempty"`
}
type SystemInformationResponse struct {
Backends []string `json:"backends"` // available backend engines
Models []SysInfoModel `json:"loaded_models"` // currently loaded models
}
type DetectionRequest struct {
BasicModelRequest
Image string `json:"image"` // URL or base64-encoded image to analyze
Prompt string `json:"prompt,omitempty"` // Text prompt (for SAM 3 PCS mode)
Points []float32 `json:"points,omitempty"` // Point coordinates as [x,y,label,...] triples (label: 1=pos, 0=neg)
Boxes []float32 `json:"boxes,omitempty"` // Box coordinates as [x1,y1,x2,y2,...] quads
Threshold float32 `json:"threshold,omitempty"` // Detection confidence threshold
}
type DetectionResponse struct {
Detections []Detection `json:"detections"`
}
type Detection struct {
X float32 `json:"x"`
Y float32 `json:"y"`
Width float32 `json:"width"`
Height float32 `json:"height"`
ClassName string `json:"class_name"`
Confidence float32 `json:"confidence,omitempty"`
Mask string `json:"mask,omitempty"` // base64-encoded PNG segmentation mask
}
// DepthRequest is the request body for the /v1/depth endpoint. It exposes the
// full Depth Anything 3 output surface; the include_* flags and exports let a
// caller ask for less work (e.g. depth only, or depth+pose without the point
// cloud).
type DepthRequest struct {
BasicModelRequest
Image string `json:"image"` // URL or base64-encoded image to analyze
Dst string `json:"dst,omitempty"` // optional output directory for exports (glb/colmap)
IncludeDepth bool `json:"include_depth,omitempty"` // return the per-pixel depth map
IncludeConfidence bool `json:"include_confidence,omitempty"` // return the per-pixel confidence map (DualDPT)
IncludePose bool `json:"include_pose,omitempty"` // return camera extrinsics/intrinsics (DualDPT)
IncludeSky bool `json:"include_sky,omitempty"` // return the per-pixel sky map (mono models)
IncludePoints bool `json:"include_points,omitempty"` // back-project to a 3D point cloud (DualDPT)
PointsConfThresh float32 `json:"points_conf_thresh,omitempty"` // keep points with confidence >= this threshold
Exports []string `json:"exports,omitempty"` // requested exports: "glb", "colmap"
}
// DepthResponse is the JSON response for the /v1/depth endpoint, mirroring the
// DepthResponse proto.
type DepthResponse struct {
Width int32 `json:"width"`
Height int32 `json:"height"`
Depth []float32 `json:"depth,omitempty"` // width*height row-major metric depth
Confidence []float32 `json:"confidence,omitempty"` // width*height row-major confidence (DualDPT)
Sky []float32 `json:"sky,omitempty"` // width*height row-major sky map (mono)
Extrinsics []float32 `json:"extrinsics,omitempty"` // 12 floats, 3x4 row-major (world-to-camera)
Intrinsics []float32 `json:"intrinsics,omitempty"` // 9 floats, 3x3 row-major
NumPoints int32 `json:"num_points,omitempty"` // number of 3D points
Points []float32 `json:"points,omitempty"` // num_points*3 xyz, world space
PointColors string `json:"point_colors,omitempty"` // base64-encoded num_points*3 uint8 rgb
ExportPaths []string `json:"export_paths,omitempty"` // paths written for the requested exports
IsMetric bool `json:"is_metric"` // depth is in metric units
}
// ─── Face recognition ──────────────────────────────────────────────
//
// FacialArea describes a bounding box for a detected face.
type FacialArea struct {
X float32 `json:"x"`
Y float32 `json:"y"`
W float32 `json:"w"`
H float32 `json:"h"`
}
// FaceVerifyRequest compares two images to decide whether they depict
// the same person. Img1 and Img2 accept URL, base64, or data-URI.
type FaceVerifyRequest struct {
BasicModelRequest
Img1 string `json:"img1"`
Img2 string `json:"img2"`
Threshold float32 `json:"threshold,omitempty"`
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
}
type FaceVerifyResponse struct {
Verified bool `json:"verified"`
Distance float32 `json:"distance"`
Threshold float32 `json:"threshold"`
Confidence float32 `json:"confidence"`
Model string `json:"model"`
Img1Area FacialArea `json:"img1_area"`
Img2Area FacialArea `json:"img2_area"`
ProcessingTimeMs float32 `json:"processing_time_ms,omitempty"`
// Liveness fields are only populated when the request set
// anti_spoofing=true. Pointers keep them fully absent from the
// JSON response otherwise, so callers can tell "not checked"
// apart from "checked and fake" (which would collapse to zero
// values with plain bool+omitempty).
Img1IsReal *bool `json:"img1_is_real,omitempty"`
Img1AntispoofScore *float32 `json:"img1_antispoof_score,omitempty"`
Img2IsReal *bool `json:"img2_is_real,omitempty"`
Img2AntispoofScore *float32 `json:"img2_antispoof_score,omitempty"`
}
// FaceAnalyzeRequest asks the backend for demographic attributes on
// every face detected in Img.
type FaceAnalyzeRequest struct {
BasicModelRequest
Img string `json:"img"`
Actions []string `json:"actions,omitempty"` // subset of {"age","gender","emotion","race"}
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
}
type FaceAnalyzeResponse struct {
Faces []FaceAnalysis `json:"faces"`
}
type FaceAnalysis struct {
Region FacialArea `json:"region"`
FaceConfidence float32 `json:"face_confidence"`
Age float32 `json:"age,omitempty"`
DominantGender string `json:"dominant_gender,omitempty"`
Gender map[string]float32 `json:"gender,omitempty"`
DominantEmotion string `json:"dominant_emotion,omitempty"`
Emotion map[string]float32 `json:"emotion,omitempty"`
DominantRace string `json:"dominant_race,omitempty"`
Race map[string]float32 `json:"race,omitempty"`
// Liveness fields — see FaceVerifyResponse for why these are pointers.
IsReal *bool `json:"is_real,omitempty"`
AntispoofScore *float32 `json:"antispoof_score,omitempty"`
}
// FaceEmbedRequest extracts a face embedding from an image. Distinct
// from /v1/embeddings (which is OpenAI-compatible and text-only); this
// endpoint accepts URL / base64 / data-URI image inputs.
type FaceEmbedRequest struct {
BasicModelRequest
Img string `json:"img"`
}
type FaceEmbedResponse struct {
Embedding []float32 `json:"embedding"`
Dim int `json:"dim"`
Model string `json:"model,omitempty"`
}
// FaceRegisterRequest enrolls a face into the 1:N recognition store.
type FaceRegisterRequest struct {
BasicModelRequest
Img string `json:"img"`
Name string `json:"name"`
Labels map[string]string `json:"labels,omitempty"`
Store string `json:"store,omitempty"` // vector store model; empty = local-store default
}
type FaceRegisterResponse struct {
ID string `json:"id"`
Name string `json:"name"`
RegisteredAt time.Time `json:"registered_at"`
}
// FaceIdentifyRequest runs 1:N recognition: embed the probe and
// return the top-K nearest registered faces.
type FaceIdentifyRequest struct {
BasicModelRequest
Img string `json:"img"`
TopK int `json:"top_k,omitempty"`
Threshold float32 `json:"threshold,omitempty"` // optional cutoff on distance
Store string `json:"store,omitempty"`
}
type FaceIdentifyResponse struct {
Matches []FaceIdentifyMatch `json:"matches"`
}
type FaceIdentifyMatch struct {
ID string `json:"id"`
Name string `json:"name"`
Labels map[string]string `json:"labels,omitempty"`
Distance float32 `json:"distance"`
Confidence float32 `json:"confidence"`
Match bool `json:"match"` // true when distance <= threshold
}
// FaceForgetRequest removes a previously-registered face by ID.
type FaceForgetRequest struct {
BasicModelRequest
ID string `json:"id"`
Store string `json:"store,omitempty"`
}
// ─── Voice (speaker) recognition ───────────────────────────────────
//
// VoiceVerifyRequest compares two audio clips and reports whether they
// were spoken by the same speaker. Audio1/Audio2 accept URL, base64,
// or data-URI (the HTTP layer materialises the bytes to a temp file
// before calling the gRPC backend).
type VoiceVerifyRequest struct {
BasicModelRequest
Audio1 string `json:"audio1"`
Audio2 string `json:"audio2"`
Threshold float32 `json:"threshold,omitempty"`
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
}
type VoiceVerifyResponse struct {
Verified bool `json:"verified"`
Distance float32 `json:"distance"`
Threshold float32 `json:"threshold"`
Confidence float32 `json:"confidence"`
Model string `json:"model"`
ProcessingTimeMs float32 `json:"processing_time_ms,omitempty"`
}
// VoiceAnalyzeRequest asks the backend for demographic attributes
// (age, gender, emotion) inferred from the audio clip.
type VoiceAnalyzeRequest struct {
BasicModelRequest
Audio string `json:"audio"`
Actions []string `json:"actions,omitempty"` // subset of {"age","gender","emotion"}
}
type VoiceAnalyzeResponse struct {
Segments []VoiceAnalysis `json:"segments"`
}
type VoiceAnalysis struct {
Start float32 `json:"start"`
End float32 `json:"end"`
Age float32 `json:"age,omitempty"`
DominantGender string `json:"dominant_gender,omitempty"`
Gender map[string]float32 `json:"gender,omitempty"`
DominantEmotion string `json:"dominant_emotion,omitempty"`
Emotion map[string]float32 `json:"emotion,omitempty"`
}
// VoiceEmbedRequest extracts a speaker embedding from an audio clip.
// Distinct from /v1/embeddings (OpenAI-compatible, text-only) — this
// endpoint accepts URL / base64 / data-URI audio inputs.
type VoiceEmbedRequest struct {
BasicModelRequest
Audio string `json:"audio"`
}
type VoiceEmbedResponse struct {
Embedding []float32 `json:"embedding"`
Dim int `json:"dim"`
Model string `json:"model,omitempty"`
}
// VoiceRegisterRequest enrolls a speaker into the 1:N identification store.
type VoiceRegisterRequest struct {
BasicModelRequest
Audio string `json:"audio"`
Name string `json:"name"`
Labels map[string]string `json:"labels,omitempty"`
Store string `json:"store,omitempty"`
}
type VoiceRegisterResponse struct {
ID string `json:"id"`
Name string `json:"name"`
RegisteredAt time.Time `json:"registered_at"`
}
// VoiceIdentifyRequest runs 1:N recognition: embed the probe and
// return the top-K nearest registered speakers.
type VoiceIdentifyRequest struct {
BasicModelRequest
Audio string `json:"audio"`
TopK int `json:"top_k,omitempty"`
Threshold float32 `json:"threshold,omitempty"`
Store string `json:"store,omitempty"`
}
type VoiceIdentifyResponse struct {
Matches []VoiceIdentifyMatch `json:"matches"`
}
type VoiceIdentifyMatch struct {
ID string `json:"id"`
Name string `json:"name"`
Labels map[string]string `json:"labels,omitempty"`
Distance float32 `json:"distance"`
Confidence float32 `json:"confidence"`
Match bool `json:"match"`
}
// VoiceForgetRequest removes a previously-registered speaker by ID.
type VoiceForgetRequest struct {
BasicModelRequest
ID string `json:"id"`
Store string `json:"store,omitempty"`
}
type ImportModelRequest struct {
URI string `json:"uri"`
Preferences json.RawMessage `json:"preferences,omitempty"`
}
// KnownBackend describes a backend that the importer knows about.
// Used by GET /backends/known to populate the import form dropdown.
type KnownBackend struct {
Name string `json:"name"`
Modality string `json:"modality"`
AutoDetect bool `json:"auto_detect"`
Description string `json:"description,omitempty"`
// Installed is true when the backend is currently present on disk — i.e. it
// appears in gallery.ListSystemBackends(systemState). Importer-registered or
// curated pref-only backends default to false unless they also show up on
// disk. The import form uses this to warn users that submitting an import
// may trigger an automatic backend download.
Installed bool `json:"installed"`
}
// SettingsResponse is the response type for settings API operations
type SettingsResponse struct {
Success bool `json:"success"`
Error string `json:"error,omitempty"`
Message string `json:"message,omitempty"`
}
// RouterDecideRequest is the input for POST /api/router/decide — the
// programmatic decision-oracle endpoint. Given the name of a router
// model (a ModelConfig that carries a `router:` block) and a prompt,
// the endpoint returns the classifier's label set plus the candidate
// model the in-band RouteModel middleware would have chosen. The
// endpoint does NOT rewrite any request, forward to a backend, or
// record a row in the decision store — it is a pure decision oracle
// for external routers that want LocalAI's classifier opinion without
// committing LocalAI to handle the request.
type RouterDecideRequest struct {
// Router is the name of the router model (a ModelConfig with a
// `router:` block). Required.
Router string `json:"router"`
// Input is the user-visible prompt text to classify. Required.
// Schema-shape extraction (chat-message concatenation, etc.) is
// the caller's responsibility — matches the Probe contract used
// by the in-band middleware.
Input string `json:"input"`
}
// RouterDecideResponse carries the classifier's decision plus the
// resolved candidate. Mirrors router.Decision with the addition of
// Candidate/Fallback so the caller learns which downstream model
// would have served the request without re-implementing the
// label-set → candidate match locally.
type RouterDecideResponse struct {
// Router echoes the requested router model.
Router string `json:"router"`
// Classifier is the classifier name that produced the decision
// (e.g. "score").
Classifier string `json:"classifier"`
// Labels is the set of active policy labels.
Labels []string `json:"labels"`
// Candidate is the model that would be routed to. Empty when no
// candidate covers Labels AND no fallback is configured.
Candidate string `json:"candidate,omitempty"`
// Fallback is true when Candidate is the router's configured
// fallback because no candidate covered Labels. Lets callers
// distinguish "matched" from "fell back" without comparing names.
Fallback bool `json:"fallback,omitempty"`
// Score is the top label's softmax probability (the
// classifier-side confidence signal).
Score float64 `json:"score"`
// LatencyMs is the classifier's wall-clock cost.
LatencyMs int64 `json:"latency_ms"`
// Cached is true when the decision came from the L2 embedding
// cache rather than a fresh classifier run.
Cached bool `json:"cached,omitempty"`
// CacheSimilarity carries the cosine similarity of the cache hit
// (0 when not cached).
CacheSimilarity float64 `json:"cache_similarity,omitempty"`
}
// PIIDecideRequest is the input for POST /api/pii/decide — the
// programmatic PII-decision oracle. External routers call it before
// dispatching a request to learn whether the content carries PII and
// what action the configured pattern set would take. The endpoint
// inspects the text and returns findings + a suggested action; it
// does NOT mutate the input, record an audit event, or rewrite any
// downstream request. The caller composes the decision with its own
// policy (mask, block, or allow).
type PIIDecideRequest struct {
// Text is the user-visible content to inspect. Required.
Text string `json:"text"`
}
// PIIDecideResponse carries the redactor's findings.
// SuggestedAction is derived from the action ordering used by the
// internal redactor (block > mask > allow) so callers don't need to
// replicate that logic.
type PIIDecideResponse struct {
// Findings is one entry per matched span — pattern id, byte
// range, and audit-safe hash prefix (never the matched value).
Findings []PIIFinding `json:"findings"`
// SuggestedAction is the strongest action across all findings:
// "block", "mask", or "allow" (no findings, or all findings
// resolved to the allow action).
SuggestedAction string `json:"suggested_action"`
// RedactedPreview is the input with mask-action spans replaced
// by their placeholders. Identical to Text when no findings or
// when the strongest action is block/allow (which don't rewrite
// content).
RedactedPreview string `json:"redacted_preview"`
}
// PIIFinding mirrors pii.Span on the wire. Pattern is the pattern id
// that matched (e.g. "email"). HashPrefix is the first 8 chars of
// sha256(matched value) — lets admins correlate recurring leaks
// without recovering the value itself.
type PIIFinding struct {
Start int `json:"start"`
End int `json:"end"`
Pattern string `json:"pattern"`
HashPrefix string `json:"hash_prefix"`
}