mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 10:28:43 -04:00
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint Adds the plumbing for image-conditioned 3D asset generation (binary glTF / GLB output), modeled on the video generation path: - backend.proto: Generate3D RPC + Generate3DRequest (staged image src, glb dst, seed/step/cfg_scale/texture_steps, quality and background enums, params map for backend-specific extras) - pkg/grpc: thread Generate3D through client, server, embed, base and the backend interfaces; connection-evicting and distributed-node wrappers (in-flight tracking + file staging) included - core/config: FLAG_3D usecase (guessed only for the trellis2cpp backend), '3d' canonical usecase string mapped to the Generate3D method, and a '3d' output modality - REST: POST /v1/3d/generations (+ unversioned alias) returning OpenAIResponse with a /generated-3d URL or b64_json; conditioning image accepted as URL, base64, or data URI; quality/background validated at the edge; .glb served as model/gltf-binary - auth: '3d' route feature (default ON); /api/instructions entry Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(trellis2cpp): add the trellis2.cpp image-to-3D backend Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2, pbr-textures branch) as a Go+purego backend, following the stablediffusion-ggml pattern: - backend/go/trellis2cpp: purego bindings to the flat C ABI (v9, asserted at startup), eager pipeline load with model-set validation (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured exactly like the upstream demo), Generate3D via t2_generate + t2_bake_glb writing a binary glTF to dst. Weight-free unit tests cover resolution/validation/param mapping — CI never downloads the multi-GB GGUF set or runs inference. - CPU SIMD variants build into per-variant directories (the shared libggml sonames collide across variants, unlike sd-ggml's flat renamed-.so scheme); run.sh picks one via /proc/cpuinfo. - CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta + latest/master image entries, bump_deps tracking of the pbr-textures branch, changed-backends.js mapping, top-level Makefile targets. - Importer: auto-detects trellis GGUF repos/URIs (registered before llama-cpp so the .gguf match isn't stolen) and expands any trellis URI to the full 10-file component set spanning the three LocalAI-io HF repos. - Gallery: trellis2-4b (full PBR + 1024 cascade) and trellis2-4b-geometry (512 untextured) with verified sha256s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(ui): 3D generation page with native GLB viewer and IndexedDB history Adds a Studio tab + /app/3d page for the new image-to-3D endpoint: - GlbViewer ports the trellis2cpp demo's dependency-free WebGL2 renderer (quaternion trackball, metallic-roughness PBR, ACES, hidden-line wireframe with a bounded index budget) and pairs it with a minimal GLB parser for the two forms t2_bake_glb emits — dense vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as normalized integers) and the opt-in UV-atlas textured form. Parsing happens before any GL so stats and errors render without WebGL2. - use3DHistory stores past generations (params, input thumbnail, and the GLB blob itself) in IndexedDB with keep-newest-20 eviction — GLBs are multi-MB binaries localStorage can't hold — and the page offers a download button for the active GLB. - Wiring: CAP_3D capability constant (FLAG_3D — the exact string /api/models/capabilities serves), threeDApi, router entries, Studio tab, vite dev proxy, en locale keys. - e2e: render-smoke entry plus a focused spec that feeds a real one-triangle vertex-PBR GLB through the parser/viewer and exercises IndexedDB persistence, selection, deletion, and API errors. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(3d): address API correctness and UX issues Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it. Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add previewable print remeshing Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download. Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(trellis2cpp): centralize remesh dependency pins Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement Generate3D stub for new proto RPC The Generate3D RPC added to backend.proto for the trellis2cpp backend made tonic's generated Backend trait require generate3_d, breaking the kokoros-grpc build. Return unimplemented like the other unsupported modalities. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
620 lines
27 KiB
Go
620 lines
27 KiB
Go
package schema
|
||
|
||
import (
|
||
"encoding/json"
|
||
"time"
|
||
|
||
gopsutil "github.com/shirou/gopsutil/v3/process"
|
||
)
|
||
|
||
type BackendMonitorRequest struct {
|
||
BasicModelRequest
|
||
}
|
||
|
||
// ModelLoadRequest asks LocalAI to pre-load a model into memory by name, so the
|
||
// first request that uses it pays no cold-start load cost. For a realtime
|
||
// pipeline model, every configured sub-model (VAD, transcription, LLM, TTS,
|
||
// sound_detection, voice_recognition) is loaded instead of the pipeline stub.
|
||
// It is the inverse of the /backend/shutdown request.
|
||
type ModelLoadRequest struct {
|
||
BasicModelRequest
|
||
}
|
||
|
||
// ModelLoadResponse reports the outcome of a /backend/load call.
|
||
type ModelLoadResponse struct {
|
||
// Loaded lists the model names actually resident in memory after the call.
|
||
// For a pipeline model these are its sub-models, not the pipeline name.
|
||
Loaded []string `json:"loaded"`
|
||
// Message is a short human-readable status ("model loaded", or an error).
|
||
Message string `json:"message"`
|
||
}
|
||
|
||
type TokenMetricsRequest struct {
|
||
BasicModelRequest
|
||
}
|
||
|
||
type BackendMonitorResponse struct {
|
||
MemoryInfo *gopsutil.MemoryInfoStat
|
||
MemoryPercent float32
|
||
CPUPercent float64
|
||
}
|
||
|
||
type GalleryResponse struct {
|
||
ID string `json:"uuid"`
|
||
StatusURL string `json:"status"`
|
||
|
||
EstimatedVRAMBytes uint64 `json:"estimated_vram_bytes,omitempty"`
|
||
EstimatedVRAMDisplay string `json:"estimated_vram_display,omitempty"`
|
||
EstimatedSizeBytes uint64 `json:"estimated_size_bytes,omitempty"`
|
||
EstimatedSizeDisplay string `json:"estimated_size_display,omitempty"`
|
||
}
|
||
|
||
type VideoRequest struct {
|
||
BasicModelRequest
|
||
Prompt string `json:"prompt" yaml:"prompt"` // text description of the video to generate
|
||
NegativePrompt string `json:"negative_prompt" yaml:"negative_prompt"` // things to avoid in the output
|
||
StartImage string `json:"start_image" yaml:"start_image"` // URL or base64 of the first frame
|
||
EndImage string `json:"end_image" yaml:"end_image"` // URL or base64 of the last frame
|
||
Audio string `json:"audio,omitempty" yaml:"audio,omitempty"` // URL or base64 audio for audio-conditioned generation
|
||
Width int32 `json:"width" yaml:"width"` // output width in pixels
|
||
Height int32 `json:"height" yaml:"height"` // output height in pixels
|
||
NumFrames int32 `json:"num_frames" yaml:"num_frames"` // total number of frames to generate
|
||
FPS int32 `json:"fps" yaml:"fps"` // frames per second
|
||
Seconds string `json:"seconds,omitempty" yaml:"seconds,omitempty"` // duration in seconds (alternative to num_frames)
|
||
Size string `json:"size,omitempty" yaml:"size,omitempty"` // WxH shorthand (e.g. "512x512")
|
||
InputReference string `json:"input_reference,omitempty" yaml:"input_reference,omitempty"` // reference image or video URL
|
||
Seed int32 `json:"seed" yaml:"seed"` // random seed for reproducibility
|
||
CFGScale float32 `json:"cfg_scale" yaml:"cfg_scale"` // classifier-free guidance scale
|
||
Step int32 `json:"step" yaml:"step"` // number of diffusion steps
|
||
ResponseFormat string `json:"response_format" yaml:"response_format"` // output format (url or b64_json)
|
||
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"` // backend-specific generation parameters
|
||
}
|
||
|
||
// @Description 3D asset generation request body. Generation is image-conditioned
|
||
// (TRELLIS.2 has no text-prompt path); the response is a binary glTF (.glb).
|
||
type Model3DRequest struct {
|
||
BasicModelRequest
|
||
Image string `json:"image" yaml:"image"` // conditioning image: URL, base64, or data URI (required)
|
||
Seed int32 `json:"seed,omitempty" yaml:"seed,omitempty"` // random seed; <=0 picks a random seed
|
||
Step int32 `json:"step,omitempty" yaml:"step,omitempty"` // flow sampling steps (backend default 12)
|
||
CFGScale float32 `json:"cfg_scale,omitempty" yaml:"cfg_scale,omitempty"` // classifier-free guidance scale (backend default 7.5)
|
||
TextureSteps int32 `json:"texture_steps,omitempty" yaml:"texture_steps,omitempty"` // texture flow sampling steps (backend default 12)
|
||
Quality string `json:"quality,omitempty" yaml:"quality,omitempty"` // mesh pipeline: auto|coarse|512|1024
|
||
Background string `json:"background,omitempty" yaml:"background,omitempty"` // background handling: auto|keep|black|white
|
||
ResponseFormat string `json:"response_format,omitempty" yaml:"response_format,omitempty"` // output format (url or b64_json)
|
||
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"` // backend-specific generation parameters
|
||
}
|
||
|
||
// @Description Print-remesh an existing trellis2.cpp GLB. The multipart mesh
|
||
// is wrapped into a watertight manifold; detail is a percentage of the source
|
||
// bounding-box diagonal and the enclosing offset is derived automatically.
|
||
type Model3DRemeshRequest struct {
|
||
BasicModelRequest
|
||
Detail float32 `json:"detail,omitempty" yaml:"detail,omitempty" form:"detail"` // detail size in percent (0.35–2.5; default 0.5)
|
||
}
|
||
|
||
// @Description TTS request body
|
||
type TTSRequest struct {
|
||
BasicModelRequest
|
||
Input string `json:"input" yaml:"input"` // text input
|
||
Voice string `json:"voice" yaml:"voice"` // voice audio file or speaker id
|
||
Backend string `json:"backend" yaml:"backend"` // backend engine override
|
||
Language string `json:"language,omitempty" yaml:"language,omitempty"` // (optional) language to use with TTS model
|
||
Format string `json:"response_format,omitempty" yaml:"response_format,omitempty"` // (optional) output format
|
||
Stream bool `json:"stream,omitempty" yaml:"stream,omitempty"` // (optional) enable streaming TTS
|
||
SampleRate int `json:"sample_rate,omitempty" yaml:"sample_rate,omitempty"` // (optional) desired output sample rate
|
||
// Speed is the OpenAI `speed` field (0.25-4.0). It is a pointer so an
|
||
// explicit `"speed": 0` (invalid, rejected with 400) is distinguishable
|
||
// from an omitted field (left at the backend default). It is normalised
|
||
// into Params["speed"] so it reaches the backend over the same channel as
|
||
// the other per-request generation parameters.
|
||
Speed *float32 `json:"speed,omitempty" yaml:"speed,omitempty"`
|
||
// Instructions is a free-form, per-request style/voice description. It maps to
|
||
// the OpenAI `instructions` field and is forwarded to the backend so expressive
|
||
// TTS models (e.g. Qwen3-TTS CustomVoice/VoiceDesign) can vary tone or designed
|
||
// voice per request instead of only via the static YAML option.
|
||
Instructions string `json:"instructions,omitempty" yaml:"instructions,omitempty"`
|
||
// Params carries optional, backend-specific per-request generation parameters
|
||
// (LocalAI extension, e.g. Chatterbox exaggeration/cfg_weight/temperature).
|
||
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"`
|
||
}
|
||
|
||
// @Description VAD request body
|
||
type VADRequest struct {
|
||
BasicModelRequest
|
||
Audio []float32 `json:"audio" yaml:"audio"` // raw audio samples as float32 PCM
|
||
}
|
||
|
||
type VADSegment struct {
|
||
Start float32 `json:"start" yaml:"start"`
|
||
End float32 `json:"end" yaml:"end"`
|
||
}
|
||
|
||
type VADResponse struct {
|
||
Segments []VADSegment `json:"segments" yaml:"segments"`
|
||
}
|
||
|
||
type StoreCommon struct {
|
||
Backend string `json:"backend,omitempty" yaml:"backend,omitempty"`
|
||
}
|
||
type StoresSet struct {
|
||
Store string `json:"store,omitempty" yaml:"store,omitempty"`
|
||
|
||
Keys [][]float32 `json:"keys" yaml:"keys"`
|
||
Values []string `json:"values" yaml:"values"`
|
||
StoreCommon
|
||
}
|
||
|
||
type StoresDelete struct {
|
||
Store string `json:"store,omitempty" yaml:"store,omitempty"`
|
||
|
||
Keys [][]float32 `json:"keys"`
|
||
StoreCommon
|
||
}
|
||
|
||
type StoresGet struct {
|
||
Store string `json:"store,omitempty" yaml:"store,omitempty"`
|
||
|
||
Keys [][]float32 `json:"keys" yaml:"keys"`
|
||
StoreCommon
|
||
}
|
||
|
||
type StoresGetResponse struct {
|
||
Keys [][]float32 `json:"keys" yaml:"keys"`
|
||
Values []string `json:"values" yaml:"values"`
|
||
}
|
||
|
||
type StoresFind struct {
|
||
Store string `json:"store,omitempty" yaml:"store,omitempty"`
|
||
|
||
Key []float32 `json:"key" yaml:"key"`
|
||
Topk int `json:"topk" yaml:"topk"`
|
||
StoreCommon
|
||
}
|
||
|
||
type StoresFindResponse struct {
|
||
Keys [][]float32 `json:"keys" yaml:"keys"`
|
||
Values []string `json:"values" yaml:"values"`
|
||
Similarities []float32 `json:"similarities" yaml:"similarities"`
|
||
}
|
||
|
||
type NodeData struct {
|
||
Name string
|
||
ID string
|
||
TunnelAddress string
|
||
ServiceID string
|
||
LastSeen time.Time
|
||
}
|
||
|
||
func (d NodeData) IsOnline() bool {
|
||
now := time.Now()
|
||
// if the node was seen in the last 40 seconds, it's online
|
||
return now.Sub(d.LastSeen) < 40*time.Second
|
||
}
|
||
|
||
type P2PNodesResponse struct {
|
||
LlamaCPPNodes []NodeData `json:"llama_cpp_nodes" yaml:"llama_cpp_nodes"`
|
||
FederatedNodes []NodeData `json:"federated_nodes" yaml:"federated_nodes"`
|
||
MLXNodes []NodeData `json:"mlx_nodes" yaml:"mlx_nodes"`
|
||
}
|
||
|
||
type SysInfoModel struct {
|
||
ID string `json:"id"`
|
||
}
|
||
|
||
type SystemInformationResponse struct {
|
||
Backends []string `json:"backends"` // available backend engines
|
||
Models []SysInfoModel `json:"loaded_models"` // currently loaded models
|
||
}
|
||
|
||
type DetectionRequest struct {
|
||
BasicModelRequest
|
||
Image string `json:"image"` // URL or base64-encoded image to analyze
|
||
Prompt string `json:"prompt,omitempty"` // Text prompt (for SAM 3 PCS mode)
|
||
Points []float32 `json:"points,omitempty"` // Point coordinates as [x,y,label,...] triples (label: 1=pos, 0=neg)
|
||
Boxes []float32 `json:"boxes,omitempty"` // Box coordinates as [x1,y1,x2,y2,...] quads
|
||
Threshold float32 `json:"threshold,omitempty"` // Detection confidence threshold
|
||
}
|
||
|
||
type DetectionResponse struct {
|
||
Detections []Detection `json:"detections"`
|
||
}
|
||
|
||
type Detection struct {
|
||
X float32 `json:"x"`
|
||
Y float32 `json:"y"`
|
||
Width float32 `json:"width"`
|
||
Height float32 `json:"height"`
|
||
ClassName string `json:"class_name"`
|
||
Confidence float32 `json:"confidence,omitempty"`
|
||
Mask string `json:"mask,omitempty"` // base64-encoded PNG segmentation mask
|
||
}
|
||
|
||
// DepthRequest is the request body for the /v1/depth endpoint. It exposes the
|
||
// full Depth Anything 3 output surface; the include_* flags and exports let a
|
||
// caller ask for less work (e.g. depth only, or depth+pose without the point
|
||
// cloud).
|
||
type DepthRequest struct {
|
||
BasicModelRequest
|
||
Image string `json:"image"` // URL or base64-encoded image to analyze
|
||
Dst string `json:"dst,omitempty"` // optional output directory for exports (glb/colmap)
|
||
IncludeDepth bool `json:"include_depth,omitempty"` // return the per-pixel depth map
|
||
IncludeConfidence bool `json:"include_confidence,omitempty"` // return the per-pixel confidence map (DualDPT)
|
||
IncludePose bool `json:"include_pose,omitempty"` // return camera extrinsics/intrinsics (DualDPT)
|
||
IncludeSky bool `json:"include_sky,omitempty"` // return the per-pixel sky map (mono models)
|
||
IncludePoints bool `json:"include_points,omitempty"` // back-project to a 3D point cloud (DualDPT)
|
||
PointsConfThresh float32 `json:"points_conf_thresh,omitempty"` // keep points with confidence >= this threshold
|
||
Exports []string `json:"exports,omitempty"` // requested exports: "glb", "colmap"
|
||
}
|
||
|
||
// DepthResponse is the JSON response for the /v1/depth endpoint, mirroring the
|
||
// DepthResponse proto.
|
||
type DepthResponse struct {
|
||
Width int32 `json:"width"`
|
||
Height int32 `json:"height"`
|
||
Depth []float32 `json:"depth,omitempty"` // width*height row-major metric depth
|
||
Confidence []float32 `json:"confidence,omitempty"` // width*height row-major confidence (DualDPT)
|
||
Sky []float32 `json:"sky,omitempty"` // width*height row-major sky map (mono)
|
||
Extrinsics []float32 `json:"extrinsics,omitempty"` // 12 floats, 3x4 row-major (world-to-camera)
|
||
Intrinsics []float32 `json:"intrinsics,omitempty"` // 9 floats, 3x3 row-major
|
||
NumPoints int32 `json:"num_points,omitempty"` // number of 3D points
|
||
Points []float32 `json:"points,omitempty"` // num_points*3 xyz, world space
|
||
PointColors string `json:"point_colors,omitempty"` // base64-encoded num_points*3 uint8 rgb
|
||
ExportPaths []string `json:"export_paths,omitempty"` // paths written for the requested exports
|
||
IsMetric bool `json:"is_metric"` // depth is in metric units
|
||
}
|
||
|
||
// ─── Face recognition ──────────────────────────────────────────────
|
||
//
|
||
// FacialArea describes a bounding box for a detected face.
|
||
type FacialArea struct {
|
||
X float32 `json:"x"`
|
||
Y float32 `json:"y"`
|
||
W float32 `json:"w"`
|
||
H float32 `json:"h"`
|
||
}
|
||
|
||
// FaceVerifyRequest compares two images to decide whether they depict
|
||
// the same person. Img1 and Img2 accept URL, base64, or data-URI.
|
||
type FaceVerifyRequest struct {
|
||
BasicModelRequest
|
||
Img1 string `json:"img1"`
|
||
Img2 string `json:"img2"`
|
||
Threshold float32 `json:"threshold,omitempty"`
|
||
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
|
||
}
|
||
|
||
type FaceVerifyResponse struct {
|
||
Verified bool `json:"verified"`
|
||
Distance float32 `json:"distance"`
|
||
Threshold float32 `json:"threshold"`
|
||
Confidence float32 `json:"confidence"`
|
||
Model string `json:"model"`
|
||
Img1Area FacialArea `json:"img1_area"`
|
||
Img2Area FacialArea `json:"img2_area"`
|
||
ProcessingTimeMs float32 `json:"processing_time_ms,omitempty"`
|
||
// Liveness fields are only populated when the request set
|
||
// anti_spoofing=true. Pointers keep them fully absent from the
|
||
// JSON response otherwise, so callers can tell "not checked"
|
||
// apart from "checked and fake" (which would collapse to zero
|
||
// values with plain bool+omitempty).
|
||
Img1IsReal *bool `json:"img1_is_real,omitempty"`
|
||
Img1AntispoofScore *float32 `json:"img1_antispoof_score,omitempty"`
|
||
Img2IsReal *bool `json:"img2_is_real,omitempty"`
|
||
Img2AntispoofScore *float32 `json:"img2_antispoof_score,omitempty"`
|
||
}
|
||
|
||
// FaceAnalyzeRequest asks the backend for demographic attributes on
|
||
// every face detected in Img.
|
||
type FaceAnalyzeRequest struct {
|
||
BasicModelRequest
|
||
Img string `json:"img"`
|
||
Actions []string `json:"actions,omitempty"` // subset of {"age","gender","emotion","race"}
|
||
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
|
||
}
|
||
|
||
type FaceAnalyzeResponse struct {
|
||
Faces []FaceAnalysis `json:"faces"`
|
||
}
|
||
|
||
type FaceAnalysis struct {
|
||
Region FacialArea `json:"region"`
|
||
FaceConfidence float32 `json:"face_confidence"`
|
||
Age float32 `json:"age,omitempty"`
|
||
DominantGender string `json:"dominant_gender,omitempty"`
|
||
Gender map[string]float32 `json:"gender,omitempty"`
|
||
DominantEmotion string `json:"dominant_emotion,omitempty"`
|
||
Emotion map[string]float32 `json:"emotion,omitempty"`
|
||
DominantRace string `json:"dominant_race,omitempty"`
|
||
Race map[string]float32 `json:"race,omitempty"`
|
||
// Liveness fields — see FaceVerifyResponse for why these are pointers.
|
||
IsReal *bool `json:"is_real,omitempty"`
|
||
AntispoofScore *float32 `json:"antispoof_score,omitempty"`
|
||
}
|
||
|
||
// FaceEmbedRequest extracts a face embedding from an image. Distinct
|
||
// from /v1/embeddings (which is OpenAI-compatible and text-only); this
|
||
// endpoint accepts URL / base64 / data-URI image inputs.
|
||
type FaceEmbedRequest struct {
|
||
BasicModelRequest
|
||
Img string `json:"img"`
|
||
}
|
||
|
||
type FaceEmbedResponse struct {
|
||
Embedding []float32 `json:"embedding"`
|
||
Dim int `json:"dim"`
|
||
Model string `json:"model,omitempty"`
|
||
}
|
||
|
||
// FaceRegisterRequest enrolls a face into the 1:N recognition store.
|
||
type FaceRegisterRequest struct {
|
||
BasicModelRequest
|
||
Img string `json:"img"`
|
||
Name string `json:"name"`
|
||
Labels map[string]string `json:"labels,omitempty"`
|
||
Store string `json:"store,omitempty"` // vector store model; empty = local-store default
|
||
}
|
||
|
||
type FaceRegisterResponse struct {
|
||
ID string `json:"id"`
|
||
Name string `json:"name"`
|
||
RegisteredAt time.Time `json:"registered_at"`
|
||
}
|
||
|
||
// FaceIdentifyRequest runs 1:N recognition: embed the probe and
|
||
// return the top-K nearest registered faces.
|
||
type FaceIdentifyRequest struct {
|
||
BasicModelRequest
|
||
Img string `json:"img"`
|
||
TopK int `json:"top_k,omitempty"`
|
||
Threshold float32 `json:"threshold,omitempty"` // optional cutoff on distance
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
type FaceIdentifyResponse struct {
|
||
Matches []FaceIdentifyMatch `json:"matches"`
|
||
}
|
||
|
||
type FaceIdentifyMatch struct {
|
||
ID string `json:"id"`
|
||
Name string `json:"name"`
|
||
Labels map[string]string `json:"labels,omitempty"`
|
||
Distance float32 `json:"distance"`
|
||
Confidence float32 `json:"confidence"`
|
||
Match bool `json:"match"` // true when distance <= threshold
|
||
}
|
||
|
||
// FaceForgetRequest removes a previously-registered face by ID.
|
||
type FaceForgetRequest struct {
|
||
BasicModelRequest
|
||
ID string `json:"id"`
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
// ─── Voice (speaker) recognition ───────────────────────────────────
|
||
//
|
||
// VoiceVerifyRequest compares two audio clips and reports whether they
|
||
// were spoken by the same speaker. Audio1/Audio2 accept URL, base64,
|
||
// or data-URI (the HTTP layer materialises the bytes to a temp file
|
||
// before calling the gRPC backend).
|
||
type VoiceVerifyRequest struct {
|
||
BasicModelRequest
|
||
Audio1 string `json:"audio1"`
|
||
Audio2 string `json:"audio2"`
|
||
Threshold float32 `json:"threshold,omitempty"`
|
||
AntiSpoofing bool `json:"anti_spoofing,omitempty"`
|
||
}
|
||
|
||
type VoiceVerifyResponse struct {
|
||
Verified bool `json:"verified"`
|
||
Distance float32 `json:"distance"`
|
||
Threshold float32 `json:"threshold"`
|
||
Confidence float32 `json:"confidence"`
|
||
Model string `json:"model"`
|
||
ProcessingTimeMs float32 `json:"processing_time_ms,omitempty"`
|
||
}
|
||
|
||
// VoiceAnalyzeRequest asks the backend for demographic attributes
|
||
// (age, gender, emotion) inferred from the audio clip.
|
||
type VoiceAnalyzeRequest struct {
|
||
BasicModelRequest
|
||
Audio string `json:"audio"`
|
||
Actions []string `json:"actions,omitempty"` // subset of {"age","gender","emotion"}
|
||
}
|
||
|
||
type VoiceAnalyzeResponse struct {
|
||
Segments []VoiceAnalysis `json:"segments"`
|
||
}
|
||
|
||
type VoiceAnalysis struct {
|
||
Start float32 `json:"start"`
|
||
End float32 `json:"end"`
|
||
Age float32 `json:"age,omitempty"`
|
||
DominantGender string `json:"dominant_gender,omitempty"`
|
||
Gender map[string]float32 `json:"gender,omitempty"`
|
||
DominantEmotion string `json:"dominant_emotion,omitempty"`
|
||
Emotion map[string]float32 `json:"emotion,omitempty"`
|
||
}
|
||
|
||
// VoiceEmbedRequest extracts a speaker embedding from an audio clip.
|
||
// Distinct from /v1/embeddings (OpenAI-compatible, text-only) — this
|
||
// endpoint accepts URL / base64 / data-URI audio inputs.
|
||
type VoiceEmbedRequest struct {
|
||
BasicModelRequest
|
||
Audio string `json:"audio"`
|
||
}
|
||
|
||
type VoiceEmbedResponse struct {
|
||
Embedding []float32 `json:"embedding"`
|
||
Dim int `json:"dim"`
|
||
Model string `json:"model,omitempty"`
|
||
}
|
||
|
||
// VoiceRegisterRequest enrolls a speaker into the 1:N identification store.
|
||
type VoiceRegisterRequest struct {
|
||
BasicModelRequest
|
||
Audio string `json:"audio"`
|
||
Name string `json:"name"`
|
||
Labels map[string]string `json:"labels,omitempty"`
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
type VoiceRegisterResponse struct {
|
||
ID string `json:"id"`
|
||
Name string `json:"name"`
|
||
RegisteredAt time.Time `json:"registered_at"`
|
||
}
|
||
|
||
// VoiceIdentifyRequest runs 1:N recognition: embed the probe and
|
||
// return the top-K nearest registered speakers.
|
||
type VoiceIdentifyRequest struct {
|
||
BasicModelRequest
|
||
Audio string `json:"audio"`
|
||
TopK int `json:"top_k,omitempty"`
|
||
Threshold float32 `json:"threshold,omitempty"`
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
type VoiceIdentifyResponse struct {
|
||
Matches []VoiceIdentifyMatch `json:"matches"`
|
||
}
|
||
|
||
type VoiceIdentifyMatch struct {
|
||
ID string `json:"id"`
|
||
Name string `json:"name"`
|
||
Labels map[string]string `json:"labels,omitempty"`
|
||
Distance float32 `json:"distance"`
|
||
Confidence float32 `json:"confidence"`
|
||
Match bool `json:"match"`
|
||
}
|
||
|
||
// VoiceForgetRequest removes a previously-registered speaker by ID.
|
||
type VoiceForgetRequest struct {
|
||
BasicModelRequest
|
||
ID string `json:"id"`
|
||
Store string `json:"store,omitempty"`
|
||
}
|
||
|
||
type ImportModelRequest struct {
|
||
URI string `json:"uri"`
|
||
Preferences json.RawMessage `json:"preferences,omitempty"`
|
||
}
|
||
|
||
// KnownBackend describes a backend that the importer knows about.
|
||
// Used by GET /backends/known to populate the import form dropdown.
|
||
type KnownBackend struct {
|
||
Name string `json:"name"`
|
||
Modality string `json:"modality"`
|
||
AutoDetect bool `json:"auto_detect"`
|
||
Description string `json:"description,omitempty"`
|
||
// Installed is true when the backend is currently present on disk — i.e. it
|
||
// appears in gallery.ListSystemBackends(systemState). Importer-registered or
|
||
// curated pref-only backends default to false unless they also show up on
|
||
// disk. The import form uses this to warn users that submitting an import
|
||
// may trigger an automatic backend download.
|
||
Installed bool `json:"installed"`
|
||
}
|
||
|
||
// SettingsResponse is the response type for settings API operations
|
||
type SettingsResponse struct {
|
||
Success bool `json:"success"`
|
||
Error string `json:"error,omitempty"`
|
||
Message string `json:"message,omitempty"`
|
||
}
|
||
|
||
// RouterDecideRequest is the input for POST /api/router/decide — the
|
||
// programmatic decision-oracle endpoint. Given the name of a router
|
||
// model (a ModelConfig that carries a `router:` block) and a prompt,
|
||
// the endpoint returns the classifier's label set plus the candidate
|
||
// model the in-band RouteModel middleware would have chosen. The
|
||
// endpoint does NOT rewrite any request, forward to a backend, or
|
||
// record a row in the decision store — it is a pure decision oracle
|
||
// for external routers that want LocalAI's classifier opinion without
|
||
// committing LocalAI to handle the request.
|
||
type RouterDecideRequest struct {
|
||
// Router is the name of the router model (a ModelConfig with a
|
||
// `router:` block). Required.
|
||
Router string `json:"router"`
|
||
// Input is the user-visible prompt text to classify. Required.
|
||
// Schema-shape extraction (chat-message concatenation, etc.) is
|
||
// the caller's responsibility — matches the Probe contract used
|
||
// by the in-band middleware.
|
||
Input string `json:"input"`
|
||
}
|
||
|
||
// RouterDecideResponse carries the classifier's decision plus the
|
||
// resolved candidate. Mirrors router.Decision with the addition of
|
||
// Candidate/Fallback so the caller learns which downstream model
|
||
// would have served the request without re-implementing the
|
||
// label-set → candidate match locally.
|
||
type RouterDecideResponse struct {
|
||
// Router echoes the requested router model.
|
||
Router string `json:"router"`
|
||
// Classifier is the classifier name that produced the decision
|
||
// (e.g. "score").
|
||
Classifier string `json:"classifier"`
|
||
// Labels is the set of active policy labels.
|
||
Labels []string `json:"labels"`
|
||
// Candidate is the model that would be routed to. Empty when no
|
||
// candidate covers Labels AND no fallback is configured.
|
||
Candidate string `json:"candidate,omitempty"`
|
||
// Fallback is true when Candidate is the router's configured
|
||
// fallback because no candidate covered Labels. Lets callers
|
||
// distinguish "matched" from "fell back" without comparing names.
|
||
Fallback bool `json:"fallback,omitempty"`
|
||
// Score is the top label's softmax probability (the
|
||
// classifier-side confidence signal).
|
||
Score float64 `json:"score"`
|
||
// LatencyMs is the classifier's wall-clock cost.
|
||
LatencyMs int64 `json:"latency_ms"`
|
||
// Cached is true when the decision came from the L2 embedding
|
||
// cache rather than a fresh classifier run.
|
||
Cached bool `json:"cached,omitempty"`
|
||
// CacheSimilarity carries the cosine similarity of the cache hit
|
||
// (0 when not cached).
|
||
CacheSimilarity float64 `json:"cache_similarity,omitempty"`
|
||
}
|
||
|
||
// PIIDecideRequest is the input for POST /api/pii/decide — the
|
||
// programmatic PII-decision oracle. External routers call it before
|
||
// dispatching a request to learn whether the content carries PII and
|
||
// what action the configured pattern set would take. The endpoint
|
||
// inspects the text and returns findings + a suggested action; it
|
||
// does NOT mutate the input, record an audit event, or rewrite any
|
||
// downstream request. The caller composes the decision with its own
|
||
// policy (mask, block, or allow).
|
||
type PIIDecideRequest struct {
|
||
// Text is the user-visible content to inspect. Required.
|
||
Text string `json:"text"`
|
||
}
|
||
|
||
// PIIDecideResponse carries the redactor's findings.
|
||
// SuggestedAction is derived from the action ordering used by the
|
||
// internal redactor (block > mask > allow) so callers don't need to
|
||
// replicate that logic.
|
||
type PIIDecideResponse struct {
|
||
// Findings is one entry per matched span — pattern id, byte
|
||
// range, and audit-safe hash prefix (never the matched value).
|
||
Findings []PIIFinding `json:"findings"`
|
||
// SuggestedAction is the strongest action across all findings:
|
||
// "block", "mask", or "allow" (no findings, or all findings
|
||
// resolved to the allow action).
|
||
SuggestedAction string `json:"suggested_action"`
|
||
// RedactedPreview is the input with mask-action spans replaced
|
||
// by their placeholders. Identical to Text when no findings or
|
||
// when the strongest action is block/allow (which don't rewrite
|
||
// content).
|
||
RedactedPreview string `json:"redacted_preview"`
|
||
}
|
||
|
||
// PIIFinding mirrors pii.Span on the wire. Pattern is the pattern id
|
||
// that matched (e.g. "email"). HashPrefix is the first 8 chars of
|
||
// sha256(matched value) — lets admins correlate recurring leaks
|
||
// without recovering the value itself.
|
||
type PIIFinding struct {
|
||
Start int `json:"start"`
|
||
End int `json:"end"`
|
||
Pattern string `json:"pattern"`
|
||
HashPrefix string `json:"hash_prefix"`
|
||
}
|