Files
LocalAI/core/http/auth/features.go
Richard Palethorpe 9058a2bb46 feat: Add 3d generation UI/API and trellis2cpp backend (#10979)
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint

Adds the plumbing for image-conditioned 3D asset generation (binary
glTF / GLB output), modeled on the video generation path:

- backend.proto: Generate3D RPC + Generate3DRequest (staged image src,
  glb dst, seed/step/cfg_scale/texture_steps, quality and background
  enums, params map for backend-specific extras)
- pkg/grpc: thread Generate3D through client, server, embed, base and
  the backend interfaces; connection-evicting and distributed-node
  wrappers (in-flight tracking + file staging) included
- core/config: FLAG_3D usecase (guessed only for the trellis2cpp
  backend), '3d' canonical usecase string mapped to the Generate3D
  method, and a '3d' output modality
- REST: POST /v1/3d/generations (+ unversioned alias) returning
  OpenAIResponse with a /generated-3d URL or b64_json; conditioning
  image accepted as URL, base64, or data URI; quality/background
  validated at the edge; .glb served as model/gltf-binary
- auth: '3d' route feature (default ON); /api/instructions entry

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(trellis2cpp): add the trellis2.cpp image-to-3D backend

Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2,
pbr-textures branch) as a Go+purego backend, following the
stablediffusion-ggml pattern:

- backend/go/trellis2cpp: purego bindings to the flat C ABI (v9,
  asserted at startup), eager pipeline load with model-set validation
  (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured
  exactly like the upstream demo), Generate3D via t2_generate +
  t2_bake_glb writing a binary glTF to dst. Weight-free unit tests
  cover resolution/validation/param mapping — CI never downloads the
  multi-GB GGUF set or runs inference.
- CPU SIMD variants build into per-variant directories (the shared
  libggml sonames collide across variants, unlike sd-ggml's flat
  renamed-.so scheme); run.sh picks one via /proc/cpuinfo.
- CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan
  amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta +
  latest/master image entries, bump_deps tracking of the pbr-textures
  branch, changed-backends.js mapping, top-level Makefile targets.
- Importer: auto-detects trellis GGUF repos/URIs (registered before
  llama-cpp so the .gguf match isn't stolen) and expands any trellis
  URI to the full 10-file component set spanning the three LocalAI-io
  HF repos.
- Gallery: trellis2-4b (full PBR + 1024 cascade) and
  trellis2-4b-geometry (512 untextured) with verified sha256s.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(ui): 3D generation page with native GLB viewer and IndexedDB history

Adds a Studio tab + /app/3d page for the new image-to-3D endpoint:

- GlbViewer ports the trellis2cpp demo's dependency-free WebGL2
  renderer (quaternion trackball, metallic-roughness PBR, ACES,
  hidden-line wireframe with a bounded index budget) and pairs it with
  a minimal GLB parser for the two forms t2_bake_glb emits — dense
  vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as
  normalized integers) and the opt-in UV-atlas textured form. Parsing
  happens before any GL so stats and errors render without WebGL2.
- use3DHistory stores past generations (params, input thumbnail, and
  the GLB blob itself) in IndexedDB with keep-newest-20 eviction —
  GLBs are multi-MB binaries localStorage can't hold — and the page
  offers a download button for the active GLB.
- Wiring: CAP_3D capability constant (FLAG_3D — the exact string
  /api/models/capabilities serves), threeDApi, router entries, Studio
  tab, vite dev proxy, en locale keys.
- e2e: render-smoke entry plus a focused spec that feeds a real
  one-triangle vertex-PBR GLB through the parser/viewer and exercises
  IndexedDB persistence, selection, deletion, and API errors.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(3d): address API correctness and UX issues

Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it.

Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(3d): add previewable print remeshing

Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download.

Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* build(trellis2cpp): centralize remesh dependency pins

Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(kokoros): implement Generate3D stub for new proto RPC

The Generate3D RPC added to backend.proto for the trellis2cpp backend
made tonic's generated Backend trait require generate3_d, breaking the
kokoros-grpc build. Return unimplemented like the other unsupported
modalities.

Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
2026-07-29 16:15:04 +02:00

203 lines
7.4 KiB
Go

package auth
// RouteFeature maps a route pattern + HTTP method to a required feature.
type RouteFeature struct {
Method string // "POST", "GET", "*" (any)
Pattern string // Echo route pattern, e.g. "/v1/chat/completions"
Feature string // Feature constant, e.g. FeatureChat
}
// RouteFeatureRegistry is the single source of truth for endpoint -> feature mappings.
// To gate a new endpoint, add an entry here -- no other file changes needed.
var RouteFeatureRegistry = []RouteFeature{
// Chat / Completions
{"POST", "/v1/chat/completions", FeatureChat},
{"POST", "/chat/completions", FeatureChat},
{"POST", "/v1/completions", FeatureChat},
{"POST", "/completions", FeatureChat},
{"POST", "/v1/engines/:model/completions", FeatureChat},
{"POST", "/v1/edits", FeatureChat},
{"POST", "/edits", FeatureChat},
// Anthropic
{"POST", "/v1/messages", FeatureChat},
{"POST", "/messages", FeatureChat},
// Open Responses
{"POST", "/v1/responses", FeatureChat},
{"POST", "/responses", FeatureChat},
{"GET", "/v1/responses", FeatureChat},
{"GET", "/responses", FeatureChat},
// Embeddings
{"POST", "/v1/embeddings", FeatureEmbeddings},
{"POST", "/embeddings", FeatureEmbeddings},
{"POST", "/v1/engines/:model/embeddings", FeatureEmbeddings},
// Images
{"POST", "/v1/images/generations", FeatureImages},
{"POST", "/images/generations", FeatureImages},
{"POST", "/v1/images/inpainting", FeatureImages},
{"POST", "/images/inpainting", FeatureImages},
// Audio transcription
{"POST", "/v1/audio/transcriptions", FeatureAudioTranscription},
{"POST", "/audio/transcriptions", FeatureAudioTranscription},
// Audio diarization (speaker turns)
{"POST", "/v1/audio/diarization", FeatureAudioDiarization},
{"POST", "/audio/diarization", FeatureAudioDiarization},
// Audio classification (sound-event tagging)
{"POST", "/v1/audio/classification", FeatureAudioClassification},
{"POST", "/audio/classification", FeatureAudioClassification},
// Audio speech / TTS
{"POST", "/v1/audio/speech", FeatureAudioSpeech},
{"POST", "/audio/speech", FeatureAudioSpeech},
{"POST", "/tts", FeatureAudioSpeech},
{"POST", "/v1/text-to-speech/:voice-id", FeatureAudioSpeech},
{"GET", "/api/voice-profiles", FeatureAudioSpeech},
{"GET", "/api/voice-profiles/:id/audio", FeatureAudioSpeech},
// VAD
{"POST", "/vad", FeatureVAD},
{"POST", "/v1/vad", FeatureVAD},
// Detection
{"POST", "/v1/detection", FeatureDetection},
// Face recognition
{"POST", "/v1/face/verify", FeatureFaceRecognition},
{"POST", "/v1/face/analyze", FeatureFaceRecognition},
{"POST", "/v1/face/embed", FeatureFaceRecognition},
{"POST", "/v1/face/register", FeatureFaceRecognition},
{"POST", "/v1/face/identify", FeatureFaceRecognition},
{"POST", "/v1/face/forget", FeatureFaceRecognition},
// Voice (speaker) recognition
{"POST", "/v1/voice/verify", FeatureVoiceRecognition},
{"POST", "/v1/voice/analyze", FeatureVoiceRecognition},
{"POST", "/v1/voice/embed", FeatureVoiceRecognition},
{"POST", "/v1/voice/register", FeatureVoiceRecognition},
{"POST", "/v1/voice/identify", FeatureVoiceRecognition},
{"POST", "/v1/voice/forget", FeatureVoiceRecognition},
// Audio transform (echo cancellation, noise suppression, voice conversion, etc.)
{"POST", "/audio/transformations", FeatureAudioTransform},
{"POST", "/audio/transform", FeatureAudioTransform},
{"GET", "/audio/transformations/stream", FeatureAudioTransform},
// Video
{"POST", "/video", FeatureVideo},
// 3D generation
{"POST", "/3d/generations", Feature3D},
{"POST", "/3d/remesh", Feature3D},
// Sound generation
{"POST", "/v1/sound-generation", FeatureSound},
// Realtime
{"GET", "/v1/realtime", FeatureRealtime},
{"POST", "/v1/realtime/sessions", FeatureRealtime},
{"POST", "/v1/realtime/transcription_session", FeatureRealtime},
{"POST", "/v1/realtime/calls", FeatureRealtime},
// MCP
{"POST", "/v1/mcp/chat/completions", FeatureMCP},
{"POST", "/mcp/v1/chat/completions", FeatureMCP},
{"POST", "/mcp/chat/completions", FeatureMCP},
// Tokenize
{"POST", "/v1/tokenize", FeatureTokenize},
// Rerank
{"POST", "/v1/rerank", FeatureRerank},
// Stores
{"POST", "/stores/set", FeatureStores},
{"POST", "/stores/delete", FeatureStores},
{"POST", "/stores/get", FeatureStores},
{"POST", "/stores/find", FeatureStores},
// Fine-tuning
{"POST", "/api/fine-tuning/jobs", FeatureFineTuning},
{"GET", "/api/fine-tuning/jobs", FeatureFineTuning},
{"GET", "/api/fine-tuning/jobs/:id", FeatureFineTuning},
{"POST", "/api/fine-tuning/jobs/:id/stop", FeatureFineTuning},
{"DELETE", "/api/fine-tuning/jobs/:id", FeatureFineTuning},
{"GET", "/api/fine-tuning/jobs/:id/progress", FeatureFineTuning},
{"GET", "/api/fine-tuning/jobs/:id/checkpoints", FeatureFineTuning},
{"POST", "/api/fine-tuning/jobs/:id/export", FeatureFineTuning},
{"GET", "/api/fine-tuning/jobs/:id/download", FeatureFineTuning},
{"POST", "/api/fine-tuning/datasets", FeatureFineTuning},
// PII analyze/redact service (the events log stays admin-gated in-handler)
{"POST", "/api/pii/analyze", FeaturePIIFilter},
{"POST", "/api/pii/redact", FeaturePIIFilter},
// Quantization
{"POST", "/api/quantization/jobs", FeatureQuantization},
{"GET", "/api/quantization/jobs", FeatureQuantization},
{"GET", "/api/quantization/jobs/:id", FeatureQuantization},
{"POST", "/api/quantization/jobs/:id/stop", FeatureQuantization},
{"DELETE", "/api/quantization/jobs/:id", FeatureQuantization},
{"GET", "/api/quantization/jobs/:id/progress", FeatureQuantization},
{"POST", "/api/quantization/jobs/:id/import", FeatureQuantization},
{"GET", "/api/quantization/jobs/:id/download", FeatureQuantization},
}
// FeatureMeta describes a feature for the admin API/UI.
type FeatureMeta struct {
Key string `json:"key"`
Label string `json:"label"`
DefaultValue bool `json:"default"`
}
// AgentFeatureMetas returns metadata for agent features.
func AgentFeatureMetas() []FeatureMeta {
return []FeatureMeta{
{FeatureAgents, "Agents", false},
{FeatureSkills, "Skills", false},
{FeatureCollections, "Collections", false},
{FeatureMCPJobs, "MCP CI Jobs", false},
{FeatureLocalAIAssistant, "LocalAI Assistant", false},
}
}
// GeneralFeatureMetas returns metadata for general features.
func GeneralFeatureMetas() []FeatureMeta {
return []FeatureMeta{
{FeatureFineTuning, "Fine-Tuning", false},
{FeatureQuantization, "Quantization", false},
}
}
// APIFeatureMetas returns metadata for API endpoint features.
func APIFeatureMetas() []FeatureMeta {
return []FeatureMeta{
{FeatureChat, "Chat Completions", true},
{FeatureImages, "Image Generation", true},
{FeatureAudioSpeech, "Audio Speech / TTS", true},
{FeatureAudioTranscription, "Audio Transcription", true},
{FeatureAudioDiarization, "Audio Diarization", true},
{FeatureAudioClassification, "Audio Classification", true},
{FeatureVAD, "Voice Activity Detection", true},
{FeatureDetection, "Detection", true},
{FeatureVideo, "Video Generation", true},
{Feature3D, "3D Generation", true},
{FeatureEmbeddings, "Embeddings", true},
{FeatureSound, "Sound Generation", true},
{FeatureRealtime, "Realtime", true},
{FeatureRerank, "Rerank", true},
{FeatureTokenize, "Tokenize", true},
{FeatureMCP, "MCP", true},
{FeatureStores, "Stores", true},
{FeatureFaceRecognition, "Face Recognition", true},
{FeatureVoiceRecognition, "Voice Recognition", true},
{FeatureAudioTransform, "Audio Transform", true},
{FeaturePIIFilter, "PII Analyze / Redact", true},
}
}