2 Commits

Include renames

Author	SHA1	Message	Date
3 people	21c5495a99	feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify (#12140) ... * feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner, ABI v27) so LocalAI can serve zero-shot named entity recognition through the existing TokenClassify gRPC method. backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with the text and labels, copies the C-owned entity array into protobuf TokenClassifyEntity messages, and frees the result. govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion bumped 26 -> 27. options.go: ner_labels, ner_threshold, ner_max_width parsed from engine_args. pkg/grpc: ClassifyModel interface and TokenClassify server handler (follows the Embedding locking pattern). core/config: vllm-cpp backend declares MethodTokenClassify and UsecaseTokenClassify. docs/content/features/vllm-cpp.md: NER section documenting the engine_args keys and the host-forward contract. Assisted-by: MAKI:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): correct NER pointer lint directive Use the govet directive for the C-owned NER array, matching the other purego pointer conversions. The array remains valid until its deferred free; the misspelled directive caused CI to flag this conversion. Assisted-by: Codex:gpt-6 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): add kev-compatible SystemOne API endpoints Add POST /v1/systemone, /v1/systemone/permute, and /v1/systemone/separate to LocalAI, mirroring the kev project's structured-extraction API. Each endpoint runs zero-shot NER over the rendered state text and builds kev-compatible answers for three question types: noul (binary entity presence), choice (pick one option), and score (pick one level). The TokenClassifyRequest proto gains a `repeated string labels` field so each question can supply its own labels at inference time, and TokenClassifier gains TokenClassifyWithLabels for per-call label selection. The vllm-cpp backend uses request labels when non-empty, falling back to configured ner_labels then the built-in defaults. Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2) are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so both servers produce the same answer shape. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): suppress gosec G404 on seeded permutation RNG The SystemOne permute endpoint uses math/rand with a caller-supplied seed for reproducible option permutations, matching kev's random.seed. gosec flags this as G404 (weak RNG). Add #nosec with a comment naming the intent: this is reproducibility, not cryptography. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(vllm-cpp): bump vllm.cpp pin to GLiNER2.5 merge commit Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER functions, and server endpoints into the LocalAI vllm-cpp backend. The ABI version (27) and Go struct mirrors already match. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): use instruction text as NER label in SystemOne handler The SystemOne handler was passing question IDs as NER labels for noul questions and bare key names for choice questions, so the model never matched any entities. Port the label mapping from vllm.cpp's ParseSystemOneBody: - noul: use the rendered instructions field (with instr alias) as the NER label, not the question ID - choice: use optionText(name, desc) — "name: description" or "name" when the description is null/empty — not the bare key - score: already correct (rendered criteria text) - permute: shuffle indices and build parallel key/label arrays so the NER call uses the optionText labels while the response is keyed by the original option names Also add the instructions field to the SystemOneQuestion schema struct (accepted alongside the instr backward-compat alias). Verified end-to-end against the real GLiNER2.5 model: noul questions now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is" (person, 0.852) where they previously returned zero entities. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>	2026-09-23 12:31:28 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	6fb9ab38aa	feat(gallery): add vllm.cpp text-generation models (#11511) ... Adds eight curated vllm-cpp entries to the model gallery. Until now the backend had gallery coverage only for MiniMax-H3 video, so serving text on it meant hand-writing engine_args. The flagship tier is what vllm.cpp gates its correctness and speed claims on: Qwen3.6-27B and Qwen3.6-35B-A3B in NVFP4, each with a speculative sibling (MTP on both, DFlash on the 27B). Qwen3-Coder-30B-A3B covers agentic tool use, and Qwen3-4B / Qwen3-0.6B in bf16 are the entries that run where NVFP4 cannot, CPU included. Three details are load-bearing rather than incidental: - The 27B entries pin revision 890bdef7. That repository was later re-quantized in place from NVFP4 to FP8 W8A8 under the same name, so an unpinned entry resolves to different weights and reports nothing. - Qwen3-Coder names tool_parser: qwen3_coder explicitly. Its dialect is byte-identical on the wire to step3p5's, so chat-template sniffing cannot separate them and auto-detection picks wrong. - enable_prefix_caching is deliberately left unset everywhere. It defaults on for dense models and off for the GDN hybrids, and that per-model default is the right answer. num_blocks is sized per model from its real KV footprint rather than copied between entries, which ranges from 20 KiB/token on the 35B to 144 KiB/token on the 4B. Docs: adds features/vllm-cpp.md covering installation, the model table, the pinning rationale and how to choose between the speculative variants, and cross-links it from the existing engine_args reference. It also records that the CUDA images are built for Blackwell only, which is narrower than vllm.cpp's own ten-architecture release and makes an otherwise cryptic "no kernel image is available" failure legible. Verified: gallery suite green; all eight decode and validate as a ModelConfig. qwen3-0.6b-vllm-cpp confirmed end to end on a real cluster, chat plus engine-parsed tool_calls. The NVFP4 entries are not yet runtime-verified: no available node has kernels for them. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-08-16 00:32:37 +02:00