mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-25 15:44:56 -04:00
21c5495a99
* feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner, ABI v27) so LocalAI can serve zero-shot named entity recognition through the existing TokenClassify gRPC method. backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with the text and labels, copies the C-owned entity array into protobuf TokenClassifyEntity messages, and frees the result. govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion bumped 26 -> 27. options.go: ner_labels, ner_threshold, ner_max_width parsed from engine_args. pkg/grpc: ClassifyModel interface and TokenClassify server handler (follows the Embedding locking pattern). core/config: vllm-cpp backend declares MethodTokenClassify and UsecaseTokenClassify. docs/content/features/vllm-cpp.md: NER section documenting the engine_args keys and the host-forward contract. Assisted-by: MAKI:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): correct NER pointer lint directive Use the govet directive for the C-owned NER array, matching the other purego pointer conversions. The array remains valid until its deferred free; the misspelled directive caused CI to flag this conversion. Assisted-by: Codex:gpt-6 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): add kev-compatible SystemOne API endpoints Add POST /v1/systemone, /v1/systemone/permute, and /v1/systemone/separate to LocalAI, mirroring the kev project's structured-extraction API. Each endpoint runs zero-shot NER over the rendered state text and builds kev-compatible answers for three question types: noul (binary entity presence), choice (pick one option), and score (pick one level). The TokenClassifyRequest proto gains a `repeated string labels` field so each question can supply its own labels at inference time, and TokenClassifier gains TokenClassifyWithLabels for per-call label selection. The vllm-cpp backend uses request labels when non-empty, falling back to configured ner_labels then the built-in defaults. Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2) are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so both servers produce the same answer shape. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): suppress gosec G404 on seeded permutation RNG The SystemOne permute endpoint uses math/rand with a caller-supplied seed for reproducible option permutations, matching kev's random.seed. gosec flags this as G404 (weak RNG). Add #nosec with a comment naming the intent: this is reproducibility, not cryptography. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(vllm-cpp): bump vllm.cpp pin to GLiNER2.5 merge commit Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER functions, and server endpoints into the LocalAI vllm-cpp backend. The ABI version (27) and Go struct mirrors already match. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): use instruction text as NER label in SystemOne handler The SystemOne handler was passing question IDs as NER labels for noul questions and bare key names for choice questions, so the model never matched any entities. Port the label mapping from vllm.cpp's ParseSystemOneBody: - noul: use the rendered instructions field (with instr alias) as the NER label, not the question ID - choice: use optionText(name, desc) — "name: description" or "name" when the description is null/empty — not the bare key - score: already correct (rendered criteria text) - permute: shuffle indices and build parallel key/label arrays so the NER call uses the optionText labels while the response is keyed by the original option names Also add the instructions field to the SystemOneQuestion schema struct (accepted alongside the instr backward-compat alias). Verified end-to-end against the real GLiNER2.5 model: noul questions now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is" (person, 0.852) where they previously returned zero entities. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
96 lines
3.8 KiB
Go
96 lines
3.8 KiB
Go
package schema
|
||
|
||
import "encoding/json"
|
||
|
||
// SystemOneRequest is the body for POST /v1/systemone,
|
||
// /v1/systemone/separate, and the inner `request` of /v1/systemone/permute.
|
||
// Mirrors the kev project's SystemOneRequest (jaredpalmer/kev serve.py).
|
||
type SystemOneRequest struct {
|
||
// State is the text (or any JSON value) to extract from. A non-string
|
||
// value is rendered to its JSON representation before NER.
|
||
State json.RawMessage `json:"state"`
|
||
// Questions maps question IDs to their definitions.
|
||
Questions map[string]SystemOneQuestion `json:"questions"`
|
||
// Model names the NER model to use. Optional.
|
||
Model string `json:"model,omitempty"`
|
||
// Threshold is the minimum entity confidence (0–1). Default 0.5.
|
||
Threshold *float32 `json:"threshold,omitempty"`
|
||
// MaxWidth is the maximum span width in tokens. Default 12.
|
||
MaxWidth *int `json:"max_width,omitempty"`
|
||
}
|
||
|
||
// SystemOneQuestion defines one question. Type is "noul", "choice", or
|
||
// "score". Instructions (or the instr alias) is human-readable instruction
|
||
// text. For noul questions the rendered instruction is the NER label.
|
||
// Criteria is:
|
||
// - noul: omitted
|
||
// - choice: a map of option_name → description (NER label is
|
||
// "name: description", or "name" when the description is null/empty)
|
||
// - score: an array of level descriptions (each rendered text is a NER label)
|
||
type SystemOneQuestion struct {
|
||
Type string `json:"type"`
|
||
Instructions json.RawMessage `json:"instructions,omitempty"`
|
||
Instr string `json:"instr,omitempty"`
|
||
Criteria json.RawMessage `json:"criteria,omitempty"`
|
||
}
|
||
|
||
// SystemOneResponse is the shared response shape for /v1/systemone and
|
||
// /v1/systemone/separate.
|
||
type SystemOneResponse struct {
|
||
Model string `json:"model"`
|
||
Answers map[string]SystemOneAnswer `json:"answers"`
|
||
Usage SystemOneUsage `json:"usage"`
|
||
LatencyMs float64 `json:"latency_ms"`
|
||
}
|
||
|
||
type SystemOneUsage struct {
|
||
InputTokens int `json:"input_tokens"`
|
||
OutputTokens int `json:"output_tokens"`
|
||
}
|
||
|
||
// SystemOneAnswer is one question's answer. The fields populated depend on
|
||
// the question type:
|
||
// - noul: Noul (float 0–1), Entities
|
||
// - choice: Choice (string), Confidence, Probabilities (map)
|
||
// - score: Score (float), Legend (map), Probabilities (map), Confidence
|
||
type SystemOneAnswer struct {
|
||
Type string `json:"type"`
|
||
Noul *float64 `json:"noul,omitempty"`
|
||
Entities []SystemOneEntity `json:"entities,omitempty"`
|
||
Choice *string `json:"choice,omitempty"`
|
||
Confidence *float64 `json:"confidence,omitempty"`
|
||
Probabilities map[string]float64 `json:"probabilities,omitempty"`
|
||
Score *float64 `json:"score,omitempty"`
|
||
Legend map[string]string `json:"legend,omitempty"`
|
||
}
|
||
|
||
type SystemOneEntity struct {
|
||
Text string `json:"text"`
|
||
Start int `json:"start"`
|
||
End int `json:"end"`
|
||
Confidence float32 `json:"confidence"`
|
||
}
|
||
|
||
// SystemOnePermuteRequest is the body for POST /v1/systemone/permute.
|
||
type SystemOnePermuteRequest struct {
|
||
Request SystemOneRequest `json:"request"`
|
||
Question string `json:"question"`
|
||
NPerm int `json:"n_perm,omitempty"`
|
||
Seed int64 `json:"seed,omitempty"`
|
||
}
|
||
|
||
// SystemOnePermuteRun is one permutation's result.
|
||
type SystemOnePermuteRun struct {
|
||
Order []string `json:"order"`
|
||
Probabilities map[string]float64 `json:"probabilities"`
|
||
Choice string `json:"choice"`
|
||
LatencyMs float64 `json:"latency_ms"`
|
||
}
|
||
|
||
// SystemOnePermuteResponse is the response for POST /v1/systemone/permute.
|
||
type SystemOnePermuteResponse struct {
|
||
Runs []SystemOnePermuteRun `json:"runs"`
|
||
ArgmaxStable bool `json:"argmax_stable"`
|
||
Spread map[string]float64 `json:"spread"`
|
||
}
|