Adira ef724a3c9d feat(api): add /v1/detokenize endpoint (#9620) ...

* feat(api): add /v1/detokenize endpoint

Closes #1649.

Mirror of the existing /v1/tokenize path, requested by @benniekiss in
the issue thread for "complete API workflow" use cases that need to
turn token IDs back into text without local processing.

- Add Detokenize gRPC RPC with DetokenizeRequest{tokens} /
  DetokenizeResponse{content} messages.
- Implement in the llama.cpp backend using common_token_to_piece, the
  same primitive TokenizeString already uses internally.
- Other backends inherit the default Unimplemented from base.Base, in
  line with how Detect, Rerank, etc. are gated per-backend.
- Wire up the Go gRPC interface, server, client, and in-process embed
  wrapper alongside their TokenizeString counterparts.
- Add the schema types, ModelDetokenize wrapper, HTTP handler, route
  registration, RouteFeatureRegistry entry (gated by FeatureTokenize so
  no new feature flag is needed), and the discovery map entry under
  ai_functions.
- Regenerated swagger reflects the new endpoint and types.
- Update authentication.md to list /v1/detokenize alongside /v1/tokenize.

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>

* test(e2e): add mock backend tests for /v1/detokenize

Add Detokenize to the mock gRPC backend and wire up two e2e tests in
the MockBackend suite: one that posts known token IDs and asserts a
non-empty content response, and a round-trip that tokenizes first then
detokenizes the returned IDs.

Addresses reviewer feedback on #9620.

Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>

* fix(kokoros): implement detokenize in the Rust backend service

The Detokenize RPC added in this PR grows the tonic-generated Backend
trait. Unlike the other languages there is nothing to inherit a default
from — Rust trait impls must list every method — so
backend/rust/kokoros failed to compile:

  error[E0046]: not all trait items implemented, missing: `detokenize`
    --> src/service.rs:72:1
  72 | impl Backend for KokorosService {

Go backends pick up the Unimplemented default from base.Base, and the
generated C++/Python servicer bases default to UNIMPLEMENTED, which is
why the Rust backend was the only one that broke. kokoros is the sole
Rust crate in the tree, so this is the full extent of the fallout.

Return Status::unimplemented("Not supported"), matching how this same
file already gates tokenize_string and ~20 other unsupported RPCs.

Fixes the tests-kokoros and backend-jobs-singlearch-4 (-cpu-kokoros)
failures on the previous head.

Assisted-by: Claude:claude-opus-5 cargo
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>

---------

Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>

2026-07-30 16:01:47 +02:00

181 KiB

Raw Blame History

View Raw