mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-12 14:22:11 -04:00
Ettore Di Giacinto
f06eb61633
grpc::ServerWriter::Write() returns false once the peer is gone, and PredictStream ignored that result at every call site. The handler kept pulling decoded tokens and writing them into a dead stream, so the llama.cpp slot stayed busy until the generation ended on its own terms. A model configured with max_tokens 0 and a large context ends on its own terms only at the context limit. On a 35B model at ~41 t/s a 120k context is about fifty minutes, and a slot held that long is a slot every other request for that model queues behind. Two abandoned requests were enough to make a node with free VRAM and a healthy control plane serve nothing: new requests timed out waiting for a slot, each timeout abandoned another generation, and the node fell further behind the longer it ran. Track the peer instead. The first failed write retires it for good, since a stream never recovers, and the RPC's own cancellation flag folds into the same predicate so the loop has one condition to test. Returning early is what frees the slot: ~server_response_reader() posts SERVER_TASK_TYPE_CANCEL for whatever is still decoding. TTSStream already checked Write(); this brings PredictStream in line. Cancellation stays cooperative and is checked between decoded results, so a batch already in flight may finish before the request stops. Assisted-by: Claude:claude-opus-5
45 lines
1.5 KiB
C++
45 lines
1.5 KiB
C++
// SPDX-License-Identifier: MIT
|
|
#pragma once
|
|
|
|
namespace llama_grpc {
|
|
|
|
// Tracks whether a server-streaming RPC still has somewhere to send tokens.
|
|
//
|
|
// grpc::ServerWriter::Write() returns false once the peer is gone, and a
|
|
// stream never recovers afterwards. Ignoring that result is not harmless: the
|
|
// handler goes on draining decoded tokens into a dead stream, so the llama.cpp
|
|
// slot stays busy for the rest of the request's token budget. A model config
|
|
// with no max_tokens and a large context turns that into tens of minutes per
|
|
// abandoned request, and the slots are exactly what every other request queues
|
|
// behind.
|
|
//
|
|
// Returning as soon as the peer is gone is what frees the slot: the handler's
|
|
// server_response_reader then goes out of scope and its destructor posts
|
|
// SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
|
|
class StreamPeer {
|
|
public:
|
|
// Records the outcome of a Write(). Once a write has failed the peer stays
|
|
// gone -- a later write cannot succeed on a broken stream.
|
|
void observe_write(bool ok) noexcept {
|
|
if (!ok) {
|
|
gone_ = true;
|
|
}
|
|
}
|
|
|
|
// Folds in the RPC's own cancellation flag, so callers have a single
|
|
// predicate to test rather than two that can disagree.
|
|
void observe_cancelled(bool cancelled) noexcept {
|
|
if (cancelled) {
|
|
gone_ = true;
|
|
}
|
|
}
|
|
|
|
bool gone() const noexcept { return gone_; }
|
|
bool alive() const noexcept { return !gone_; }
|
|
|
|
private:
|
|
bool gone_ = false;
|
|
};
|
|
|
|
} // namespace llama_grpc
|