From c60a2e47d6bf940592a2c76af3987975bf1ee745 Mon Sep 17 00:00:00 2001 From: mudler-agent Date: Sat, 24 Oct 2026 22:47:04 +0200 Subject: [PATCH] feat(backends): timestral.cpp forecast backend and /v1/forecast API (#12885) * feat(grpc): add a Forecast RPC for time-series models Add Forecast to the backend service with columnar request and response messages: flat row-major arrays, NaN for missing values, and one message per series so a caller can send series of different shapes in one call. The response keeps the request order. Wire the RPC through the server, client, embedded client, connection evicting client, in-flight tracking client and the base backend, which answers unimplemented. core/backend gains a Forecast helper that stamps the model identity and records a trace, like the other single-purpose RPCs. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(config): add the forecast usecase FLAG_FORECAST marks a model as a time-series forecaster. The timestral-cpp backend declares it as its default usecase, so a config with that backend needs no extra lines. The backend is also listed as a non-text backend, so a forecast model is never offered as a chat model. It shows up in the gallery filter and in the capability list. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(api): add POST /v1/forecast Take one or more numeric series and return quantile trajectories for the next prediction_length steps. Targets are [variates][steps] or a flat array for one variate; JSON null marks a missing value. Past and known covariates and categorical codes are checked for length before the backend sees them. Quantiles default to 0.1, 0.5 and 0.9 in the backend. The route is gated by a new default-on forecast API feature. The endpoint is listed in the swagger spec, the /api/instructions index, the UI model filter and the capability symbols. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(backends): add the timestral-cpp forecast backend Wrap the timestral.cpp C ABI with purego, the way the other ggml backends do: libtimestral is built from a pinned commit with ggml embedded, and the Go binary loads it at start. Forecast validates the counts against the array lengths, groups the series by shape, makes one library call per group and puts the results back in the order the caller sent them. Quantiles default to 0.1, 0.5 and 0.9, and a level the model does not have is refused with the list of valid ones. A mutex guards the handle because the library is not reentrant. Request size is bounded by two model options. The unit specs use a fake library. The integration specs and a gRPC spec run against the 5M F32 model that test.sh downloads and checks against a fixed SHA-256. Assisted-by: Claude:claude-sonnet-5-5 go-vet * build(backends): register timestral-cpp in the build and the library Add the CPU image to the backend matrix for amd64 and arm64, plus an Apple silicon entry that is not built or tested yet. The image has no GPU keys because the engine runs on the CPU only. Wire the make targets, the path filter and the dependency bump bot. The library entries were generated with backendcatalog. The images are not published yet, so the release entries are unpinned until a maintainer runs --repin. The gallery reader test now counts the names in the library instead of a fixed number, so adding a backend does not break it. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(gallery): add the Timestral forecast models Add Timestral 5M, 28M, 123M and 209M as F32 entries, each with its F16 build as a variant. Q8_0 and the K-quants are left out until the quality question is settled. Each entry runs on the timestral-cpp backend, on the CPU only. The weights use Mistral's Modified MIT licence, which bars use by companies above a monthly revenue threshold. The first paragraph after the summary says so, the licence file is installed next to the weights, and the revenue-restricted-license tag makes the entries filterable. The memory each build needs has not been measured, so no size is set. Assisted-by: Claude:claude-sonnet-5-5 * docs(forecast): document /v1/forecast and the Timestral models Add a feature page with the request and response shapes, the error cases, the model options and the licence note. List the engine in the README, the compatibility table and the backend categories. Assisted-by: Claude:claude-sonnet-5-5 --------- Co-authored-by: Ettore Di Giacinto --- .github/backend-matrix.yml | 35 ++ .github/workflows/bump_deps.yaml | 4 + Makefile | 8 +- README.md | 1 + backend/backend.proto | 49 +++ backend/go/timestral-cpp/.gitignore | 21 + backend/go/timestral-cpp/Makefile | 80 ++++ backend/go/timestral-cpp/abi.go | 71 ++++ backend/go/timestral-cpp/abi_test.go | 39 ++ backend/go/timestral-cpp/fakelib_test.go | 148 +++++++ backend/go/timestral-cpp/gotimestralcpp.go | 218 ++++++++++ .../go/timestral-cpp/gotimestralcpp_test.go | 305 ++++++++++++++ .../go/timestral-cpp/grpc_integration_test.go | 106 +++++ backend/go/timestral-cpp/integration_test.go | 218 ++++++++++ backend/go/timestral-cpp/main.go | 40 ++ backend/go/timestral-cpp/options.go | 84 ++++ backend/go/timestral-cpp/options_test.go | 39 ++ backend/go/timestral-cpp/package.sh | 40 ++ backend/go/timestral-cpp/plan.go | 182 +++++++++ backend/go/timestral-cpp/run.sh | 21 + backend/go/timestral-cpp/test.sh | 35 ++ .../go/timestral-cpp/timestral_suite_test.go | 13 + backend/index.yaml | 108 +++++ core/backend/forecast.go | 80 ++++ .../backend/model_identity_modalities_test.go | 11 + core/config/backend_capabilities.go | 13 + core/config/backend_capabilities_test.go | 31 ++ core/config/meta/constants.go | 1 + core/config/model_capabilities.go | 1 + core/config/model_config.go | 13 +- core/config/model_config_filter_test.go | 2 +- .../gallery/backend_library_generated_test.go | 10 +- core/http/auth/features.go | 4 + core/http/auth/features_forecast_test.go | 24 ++ core/http/auth/permissions.go | 3 +- .../endpoints/localai/api_instructions.go | 6 + .../localai/api_instructions_test.go | 3 +- core/http/endpoints/localai/forecast.go | 221 ++++++++++ core/http/endpoints/localai/forecast_test.go | 179 ++++++++ .../react-ui/public/locales/en/models.json | 1 + core/http/react-ui/src/pages/Models.jsx | 1 + core/http/react-ui/src/utils/capabilities.js | 1 + core/http/routes/localai.go | 6 + core/http/routes/model_listing_routes_test.go | 1 + core/http/routes/ui_api.go | 1 + core/schema/localai.go | 73 ++++ core/services/nodes/health_mock_test.go | 3 + core/services/nodes/inflight.go | 6 + core/services/nodes/inflight_test.go | 4 + core/trace/backend_trace.go | 1 + docs/content/features/backends.md | 1 + docs/content/features/forecast.md | 131 ++++++ docs/content/reference/compatibility-table.md | 1 + gallery/backends/index.yaml | 150 +++++++ gallery/index.yaml | 316 ++++++++++++++ gallery/models/index.yaml | 384 ++++++++++++++++++ pkg/grpc/backend.go | 1 + pkg/grpc/base/base.go | 4 + pkg/grpc/client.go | 17 + pkg/grpc/embed.go | 4 + pkg/grpc/interface.go | 1 + pkg/grpc/model_identity_modalities_test.go | 6 + pkg/grpc/server.go | 15 + pkg/model/connection_evicting_client.go | 6 + scripts/lib/backend-filter.mjs | 7 + swagger/docs.go | 191 +++++++++ swagger/swagger.json | 191 +++++++++ swagger/swagger.yaml | 135 ++++++ 68 files changed, 4118 insertions(+), 8 deletions(-) create mode 100644 backend/go/timestral-cpp/.gitignore create mode 100644 backend/go/timestral-cpp/Makefile create mode 100644 backend/go/timestral-cpp/abi.go create mode 100644 backend/go/timestral-cpp/abi_test.go create mode 100644 backend/go/timestral-cpp/fakelib_test.go create mode 100644 backend/go/timestral-cpp/gotimestralcpp.go create mode 100644 backend/go/timestral-cpp/gotimestralcpp_test.go create mode 100644 backend/go/timestral-cpp/grpc_integration_test.go create mode 100644 backend/go/timestral-cpp/integration_test.go create mode 100644 backend/go/timestral-cpp/main.go create mode 100644 backend/go/timestral-cpp/options.go create mode 100644 backend/go/timestral-cpp/options_test.go create mode 100755 backend/go/timestral-cpp/package.sh create mode 100644 backend/go/timestral-cpp/plan.go create mode 100755 backend/go/timestral-cpp/run.sh create mode 100755 backend/go/timestral-cpp/test.sh create mode 100644 backend/go/timestral-cpp/timestral_suite_test.go create mode 100644 core/backend/forecast.go create mode 100644 core/http/auth/features_forecast_test.go create mode 100644 core/http/endpoints/localai/forecast.go create mode 100644 core/http/endpoints/localai/forecast_test.go create mode 100644 docs/content/features/forecast.md diff --git a/.github/backend-matrix.yml b/.github/backend-matrix.yml index 293bc6978..6fd62baf6 100644 --- a/.github/backend-matrix.yml +++ b/.github/backend-matrix.yml @@ -4363,6 +4363,35 @@ include: dockerfile: "./backend/Dockerfile.golang" context: "./" ubuntu-version: '2404' + # timestral-cpp (CPU only: the engine has no GPU path) + - build-type: '' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + platform-tag: 'amd64' + tag-latest: 'auto' + tag-suffix: '-cpu-timestral-cpp' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "timestral-cpp" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: '' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/arm64' + platform-tag: 'arm64' + tag-latest: 'auto' + tag-suffix: '-cpu-timestral-cpp' + runs-on: 'ubuntu-24.04-arm' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "timestral-cpp" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' # parakeet-cpp - build-type: '' cuda-major-version: "" @@ -6704,6 +6733,12 @@ includeDarwin: tag-suffix: "-metal-darwin-arm64-voice-detect" build-type: "metal" lang: "go" + # timestral-cpp: CPU build for Apple silicon (the engine has no Metal path). + # Not built or tested on macOS yet. + - backend: "timestral-cpp" + tag-suffix: "-metal-darwin-arm64-timestral-cpp" + build-type: "metal" + lang: "go" - backend: "face-detect" tag-suffix: "-metal-darwin-arm64-face-detect" build-type: "metal" diff --git a/.github/workflows/bump_deps.yaml b/.github/workflows/bump_deps.yaml index de3a8b74d..487a52d33 100644 --- a/.github/workflows/bump_deps.yaml +++ b/.github/workflows/bump_deps.yaml @@ -78,6 +78,10 @@ jobs: variable: "FACEDETECT_VERSION" branch: "master" file: "backend/go/face-detect/Makefile" + - repository: "mudler/timestral.cpp" + variable: "TIMESTRAL_VERSION" + branch: "main" + file: "backend/go/timestral-cpp/Makefile" - repository: "mudler/depth-anything.cpp" variable: "DEPTHANYTHING_VERSION" branch: "master" diff --git a/Makefile b/Makefile index 8739e1dac..f5d56cbd6 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ # Disable parallel execution for backend builds -.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/qdrant-store backends/cloud-proxy backends/localai-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/mlx-video backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin +.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/qdrant-store backends/cloud-proxy backends/localai-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/mlx-video backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/timestral-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin .NOTPARALLEL: backends/whisper-medusa .NOTPARALLEL: backends/funasr @@ -658,6 +658,7 @@ prepare-test-extra: protogen-python $(MAKE) -C backend/rust/kokoros kokoros-grpc $(MAKE) -C backend/go/rfdetr-cpp $(MAKE) -C backend/go/locate-anything-cpp + $(MAKE) -C backend/go/timestral-cpp $(MAKE) -C backend/go/trellis2cpp $(MAKE) -C backend/go/kimodocpp $(MAKE) -C backend/go/gemxcpp @@ -696,6 +697,7 @@ test-extra: prepare-test-extra $(MAKE) -C backend/go/rfdetr-cpp test $(MAKE) -C backend/go/locate-anything-cpp test $(MAKE) -C backend/go/depth-anything-cpp test + $(MAKE) -C backend/go/timestral-cpp test $(MAKE) -C backend/go/supertonic test $(MAKE) -C backend/go/vllm-cpp test $(MAKE) -C backend/go/nemo-speech-cpp test @@ -1368,6 +1370,7 @@ BACKEND_PARAKEET_CPP = parakeet-cpp|golang|.|false|true BACKEND_MOSS_TRANSCRIBE_CPP = moss-transcribe-cpp|golang|.|false|true BACKEND_NEMO_SPEECH_CPP = nemo-speech-cpp|golang|.|false|true BACKEND_DEPTH_ANYTHING_CPP = depth-anything-cpp|golang|.|false|true +BACKEND_TIMESTRAL_CPP = timestral-cpp|golang|.|false|true BACKEND_VOXTRAL = voxtral|golang|.|false|true BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true @@ -1491,6 +1494,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_PARAKEET_CPP))) $(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TRANSCRIBE_CPP))) $(eval $(call generate-docker-build-target,$(BACKEND_NEMO_SPEECH_CPP))) $(eval $(call generate-docker-build-target,$(BACKEND_DEPTH_ANYTHING_CPP))) +$(eval $(call generate-docker-build-target,$(BACKEND_TIMESTRAL_CPP))) $(eval $(call generate-docker-build-target,$(BACKEND_VOXTRAL))) $(eval $(call generate-docker-build-target,$(BACKEND_OPUS))) $(eval $(call generate-docker-build-target,$(BACKEND_RERANKERS))) @@ -1548,7 +1552,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC))) docker-save-%: backend-images docker save local-ai-backend:$* -o backend-images/$*.tar -docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-localai-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-nemo-speech-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-qdrant-store docker-build-audio-cpp +docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-localai-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-nemo-speech-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-qdrant-store docker-build-audio-cpp docker-build-timestral-cpp docker-build-backends: docker-build-whisper-medusa docker-build-backends: docker-build-funasr diff --git a/README.md b/README.md index 7548f2c4f..8a6982b09 100644 --- a/README.md +++ b/README.md @@ -254,6 +254,7 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native | [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) | Open-vocabulary object detection and visual grounding (LocateAnything-3B) | | [depth-anything.cpp](https://github.com/mudler/depth-anything.cpp) | Depth Anything 3 monocular metric depth + camera pose estimation | | [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace), replacing the Python insightface backend | +| [timestral.cpp](https://github.com/mudler/timestral.cpp) | Mistral Timestral time-series forecasting: quantile forecasts with covariates, on the CPU | | [free-splatter.cpp](https://github.com/localai-org/free-splatter.cpp) | Pose-free 3D reconstruction (FreeSplatter): turns a handful of plain photos into 3D Gaussians, no camera poses or GPU required | | [trellis2.cpp](https://github.com/localai-org/trellis2cpp) | C++/GGML port of Microsoft TRELLIS.2: single-image to textured 3D mesh (GLB with PBR materials) | | [gem-x.cpp](https://github.com/localai-org/gem-x.cpp) | Live human motion capture with SOMA and SMPL pose streams (CPU/Vulkan) | diff --git a/backend/backend.proto b/backend/backend.proto index cf15088e6..2d3f0802b 100644 --- a/backend/backend.proto +++ b/backend/backend.proto @@ -48,6 +48,9 @@ service Backend { // (e.g. CED over the AudioSet ontology) on a clip and returns scored labels. rpc SoundDetection(SoundDetectionRequest) returns (SoundDetectionResponse) {} rpc Depth(DepthRequest) returns (DepthResponse) {} + // Forecast runs a time-series forecasting model (timestral.cpp) on one or more + // numeric series and returns quantile trajectories for the next steps. + rpc Forecast(ForecastRequest) returns (ForecastResponse) {} rpc FaceVerify(FaceVerifyRequest) returns (FaceVerifyResponse) {} rpc FaceAnalyze(FaceAnalyzeRequest) returns (FaceAnalyzeResponse) {} rpc VoiceVerify(VoiceVerifyRequest) returns (VoiceVerifyResponse) {} @@ -1090,6 +1093,52 @@ message DepthResponse { bool is_metric = 12; // depth is in metric units } +// --- Time-series forecasting messages (timestral.cpp) --- +// +// Arrays are flat and row-major. NaN marks a missing value. One request may +// carry series of different shapes; the backend groups them by shape and +// returns the results in the order the series were sent. + +message ForecastSeries { + string id = 1; // echoed back in the matching result + int32 num_variates = 2; // V + int32 length = 3; // T, context steps per variate + repeated float target = 4; // V*T, NaN = missing + int32 num_past_covariates = 5; // C_past + repeated float past_covariates = 6; // C_past*T + int32 num_known_covariates = 7; // C_known + repeated float known_covariates = 8; // C_known*(T+H) + int32 num_past_categorical = 9; // CC_past + repeated double past_categorical = 10; // CC_past*T, NaN = missing code + int32 num_known_categorical = 11; // CC_known + repeated double known_categorical = 12; // CC_known*(T+H), NaN = missing code +} + +message ForecastRequest { + repeated ForecastSeries series = 1; + int32 prediction_length = 2; // H, steps to forecast + repeated double quantiles = 3; // empty = backend default + int32 max_context = 4; // 0 = model default + optional bool symmetric_averaging = 5; // unset = model default + optional bool clip_nonnegative = 6; // unset = model default + // ModelIdentity names the model this request is for; see + // PredictOptions.ModelIdentity for the full rationale. Empty means "no + // identity supplied" and backends MUST skip the check. + string ModelIdentity = 7; +} + +message ForecastSeriesResult { + string id = 1; + int32 num_variates = 2; // V + repeated float values = 3; // V*Q*H row-major +} + +message ForecastResponse { + int32 prediction_length = 1; + repeated double quantile_levels = 2; // levels returned, ascending + repeated ForecastSeriesResult series = 3; // same order as the request +} + // --- Face recognition messages --- message FacialArea { diff --git a/backend/go/timestral-cpp/.gitignore b/backend/go/timestral-cpp/.gitignore new file mode 100644 index 000000000..1d2cbe785 --- /dev/null +++ b/backend/go/timestral-cpp/.gitignore @@ -0,0 +1,21 @@ +# Fetched upstream sources +sources/ + +# CMake build directories +build*/ + +# Build artifacts staged in this directory by the Makefile (copied from +# sources/ or symlinked for local development). +*.so +*.so.* +*.dylib +timestral_capi.h + +# Test model downloaded by test.sh +test-models/ + +# Compiled backend binary +timestral-cpp-grpc + +# Packaging output +package/ diff --git a/backend/go/timestral-cpp/Makefile b/backend/go/timestral-cpp/Makefile new file mode 100644 index 000000000..de8e4f195 --- /dev/null +++ b/backend/go/timestral-cpp/Makefile @@ -0,0 +1,80 @@ +# timestral-cpp backend Makefile. +# +# The upstream pin is TIMESTRAL_VERSION below, in this file so the dependency +# bump bot can find and update it. +# +# Local dev shortcut: if you already have an out-of-tree timestral.cpp build, +# symlink the .so + header into this directory and skip the clone/cmake steps: +# +# ln -sf /path/to/timestral.cpp/build-shared/libtimestral.so . +# ln -sf /path/to/timestral.cpp/include/timestral_capi.h . +# go build -o timestral-cpp-grpc . +# +# The default target does the clone-at-pin and cmake build, so CI needs no +# side checkout. + +TIMESTRAL_VERSION?=6c0ee070de54e9b2611ea90696dee3a41178700c +TIMESTRAL_REPO?=https://github.com/mudler/timestral.cpp + +GOCMD?=go +GO_TAGS?= +JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4) + +NATIVE?=false + +# Build ggml statically into libtimestral (PIC) so the shared library is +# self-contained: dlopen needs no libggml*.so next to it, only system libs +# (libstdc++, libgomp, libc) that the runtime image already provides. +# +# The engine runs on the CPU only. BUILD_TYPE is therefore not forwarded: a +# metal, cublas or vulkan build type still produces the same CPU library. +CMAKE_ARGS?=-DCMAKE_BUILD_TYPE=Release -DTIMESTRAL_SHARED=ON -DTIMESTRAL_BUILD_CLI=OFF -DTIMESTRAL_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON + +ifeq ($(NATIVE),false) + CMAKE_ARGS+=-DGGML_NATIVE=OFF +endif + +.PHONY: timestral-cpp-grpc package build clean purge test all + +all: timestral-cpp-grpc + +# Clone the upstream source at the pinned commit, with its ggml submodule. The +# directory is the target, so make clones only when it is missing. After a +# TIMESTRAL_VERSION bump, run 'make purge && make' to fetch again. +sources/timestral.cpp: + mkdir -p sources/timestral.cpp + cd sources/timestral.cpp && \ + git init -q && \ + git remote add origin $(TIMESTRAL_REPO) && \ + git fetch --depth 1 origin $(TIMESTRAL_VERSION) && \ + git checkout FETCH_HEAD && \ + git submodule update --init --recursive --depth 1 --single-branch + +# Build the shared library out of tree, then stage it and the header next to +# the Go sources, so purego.Dlopen("libtimestral.so") finds it. +libtimestral.so: sources/timestral.cpp + cmake -B sources/timestral.cpp/build-shared -S sources/timestral.cpp $(CMAKE_ARGS) + cmake --build sources/timestral.cpp/build-shared --config Release -j$(JOBS) --target timestral + cp -fv sources/timestral.cpp/build-shared/libtimestral.so* ./ 2>/dev/null || true + cp -fv sources/timestral.cpp/build-shared/libtimestral.dylib ./ 2>/dev/null || true + cp -fv sources/timestral.cpp/include/timestral_capi.h ./ + +timestral-cpp-grpc: libtimestral.so main.go abi.go gotimestralcpp.go options.go plan.go + CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o timestral-cpp-grpc . + +package: timestral-cpp-grpc + bash package.sh + +build: package + +# The pure-Go specs always run. The integration specs need +# TIMESTRAL_BACKEND_TEST_MODEL and a built library; test.sh fetches the 5M F32 +# model and sets both. Without them the integration specs skip. +test: + bash test.sh + +clean: purge + rm -rf libtimestral.so* libtimestral.dylib timestral_capi.h package timestral-cpp-grpc + +purge: + rm -rf sources/timestral.cpp diff --git a/backend/go/timestral-cpp/abi.go b/backend/go/timestral-cpp/abi.go new file mode 100644 index 000000000..8b651c78f --- /dev/null +++ b/backend/go/timestral-cpp/abi.go @@ -0,0 +1,71 @@ +package main + +import ( + "unsafe" + + "github.com/ebitengine/purego" +) + +// purego-bound entry points from libtimestral.so. Names match +// timestral_capi.h exactly, so `nm libtimestral.so | grep timestral_capi` is +// enough to spot drift. +// +// The context and the malloc'd forecast buffer are uintptr so the raw pointer +// comes back and can be released with timestral_capi_free_buffer. A float* +// return typed as a Go slice would be copied and the C buffer leaked. +var ( + CppAbiVersion func() int32 + CppLoad func(ggufPath string, threads int32) uintptr + CppFree func(ctx uintptr) + CppLastError func(ctx uintptr) string + CppNumQuantiles func(ctx uintptr) int32 + CppQuantileLevel func(ctx uintptr, i int32) float64 + CppForecast func(ctx uintptr, req unsafe.Pointer, outQ unsafe.Pointer) uintptr + CppFreeBuffer func(p uintptr) + abiVersionRequired int32 = 1 +) + +// forecastRequest mirrors timestral_forecast_request (ABI version 1). Go lays +// the struct out like a 64-bit C compiler does, and abi_test.go checks every +// offset. The pointer fields are unsafe.Pointer, not uintptr, so the buffers +// they name stay reachable for the garbage collector until the call returns. +type forecastRequest struct { + context unsafe.Pointer // B*V*T float32, NaN = missing + observationMask unsafe.Pointer // B*V*T bytes or nil; not used by this backend + B, V, T int32 + predictionLen int32 + pastCov unsafe.Pointer // B*c_past*T float32 + cPast int32 + knownCov unsafe.Pointer // B*c_known*(T+H) float32 + cKnown int32 + catPast unsafe.Pointer // B*cc_past*T float64 + ccPast int32 + catKnown unsafe.Pointer // B*cc_known*(T+H) float64 + ccKnown int32 + symmetricAvg int32 // -1 model default, 0, 1 + clipNonneg int32 // -1 model default, 0, 1 + maxContext int32 // 0 model default +} + +// libFunc pairs a Go function variable with its C symbol. +type libFunc struct { + ptr any + name string +} + +// registerLibFuncs binds the eight timestral_capi_* entry points of an opened +// library. main.go and the integration tests share it. +func registerLibFuncs(lib uintptr) { + for _, lf := range []libFunc{ + {&CppAbiVersion, "timestral_capi_abi_version"}, + {&CppLoad, "timestral_capi_load"}, + {&CppFree, "timestral_capi_free"}, + {&CppLastError, "timestral_capi_last_error"}, + {&CppNumQuantiles, "timestral_capi_num_quantiles"}, + {&CppQuantileLevel, "timestral_capi_quantile_level"}, + {&CppForecast, "timestral_capi_forecast"}, + {&CppFreeBuffer, "timestral_capi_free_buffer"}, + } { + purego.RegisterLibFunc(lf.ptr, lib, lf.name) + } +} diff --git a/backend/go/timestral-cpp/abi_test.go b/backend/go/timestral-cpp/abi_test.go new file mode 100644 index 000000000..fa671f2af --- /dev/null +++ b/backend/go/timestral-cpp/abi_test.go @@ -0,0 +1,39 @@ +package main + +import ( + "unsafe" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// The offsets below were read from the compiler for timestral_forecast_request +// in timestral_capi.h (ABI version 1) on a 64-bit target. If the library moves a +// field, the ABI version must change and this test must fail first. +var _ = Describe("forecastRequest layout", func() { + It("matches timestral_forecast_request", func() { + var r forecastRequest + Expect(unsafe.Sizeof(r)).To(Equal(uintptr(104))) + for name, want := range map[string][2]uintptr{ + "context": {unsafe.Offsetof(r.context), 0}, + "observationMask": {unsafe.Offsetof(r.observationMask), 8}, + "B": {unsafe.Offsetof(r.B), 16}, + "V": {unsafe.Offsetof(r.V), 20}, + "T": {unsafe.Offsetof(r.T), 24}, + "predictionLen": {unsafe.Offsetof(r.predictionLen), 28}, + "pastCov": {unsafe.Offsetof(r.pastCov), 32}, + "cPast": {unsafe.Offsetof(r.cPast), 40}, + "knownCov": {unsafe.Offsetof(r.knownCov), 48}, + "cKnown": {unsafe.Offsetof(r.cKnown), 56}, + "catPast": {unsafe.Offsetof(r.catPast), 64}, + "ccPast": {unsafe.Offsetof(r.ccPast), 72}, + "catKnown": {unsafe.Offsetof(r.catKnown), 80}, + "ccKnown": {unsafe.Offsetof(r.ccKnown), 88}, + "symmetricAvg": {unsafe.Offsetof(r.symmetricAvg), 92}, + "clipNonneg": {unsafe.Offsetof(r.clipNonneg), 96}, + "maxContext": {unsafe.Offsetof(r.maxContext), 100}, + } { + Expect(want[0]).To(Equal(want[1]), "offset of %s", name) + } + }) +}) diff --git a/backend/go/timestral-cpp/fakelib_test.go b/backend/go/timestral-cpp/fakelib_test.go new file mode 100644 index 000000000..634f2788d --- /dev/null +++ b/backend/go/timestral-cpp/fakelib_test.go @@ -0,0 +1,148 @@ +package main + +import ( + "math" + "sync" + "sync/atomic" + "syscall" + "time" + "unsafe" +) + +// fakeLib replaces the package-level Cpp* function variables so the backend can +// be tested without libtimestral.so. Forecast output encodes where each value +// came from: see value(). +type fakeLib struct { + mu sync.Mutex + + levels []float64 + abiVersion int32 + loadFails bool + forecastFail bool + delay time.Duration + + loads, frees, freeBuffers int + lastErr string + calls []fakeCall + mapped map[uintptr][]byte // buffers handed out, outside the Go heap like malloc'd C memory + + inFlight, maxInFlight atomic.Int32 +} + +type fakeCall struct { + B, V, T, H, cPast, cKnown, ccPast, ccKnown int32 + sym, clip, maxContext int32 + threads int32 + context []float32 + knownCov []float32 + catPast []float64 +} + +// value is the forecast value for batch row b, variate v, step h, quantile q: +// the first context value of that row and variate, then the position. +func value(first float32, h, q int) float32 { return first*1000 + float32(h*10+q) } + +func (f *fakeLib) install() { + CppAbiVersion = func() int32 { return f.abiVersion } + CppLoad = func(path string, threads int32) uintptr { + f.mu.Lock() + defer f.mu.Unlock() + if f.loadFails { + f.lastErr = "cannot read model" + return 0 + } + f.loads++ + f.calls = append(f.calls, fakeCall{threads: threads}) + return 0x1000 + } + CppFree = func(uintptr) { f.mu.Lock(); f.frees++; f.mu.Unlock() } + CppLastError = func(uintptr) string { return f.lastErr } + CppNumQuantiles = func(uintptr) int32 { return int32(len(f.levels)) } + CppQuantileLevel = func(_ uintptr, i int32) float64 { return f.levels[i] } + CppFreeBuffer = func(p uintptr) { + f.mu.Lock() + defer f.mu.Unlock() + f.freeBuffers++ + if m, ok := f.mapped[p]; ok { + _ = syscall.Munmap(m) + delete(f.mapped, p) + } + } + CppForecast = func(_ uintptr, req unsafe.Pointer, outQ unsafe.Pointer) uintptr { + cur := f.inFlight.Add(1) + defer f.inFlight.Add(-1) + for { + prev := f.maxInFlight.Load() + if cur <= prev || f.maxInFlight.CompareAndSwap(prev, cur) { + break + } + } + if f.delay > 0 { + time.Sleep(f.delay) + } + r := (*forecastRequest)(req) + call := fakeCall{ + B: r.B, V: r.V, T: r.T, H: r.predictionLen, + cPast: r.cPast, cKnown: r.cKnown, ccPast: r.ccPast, ccKnown: r.ccKnown, + sym: r.symmetricAvg, clip: r.clipNonneg, maxContext: r.maxContext, + } + ctx := unsafe.Slice((*float32)(r.context), int(r.B*r.V*r.T)) + call.context = append([]float32(nil), ctx...) + if r.knownCov != nil { + call.knownCov = append([]float32(nil), unsafe.Slice((*float32)(r.knownCov), int(r.B*r.cKnown*(r.T+r.predictionLen)))...) + } + if r.catPast != nil { + call.catPast = append([]float64(nil), unsafe.Slice((*float64)(r.catPast), int(r.B*r.ccPast*r.T))...) + } + + f.mu.Lock() + defer f.mu.Unlock() + f.calls = append(f.calls, call) + if f.forecastFail { + f.lastErr = "invalid request" + return 0 + } + q := len(f.levels) + out := make([]float32, 0, int(r.B*r.V*r.predictionLen)*q) + for b := int32(0); b < r.B; b++ { + for v := int32(0); v < r.V; v++ { + first := ctx[(b*r.V+v)*r.T] + if math.IsNaN(float64(first)) { + first = 0 + } + for h := 0; h < int(r.predictionLen); h++ { + for qi := 0; qi < q; qi++ { + out = append(out, value(first, h, qi)) + } + } + } + } + // Hand the output over in memory the Go runtime does not own, like the + // malloc'd buffer of the real library. + mem, err := syscall.Mmap(-1, 0, len(out)*4, syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_ANON|syscall.MAP_PRIVATE) + if err != nil { + panic(err) + } + copy(unsafe.Slice((*float32)(unsafe.Pointer(&mem[0])), len(out)), out) + p := uintptr(unsafe.Pointer(&mem[0])) + if f.mapped == nil { + f.mapped = map[uintptr][]byte{} + } + f.mapped[p] = mem + *(*int32)(outQ) = int32(q) + return p + } +} + +// forecastCalls returns the recorded Forecast calls, without the load records. +func (f *fakeLib) forecastCalls() []fakeCall { + f.mu.Lock() + defer f.mu.Unlock() + var out []fakeCall + for _, c := range f.calls { + if c.B > 0 { + out = append(out, c) + } + } + return out +} diff --git a/backend/go/timestral-cpp/gotimestralcpp.go b/backend/go/timestral-cpp/gotimestralcpp.go new file mode 100644 index 000000000..e25e3b075 --- /dev/null +++ b/backend/go/timestral-cpp/gotimestralcpp.go @@ -0,0 +1,218 @@ +package main + +import ( + "fmt" + "os" + "path/filepath" + "runtime" + "sync" + "unsafe" + + "github.com/mudler/LocalAI/pkg/grpc/base" + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + "github.com/mudler/xlog" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// TimestralCpp implements the Forecast RPC of the Backend gRPC service over +// libtimestral.so. The C handle is not reentrant: the server serializes calls +// through base.SingleThread, and mu repeats that guard for Forecast and Free so +// the handle is safe even when the type is used without the server. +type TimestralCpp struct { + base.SingleThread + + mu sync.Mutex + ctxPtr uintptr + opts loadOptions + levels []float64 // quantile levels of the loaded model, ascending +} + +func (t *TimestralCpp) Load(opts *pb.ModelOptions) error { + model := opts.ModelFile + if model == "" { + model = opts.ModelPath + } + if !filepath.IsAbs(model) && opts.ModelPath != "" && model != opts.ModelPath { + model = filepath.Join(opts.ModelPath, model) + } + if model == "" { + return fmt.Errorf("timestral-cpp: ModelFile is required") + } + if st, err := os.Stat(model); err != nil || st.IsDir() { + return fmt.Errorf("timestral-cpp: model file %q is not readable", model) + } + + if v := CppAbiVersion(); v != abiVersionRequired { + return fmt.Errorf("timestral-cpp: libtimestral ABI version %d, this backend needs %d", v, abiVersionRequired) + } + + threads := opts.Threads + if threads <= 0 { + threads = defaultThreads + } + + t.mu.Lock() + defer t.mu.Unlock() + t.freeLocked() + + // last_error(NULL) is thread-local, and a goroutine may move to another OS + // thread between two calls into C. Pin the thread across the load and the + // error read. + runtime.LockOSThread() + ctx := CppLoad(model, threads) + if ctx == 0 { + msg := CppLastError(0) + runtime.UnlockOSThread() + return fmt.Errorf("timestral-cpp: load %q failed: %s", filepath.Base(model), msg) + } + runtime.UnlockOSThread() + + n := int(CppNumQuantiles(ctx)) + if n <= 0 { + msg := CppLastError(ctx) + CppFree(ctx) + return fmt.Errorf("timestral-cpp: model reports no quantile levels: %s", msg) + } + levels := make([]float64, n) + for i := range levels { + levels[i] = CppQuantileLevel(ctx, int32(i)) + } + + t.ctxPtr = ctx + t.levels = levels + t.opts = parseOptions(opts.Options) + xlog.Info("timestral-cpp: model loaded", "model", filepath.Base(model), "threads", threads, + "quantile_levels", len(levels), "abi", abiVersionRequired) + return nil +} + +func (t *TimestralCpp) Free() error { + t.mu.Lock() + defer t.mu.Unlock() + t.freeLocked() + return nil +} + +func (t *TimestralCpp) freeLocked() { + if t.ctxPtr != 0 { + CppFree(t.ctxPtr) + t.ctxPtr = 0 + } + t.levels = nil +} + +// Forecast groups the series by shape, runs one ABI call per group and returns +// the results in the order the series were sent. +func (t *TimestralCpp) Forecast(req *pb.ForecastRequest) (pb.ForecastResponse, error) { + t.mu.Lock() + defer t.mu.Unlock() + + if t.ctxPtr == 0 { + return pb.ForecastResponse{}, status.Error(codes.FailedPrecondition, "timestral-cpp: no model loaded") + } + if req.GetPredictionLength() < 1 { + return pb.ForecastResponse{}, invalidArgument("prediction_length must be at least 1") + } + if req.GetMaxContext() < 0 { + return pb.ForecastResponse{}, invalidArgument("max_context must not be negative") + } + h := req.GetPredictionLength() + if err := validateSeries(req.GetSeries(), h, t.opts); err != nil { + return pb.ForecastResponse{}, err + } + sel, levels, err := selectQuantiles(req.GetQuantiles(), t.opts.defaultQuantiles, t.levels) + if err != nil { + return pb.ForecastResponse{}, err + } + + maxContext := req.GetMaxContext() + if maxContext == 0 { + maxContext = t.opts.maxContext + } + + results := make([]*pb.ForecastSeriesResult, len(req.GetSeries())) + for _, g := range planGroups(req.GetSeries()) { + out, err := t.forecastGroup(req, g, h, maxContext) + if err != nil { + return pb.ForecastResponse{}, err + } + for i, r := range reshape(out, req.GetSeries(), g, int(h), len(t.levels), sel) { + results[i] = r + } + } + + return pb.ForecastResponse{PredictionLength: h, QuantileLevels: levels, Series: results}, nil +} + +// flagValue maps an optional bool to the ABI flag: -1 model default, 0 or 1. +func flagValue(b *bool) int32 { + switch { + case b == nil: + return -1 + case *b: + return 1 + default: + return 0 + } +} + +func floatPtr(s []float32) unsafe.Pointer { + if len(s) == 0 { + return nil + } + return unsafe.Pointer(&s[0]) +} + +func doublePtr(s []float64) unsafe.Pointer { + if len(s) == 0 { + return nil + } + return unsafe.Pointer(&s[0]) +} + +// forecastGroup runs one ABI call and returns a Go copy of the [B,V,H,Q] +// output. The C buffer is freed before it returns. +func (t *TimestralCpp) forecastGroup(req *pb.ForecastRequest, g group, h, maxContext int32) ([]float32, error) { + b := buildBatch(req.GetSeries(), g) + cr := forecastRequest{ + context: floatPtr(b.context), + B: int32(len(g.idx)), + V: g.shape.V, + T: g.shape.T, + predictionLen: h, + pastCov: floatPtr(b.pastCov), + cPast: g.shape.cPast, + knownCov: floatPtr(b.knownCov), + cKnown: g.shape.cKnown, + catPast: doublePtr(b.catPast), + ccPast: g.shape.ccPast, + catKnown: doublePtr(b.catKnown), + ccKnown: g.shape.ccKnown, + symmetricAvg: flagValue(req.SymmetricAveraging), + clipNonneg: flagValue(req.ClipNonnegative), + maxContext: maxContext, + } + + var q int32 + ptr := CppForecast(t.ctxPtr, unsafe.Pointer(&cr), unsafe.Pointer(&q)) + // The buffers behind cr must outlive the call: they are only reachable from + // the C side while it runs. + runtime.KeepAlive(b) + runtime.KeepAlive(&cr) + if ptr == 0 { + // The ABI rejects what it cannot serve (for example covariates the model + // has no input for), so the message is about the request. + return nil, invalidArgument("forecast failed: %s", CppLastError(t.ctxPtr)) + } + defer CppFreeBuffer(ptr) + + if int(q) != len(t.levels) { + return nil, status.Errorf(codes.Internal, "timestral-cpp: library returned %d quantiles, model reports %d", q, len(t.levels)) + } + n := len(g.idx) * int(g.shape.V) * int(h) * int(q) + src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), n) //nolint:govet // C-owned malloc'd buffer, not Go-GC memory; freed by CppFreeBuffer above + out := make([]float32, n) + copy(out, src) + return out, nil +} diff --git a/backend/go/timestral-cpp/gotimestralcpp_test.go b/backend/go/timestral-cpp/gotimestralcpp_test.go new file mode 100644 index 000000000..ce5883e6c --- /dev/null +++ b/backend/go/timestral-cpp/gotimestralcpp_test.go @@ -0,0 +1,305 @@ +package main + +import ( + "math" + "os" + "path/filepath" + "sync" + "time" + + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +var modelLevels = []float64{0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99} + +// series builds a series of v variates and t steps. Variate j starts at +// first+j, so a result can be traced back to its input. +func series(id string, first float32, v, t int) *pb.ForecastSeries { + s := &pb.ForecastSeries{Id: id, NumVariates: int32(v), Length: int32(t)} + for j := 0; j < v; j++ { + for i := 0; i < t; i++ { + s.Target = append(s.Target, first+float32(j)) + } + } + return s +} + +func boolPtr(b bool) *bool { return &b } + +func errCode(err error) codes.Code { + st, ok := status.FromError(err) + ExpectWithOffset(1, ok).To(BeTrue(), "not a gRPC status error: %v", err) + return st.Code() +} + +var _ = Describe("TimestralCpp", func() { + var ( + fake *fakeLib + model string + tc *TimestralCpp + ) + + loadOpts := func(options ...string) *pb.ModelOptions { + return &pb.ModelOptions{ModelFile: model, Threads: 3, Options: options} + } + + BeforeEach(func() { + fake = &fakeLib{levels: modelLevels, abiVersion: 1} + fake.install() + model = filepath.Join(GinkgoT().TempDir(), "timestral-test.gguf") + Expect(os.WriteFile(model, []byte("gguf"), 0o600)).To(Succeed()) + tc = &TimestralCpp{} + }) + + Describe("Load", func() { + It("loads the model, reads the quantile levels and passes the thread budget", func() { + Expect(tc.Load(loadOpts())).To(Succeed()) + Expect(tc.levels).To(Equal(modelLevels)) + Expect(fake.calls[0].threads).To(Equal(int32(3))) + }) + + It("uses 4 threads when LocalAI gives no budget", func() { + Expect(tc.Load(&pb.ModelOptions{ModelFile: model})).To(Succeed()) + Expect(fake.calls[0].threads).To(Equal(int32(4))) + }) + + It("refuses an empty path and a missing file", func() { + Expect(tc.Load(&pb.ModelOptions{})).To(MatchError(ContainSubstring("ModelFile is required"))) + Expect(tc.Load(&pb.ModelOptions{ModelFile: model + ".missing"})).To(MatchError(ContainSubstring("not readable"))) + Expect(fake.loads).To(BeZero()) + }) + + It("refuses a library with another ABI version", func() { + fake.abiVersion = 2 + Expect(tc.Load(loadOpts())).To(MatchError(ContainSubstring("ABI version 2"))) + Expect(fake.loads).To(BeZero()) + }) + + It("returns the library's own error when the load fails", func() { + fake.loadFails = true + Expect(tc.Load(loadOpts())).To(MatchError(ContainSubstring("cannot read model"))) + }) + + It("frees the previous handle on a second load", func() { + Expect(tc.Load(loadOpts())).To(Succeed()) + Expect(tc.Load(loadOpts())).To(Succeed()) + Expect(fake.frees).To(Equal(1)) + }) + }) + + Describe("Free", func() { + It("frees the handle once and makes Forecast fail cleanly", func() { + Expect(tc.Load(loadOpts())).To(Succeed()) + Expect(tc.Free()).To(Succeed()) + Expect(tc.Free()).To(Succeed()) + Expect(fake.frees).To(Equal(1)) + _, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}}) + Expect(errCode(err)).To(Equal(codes.FailedPrecondition)) + }) + }) + + Describe("Forecast", func() { + BeforeEach(func() { + Expect(tc.Load(loadOpts())).To(Succeed()) + }) + + It("returns the default quantiles 0.1, 0.5 and 0.9 when none are requested", func() { + res, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 2, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}}) + Expect(err).ToNot(HaveOccurred()) + Expect(res.QuantileLevels).To(Equal([]float64{0.1, 0.5, 0.9})) + Expect(res.PredictionLength).To(Equal(int32(2))) + }) + + It("honours the default_quantiles option", func() { + Expect(tc.Load(loadOpts("default_quantiles:0.25,0.75"))).To(Succeed()) + res, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}}) + Expect(err).ToNot(HaveOccurred()) + Expect(res.QuantileLevels).To(Equal([]float64{0.25, 0.75})) + }) + + It("reshapes the ABI [B,V,H,Q] output to [V][Q][H] for the chosen quantiles", func() { + req := &pb.ForecastRequest{ + PredictionLength: 3, + Quantiles: []float64{0.9, 0.1, 0.9}, // unsorted and repeated on purpose + Series: []*pb.ForecastSeries{series("a", 2, 2, 5)}, + } + res, err := tc.Forecast(req) + Expect(err).ToNot(HaveOccurred()) + Expect(res.QuantileLevels).To(Equal([]float64{0.1, 0.9}), "ascending, no duplicates") + sr := res.Series[0] + Expect(sr.NumVariates).To(Equal(int32(2))) + var want []float32 + for v := 0; v < 2; v++ { + for _, q := range []int{1, 5} { // indexes of 0.1 and 0.9 in the model levels + for h := 0; h < 3; h++ { + want = append(want, value(float32(2+v), h, q)) + } + } + } + Expect(sr.Values).To(Equal(want)) + }) + + It("accepts a requested level within 1e-6 and rejects one further away", func() { + _, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 1, Quantiles: []float64{0.5 + 5e-7}, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}}) + Expect(err).ToNot(HaveOccurred()) + + _, err = tc.Forecast(&pb.ForecastRequest{PredictionLength: 1, Quantiles: []float64{0.4}, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}}) + Expect(errCode(err)).To(Equal(codes.InvalidArgument)) + Expect(err.Error()).To(ContainSubstring("valid levels: 0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99")) + }) + + It("groups series by shape, runs one call per group and keeps the request order", func() { + req := &pb.ForecastRequest{ + PredictionLength: 2, + Quantiles: []float64{0.5}, + Series: []*pb.ForecastSeries{ + series("long-1", 1, 1, 6), + series("short", 2, 1, 4), + series("long-2", 3, 1, 6), + series("wide", 4, 2, 6), + }, + } + res, err := tc.Forecast(req) + Expect(err).ToNot(HaveOccurred()) + + calls := fake.forecastCalls() + Expect(calls).To(HaveLen(3), "shapes: 1x6, 1x4, 2x6") + Expect(calls[0].B).To(Equal(int32(2))) + Expect(calls[0].context).To(Equal([]float32{1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 3}), "long-1 then long-2") + Expect(calls[1].B).To(Equal(int32(1))) + Expect(calls[1].T).To(Equal(int32(4))) + Expect(calls[2].V).To(Equal(int32(2))) + + Expect(res.Series).To(HaveLen(4)) + for i, want := range []struct { + id string + first float32 + v int32 + }{{"long-1", 1, 1}, {"short", 2, 1}, {"long-2", 3, 1}, {"wide", 4, 2}} { + sr := res.Series[i] + Expect(sr.Id).To(Equal(want.id)) + Expect(sr.NumVariates).To(Equal(want.v)) + Expect(sr.Values[0]).To(Equal(value(want.first, 0, 3)), "series %s carries its own input", want.id) + Expect(sr.Values).To(HaveLen(int(want.v) * 2)) + } + }) + + It("passes covariates through to the library per series", func() { + s1 := series("a", 1, 1, 3) + s1.NumKnownCovariates = 1 + s1.KnownCovariates = []float32{1, 2, 3, 4, 5} // T+H = 3+2 + s1.NumPastCategorical = 1 + s1.PastCategorical = []float64{7, math.NaN(), 9} + s2 := series("b", 2, 1, 3) + s2.NumKnownCovariates = 1 + s2.KnownCovariates = []float32{6, 7, 8, 9, 10} + s2.NumPastCategorical = 1 + s2.PastCategorical = []float64{1, 2, 3} + _, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 2, Series: []*pb.ForecastSeries{s1, s2}}) + Expect(err).ToNot(HaveOccurred()) + calls := fake.forecastCalls() + Expect(calls).To(HaveLen(1)) + Expect(calls[0].cKnown).To(Equal(int32(1))) + Expect(calls[0].knownCov).To(Equal([]float32{1, 2, 3, 4, 5, 6, 7, 8, 9, 10})) + Expect(calls[0].ccPast).To(Equal(int32(1))) + Expect(calls[0].catPast[0]).To(Equal(7.0)) + Expect(math.IsNaN(calls[0].catPast[1])).To(BeTrue()) + }) + + It("sends -1 for absent averaging and clipping flags and the value when set", func() { + base := func() *pb.ForecastRequest { + return &pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}} + } + _, err := tc.Forecast(base()) + Expect(err).ToNot(HaveOccurred()) + r := base() + r.SymmetricAveraging, r.ClipNonnegative = boolPtr(true), boolPtr(false) + r.MaxContext = 128 + _, err = tc.Forecast(r) + Expect(err).ToNot(HaveOccurred()) + + calls := fake.forecastCalls() + Expect(calls[0].sym).To(Equal(int32(-1))) + Expect(calls[0].clip).To(Equal(int32(-1))) + Expect(calls[0].maxContext).To(BeZero()) + Expect(calls[1].sym).To(Equal(int32(1))) + Expect(calls[1].clip).To(Equal(int32(0))) + Expect(calls[1].maxContext).To(Equal(int32(128))) + }) + + It("falls back to the max_context option", func() { + Expect(tc.Load(loadOpts("max_context:64"))).To(Succeed()) + _, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}}) + Expect(err).ToNot(HaveOccurred()) + Expect(fake.forecastCalls()[0].maxContext).To(Equal(int32(64))) + }) + + It("frees the library buffer exactly once per call", func() { + req := &pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 4), series("b", 1, 1, 5)}} + _, err := tc.Forecast(req) + Expect(err).ToNot(HaveOccurred()) + Expect(fake.freeBuffers).To(Equal(2), "one buffer per shape group") + }) + + It("reports the library error as InvalidArgument and frees nothing", func() { + fake.forecastFail = true + _, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}}) + Expect(errCode(err)).To(Equal(codes.InvalidArgument)) + Expect(err.Error()).To(ContainSubstring("invalid request")) + Expect(fake.freeBuffers).To(BeZero()) + }) + + DescribeTable("rejects a malformed request before calling the library", + func(mutate func(*pb.ForecastRequest), fragment string) { + req := &pb.ForecastRequest{PredictionLength: 2, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}} + mutate(req) + _, err := tc.Forecast(req) + Expect(errCode(err)).To(Equal(codes.InvalidArgument)) + Expect(err.Error()).To(ContainSubstring(fragment)) + Expect(fake.forecastCalls()).To(BeEmpty()) + }, + Entry("no series", func(r *pb.ForecastRequest) { r.Series = nil }, "no series"), + Entry("a zero horizon", func(r *pb.ForecastRequest) { r.PredictionLength = 0 }, "prediction_length"), + Entry("a negative max_context", func(r *pb.ForecastRequest) { r.MaxContext = -1 }, "max_context"), + Entry("a target of the wrong length", func(r *pb.ForecastRequest) { r.Series[0].Target = r.Series[0].Target[:3] }, "target holds 3 values, expected 4"), + Entry("known covariates that stop at the context end", func(r *pb.ForecastRequest) { + r.Series[0].NumKnownCovariates = 1 + r.Series[0].KnownCovariates = []float32{1, 2, 3, 4} + }, "known_covariates holds 4 values, expected 6"), + Entry("zero variates", func(r *pb.ForecastRequest) { r.Series[0].NumVariates = 0 }, "at least one variate"), + Entry("a negative covariate count", func(r *pb.ForecastRequest) { r.Series[0].NumPastCovariates = -1 }, "negative"), + ) + + It("enforces the per-request series and point limits", func() { + Expect(tc.Load(loadOpts("max_series:2", "max_total_points:10"))).To(Succeed()) + three := &pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 2), series("b", 1, 1, 2), series("c", 1, 1, 2)}} + _, err := tc.Forecast(three) + Expect(err).To(MatchError(ContainSubstring("exceed the limit of 2"))) + big := &pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 6), series("b", 1, 1, 6)}} + _, err = tc.Forecast(big) + Expect(err).To(MatchError(ContainSubstring("max_total_points"))) + Expect(fake.forecastCalls()).To(BeEmpty()) + }) + + It("never runs two library calls at once", func() { + fake.delay = 20 * time.Millisecond + var wg sync.WaitGroup + for i := 0; i < 6; i++ { + wg.Add(1) + go func() { + defer GinkgoRecover() + defer wg.Done() + _, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 1, Series: []*pb.ForecastSeries{series("a", 1, 1, 4)}}) + Expect(err).ToNot(HaveOccurred()) + }() + } + wg.Wait() + Expect(fake.maxInFlight.Load()).To(Equal(int32(1))) + Expect(fake.forecastCalls()).To(HaveLen(6)) + }) + }) +}) diff --git a/backend/go/timestral-cpp/grpc_integration_test.go b/backend/go/timestral-cpp/grpc_integration_test.go new file mode 100644 index 000000000..f227f3432 --- /dev/null +++ b/backend/go/timestral-cpp/grpc_integration_test.go @@ -0,0 +1,106 @@ +package main + +// Spawns the built timestral-cpp-grpc binary and calls it over gRPC with the +// 5M F32 model. This covers what the in-process specs cannot: the server +// registration, the model identity check, the lock around the RPC and the +// mapping of a bad request to InvalidArgument. It skips unless the binary, the +// library and TIMESTRAL_BACKEND_TEST_MODEL are all present. + +import ( + "context" + "fmt" + "net" + "os" + "os/exec" + "path/filepath" + "time" + + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/status" +) + +var _ = Describe("timestral-cpp gRPC server", func() { + It("loads a model and answers Forecast", func() { + modelPath := os.Getenv("TIMESTRAL_BACKEND_TEST_MODEL") + if modelPath == "" { + Skip("TIMESTRAL_BACKEND_TEST_MODEL is not set") + } + binary, err := filepath.Abs("./timestral-cpp-grpc") + Expect(err).ToNot(HaveOccurred()) + if _, err := os.Stat(binary); err != nil { + Skip("backend binary not built: run `make timestral-cpp-grpc` first") + } + lib := os.Getenv("TIMESTRAL_LIBRARY") + if lib == "" { + lib, _ = filepath.Abs("./libtimestral.so") + } + if _, err := os.Stat(lib); err != nil { + Skip("libtimestral.so not found: " + lib) + } + + l, err := net.Listen("tcp", "127.0.0.1:0") + Expect(err).ToNot(HaveOccurred()) + addr := l.Addr().String() + Expect(l.Close()).To(Succeed()) + + cmd := exec.Command(binary, "--addr", addr) + cmd.Env = append(os.Environ(), "TIMESTRAL_LIBRARY="+lib) + cmd.Stdout, cmd.Stderr = os.Stderr, os.Stderr + Expect(cmd.Start()).To(Succeed()) + defer func() { + _ = cmd.Process.Kill() + _, _ = cmd.Process.Wait() + }() + + deadline := time.Now().Add(10 * time.Second) + for { + c, err := net.DialTimeout("tcp", addr, 200*time.Millisecond) + if err == nil { + _ = c.Close() + break + } + if time.Now().After(deadline) { + Fail(fmt.Sprintf("backend did not listen on %s within 10s", addr)) + } + time.Sleep(100 * time.Millisecond) + } + + conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials())) + Expect(err).ToNot(HaveOccurred()) + defer func() { _ = conn.Close() }() + client := pb.NewBackendClient(conn) + + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + + loaded, err := client.LoadModel(ctx, &pb.ModelOptions{Model: "timestral-5m", ModelFile: modelPath, Threads: 2}) + Expect(err).ToNot(HaveOccurred()) + Expect(loaded.Success).To(BeTrue()) + + res, err := client.Forecast(ctx, &pb.ForecastRequest{ + ModelIdentity: "timestral-5m", + PredictionLength: 8, + Series: []*pb.ForecastSeries{waveSeries("a", 96, 12, 0), waveSeries("b", 48, 6, 2)}, + }) + Expect(err).ToNot(HaveOccurred()) + Expect(res.QuantileLevels).To(Equal([]float64{0.1, 0.5, 0.9})) + Expect(res.Series).To(HaveLen(2)) + Expect(res.Series[0].Id).To(Equal("a")) + Expect(res.Series[1].Id).To(Equal("b")) + Expect(res.Series[0].Values).To(HaveLen(1 * 3 * 8)) + + _, err = client.Forecast(ctx, &pb.ForecastRequest{PredictionLength: 8, Quantiles: []float64{0.42}, Series: []*pb.ForecastSeries{waveSeries("a", 96, 12, 0)}}) + st, ok := status.FromError(err) + Expect(ok).To(BeTrue()) + Expect(st.Code()).To(Equal(codes.InvalidArgument)) + Expect(st.Message()).To(ContainSubstring("valid levels")) + + _, err = client.Forecast(ctx, &pb.ForecastRequest{ModelIdentity: "another-model", PredictionLength: 8, Series: []*pb.ForecastSeries{waveSeries("a", 96, 12, 0)}}) + Expect(err).To(HaveOccurred(), "a request for another model must be refused") + }) +}) diff --git a/backend/go/timestral-cpp/integration_test.go b/backend/go/timestral-cpp/integration_test.go new file mode 100644 index 000000000..8075924dd --- /dev/null +++ b/backend/go/timestral-cpp/integration_test.go @@ -0,0 +1,218 @@ +package main + +// Integration specs against the real libtimestral.so and the 5M F32 model. +// They need TIMESTRAL_BACKEND_TEST_MODEL (path of the GGUF) and a loadable +// library (TIMESTRAL_LIBRARY, or libtimestral.so on LD_LIBRARY_PATH). When +// either is missing the specs skip; they never pass silently. test.sh fetches +// the model and checks its SHA-256. +// +// The weights are real, so the specs check behaviour and invariants (shape, +// finite values, ordered quantiles, repeatability, request order), not the +// forecast values. + +import ( + "math" + "os" + "sync" + + "github.com/ebitengine/purego" + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var ( + realLibOnce sync.Once + realLib uintptr + realLibErr error +) + +func openRealLib() (uintptr, error) { + realLibOnce.Do(func() { + name := os.Getenv("TIMESTRAL_LIBRARY") + if name == "" { + name = "libtimestral.so" + } + realLib, realLibErr = purego.Dlopen(name, purego.RTLD_NOW|purego.RTLD_GLOBAL) + }) + return realLib, realLibErr +} + +// wave is a smooth, strictly positive series with a given period. +func wave(n int, period float64, offset float64) []float32 { + out := make([]float32, n) + for i := range out { + out[i] = float32(10 + offset + 3*math.Sin(2*math.Pi*float64(i)/period)) + } + return out +} + +func waveSeries(id string, n int, period, offset float64) *pb.ForecastSeries { + return &pb.ForecastSeries{Id: id, NumVariates: 1, Length: int32(n), Target: wave(n, period, offset)} +} + +var _ = Describe("timestral.cpp integration", Ordered, func() { + var tc *TimestralCpp + + BeforeAll(func() { + modelPath := os.Getenv("TIMESTRAL_BACKEND_TEST_MODEL") + if modelPath == "" { + Skip("TIMESTRAL_BACKEND_TEST_MODEL is not set; run test.sh to fetch the 5M F32 model") + } + if _, err := os.Stat(modelPath); err != nil { + Skip("model file not found: " + modelPath) + } + lib, err := openRealLib() + if err != nil { + Skip("libtimestral.so is not loadable: " + err.Error()) + } + registerLibFuncs(lib) + + tc = &TimestralCpp{} + Expect(tc.Load(&pb.ModelOptions{ModelFile: modelPath, Threads: 2})).To(Succeed()) + }) + + AfterAll(func() { + if tc != nil { + Expect(tc.Free()).To(Succeed()) + } + }) + + forecast := func(req *pb.ForecastRequest) *pb.ForecastResponse { + res, err := tc.Forecast(req) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) + return &res + } + + expectSane := func(res *pb.ForecastResponse, h int) { + q := len(res.QuantileLevels) + for _, sr := range res.Series { + ExpectWithOffset(1, sr.Values).To(HaveLen(int(sr.NumVariates) * q * h)) + for v := 0; v < int(sr.NumVariates); v++ { + for step := 0; step < h; step++ { + prev := math.Inf(-1) + for qi := 0; qi < q; qi++ { + x := float64(sr.Values[(v*q+qi)*h+step]) + ExpectWithOffset(1, math.IsNaN(x) || math.IsInf(x, 0)).To(BeFalse(), "finite") + ExpectWithOffset(1, x).To(BeNumerically(">=", prev-1e-4), "quantiles do not cross at step %d", step) + prev = x + } + } + } + } + } + + It("exposes ascending quantile levels that include 0.1, 0.5 and 0.9", func() { + Expect(len(tc.levels)).To(BeNumerically(">", 3)) + for i := 1; i < len(tc.levels); i++ { + Expect(tc.levels[i]).To(BeNumerically(">", tc.levels[i-1])) + } + _, levels, err := selectQuantiles(nil, defaultQuantiles, tc.levels) + Expect(err).ToNot(HaveOccurred()) + Expect(levels).To(Equal([]float64{0.1, 0.5, 0.9})) + }) + + It("forecasts one univariate series with the default quantiles", func() { + res := forecast(&pb.ForecastRequest{PredictionLength: 12, Series: []*pb.ForecastSeries{waveSeries("a", 96, 12, 0)}}) + Expect(res.PredictionLength).To(Equal(int32(12))) + Expect(res.QuantileLevels).To(Equal([]float64{0.1, 0.5, 0.9})) + Expect(res.Series).To(HaveLen(1)) + expectSane(res, 12) + }) + + It("treats NaN in the target as missing and still returns finite values", func() { + s := waveSeries("gap", 96, 12, 0) + s.Target[40], s.Target[41] = float32(math.NaN()), float32(math.NaN()) + res := forecast(&pb.ForecastRequest{PredictionLength: 6, Series: []*pb.ForecastSeries{s}}) + expectSane(res, 6) + }) + + It("forecasts a two-variate series", func() { + s := &pb.ForecastSeries{Id: "mv", NumVariates: 2, Length: 64} + s.Target = append(wave(64, 8, 0), wave(64, 16, 5)...) + res := forecast(&pb.ForecastRequest{PredictionLength: 5, Series: []*pb.ForecastSeries{s}}) + Expect(res.Series[0].NumVariates).To(Equal(int32(2))) + expectSane(res, 5) + }) + + It("is repeatable: the same request gives byte-identical output", func() { + req := func() *pb.ForecastRequest { + return &pb.ForecastRequest{PredictionLength: 8, Series: []*pb.ForecastSeries{waveSeries("a", 80, 10, 0)}} + } + Expect(forecast(req()).Series[0].Values).To(Equal(forecast(req()).Series[0].Values)) + }) + + It("keeps the request order when series differ in length", func() { + long := waveSeries("long", 96, 12, 0) + short := waveSeries("short", 40, 8, 3) + longAgain := waveSeries("long-2", 96, 12, 4) + + alone := func(s *pb.ForecastSeries) []float32 { + return forecast(&pb.ForecastRequest{PredictionLength: 6, Series: []*pb.ForecastSeries{s}}).Series[0].Values + } + wantLong, wantShort, wantLong2 := alone(long), alone(short), alone(longAgain) + + res := forecast(&pb.ForecastRequest{PredictionLength: 6, Series: []*pb.ForecastSeries{long, short, longAgain}}) + Expect(res.Series).To(HaveLen(3)) + Expect([]string{res.Series[0].Id, res.Series[1].Id, res.Series[2].Id}).To(Equal([]string{"long", "short", "long-2"})) + Expect(res.Series[1].Values).To(Equal(wantShort), "a series in a mixed request gives what it gives alone") + Expect(res.Series[0].Values).To(HaveLen(len(wantLong))) + Expect(res.Series[2].Values).To(HaveLen(len(wantLong2))) + expectSane(res, 6) + }) + + It("forecasts with past and known covariates and categorical codes", func() { + const n, h = 64, 6 + s := waveSeries("cov", n, 8, 0) + s.NumPastCovariates = 1 + s.PastCovariates = wave(n, 8, 0) + s.NumKnownCovariates = 1 + s.KnownCovariates = wave(n+h, 8, 0) + s.NumKnownCategorical = 1 + for i := 0; i < n+h; i++ { + s.KnownCategorical = append(s.KnownCategorical, float64(i%3)) + } + s.NumPastCategorical = 1 + for i := 0; i < n; i++ { + s.PastCategorical = append(s.PastCategorical, float64(i%3)) + } + res := forecast(&pb.ForecastRequest{PredictionLength: h, Series: []*pb.ForecastSeries{s}}) + expectSane(res, h) + }) + + It("returns only the requested quantiles, ascending", func() { + res := forecast(&pb.ForecastRequest{PredictionLength: 4, Quantiles: []float64{0.9, 0.5}, Series: []*pb.ForecastSeries{waveSeries("a", 64, 8, 0)}}) + Expect(res.QuantileLevels).To(Equal([]float64{0.5, 0.9})) + expectSane(res, 4) + }) + + It("accepts a max_context shorter than the history", func() { + res := forecast(&pb.ForecastRequest{PredictionLength: 4, MaxContext: 32, Series: []*pb.ForecastSeries{waveSeries("a", 128, 8, 0)}}) + expectSane(res, 4) + }) + + It("survives parallel callers", func() { + var wg sync.WaitGroup + for i := 0; i < 8; i++ { + wg.Add(1) + go func(i int) { + defer GinkgoRecover() + defer wg.Done() + res, err := tc.Forecast(&pb.ForecastRequest{PredictionLength: 4, Series: []*pb.ForecastSeries{waveSeries("p", 48+i, 8, float64(i))}}) + Expect(err).ToNot(HaveOccurred()) + Expect(res.Series).To(HaveLen(1)) + }(i) + } + wg.Wait() + }) + + It("refuses a file that is not a timestral model and keeps the error text", func() { + bad := GinkgoT().TempDir() + "/not-a-model.gguf" + Expect(os.WriteFile(bad, []byte("this is not a gguf file"), 0o600)).To(Succeed()) + other := &TimestralCpp{} + err := other.Load(&pb.ModelOptions{ModelFile: bad}) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("load")) + Expect(err.Error()).ToNot(HaveSuffix(": "), "the library's own message is included") + }) +}) diff --git a/backend/go/timestral-cpp/main.go b/backend/go/timestral-cpp/main.go new file mode 100644 index 000000000..040b7820b --- /dev/null +++ b/backend/go/timestral-cpp/main.go @@ -0,0 +1,40 @@ +package main + +// Started internally by LocalAI - one gRPC server per loaded model. +// +// Loads libtimestral.so via purego and registers the flat C-API entry points +// declared in timestral_capi.h. The library name can be overridden with +// TIMESTRAL_LIBRARY (the same convention as the other purego backends); the +// default looks for the .so next to this binary, which run.sh puts on +// LD_LIBRARY_PATH. +import ( + "flag" + "fmt" + "os" + + "github.com/ebitengine/purego" + grpc "github.com/mudler/LocalAI/pkg/grpc" +) + +var ( + addr = flag.String("addr", "localhost:50051", "the address to connect to") +) + +func main() { + libName := os.Getenv("TIMESTRAL_LIBRARY") + if libName == "" { + libName = "libtimestral.so" + } + + lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL) + if err != nil { + panic(fmt.Errorf("timestral-cpp: dlopen %q: %w", libName, err)) + } + registerLibFuncs(lib) + + flag.Parse() + + if err := grpc.StartServer(*addr, &TimestralCpp{}); err != nil { + panic(err) + } +} diff --git a/backend/go/timestral-cpp/options.go b/backend/go/timestral-cpp/options.go new file mode 100644 index 000000000..d0bedab35 --- /dev/null +++ b/backend/go/timestral-cpp/options.go @@ -0,0 +1,84 @@ +package main + +import ( + "strconv" + "strings" +) + +const ( + // defaultMaxSeries and defaultMaxPoints bound one request, so a single call + // cannot ask for an unbounded batch. Both can be raised in the model options. + defaultMaxSeries = 256 + defaultMaxPoints = 2_000_000 + // defaultThreads is used when LocalAI passes no thread budget. + defaultThreads = 4 +) + +// defaultQuantiles are returned when a request names none. +var defaultQuantiles = []float64{0.1, 0.5, 0.9} + +// loadOptions holds the parsed model-level options. +type loadOptions struct { + defaultQuantiles []float64 + maxContext int32 + maxSeries int + maxPoints int +} + +func splitOption(o string) (key, value string, ok bool) { + i := strings.Index(o, ":") + if i < 0 { + return "", "", false + } + return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true +} + +// parseOptions reads the backend "key:value" option list. Unknown keys and +// malformed values are ignored and keep the default. Keys: +// +// default_quantiles comma list of levels, default 0.1,0.5,0.9 +// max_context history cap used when a request sets none, 0 = model default +// max_series most series in one request +// max_total_points most input values in one request +func parseOptions(opts []string) loadOptions { + o := loadOptions{ + defaultQuantiles: append([]float64(nil), defaultQuantiles...), + maxSeries: defaultMaxSeries, + maxPoints: defaultMaxPoints, + } + for _, oo := range opts { + key, value, ok := splitOption(oo) + if !ok { + continue + } + switch key { + case "default_quantiles": + var levels []float64 + valid := true + for _, part := range strings.Split(value, ",") { + f, err := strconv.ParseFloat(strings.TrimSpace(part), 64) + if err != nil || f <= 0 || f >= 1 { + valid = false + break + } + levels = append(levels, f) + } + if valid && len(levels) > 0 { + o.defaultQuantiles = levels + } + case "max_context": + if n, err := strconv.ParseInt(value, 10, 32); err == nil && n >= 0 { + o.maxContext = int32(n) + } + case "max_series": + if n, err := strconv.Atoi(value); err == nil && n > 0 { + o.maxSeries = n + } + case "max_total_points": + if n, err := strconv.Atoi(value); err == nil && n > 0 { + o.maxPoints = n + } + } + } + return o +} diff --git a/backend/go/timestral-cpp/options_test.go b/backend/go/timestral-cpp/options_test.go new file mode 100644 index 000000000..a0b1b2642 --- /dev/null +++ b/backend/go/timestral-cpp/options_test.go @@ -0,0 +1,39 @@ +package main + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("parseOptions", func() { + It("defaults to quantiles 0.1, 0.5, 0.9 and the request limits", func() { + o := parseOptions(nil) + Expect(o.defaultQuantiles).To(Equal([]float64{0.1, 0.5, 0.9})) + Expect(o.maxContext).To(BeZero()) + Expect(o.maxSeries).To(Equal(256)) + Expect(o.maxPoints).To(Equal(2_000_000)) + }) + + It("reads every key", func() { + o := parseOptions([]string{ + "default_quantiles: 0.25, 0.75", "max_context:512", "max_series:8", "max_total_points:1000", "unknown:x", "novalue", + }) + Expect(o.defaultQuantiles).To(Equal([]float64{0.25, 0.75})) + Expect(o.maxContext).To(Equal(int32(512))) + Expect(o.maxSeries).To(Equal(8)) + Expect(o.maxPoints).To(Equal(1000)) + }) + + It("keeps the default for malformed values", func() { + o := parseOptions([]string{"default_quantiles:0.1,abc", "max_context:-1", "max_series:0", "max_total_points:x"}) + Expect(o.defaultQuantiles).To(Equal([]float64{0.1, 0.5, 0.9})) + Expect(o.maxContext).To(BeZero()) + Expect(o.maxSeries).To(Equal(256)) + Expect(o.maxPoints).To(Equal(2_000_000)) + }) + + It("rejects default levels outside (0,1)", func() { + Expect(parseOptions([]string{"default_quantiles:0,0.5"}).defaultQuantiles).To(Equal([]float64{0.1, 0.5, 0.9})) + Expect(parseOptions([]string{"default_quantiles:1"}).defaultQuantiles).To(Equal([]float64{0.1, 0.5, 0.9})) + }) +}) diff --git a/backend/go/timestral-cpp/package.sh b/backend/go/timestral-cpp/package.sh new file mode 100755 index 000000000..392318fb1 --- /dev/null +++ b/backend/go/timestral-cpp/package.sh @@ -0,0 +1,40 @@ +#!/bin/bash +# +# Bundle the timestral-cpp-grpc binary, libtimestral, the core runtime libs +# (libc, libstdc++, libgomp and ld.so) and the licence files, so the package is +# self-contained. The engine runs on the CPU only, so no GPU libraries are +# packaged. run.sh routes the (CGO_ENABLED=0) binary through lib/ld.so so the +# packaged libc is used instead of the host's. + +set -e + +CURDIR=$(dirname "$(realpath "$0")") + +mkdir -p "$CURDIR/package/lib" + +cp -avf "$CURDIR/timestral-cpp-grpc" "$CURDIR/package/" +cp -avf "$CURDIR/run.sh" "$CURDIR/package/" + +# libtimestral and any soname symlinks. On Linux this is libtimestral.so[.X.Y]; +# on macOS it is libtimestral.dylib. run.sh points TIMESTRAL_LIBRARY at lib/. +cp -avf "$CURDIR"/libtimestral.so* "$CURDIR/package/lib/" 2>/dev/null || true +cp -avf "$CURDIR"/libtimestral.dylib "$CURDIR/package/lib/" 2>/dev/null || true +if ! ls "$CURDIR"/package/lib/libtimestral.* >/dev/null 2>&1; then + echo "ERROR: libtimestral shared library not found in $CURDIR, run 'make' first" >&2 + exit 1 +fi + +# The engine is MIT. Its model code follows an Apache-2.0 reference +# implementation, and that licence asks for the notice to travel with copies. +for f in LICENSE NOTICE; do + if [ -f "$CURDIR/sources/timestral.cpp/$f" ]; then + cp -avf "$CURDIR/sources/timestral.cpp/$f" "$CURDIR/package/timestral.cpp-$f" + fi +done + +# Copy the core runtime libs libtimestral links against, plus the matching +# dynamic loader as lib/ld.so. +source "$CURDIR/../../../scripts/build/package-system-libs.sh" "$CURDIR/package/lib" "" + +echo "Packaging completed successfully" +ls -liah "$CURDIR/package/" "$CURDIR/package/lib/" diff --git a/backend/go/timestral-cpp/plan.go b/backend/go/timestral-cpp/plan.go new file mode 100644 index 000000000..b9e3b6b52 --- /dev/null +++ b/backend/go/timestral-cpp/plan.go @@ -0,0 +1,182 @@ +package main + +import ( + "fmt" + "math" + "sort" + "strings" + + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// quantileTolerance is how far a requested level may be from a model level. +// The model returns the levels it was trained on and nothing in between. +const quantileTolerance = 1e-6 + +func invalidArgument(format string, args ...any) error { + return status.Errorf(codes.InvalidArgument, "timestral-cpp: "+format, args...) +} + +// shape is everything one ABI call needs equal across its series: the ABI takes +// one [B,V,T] block with no padding fields. +type shape struct { + V, T int32 + cPast, cKnown int32 + ccPast, ccKnown int32 +} + +func shapeOf(s *pb.ForecastSeries) shape { + return shape{ + V: s.GetNumVariates(), T: s.GetLength(), + cPast: s.GetNumPastCovariates(), cKnown: s.GetNumKnownCovariates(), + ccPast: s.GetNumPastCategorical(), ccKnown: s.GetNumKnownCategorical(), + } +} + +// group is a run of series that share a shape. idx holds their positions in the +// request, so results can go back to the order the caller sent. +type group struct { + shape shape + idx []int +} + +// validateSeries checks counts against array lengths for every series and the +// request limits. The HTTP layer checks the same things, but a gRPC caller may +// not, and the ABI reads caller buffers without a length field. +func validateSeries(series []*pb.ForecastSeries, h int32, opts loadOptions) error { + if len(series) == 0 { + return invalidArgument("no series") + } + if len(series) > opts.maxSeries { + return invalidArgument("%d series exceed the limit of %d (option max_series)", len(series), opts.maxSeries) + } + total := 0 + for i, s := range series { + sh := shapeOf(s) + if sh.V < 1 || sh.T < 1 { + return invalidArgument("series %d: need at least one variate and one step", i) + } + if sh.cPast < 0 || sh.cKnown < 0 || sh.ccPast < 0 || sh.ccKnown < 0 { + return invalidArgument("series %d: negative covariate count", i) + } + th := int64(sh.T) + int64(h) + for _, c := range []struct { + name string + got, want int64 + }{ + {"target", int64(len(s.GetTarget())), int64(sh.V) * int64(sh.T)}, + {"past_covariates", int64(len(s.GetPastCovariates())), int64(sh.cPast) * int64(sh.T)}, + {"known_covariates", int64(len(s.GetKnownCovariates())), int64(sh.cKnown) * th}, + {"past_categorical", int64(len(s.GetPastCategorical())), int64(sh.ccPast) * int64(sh.T)}, + {"known_categorical", int64(len(s.GetKnownCategorical())), int64(sh.ccKnown) * th}, + } { + if c.got != c.want { + return invalidArgument("series %d: %s holds %d values, expected %d", i, c.name, c.got, c.want) + } + total += int(c.got) + } + if total > opts.maxPoints { + return invalidArgument("request exceeds %d input values (option max_total_points)", opts.maxPoints) + } + } + return nil +} + +// planGroups splits the series by shape, in order of first appearance. +func planGroups(series []*pb.ForecastSeries) []group { + var groups []group + at := map[shape]int{} + for i, s := range series { + sh := shapeOf(s) + g, ok := at[sh] + if !ok { + g = len(groups) + at[sh] = g + groups = append(groups, group{shape: sh}) + } + groups[g].idx = append(groups[g].idx, i) + } + return groups +} + +// batch holds the concatenated buffers of one group, ready for one ABI call. +// Every series keeps its own [rows][steps] layout, so plain concatenation gives +// the ABI's B-major layout. +type batch struct { + context []float32 + pastCov []float32 + knownCov []float32 + catPast []float64 + catKnown []float64 +} + +func buildBatch(series []*pb.ForecastSeries, g group) batch { + var b batch + for _, i := range g.idx { + s := series[i] + b.context = append(b.context, s.GetTarget()...) + b.pastCov = append(b.pastCov, s.GetPastCovariates()...) + b.knownCov = append(b.knownCov, s.GetKnownCovariates()...) + b.catPast = append(b.catPast, s.GetPastCategorical()...) + b.catKnown = append(b.catKnown, s.GetKnownCategorical()...) + } + return b +} + +// selectQuantiles maps the requested levels (or the defaults when none are +// requested) onto indexes of the model levels. The result is ascending and has +// no duplicates. A level the model does not have is an error that lists the +// valid ones. +func selectQuantiles(requested, defaults, modelLevels []float64) (idx []int, levels []float64, err error) { + want := requested + if len(want) == 0 { + want = defaults + } + seen := map[int]bool{} + for _, q := range want { + found := -1 + for i, l := range modelLevels { + if math.Abs(l-q) <= quantileTolerance { + found = i + break + } + } + if found < 0 { + strs := make([]string, len(modelLevels)) + for i, l := range modelLevels { + strs[i] = fmt.Sprintf("%g", l) + } + return nil, nil, invalidArgument("quantile %g is not a level of this model; valid levels: %s", q, strings.Join(strs, ", ")) + } + seen[found] = true + } + for i := range seen { + idx = append(idx, i) + } + sort.Ints(idx) + for _, i := range idx { + levels = append(levels, modelLevels[i]) + } + return idx, levels, nil +} + +// reshape turns one group's ABI output [B,V,H,Q] into one result per series +// with values laid out [V][Q'][H], where Q' are the selected quantiles. +func reshape(out []float32, series []*pb.ForecastSeries, g group, h, modelQ int, sel []int) map[int]*pb.ForecastSeriesResult { + results := make(map[int]*pb.ForecastSeriesResult, len(g.idx)) + v := int(g.shape.V) + for b, i := range g.idx { + vals := make([]float32, 0, v*len(sel)*h) + for vi := 0; vi < v; vi++ { + for _, q := range sel { + for step := 0; step < h; step++ { + vals = append(vals, out[((b*v+vi)*h+step)*modelQ+q]) + } + } + } + results[i] = &pb.ForecastSeriesResult{Id: series[i].GetId(), NumVariates: g.shape.V, Values: vals} + } + return results +} diff --git a/backend/go/timestral-cpp/run.sh b/backend/go/timestral-cpp/run.sh new file mode 100755 index 000000000..e766dd252 --- /dev/null +++ b/backend/go/timestral-cpp/run.sh @@ -0,0 +1,21 @@ +#!/bin/bash +set -e + +CURDIR=$(dirname "$(realpath "$0")") + +if [ "$(uname)" = "Darwin" ]; then + export DYLD_LIBRARY_PATH="$CURDIR/lib:$CURDIR:${DYLD_LIBRARY_PATH:-}" + export TIMESTRAL_LIBRARY="$CURDIR/lib/libtimestral.dylib" +else + export LD_LIBRARY_PATH="$CURDIR/lib:$CURDIR:${LD_LIBRARY_PATH:-}" + export TIMESTRAL_LIBRARY="$CURDIR/lib/libtimestral.so" +fi + +# If a self-contained ld.so was packaged, route through it so the packaged +# libc and libstdc++ are used instead of the host's. Linux only. +if [ -f "$CURDIR/lib/ld.so" ]; then + echo "Using lib/ld.so" + exec "$CURDIR/lib/ld.so" "$CURDIR/timestral-cpp-grpc" "$@" +fi + +exec "$CURDIR/timestral-cpp-grpc" "$@" diff --git a/backend/go/timestral-cpp/test.sh b/backend/go/timestral-cpp/test.sh new file mode 100755 index 000000000..1ed37d01f --- /dev/null +++ b/backend/go/timestral-cpp/test.sh @@ -0,0 +1,35 @@ +#!/bin/bash +set -e + +CURDIR=$(dirname "$(realpath "$0")") +cd "$CURDIR" + +echo "Running timestral-cpp backend tests..." + +# The pure-Go specs always run. The integration specs run when libtimestral.so +# is built in this directory (make libtimestral.so) or TIMESTRAL_LIBRARY names +# one. They use the 5M F32 model (22 MB) from the public GGUF repository. The +# download is checked against a fixed SHA-256, taken from the repository's +# SHA256SUMS file, so a changed or truncated file stops the run. +TIMESTRAL_MODEL_DIR="${TIMESTRAL_MODEL_DIR:-$CURDIR/test-models}" +TIMESTRAL_MODEL_FILE="timestral-5m-f32.gguf" +TIMESTRAL_MODEL_URL="${TIMESTRAL_MODEL_URL:-https://huggingface.co/mudler/timestral-cpp-gguf/resolve/main/5M/timestral-5m-f32.gguf}" +TIMESTRAL_MODEL_SHA256="c320276d8db95fb72fe5f4e043147e29142a1bf197c1b45daf6736780f6736ed" + +if [ -n "${TIMESTRAL_LIBRARY:-}" ] || ls "$CURDIR"/libtimestral.so* >/dev/null 2>&1; then + mkdir -p "$TIMESTRAL_MODEL_DIR" + MODEL="$TIMESTRAL_MODEL_DIR/$TIMESTRAL_MODEL_FILE" + if [ ! -f "$MODEL" ]; then + echo "Downloading $TIMESTRAL_MODEL_FILE (22 MB)..." + # -C - resumes a partial download. + curl -fL -C - -o "$MODEL" "$TIMESTRAL_MODEL_URL" --progress-bar + fi + echo "$TIMESTRAL_MODEL_SHA256 $MODEL" | sha256sum -c - + export TIMESTRAL_BACKEND_TEST_MODEL="$MODEL" +else + echo "libtimestral.so not found: the integration specs will skip." +fi + +LD_LIBRARY_PATH="$CURDIR:${LD_LIBRARY_PATH:-}" go test -v -count=1 -timeout 20m . + +echo "timestral-cpp tests completed." diff --git a/backend/go/timestral-cpp/timestral_suite_test.go b/backend/go/timestral-cpp/timestral_suite_test.go new file mode 100644 index 000000000..c59fa7b1e --- /dev/null +++ b/backend/go/timestral-cpp/timestral_suite_test.go @@ -0,0 +1,13 @@ +package main + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestTimestralCpp(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "timestral-cpp Backend Suite") +} diff --git a/backend/index.yaml b/backend/index.yaml index 1bf606e65..8291f8f0b 100644 --- a/backend/index.yaml +++ b/backend/index.yaml @@ -29389,3 +29389,111 @@ uri: quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-funasr mirrors: - localai/localai-backends:master-metal-darwin-arm64-funasr +- name: timestral-cpp + alias: timestral-cpp + license: mit + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + urls: + - https://github.com/mudler/timestral.cpp + tags: + - forecasting + - time-series + - CPU + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp +- name: timestral-cpp-development + alias: timestral-cpp + license: mit + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + urls: + - https://github.com/mudler/timestral.cpp + tags: + - forecasting + - time-series + - CPU + capabilities: + default: cpu-timestral-cpp-development + metal: metal-timestral-cpp-development +- name: cpu-timestral-cpp + alias: timestral-cpp + license: mit + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + urls: + - https://github.com/mudler/timestral.cpp + tags: + - forecasting + - time-series + - CPU + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + uri: quay.io/go-skynet/local-ai-backends:latest-cpu-timestral-cpp + mirrors: + - localai/localai-backends:latest-cpu-timestral-cpp +- name: cpu-timestral-cpp-development + alias: timestral-cpp + license: mit + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + urls: + - https://github.com/mudler/timestral.cpp + tags: + - forecasting + - time-series + - CPU + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + uri: quay.io/go-skynet/local-ai-backends:master-cpu-timestral-cpp + mirrors: + - localai/localai-backends:master-cpu-timestral-cpp +- name: metal-timestral-cpp + alias: timestral-cpp + license: mit + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + urls: + - https://github.com/mudler/timestral.cpp + tags: + - forecasting + - time-series + - CPU + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + uri: quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-timestral-cpp + mirrors: + - localai/localai-backends:latest-metal-darwin-arm64-timestral-cpp +- name: metal-timestral-cpp-development + alias: timestral-cpp + license: mit + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + urls: + - https://github.com/mudler/timestral.cpp + tags: + - forecasting + - time-series + - CPU + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + uri: quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-timestral-cpp + mirrors: + - localai/localai-backends:master-metal-darwin-arm64-timestral-cpp diff --git a/core/backend/forecast.go b/core/backend/forecast.go new file mode 100644 index 000000000..7fb2b97cd --- /dev/null +++ b/core/backend/forecast.go @@ -0,0 +1,80 @@ +package backend + +import ( + "context" + "fmt" + "time" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/trace" + "github.com/mudler/LocalAI/pkg/grpc/proto" + "github.com/mudler/LocalAI/pkg/model" +) + +// Forecast runs a time-series forecasting model on the series in the request +// and returns quantile trajectories for the next prediction_length steps. +func Forecast( + ctx context.Context, + in *proto.ForecastRequest, + loader *model.ModelLoader, + appConfig *config.ApplicationConfig, + modelConfig config.ModelConfig, +) (*proto.ForecastResponse, error) { + opts := ModelOptions(modelConfig, appConfig) + forecastModel, err := loader.Load(opts...) + if err != nil { + recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil) + return nil, err + } + + if forecastModel == nil { + return nil, fmt.Errorf("could not load forecast model") + } + release, err := AcquireGlobalBackendSlot() + if err != nil { + return nil, err + } + defer release() + + summary := fmt.Sprintf("%d series, horizon %d", len(in.GetSeries()), in.GetPredictionLength()) + + var startTime time.Time + var traceID string + if appConfig.EnableTracing { + trace.InitBackendTracingIfEnabled(appConfig.TracingMaxItems, appConfig.TracingMaxBodyBytes) + startTime = time.Now() + traceID = trace.BeginBackendTrace(trace.BackendTrace{Timestamp: startTime, Type: trace.BackendTraceForecast, ModelName: modelConfig.Name, Backend: modelConfig.Backend, Summary: summary}) + } + defer trace.CancelBackendTrace(traceID) + + // Stamped here for the same reason as in rerank.go: the caller builds the + // request without a ModelConfig, this function has the one that loaded. + in.ModelIdentity = modelConfig.Model + + res, err := forecastModel.Forecast(ctx, in) + + if appConfig.EnableTracing { + errStr := "" + if err != nil { + errStr = err.Error() + } + + trace.RecordBackendTrace(trace.BackendTrace{ + ID: traceID, + Timestamp: startTime, + Duration: time.Since(startTime), + Type: trace.BackendTraceForecast, + ModelName: modelConfig.Name, + Backend: modelConfig.Backend, + Summary: summary, + Error: errStr, + Data: map[string]any{ + "series": len(in.GetSeries()), + "prediction_length": in.GetPredictionLength(), + "quantiles": in.GetQuantiles(), + }, + }) + } + + return res, err +} diff --git a/core/backend/model_identity_modalities_test.go b/core/backend/model_identity_modalities_test.go index 39d7930e2..1c2f32201 100644 --- a/core/backend/model_identity_modalities_test.go +++ b/core/backend/model_identity_modalities_test.go @@ -97,6 +97,11 @@ func (r *recordingBackend) Depth(_ context.Context, in *pb.DepthRequest, _ ...gg return &pb.DepthResponse{}, nil } +func (r *recordingBackend) Forecast(_ context.Context, in *pb.ForecastRequest, _ ...ggrpc.CallOption) (*pb.ForecastResponse, error) { + r.record("Forecast", in) + return &pb.ForecastResponse{}, nil +} + func (r *recordingBackend) VAD(_ context.Context, in *pb.VADRequest, _ ...ggrpc.CallOption) (*pb.VADResponse, error) { r.record("VAD", in) return &pb.VADResponse{}, nil @@ -247,6 +252,12 @@ var _ = Describe("modality requests carry the model identity", func() { expectIdentity("Depth") }) + It("Forecast", func() { + _, err := backend.Forecast(ctx, &pb.ForecastRequest{PredictionLength: 1}, loader, appCfg, cfg) + Expect(err).ToNot(HaveOccurred()) + expectIdentity("Forecast") + }) + It("VAD", func() { _, err := backend.VAD(&schema.VADRequest{Audio: []float32{0}}, ctx, loader, appCfg, cfg) Expect(err).ToNot(HaveOccurred()) diff --git a/core/config/backend_capabilities.go b/core/config/backend_capabilities.go index cfaa7cec1..cd728d863 100644 --- a/core/config/backend_capabilities.go +++ b/core/config/backend_capabilities.go @@ -28,6 +28,7 @@ const ( UsecaseRerank = "rerank" UsecaseDetection = "detection" UsecaseDepth = "depth" + UsecaseForecast = "forecast" UsecaseVAD = "vad" UsecaseAudioTransform = "audio_transform" UsecaseDiarization = "diarization" @@ -60,6 +61,7 @@ const ( MethodTokenizeString GRPCMethod = "TokenizeString" MethodDetect GRPCMethod = "Detect" MethodDepth GRPCMethod = "Depth" + MethodForecast GRPCMethod = "Forecast" MethodRerank GRPCMethod = "Rerank" MethodVAD GRPCMethod = "VAD" MethodAudioTransform GRPCMethod = "AudioTransform" @@ -181,6 +183,11 @@ var UsecaseInfoMap = map[string]UsecaseInfo{ GRPCMethod: MethodDepth, Description: "Per-pixel metric depth, camera pose and 3D point cloud via the Depth RPC (Depth Anything 3).", }, + UsecaseForecast: { + Flag: FLAG_FORECAST, + GRPCMethod: MethodForecast, + Description: "Time-series quantile forecasting via the Forecast RPC (timestral.cpp).", + }, UsecaseVAD: { Flag: FLAG_VAD, GRPCMethod: MethodVAD, @@ -814,6 +821,12 @@ var BackendCapabilities = map[string]BackendCapability{ AcceptsImages: true, Description: "Depth Anything 3 C++ — per-pixel metric depth, camera pose and 3D point cloud", }, + "timestral-cpp": { + GRPCMethods: []GRPCMethod{MethodForecast}, + PossibleUsecases: []string{UsecaseForecast}, + DefaultUsecases: []string{UsecaseForecast}, + Description: "timestral.cpp (C++/ggml, CPU only): quantile forecasts for numeric time series with optional covariates", + }, // --- Face and speaker recognition backends --- "insightface": { diff --git a/core/config/backend_capabilities_test.go b/core/config/backend_capabilities_test.go index bcf9cb34f..73005e8b3 100644 --- a/core/config/backend_capabilities_test.go +++ b/core/config/backend_capabilities_test.go @@ -471,3 +471,34 @@ var _ = Describe("GEM-X motion capability", func() { Expect((&ModelConfig{Backend: "llama-cpp"}).HasUsecases(FLAG_MOTION)).To(BeFalse()) }) }) + +var _ = Describe("timestral forecast capability", func() { + It("discovers forecast on the backend and keeps the model out of chat", func() { + Expect(DefaultUsecasesForBackendCap("timestral-cpp")).To(Equal([]string{UsecaseForecast})) + cfg := &ModelConfig{Backend: "timestral-cpp"} + Expect(cfg.HasUsecases(FLAG_FORECAST)).To(BeTrue()) + Expect(cfg.HasUsecases(FLAG_CHAT)).To(BeFalse()) + Expect(cfg.HasUsecases(FLAG_COMPLETION)).To(BeFalse()) + Expect(cfg.HasUsecases(FLAG_EMBEDDINGS)).To(BeFalse()) + Expect(cfg.HasUsecases(FLAG_DEPTH)).To(BeFalse()) + Expect(cfg.Capabilities()).To(ContainElement(UsecaseForecast)) + Expect(cfg.Capabilities()).NotTo(ContainElement(UsecaseChat)) + }) + + It("keeps chat models and other backends out of forecast", func() { + Expect((&ModelConfig{Backend: "llama-cpp"}).HasUsecases(FLAG_FORECAST)).To(BeFalse()) + chat := &ModelConfig{Backend: "timestral-cpp"} + chat.TemplateConfig.UseTokenizerTemplate = true + Expect(chat.HasUsecases(FLAG_CHAT)).To(BeFalse(), "a timestral model never serves text") + }) + + It("maps the usecase to its flag, RPC and string form", func() { + info := UsecaseInfoMap[UsecaseForecast] + Expect(info.Flag).To(Equal(FLAG_FORECAST)) + Expect(info.GRPCMethod).To(Equal(MethodForecast)) + Expect(GetAllModelConfigUsecases()).To(HaveKeyWithValue("FLAG_FORECAST", FLAG_FORECAST)) + flags := GetUsecasesFromYAML([]string{"forecast"}) + Expect(flags).NotTo(BeNil()) + Expect(*flags).To(Equal(FLAG_FORECAST)) + }) +}) diff --git a/core/config/meta/constants.go b/core/config/meta/constants.go index 008afe64f..8d95edfec 100644 --- a/core/config/meta/constants.go +++ b/core/config/meta/constants.go @@ -66,6 +66,7 @@ var UsecaseOptions = []FieldOption{ {Value: "vision", Label: "Vision"}, {Value: "detection", Label: "Detection"}, {Value: "depth", Label: "Depth"}, + {Value: "forecast", Label: "Forecast"}, {Value: "face_recognition", Label: "Face Recognition"}, {Value: "transcript", Label: "Transcript"}, {Value: "diarization", Label: "Diarization"}, diff --git a/core/config/model_capabilities.go b/core/config/model_capabilities.go index 5d99510f3..e4d1aa648 100644 --- a/core/config/model_capabilities.go +++ b/core/config/model_capabilities.go @@ -169,6 +169,7 @@ func (c *ModelConfig) Capabilities() []string { add(c.HasUsecases(FLAG_VAD), UsecaseVAD) add(c.HasUsecases(FLAG_DETECTION), UsecaseDetection) add(c.HasUsecases(FLAG_DEPTH), UsecaseDepth) + add(c.HasUsecases(FLAG_FORECAST), UsecaseForecast) add(c.HasUsecases(FLAG_AUDIO_TRANSFORM), UsecaseAudioTransform) add(c.HasUsecases(FLAG_DIARIZATION), UsecaseDiarization) add(c.HasUsecases(FLAG_SOUND_CLASSIFICATION), UsecaseSoundClassification) diff --git a/core/config/model_config.go b/core/config/model_config.go index 3b56213ea..97845a6a9 100644 --- a/core/config/model_config.go +++ b/core/config/model_config.go @@ -2405,6 +2405,9 @@ const ( // guards against two flags silently sharing one. FLAG_MOTION ModelConfigUsecase = 1 << 27 + // Marks a model as a time-series forecaster served by the Forecast RPC. + FLAG_FORECAST ModelConfigUsecase = 1 << 28 + // Common Subsets FLAG_LLM ModelConfigUsecase = FLAG_CHAT | FLAG_COMPLETION | FLAG_EDIT ) @@ -2470,6 +2473,7 @@ func GetAllModelConfigUsecases() map[string]ModelConfigUsecase { "FLAG_DECISIONS": FLAG_DECISIONS, "FLAG_UPSCALE": FLAG_UPSCALE, "FLAG_MOTION": FLAG_MOTION, + "FLAG_FORECAST": FLAG_FORECAST, } } @@ -2536,7 +2540,7 @@ func (c *ModelConfig) GuessUsecases(u ModelConfigUsecase) bool { "whisper", "piper", "kokoro", "diffusers", "stablediffusion", "stablediffusion-ggml", "rerankers", "silero-vad", "rfdetr", "insightface", "speaker-recognition", - "transformers-musicgen", "ace-step", "acestep-cpp", + "transformers-musicgen", "ace-step", "acestep-cpp", "timestral-cpp", } if (u & FLAG_CHAT) == FLAG_CHAT { @@ -2645,6 +2649,13 @@ func (c *ModelConfig) GuessUsecases(u ModelConfigUsecase) bool { } } + if (u & FLAG_FORECAST) == FLAG_FORECAST { + forecastBackends := []string{"timestral-cpp"} + if !slices.Contains(forecastBackends, c.Backend) { + return false + } + } + if (u & FLAG_3D) == FLAG_3D { threeDBackends := []string{"trellis2cpp"} if !slices.Contains(threeDBackends, c.Backend) { diff --git a/core/config/model_config_filter_test.go b/core/config/model_config_filter_test.go index 25be32a15..f9506c6f2 100644 --- a/core/config/model_config_filter_test.go +++ b/core/config/model_config_filter_test.go @@ -13,7 +13,7 @@ var _ = Describe("FilterMayMatchLooseFiles", func() { FLAG_CHAT, FLAG_COMPLETION, FLAG_EDIT, FLAG_EMBEDDINGS, FLAG_RERANK, FLAG_IMAGE, FLAG_TRANSCRIPT, FLAG_TTS, FLAG_SOUND_GENERATION, FLAG_TOKENIZE, FLAG_VAD, FLAG_VIDEO, FLAG_DETECTION, FLAG_VISION, FLAG_FACE_RECOGNITION, FLAG_SPEAKER_RECOGNITION, - FLAG_AUDIO_TRANSFORM, FLAG_DIARIZATION, FLAG_DEPTH, FLAG_3D, FLAG_UPSCALE, FLAG_LLM, + FLAG_AUDIO_TRANSFORM, FLAG_DIARIZATION, FLAG_DEPTH, FLAG_FORECAST, FLAG_3D, FLAG_UPSCALE, FLAG_LLM, } { Expect(FilterMayMatchLooseFiles(BuildUsecaseFilterFn(flag))).To(BeFalse(), "flag %d", flag) } diff --git a/core/gallery/backend_library_generated_test.go b/core/gallery/backend_library_generated_test.go index 57676a6b3..87b11526a 100644 --- a/core/gallery/backend_library_generated_test.go +++ b/core/gallery/backend_library_generated_test.go @@ -104,8 +104,14 @@ func TestLibraryGalleryReadsTheBackendLibrary(t *testing.T) { official.Library = true official.URL = srv.URL + "/backends/index.yaml" backends := names(official) - if len(backends) != goldenGalleryNames { - t.Fatalf("the library gave %d backends, want %d", len(backends), goldenGalleryNames) + // One backend per distinct name: platform images of one backend share it. + // The count follows the library, so adding a backend does not break this. + wantNames := map[string]bool{} + for _, r := range loadRealBackendLibrary(t) { + wantNames[r.Name] = true + } + if len(backends) != len(wantNames) { + t.Fatalf("the library gave %d backends, want %d", len(backends), len(wantNames)) } llama := backends.FindByName("llama-cpp") if llama == nil || !llama.IsMeta() || llama.CapabilitiesMap["nvidia"] != "cuda12-llama-cpp" || llama.Alias != "llama-cpp" { diff --git a/core/http/auth/features.go b/core/http/auth/features.go index d4b3d3fbf..79b4732f0 100644 --- a/core/http/auth/features.go +++ b/core/http/auth/features.go @@ -116,6 +116,9 @@ var RouteFeatureRegistry = []RouteFeature{ // Detection {"POST", "/v1/detection", FeatureDetection}, + // Time-series forecasting + {"POST", "/v1/forecast", FeatureForecast}, + // Decisions API (OpenAI and SystemOne wire contracts) {"POST", "/v1/decisions", FeatureDecisions}, {"POST", "/v1/systemone", FeatureDecisions}, @@ -274,6 +277,7 @@ func APIFeatureMetas() []FeatureMeta { {FeatureAudioTransform, "Audio Transform", true}, {FeaturePIIFilter, "PII Analyze / Redact", true}, {FeatureDecisions, "Decisions", true}, + {FeatureForecast, "Forecast", true}, {FeatureChatHistory, "Chat History", true}, } } diff --git a/core/http/auth/features_forecast_test.go b/core/http/auth/features_forecast_test.go new file mode 100644 index 000000000..19f737610 --- /dev/null +++ b/core/http/auth/features_forecast_test.go @@ -0,0 +1,24 @@ +package auth_test + +import ( + . "github.com/mudler/LocalAI/core/http/auth" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("Forecast feature registration", func() { + It("gates /v1/forecast behind a default-on API feature", func() { + Expect(APIFeatures).To(ContainElement(FeatureForecast)) + + patterns := []string{} + for _, route := range RouteFeatureRegistry { + if route.Feature == FeatureForecast { + Expect(route.Method).To(Equal("POST")) + patterns = append(patterns, route.Pattern) + } + } + Expect(patterns).To(ConsistOf("/v1/forecast")) + + Expect(APIFeatureMetas()).To(ContainElement(FeatureMeta{Key: FeatureForecast, Label: "Forecast", DefaultValue: true})) + }) +}) diff --git a/core/http/auth/permissions.go b/core/http/auth/permissions.go index a0ad02e48..75d844bfe 100644 --- a/core/http/auth/permissions.go +++ b/core/http/auth/permissions.go @@ -62,6 +62,7 @@ const ( FeatureVoiceRecognition = "voice_recognition" FeatureAudioTransform = "audio_transform" FeatureDecisions = "decisions" + FeatureForecast = "forecast" // FeaturePIIFilter gates the synchronous PII analyze/redact service // (POST /api/pii/{analyze,redact}). Default ON like the other API // features; the admin-only events log is gated separately in-handler. @@ -82,7 +83,7 @@ var APIFeatures = []string{ FeatureVAD, FeatureDetection, FeatureVideo, Feature3D, FeatureMotion, FeatureEmbeddings, FeatureSound, FeatureRealtime, FeatureModeration, FeatureRerank, FeatureTokenize, FeatureMCP, FeatureStores, FeatureFaceRecognition, FeatureVoiceRecognition, FeatureAudioTransform, - FeaturePIIFilter, FeatureDecisions, + FeaturePIIFilter, FeatureDecisions, FeatureForecast, FeatureChatHistory, } diff --git a/core/http/endpoints/localai/api_instructions.go b/core/http/endpoints/localai/api_instructions.go index 0164b6762..4175c56f7 100644 --- a/core/http/endpoints/localai/api_instructions.go +++ b/core/http/endpoints/localai/api_instructions.go @@ -99,6 +99,12 @@ var instructionDefs = []instructionDef{ Tags: []string{"3d"}, Intro: "POST /3d/generations accepts a conditioning image as public URL, base64, or data URI (no text prompt) and returns one .glb asset as a URL under /generated-3d or as b64_json. quality selects the mesh pipeline (auto|coarse|512|1024); background controls solid-background removal (auto|keep|black|white); step, texture_steps, and cfg_scale tune the flow sampling. POST /3d/remesh accepts multipart model, mesh (GLB), and a single detail percentage to return a watertight print-ready GLB; the enclosing offset is derived automatically. POST /3d/animate accepts named inputs with type and data, plus string-valued params. Consult the selected model's three_d_operations from /v1/models/capabilities for its required inputs, parameters, and output type. Kimodo produces text-conditioned animated skeletons without a mesh or skin; animation inputs are not universally text-only.", }, + { + Name: "forecast", + Description: "Time-series forecasting: quantile trajectories for numeric series, with optional covariates", + Tags: []string{"forecast"}, + Intro: "POST /v1/forecast accepts { model, prediction_length, series: [{ id, target, past_covariates, known_covariates, past_categorical, known_categorical }], quantiles, max_context }. target is [variates][steps], or a flat array for one variate; null marks a missing value. known_* rows hold steps + prediction_length values. Series of different shapes may share one request: the response keeps the request order. quantiles defaults to 0.1, 0.5 and 0.9 and must be levels the model was trained on. Forecast models serve this route only; they do not appear as chat models. CPU only.", + }, { Name: "face-recognition", Description: "Face verification (1:1), identification (1:N), embedding, and demographic analysis", diff --git a/core/http/endpoints/localai/api_instructions_test.go b/core/http/endpoints/localai/api_instructions_test.go index cb5e29c84..39210f530 100644 --- a/core/http/endpoints/localai/api_instructions_test.go +++ b/core/http/endpoints/localai/api_instructions_test.go @@ -39,7 +39,7 @@ var _ = Describe("API Instructions Endpoints", func() { instructions, ok := resp["instructions"].([]any) Expect(ok).To(BeTrue()) - Expect(instructions).To(HaveLen(27)) + Expect(instructions).To(HaveLen(28)) // Verify each instruction has required fields and correct URL format for _, s := range instructions { @@ -77,6 +77,7 @@ var _ = Describe("API Instructions Endpoints", func() { "monitoring", "agents", "face-recognition", + "forecast", "usage-and-billing", "pii-filtering", "middleware-admin", diff --git a/core/http/endpoints/localai/forecast.go b/core/http/endpoints/localai/forecast.go new file mode 100644 index 000000000..b0fa06bde --- /dev/null +++ b/core/http/endpoints/localai/forecast.go @@ -0,0 +1,221 @@ +package localai + +import ( + "fmt" + "math" + "net/http" + + "github.com/labstack/echo/v4" + "github.com/mudler/LocalAI/core/backend" + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/http/middleware" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/pkg/grpc/proto" + "github.com/mudler/LocalAI/pkg/model" + "github.com/mudler/xlog" +) + +// ForecastEndpoint forecasts the next steps of one or more numeric time series +// and returns quantile trajectories per series and variate. +// @Summary Forecasts numeric time series as quantile trajectories. +// @Tags forecast +// @Param request body schema.ForecastRequest true "query params" +// @Success 200 {object} schema.ForecastResponse "Response" +// @Router /v1/forecast [post] +func ForecastEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig) echo.HandlerFunc { + return func(c echo.Context) error { + input, ok := c.Get(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST).(*schema.ForecastRequest) + if !ok || input.Model == "" { + return echo.ErrBadRequest + } + + cfg, ok := c.Get(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG).(*config.ModelConfig) + if !ok || cfg == nil { + return echo.ErrBadRequest + } + + req, err := buildForecastRequest(input) + if err != nil { + return err + } + + xlog.Debug("Forecast", "series", len(req.Series), "horizon", req.PredictionLength, "backend", cfg.Backend) + + res, err := backend.Forecast(c.Request().Context(), req, ml, appConfig, *cfg) + if err != nil { + return mapBackendError(err) + } + + response, err := buildForecastResponse(input.Model, req, res) + if err != nil { + return err + } + return c.JSON(http.StatusOK, response) + } +} + +func forecastBadRequest(format string, args ...any) error { + return echo.NewHTTPError(http.StatusBadRequest, fmt.Sprintf(format, args...)) +} + +// forecastValue converts one JSON number to the float32 the backend takes. A +// nil value (JSON null) is a missing value and becomes NaN. +func forecastValue(v *float64, field string, row, step int) (float32, error) { + if v == nil { + return float32(math.NaN()), nil + } + if math.IsNaN(*v) || math.IsInf(*v, 0) || math.Abs(*v) > math.MaxFloat32 { + return 0, forecastBadRequest("%s[%d][%d]: value %v is not a finite float32", field, row, step, *v) + } + return float32(*v), nil +} + +// flattenForecastRows checks that every row holds steps values and returns the +// rows as one row-major slice: float32 for numeric rows, float64 for +// categorical rows, which must hold integer codes. The other slice is nil. +func flattenForecastRows(rows schema.ForecastRows, field string, steps int, categorical bool) ([]float32, []float64, error) { + var f32 []float32 + var f64 []float64 + if categorical { + f64 = make([]float64, 0, len(rows)*steps) + } else { + f32 = make([]float32, 0, len(rows)*steps) + } + for r, row := range rows { + if len(row) != steps { + return nil, nil, forecastBadRequest("%s[%d] has %d values, expected %d", field, r, len(row), steps) + } + for t, v := range row { + if categorical { + switch { + case v == nil: + f64 = append(f64, math.NaN()) + case math.IsNaN(*v) || math.IsInf(*v, 0) || *v != math.Trunc(*v): + return nil, nil, forecastBadRequest("%s[%d][%d]: %v is not an integer code", field, r, t, *v) + default: + f64 = append(f64, *v) + } + continue + } + x, err := forecastValue(v, field, r, t) + if err != nil { + return nil, nil, err + } + f32 = append(f32, x) + } + } + return f32, f64, nil +} + +// buildForecastRequest validates the JSON request and packs it into the +// columnar proto message. Every problem the caller can fix is a 400. +func buildForecastRequest(in *schema.ForecastRequest) (*proto.ForecastRequest, error) { + if in.PredictionLength < 1 { + return nil, forecastBadRequest("prediction_length must be at least 1") + } + if in.MaxContext < 0 { + return nil, forecastBadRequest("max_context must not be negative") + } + if len(in.Series) == 0 { + return nil, forecastBadRequest("series must hold at least one series") + } + for _, q := range in.Quantiles { + if math.IsNaN(q) || q <= 0 || q >= 1 { + return nil, forecastBadRequest("quantiles must be between 0 and 1, exclusive") + } + } + + h := in.PredictionLength + req := &proto.ForecastRequest{ + PredictionLength: int32(h), + Quantiles: in.Quantiles, + MaxContext: int32(in.MaxContext), + SymmetricAveraging: in.SymmetricAveraging, + ClipNonnegative: in.ClipNonnegative, + Series: make([]*proto.ForecastSeries, 0, len(in.Series)), + } + + for i, s := range in.Series { + name := fmt.Sprintf("series[%d]", i) + if len(s.Target) == 0 { + return nil, forecastBadRequest("%s.target must hold at least one variate", name) + } + t := len(s.Target[0]) + if t == 0 { + return nil, forecastBadRequest("%s.target must hold at least one step", name) + } + target, _, err := flattenForecastRows(schema.ForecastRows(s.Target), name+".target", t, false) + if err != nil { + return nil, err + } + past, _, err := flattenForecastRows(s.PastCovariates, name+".past_covariates", t, false) + if err != nil { + return nil, err + } + known, _, err := flattenForecastRows(s.KnownCovariates, name+".known_covariates", t+h, false) + if err != nil { + return nil, err + } + _, catPast, err := flattenForecastRows(s.PastCategorical, name+".past_categorical", t, true) + if err != nil { + return nil, err + } + _, catKnown, err := flattenForecastRows(s.KnownCategorical, name+".known_categorical", t+h, true) + if err != nil { + return nil, err + } + req.Series = append(req.Series, &proto.ForecastSeries{ + Id: s.ID, + NumVariates: int32(len(s.Target)), + Length: int32(t), + Target: target, + NumPastCovariates: int32(len(s.PastCovariates)), + PastCovariates: past, + NumKnownCovariates: int32(len(s.KnownCovariates)), + KnownCovariates: known, + NumPastCategorical: int32(len(s.PastCategorical)), + PastCategorical: catPast, + NumKnownCategorical: int32(len(s.KnownCategorical)), + KnownCategorical: catKnown, + }) + } + return req, nil +} + +// buildForecastResponse reshapes the flat V*Q*H backend values into one +// [Q][H] block per variate. The backend answers in request order, so the +// series index is the position in the response. +func buildForecastResponse(modelName string, req *proto.ForecastRequest, res *proto.ForecastResponse) (*schema.ForecastResponse, error) { + h := int(res.GetPredictionLength()) + levels := res.GetQuantileLevels() + q := len(levels) + if len(res.GetSeries()) != len(req.GetSeries()) { + return nil, fmt.Errorf("forecast backend returned %d series for %d requested", len(res.GetSeries()), len(req.GetSeries())) + } + + out := &schema.ForecastResponse{ + Object: "forecast", + Model: modelName, + PredictionLength: h, + QuantileLevels: levels, + Data: make([]schema.ForecastSeriesResult, 0, len(res.GetSeries())), + } + for i, sr := range res.GetSeries() { + v := int(sr.GetNumVariates()) + vals := sr.GetValues() + if v*q*h != len(vals) { + return nil, fmt.Errorf("forecast backend returned %d values for series %d, expected %d", len(vals), i, v*q*h) + } + item := schema.ForecastSeriesResult{Index: i, ID: sr.GetId(), Variates: make([]schema.ForecastVariate, 0, v)} + for vi := 0; vi < v; vi++ { + rows := make([][]float32, q) + for qi := 0; qi < q; qi++ { + start := (vi*q + qi) * h + rows[qi] = vals[start : start+h : start+h] + } + item.Variates = append(item.Variates, schema.ForecastVariate{Index: vi, Values: rows}) + } + out.Data = append(out.Data, item) + } + return out, nil +} diff --git a/core/http/endpoints/localai/forecast_test.go b/core/http/endpoints/localai/forecast_test.go new file mode 100644 index 000000000..4e789f75c --- /dev/null +++ b/core/http/endpoints/localai/forecast_test.go @@ -0,0 +1,179 @@ +package localai + +import ( + "encoding/json" + "math" + "net/http" + + "github.com/labstack/echo/v4" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/pkg/grpc/proto" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func parseForecastRequest(body string) *schema.ForecastRequest { + var req schema.ForecastRequest + ExpectWithOffset(1, json.Unmarshal([]byte(body), &req)).To(Succeed()) + return &req +} + +func expectBadRequest(err error, fragment string) { + ExpectWithOffset(1, err).To(HaveOccurred()) + var httpErr *echo.HTTPError + ExpectWithOffset(1, err).To(BeAssignableToTypeOf(httpErr)) + httpErr = err.(*echo.HTTPError) + ExpectWithOffset(1, httpErr.Code).To(Equal(http.StatusBadRequest)) + ExpectWithOffset(1, httpErr.Message).To(ContainSubstring(fragment)) +} + +var _ = Describe("forecast request parsing", func() { + It("reads null as a missing value and a flat target as one variate", func() { + req := parseForecastRequest(`{"model":"m","prediction_length":2,"series":[{"target":[1,null,3]}]}`) + Expect(req.Series[0].Target).To(HaveLen(1)) + Expect(req.Series[0].Target[0]).To(HaveLen(3)) + Expect(req.Series[0].Target[0][1]).To(BeNil()) + + out, err := buildForecastRequest(req) + Expect(err).ToNot(HaveOccurred()) + s := out.Series[0] + Expect(s.NumVariates).To(Equal(int32(1))) + Expect(s.Length).To(Equal(int32(3))) + Expect(s.Target[0]).To(Equal(float32(1))) + Expect(math.IsNaN(float64(s.Target[1]))).To(BeTrue()) + Expect(s.Target[2]).To(Equal(float32(3))) + }) + + It("rejects a target that is neither numbers nor arrays of numbers", func() { + var req schema.ForecastRequest + Expect(json.Unmarshal([]byte(`{"series":[{"target":"abc"}]}`), &req)).ToNot(Succeed()) + Expect(json.Unmarshal([]byte(`{"series":[{"target":[["a"]]}]}`), &req)).ToNot(Succeed()) + }) + + It("rejects the literal NaN, which is not JSON", func() { + var req schema.ForecastRequest + Expect(json.Unmarshal([]byte(`{"series":[{"target":[1,NaN]}]}`), &req)).ToNot(Succeed()) + }) +}) + +var _ = Describe("forecast request packing", func() { + const full = `{ + "model":"m","prediction_length":2,"quantiles":[0.1,0.9],"max_context":64, + "symmetric_averaging":true, + "series":[{ + "id":"a", + "target":[[1,2,3],[4,5,6]], + "past_covariates":[[0,1,0]], + "known_covariates":[[0,1,0,1,1]], + "past_categorical":[[1,null,2]], + "known_categorical":[[1,1,2,2,3]] + }]}` + + It("packs every field into the columnar message", func() { + out, err := buildForecastRequest(parseForecastRequest(full)) + Expect(err).ToNot(HaveOccurred()) + Expect(out.PredictionLength).To(Equal(int32(2))) + Expect(out.Quantiles).To(Equal([]float64{0.1, 0.9})) + Expect(out.MaxContext).To(Equal(int32(64))) + Expect(out.SymmetricAveraging).ToNot(BeNil()) + Expect(*out.SymmetricAveraging).To(BeTrue()) + Expect(out.ClipNonnegative).To(BeNil(), "an absent flag stays unset so the model default applies") + + s := out.Series[0] + Expect(s.Id).To(Equal("a")) + Expect(s.NumVariates).To(Equal(int32(2))) + Expect(s.Target).To(Equal([]float32{1, 2, 3, 4, 5, 6})) + Expect(s.NumPastCovariates).To(Equal(int32(1))) + Expect(s.PastCovariates).To(Equal([]float32{0, 1, 0})) + Expect(s.NumKnownCovariates).To(Equal(int32(1))) + Expect(s.KnownCovariates).To(HaveLen(5)) + Expect(s.NumPastCategorical).To(Equal(int32(1))) + Expect(math.IsNaN(s.PastCategorical[1])).To(BeTrue()) + Expect(s.KnownCategorical).To(Equal([]float64{1, 1, 2, 2, 3})) + }) + + DescribeTable("returns 400 for", + func(body, fragment string) { + _, err := buildForecastRequest(parseForecastRequest(body)) + expectBadRequest(err, fragment) + }, + Entry("a zero horizon", `{"prediction_length":0,"series":[{"target":[1]}]}`, "prediction_length"), + Entry("no series", `{"prediction_length":1,"series":[]}`, "series"), + Entry("an empty target", `{"prediction_length":1,"series":[{"target":[]}]}`, "target"), + Entry("a ragged target", `{"prediction_length":1,"series":[{"target":[[1,2],[3]]}]}`, "series[0].target[1]"), + Entry("a negative max_context", `{"prediction_length":1,"max_context":-1,"series":[{"target":[1]}]}`, "max_context"), + Entry("a quantile outside (0,1)", `{"prediction_length":1,"quantiles":[1],"series":[{"target":[1]}]}`, "quantiles"), + Entry("past covariates of the wrong length", `{"prediction_length":1,"series":[{"target":[1,2],"past_covariates":[[1]]}]}`, "past_covariates"), + Entry("known covariates one step short", `{"prediction_length":2,"series":[{"target":[1,2],"known_covariates":[[1,2,3]]}]}`, "expected 4"), + Entry("a fractional categorical code", `{"prediction_length":1,"series":[{"target":[1],"past_categorical":[[1.5]]}]}`, "integer code"), + Entry("a value beyond float32", `{"prediction_length":1,"series":[{"target":[1e300]}]}`, "float32"), + Entry("the second series being bad", `{"prediction_length":1,"series":[{"target":[1]},{"target":[]}]}`, "series[1]"), + ) + + It("accepts series of different lengths in one request", func() { + out, err := buildForecastRequest(parseForecastRequest( + `{"prediction_length":1,"series":[{"target":[1,2,3]},{"target":[1,2]}]}`)) + Expect(err).ToNot(HaveOccurred()) + Expect(out.Series[0].Length).To(Equal(int32(3))) + Expect(out.Series[1].Length).To(Equal(int32(2))) + }) +}) + +var _ = Describe("forecast response", func() { + It("slices the flat V*Q*H values into one [Q][H] block per variate", func() { + req := &proto.ForecastRequest{Series: []*proto.ForecastSeries{{Id: "a"}, {Id: "b"}}} + // Each value encodes series, variate, quantile and step as sVQH. + code := func(s, v, q, h int) float32 { return float32(s*1000 + v*100 + q*10 + h) } + res := &proto.ForecastResponse{PredictionLength: 3, QuantileLevels: []float64{0.1, 0.9}} + for s, nv := range []int{2, 1} { + sr := &proto.ForecastSeriesResult{NumVariates: int32(nv), Id: []string{"a", "b"}[s]} + for v := 0; v < nv; v++ { + for q := 0; q < 2; q++ { + for h := 0; h < 3; h++ { + sr.Values = append(sr.Values, code(s, v, q, h)) + } + } + } + res.Series = append(res.Series, sr) + } + + out, err := buildForecastResponse("m", req, res) + Expect(err).ToNot(HaveOccurred()) + Expect(out.Object).To(Equal("forecast")) + Expect(out.Model).To(Equal("m")) + Expect(out.PredictionLength).To(Equal(3)) + Expect(out.QuantileLevels).To(Equal([]float64{0.1, 0.9})) + Expect(out.Data).To(HaveLen(2)) + Expect(out.Data[0].ID).To(Equal("a")) + Expect(out.Data[1].Index).To(Equal(1)) + Expect(out.Data[0].Variates).To(HaveLen(2)) + Expect(out.Data[0].Variates[1].Values[1]).To(Equal([]float32{code(0, 1, 1, 0), code(0, 1, 1, 1), code(0, 1, 1, 2)})) + Expect(out.Data[1].Variates[0].Values[0]).To(Equal([]float32{code(1, 0, 0, 0), code(1, 0, 0, 1), code(1, 0, 0, 2)})) + }) + + It("serialises with the documented JSON field names", func() { + req := &proto.ForecastRequest{Series: []*proto.ForecastSeries{{}}} + res := &proto.ForecastResponse{ + PredictionLength: 2, QuantileLevels: []float64{0.5}, + Series: []*proto.ForecastSeriesResult{{Id: "x", NumVariates: 1, Values: []float32{1.5, 2.5}}}, + } + out, err := buildForecastResponse("m", req, res) + Expect(err).ToNot(HaveOccurred()) + raw, err := json.Marshal(out) + Expect(err).ToNot(HaveOccurred()) + Expect(string(raw)).To(MatchJSON(`{ + "object":"forecast","model":"m","prediction_length":2,"quantile_levels":[0.5], + "data":[{"index":0,"id":"x","variates":[{"index":0,"values":[[1.5,2.5]]}]}]}`)) + }) + + It("fails loudly when the backend answers with the wrong shape", func() { + req := &proto.ForecastRequest{Series: []*proto.ForecastSeries{{}}} + _, err := buildForecastResponse("m", req, &proto.ForecastResponse{}) + Expect(err).To(HaveOccurred()) + _, err = buildForecastResponse("m", req, &proto.ForecastResponse{ + PredictionLength: 2, QuantileLevels: []float64{0.5}, + Series: []*proto.ForecastSeriesResult{{NumVariates: 1, Values: []float32{1}}}, + }) + Expect(err).To(HaveOccurred()) + }) +}) diff --git a/core/http/react-ui/public/locales/en/models.json b/core/http/react-ui/public/locales/en/models.json index e2b1b84ea..df2db5439 100644 --- a/core/http/react-ui/public/locales/en/models.json +++ b/core/http/react-ui/public/locales/en/models.json @@ -100,6 +100,7 @@ }, "filters": { "motion": "Motion Capture", + "forecast": "Forecasting", "all": "All", "llm": "Chat", "image": "Image", diff --git a/core/http/react-ui/src/pages/Models.jsx b/core/http/react-ui/src/pages/Models.jsx index 0bfafca86..7dc7ebc6f 100644 --- a/core/http/react-ui/src/pages/Models.jsx +++ b/core/http/react-ui/src/pages/Models.jsx @@ -98,6 +98,7 @@ const FILTERS = [ { key: '3d', labelKey: 'filters.threed', icon: 'cube' }, { key: '3d_animation', labelKey: 'filters.threedAnimation', icon: 'walk' }, { key: 'motion', labelKey: 'filters.motion', icon: 'walk' }, + { key: 'forecast', labelKey: 'filters.forecast', icon: 'chart-line' }, { key: 'multimodal', labelKey: 'filters.multimodal', icon: 'shapes' }, { key: 'vision', labelKey: 'filters.vision', icon: 'eye' }, { key: 'tts', labelKey: 'filters.tts', icon: 'mic' }, diff --git a/core/http/react-ui/src/utils/capabilities.js b/core/http/react-ui/src/utils/capabilities.js index 6bb64c030..b2cc7a90d 100644 --- a/core/http/react-ui/src/utils/capabilities.js +++ b/core/http/react-ui/src/utils/capabilities.js @@ -36,3 +36,4 @@ export const CAP_DECISIONS = 'FLAG_DECISIONS' export const CAP_TOKEN_CLASSIFY = 'FLAG_TOKEN_CLASSIFY' export const CAP_MOTION = 'FLAG_MOTION' +export const CAP_FORECAST = 'FLAG_FORECAST' diff --git a/core/http/routes/localai.go b/core/http/routes/localai.go index 3cd6ad8b7..fd39b2bbd 100644 --- a/core/http/routes/localai.go +++ b/core/http/routes/localai.go @@ -171,6 +171,12 @@ func RegisterLocalAIRoutes(router *echo.Echo, requestExtractor.BuildFilteredFirstAvailableDefaultModel(config.BuildUsecaseFilterFn(config.FLAG_DEPTH)), requestExtractor.SetModelAndConfig(func() schema.LocalAIRequest { return new(schema.DepthRequest) })) + forecastHandler := localai.ForecastEndpoint(cl, ml, appConfig) + router.POST("/v1/forecast", + forecastHandler, + requestExtractor.BuildFilteredFirstAvailableDefaultModel(config.BuildUsecaseFilterFn(config.FLAG_FORECAST)), + requestExtractor.SetModelAndConfig(func() schema.LocalAIRequest { return new(schema.ForecastRequest) })) + // Face recognition endpoints faceMw := []echo.MiddlewareFunc{ requestExtractor.BuildFilteredFirstAvailableDefaultModel(config.BuildUsecaseFilterFn(config.FLAG_FACE_RECOGNITION)), diff --git a/core/http/routes/model_listing_routes_test.go b/core/http/routes/model_listing_routes_test.go index 05d2cc5aa..ae6b47df3 100644 --- a/core/http/routes/model_listing_routes_test.go +++ b/core/http/routes/model_listing_routes_test.go @@ -41,6 +41,7 @@ var routeUsecaseFilters = []struct { {"upscale", config.FLAG_UPSCALE}, {"detection", config.FLAG_DETECTION}, {"depth", config.FLAG_DEPTH}, + {"forecast", config.FLAG_FORECAST}, {"face recognition", config.FLAG_FACE_RECOGNITION}, {"speaker recognition", config.FLAG_SPEAKER_RECOGNITION}, {"audio transform", config.FLAG_AUDIO_TRANSFORM}, diff --git a/core/http/routes/ui_api.go b/core/http/routes/ui_api.go index 28aef9a60..cef357b6a 100644 --- a/core/http/routes/ui_api.go +++ b/core/http/routes/ui_api.go @@ -60,6 +60,7 @@ var usecaseFilters = map[string]config.ModelConfigUsecase{ config.UsecaseEmbeddings: config.FLAG_EMBEDDINGS, config.UsecaseRerank: config.FLAG_RERANK, config.UsecaseDetection: config.FLAG_DETECTION, + config.UsecaseForecast: config.FLAG_FORECAST, config.UsecaseVAD: config.FLAG_VAD, config.UsecaseAudioTransform: config.FLAG_AUDIO_TRANSFORM, config.UsecaseDiarization: config.FLAG_DIARIZATION, diff --git a/core/schema/localai.go b/core/schema/localai.go index 479cdfbc3..10c08edf6 100644 --- a/core/schema/localai.go +++ b/core/schema/localai.go @@ -2,6 +2,7 @@ package schema import ( "encoding/json" + "fmt" "time" gopsutil "github.com/shirou/gopsutil/v3/process" @@ -281,6 +282,78 @@ type DepthRequest struct { Exports []string `json:"exports,omitempty"` // requested exports: "glb", "colmap" } +// ForecastRows is a list of equally long numeric rows (variates or +// covariates), oldest step first. A JSON null is a missing value and is held +// as a nil pointer. +type ForecastRows [][]*float64 + +// ForecastTarget is ForecastRows that also accepts a flat array as shorthand +// for a single variate: [1, 2, 3] reads as [[1, 2, 3]]. +type ForecastTarget ForecastRows + +// UnmarshalJSON accepts both the nested form and the flat shorthand. +func (t *ForecastTarget) UnmarshalJSON(data []byte) error { + var rows [][]*float64 + if err := json.Unmarshal(data, &rows); err == nil { + *t = rows + return nil + } + var flat []*float64 + if err := json.Unmarshal(data, &flat); err != nil { + return fmt.Errorf("target must be an array of numbers or an array of arrays of numbers") + } + *t = ForecastTarget{flat} + return nil +} + +// ForecastSeries is one time series of a /v1/forecast request. +type ForecastSeries struct { + ID string `json:"id,omitempty"` // echoed back in the matching result + Target ForecastTarget `json:"target"` // [V][T] or [T], null = missing + // PastCovariates is [C][T]: numeric covariates known only for the past. + PastCovariates ForecastRows `json:"past_covariates,omitempty"` + // KnownCovariates is [C][T+H]: numeric covariates known for the future too. + KnownCovariates ForecastRows `json:"known_covariates,omitempty"` + // PastCategorical is [C][T]: integer category codes for the past. + PastCategorical ForecastRows `json:"past_categorical,omitempty"` + // KnownCategorical is [C][T+H]: integer category codes for the future too. + KnownCategorical ForecastRows `json:"known_categorical,omitempty"` +} + +// ForecastRequest is the request body for the /v1/forecast endpoint. +type ForecastRequest struct { + BasicModelRequest + PredictionLength int `json:"prediction_length"` // steps to forecast + Quantiles []float64 `json:"quantiles,omitempty"` // levels to return; default 0.1, 0.5, 0.9 + MaxContext int `json:"max_context,omitempty"` // cap on history used; 0 = model default + SymmetricAveraging *bool `json:"symmetric_averaging,omitempty"` // null = model default + ClipNonnegative *bool `json:"clip_nonnegative,omitempty"` // null = model default + Series []ForecastSeries `json:"series"` +} + +// ForecastVariate holds the forecast of one variate. +type ForecastVariate struct { + Index int `json:"index"` + // Values is [Q][H]: Values[q][h] is quantile_levels[q] at step h. + Values [][]float32 `json:"values"` +} + +// ForecastSeriesResult is the forecast of one input series. +type ForecastSeriesResult struct { + Index int `json:"index"` // position of the series in the request + ID string `json:"id,omitempty"` + Variates []ForecastVariate `json:"variates"` +} + +// ForecastResponse is the JSON response for the /v1/forecast endpoint. +type ForecastResponse struct { + Object string `json:"object"` + Model string `json:"model"` + PredictionLength int `json:"prediction_length"` + QuantileLevels []float64 `json:"quantile_levels"` + Data []ForecastSeriesResult `json:"data"` +} + // DepthResponse is the JSON response for the /v1/depth endpoint, mirroring the // DepthResponse proto. type DepthResponse struct { diff --git a/core/services/nodes/health_mock_test.go b/core/services/nodes/health_mock_test.go index 330871570..2e86ae8b2 100644 --- a/core/services/nodes/health_mock_test.go +++ b/core/services/nodes/health_mock_test.go @@ -185,6 +185,9 @@ func (c *fakeBackendClient) SoundDetection(_ context.Context, _ *pb.SoundDetecti func (c *fakeBackendClient) Depth(_ context.Context, _ *pb.DepthRequest, _ ...ggrpc.CallOption) (*pb.DepthResponse, error) { return nil, nil } +func (c *fakeBackendClient) Forecast(_ context.Context, _ *pb.ForecastRequest, _ ...ggrpc.CallOption) (*pb.ForecastResponse, error) { + return nil, nil +} func (c *fakeBackendClient) FaceVerify(_ context.Context, _ *pb.FaceVerifyRequest, _ ...ggrpc.CallOption) (*pb.FaceVerifyResponse, error) { return nil, nil } diff --git a/core/services/nodes/inflight.go b/core/services/nodes/inflight.go index afe798f66..b44ae38b6 100644 --- a/core/services/nodes/inflight.go +++ b/core/services/nodes/inflight.go @@ -203,6 +203,12 @@ func (c *InFlightTrackingClient) Depth(ctx context.Context, in *pb.DepthRequest, return res, c.reconcile(err) } +func (c *InFlightTrackingClient) Forecast(ctx context.Context, in *pb.ForecastRequest, opts ...ggrpc.CallOption) (*pb.ForecastResponse, error) { + defer c.track(ctx)() + res, err := c.inner.Forecast(ctx, in, opts...) + return res, c.reconcile(err) +} + func (c *InFlightTrackingClient) Rerank(ctx context.Context, in *pb.RerankRequest, opts ...ggrpc.CallOption) (*pb.RerankResult, error) { defer c.track(ctx)() res, err := c.inner.Rerank(ctx, in, opts...) diff --git a/core/services/nodes/inflight_test.go b/core/services/nodes/inflight_test.go index 0952c6831..5c2e9f2f3 100644 --- a/core/services/nodes/inflight_test.go +++ b/core/services/nodes/inflight_test.go @@ -122,6 +122,10 @@ func (f *fakeGRPCBackend) Depth(_ context.Context, _ *pb.DepthRequest, _ ...ggrp return &pb.DepthResponse{}, nil } +func (f *fakeGRPCBackend) Forecast(_ context.Context, _ *pb.ForecastRequest, _ ...ggrpc.CallOption) (*pb.ForecastResponse, error) { + return &pb.ForecastResponse{}, nil +} + func (f *fakeGRPCBackend) FaceVerify(_ context.Context, _ *pb.FaceVerifyRequest, _ ...ggrpc.CallOption) (*pb.FaceVerifyResponse, error) { return &pb.FaceVerifyResponse{}, nil } diff --git a/core/trace/backend_trace.go b/core/trace/backend_trace.go index 5cf6299bc..f822e20ee 100644 --- a/core/trace/backend_trace.go +++ b/core/trace/backend_trace.go @@ -38,6 +38,7 @@ const ( BackendTraceTokenize BackendTraceType = "tokenize" BackendTraceDetection BackendTraceType = "detection" BackendTraceDepth BackendTraceType = "depth" + BackendTraceForecast BackendTraceType = "forecast" BackendTraceFaceVerify BackendTraceType = "face_verify" BackendTraceFaceAnalyze BackendTraceType = "face_analyze" BackendTraceVoiceVerify BackendTraceType = "voice_verify" diff --git a/docs/content/features/backends.md b/docs/content/features/backends.md index 8591b6250..38a9e6775 100644 --- a/docs/content/features/backends.md +++ b/docs/content/features/backends.md @@ -441,6 +441,7 @@ LocalAI supports various types of backends: - **Image & Video Generation Backends**: For diffusion and audio-conditioned avatar models (e.g., stable-diffusion.cpp, diffusers, vLLM-Omni, [MLX-Video on Apple Silicon]({{%relref "features/video-generation" %}}), [LongCat-Video]({{%relref "features/video-generation" %}}), [vllm.cpp / MiniMax-H3]({{%relref "features/video-generation" %}})) - **3D Generation Backends**: For image-to-3D mesh generation ([trellis2.cpp]({{%relref "features/3d-generation" %}}) — Microsoft TRELLIS.2, producing GLB assets with PBR textures) - **Vision & Detection Backends**: For object detection, segmentation, depth, and face/voice recognition (e.g., rf-detr.cpp, locate-anything.cpp, sam3.cpp, insightface) +- **Time-Series Backends**: For quantile forecasting of numeric series ([timestral.cpp]({{%relref "features/forecast" %}}), Mistral's Timestral models) - **Audio Processing Backends**: For voice activity detection and audio enhancement (e.g., Silero VAD, LocalVQE, [audio.cpp]({{%relref "features/audio-cpp" %}})) - **Source Separation & Voice Conversion Backends**: For splitting a mix into named stems (vocals, drums, bass) and for converting speech or singing to a target voice (e.g., [audio.cpp]({{%relref "features/audio-cpp" %}})) - **Utility Backends**: For reranking, PII/NER token classification, fine-tuning, quantization, and vector storage (e.g., rerankers, privacy-filter.cpp, TRL, local-store, valkey-store, qdrant-store) diff --git a/docs/content/features/forecast.md b/docs/content/features/forecast.md new file mode 100644 index 000000000..609ce6096 --- /dev/null +++ b/docs/content/features/forecast.md @@ -0,0 +1,131 @@ ++++ +disableToc = false +title = "Time-Series Forecasting" +weight = 42 +url = "/features/forecast/" ++++ + +LocalAI forecasts numeric time series with [timestral.cpp](https://github.com/mudler/timestral.cpp), a C++/ggml port of Mistral's Timestral models. You send one or more series, with optional covariates, to `POST /v1/forecast` and get quantile trajectories for the next steps. + +A forecast model is not a chat model. It has its own usecase, `forecast`, and does not appear in the chat model picker. It shows in the gallery filter and in the `/v1/models` capability list. + +> **Licence.** The Timestral weights use Mistral's Modified MIT licence. You may not use them if the global consolidated monthly revenue of your company (or your employer) exceeded USD 20 million in the preceding month, unless you get a commercial licence from Mistral AI. The restriction covers derivatives, so it also applies to the GGUF conversions in the gallery. Each gallery entry installs the licence text next to the weights. The timestral.cpp code is MIT. + +## Models + +| Gallery entry | Parameters | Builds | +|---|---|---| +| `timestral-5m` | 5 million | F32 (default), F16 | +| `timestral-28m` | 28 million | F32 (default), F16 | +| `timestral-123m` | 123 million | F32 (default), F16 | +| `timestral-209m` | 209 million | F32 (default), F16 | + +```bash +local-ai models install timestral-28m +``` + +The default build is F32. Install the F16 entry, for example `timestral-28m-f16`, for a build of about half the size. Quantized builds below F16 are not offered yet. + +The engine runs on the CPU only. It is slower than the PyTorch reference at long context lengths, so keep the context short where you can and allow a long request timeout. + +How much memory each build needs has not been measured yet, so the gallery sets no size. + +## Request + +```bash +curl http://localhost:8080/v1/forecast \ + -H "Content-Type: application/json" \ + -d '{ + "model": "timestral-28m", + "prediction_length": 4, + "series": [ + { "id": "store-12", "target": [3.1, 3.4, null, 3.9, 4.2, 4.0, 4.4, 4.8] } + ] + }' +``` + +| Field | Type | Meaning | +|---|---|---| +| `model` | string | Name of a forecast model. | +| `prediction_length` | integer, at least 1 | Steps to forecast. | +| `quantiles` | list of numbers | Levels to return. Each must be a level the model was trained on. Default: `0.1`, `0.5`, `0.9`. | +| `max_context` | integer | Most history steps the model uses. `0` or absent keeps the model default. | +| `symmetric_averaging`, `clip_nonnegative` | boolean or `null` | `null` or absent keeps the model default. | +| `series` | list | The series to forecast. | + +Each entry of `series` has these fields: + +| Field | Shape | Meaning | +|---|---|---| +| `id` | string | Optional. Returned with the result. | +| `target` | `[variates][steps]`, or `[steps]` for one variate | History, oldest step first. `null` marks a missing value. | +| `past_covariates` | `[covariates][steps]` | Numeric covariates known only for the history. | +| `known_covariates` | `[covariates][steps + prediction_length]` | Numeric covariates that are also known for the future. | +| `past_categorical` | `[covariates][steps]` | Integer category codes for the history. `null` marks a missing code. | +| `known_categorical` | `[covariates][steps + prediction_length]` | Integer category codes that are also known for the future. | + +Series in one request can have different lengths and different numbers of variates or covariates. LocalAI groups them by shape, runs one forecast for each group, and returns the results in the order you sent the series. + +## Response + +```json +{ + "object": "forecast", + "model": "timestral-28m", + "prediction_length": 4, + "quantile_levels": [0.1, 0.5, 0.9], + "data": [ + { + "index": 0, + "id": "store-12", + "variates": [ + { "index": 0, "values": [[4.4, 4.5, 4.6, 4.6], [4.7, 4.8, 4.9, 5.0], [5.0, 5.2, 5.3, 5.4]] } + ] + } + ] +} +``` + +`values[q][h]` is the value of level `quantile_levels[q]` at step `h`. Step 0 is the first step after the last step of the history. The numbers above are an example of the shape, not real output. + +## Errors + +The endpoint returns `400` when: + +- `prediction_length` is less than 1, or `series` is empty. +- A row has the wrong number of steps, for example a known covariate that stops at the end of the history. +- A value is not a finite number, or a categorical code is not an integer. `NaN` and infinity are not valid JSON; use `null` for a missing value. +- A requested quantile is not a level of the model. The error lists the valid levels. +- The request is larger than the model limits. + +## Model options + +Set these under `options:` in the model configuration: + +```yaml +name: timestral-28m +backend: timestral-cpp +parameters: + model: timestral-28m-f32.gguf +threads: 4 +options: + - default_quantiles:0.1,0.5,0.9 + - max_context:0 + - max_series:256 + - max_total_points:2000000 +``` + +| Option | Default | Meaning | +|---|---|---| +| `default_quantiles` | `0.1,0.5,0.9` | Levels returned when a request names none. | +| `max_context` | `0` | History cap used when a request sets none. `0` keeps the model default. | +| `max_series` | `256` | Most series in one request. | +| `max_total_points` | `2000000` | Most input values in one request. | + +`threads` sets the number of CPU threads for each forecast. + +The library is not reentrant, so LocalAI runs one forecast at a time for each loaded model. + +## Platforms + +The backend image is built for Linux on amd64 and arm64. A build for Apple silicon is wired into the build matrix, but it has not been built or tested yet. diff --git a/docs/content/reference/compatibility-table.md b/docs/content/reference/compatibility-table.md index 127de9e94..9585a1e6d 100644 --- a/docs/content/reference/compatibility-table.md +++ b/docs/content/reference/compatibility-table.md @@ -107,6 +107,7 @@ All backends listed here can be installed on demand from the [Backend Gallery]({ | [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) | Open-vocabulary object detection and visual grounding (LocateAnything-3B) in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T | | [depth-anything.cpp](https://github.com/mudler/depth-anything.cpp) | Depth Anything 3 monocular metric depth + camera pose in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T | | [sam3.cpp](https://github.com/PABannier/sam3.cpp) | Segment Anything (SAM 3/2/EdgeTAM) with text/point/box prompts in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T | +| [timestral.cpp](https://github.com/mudler/timestral.cpp) | Mistral Timestral time-series forecasting through `/v1/forecast`, in C/C++ using GGML | CPU (Linux amd64 and arm64) | | [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Native face detection, recognition, embedding, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T | | [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Native speaker (voice) recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T | | [insightface](https://github.com/deepinsight/insightface) | Face verification, embedding, and anti-spoofing liveness (ONNX Runtime) | CPU, CUDA 12 | diff --git a/gallery/backends/index.yaml b/gallery/backends/index.yaml index 383d88dd4..d9d0d0eb4 100644 --- a/gallery/backends/index.yaml +++ b/gallery/backends/index.yaml @@ -43107,3 +43107,153 @@ urls: - https://github.com/modelscope/FunASR version: 0.0.0-development +- alias: timestral-cpp + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + channel: latest + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + id: timestral-cpp + kind: backend + license: mit + name: timestral-cpp + schemaVersion: 1 + tags: + - forecasting + - time-series + - CPU + urls: + - https://github.com/mudler/timestral.cpp + version: 0.0.0-latest +- alias: timestral-cpp + capabilities: + default: cpu-timestral-cpp-development + metal: metal-timestral-cpp-development + channel: development + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + id: timestral-cpp-development + kind: backend + license: mit + name: timestral-cpp-development + schemaVersion: 1 + tags: + - forecasting + - time-series + - CPU + urls: + - https://github.com/mudler/timestral.cpp + version: 0.0.0-development +- alias: timestral-cpp + arch: amd64 + artifact: oci://quay.io/go-skynet/local-ai-backends:latest-cpu-timestral-cpp + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + channel: latest + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + id: cpu-timestral-cpp + kind: backend + license: mit + mirrors: + - oci://localai/localai-backends:latest-cpu-timestral-cpp + mutable: true + name: cpu-timestral-cpp + os: linux + schemaVersion: 1 + tags: + - forecasting + - time-series + - CPU + urls: + - https://github.com/mudler/timestral.cpp + version: 0.0.0-latest +- alias: timestral-cpp + arch: amd64 + artifact: oci://quay.io/go-skynet/local-ai-backends:master-cpu-timestral-cpp + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + channel: development + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + id: cpu-timestral-cpp-development + kind: backend + license: mit + mirrors: + - oci://localai/localai-backends:master-cpu-timestral-cpp + mutable: true + name: cpu-timestral-cpp-development + os: linux + schemaVersion: 1 + tags: + - forecasting + - time-series + - CPU + urls: + - https://github.com/mudler/timestral.cpp + version: 0.0.0-development +- alias: timestral-cpp + arch: arm64 + artifact: oci://quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-timestral-cpp + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + channel: latest + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + id: metal-timestral-cpp + kind: backend + license: mit + mirrors: + - oci://localai/localai-backends:latest-metal-darwin-arm64-timestral-cpp + mutable: true + name: metal-timestral-cpp + os: darwin + schemaVersion: 1 + tags: + - forecasting + - time-series + - CPU + urls: + - https://github.com/mudler/timestral.cpp + version: 0.0.0-latest +- alias: timestral-cpp + arch: arm64 + artifact: oci://quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-timestral-cpp + capabilities: + default: cpu-timestral-cpp + metal: metal-timestral-cpp + channel: development + description: | + timestral.cpp is a C++/ggml port of Mistral's Timestral time-series forecasting models. + It takes numeric series, with optional covariates, and returns quantile forecasts through + POST /v1/forecast. CPU only. The code is MIT; the model weights have their own licence. + id: metal-timestral-cpp-development + kind: backend + license: mit + mirrors: + - oci://localai/localai-backends:master-metal-darwin-arm64-timestral-cpp + mutable: true + name: metal-timestral-cpp-development + os: darwin + schemaVersion: 1 + tags: + - forecasting + - time-series + - CPU + urls: + - https://github.com/mudler/timestral.cpp + version: 0.0.0-development diff --git a/gallery/index.yaml b/gallery/index.yaml index 6742070f7..c00a702a4 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -31934,6 +31934,322 @@ parameters: model: depth-anything2-metric-vkitti-large-q4_k.gguf variants: [] +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: timestral-5m + description: | + Timestral 5M (5 million parameters) time-series forecaster by Mistral AI, + served by the native timestral.cpp backend (C++/ggml, CPU only). Given one or + more numeric series, with optional covariates, it returns quantile forecasts + through POST /v1/forecast. Full precision (F32). The CPU engine is slower than + the PyTorch reference at long context lengths. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-5m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-5M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + files: + - filename: timestral-5m-f32.gguf + sha256: c320276d8db95fb72fe5f4e043147e29142a1bf197c1b45daf6736780f6736ed + uri: huggingface://mudler/timestral-cpp-gguf/5M/timestral-5m-f32.gguf + - filename: LICENSE-timestral-5m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/5M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-5m-f32.gguf + variants: + - model: timestral-5m-f16 +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: timestral-5m-f16 + description: | + Timestral 5M (5 million parameters) time-series forecaster by Mistral AI, + half precision (F16) build, about half the size of the F32 build. Served by the + native timestral.cpp backend (C++/ggml, CPU only) through POST /v1/forecast. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-5m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-5M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + files: + - filename: timestral-5m-f16.gguf + sha256: 933e2e518300e97e7799deb1920dd4e539da89a90c743b663cda772ee79d4a38 + uri: huggingface://mudler/timestral-cpp-gguf/5M/timestral-5m-f16.gguf + - filename: LICENSE-timestral-5m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/5M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-5m-f16.gguf + variants: [] +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: timestral-28m + description: | + Timestral 28M (28 million parameters) time-series forecaster by Mistral AI, + served by the native timestral.cpp backend (C++/ggml, CPU only). Given one or + more numeric series, with optional covariates, it returns quantile forecasts + through POST /v1/forecast. Full precision (F32). The CPU engine is slower than + the PyTorch reference at long context lengths. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-28m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-28M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + files: + - filename: timestral-28m-f32.gguf + sha256: fbb5a8dcb8b224a45b149dc802428beca20504e71f585d942b0530f67f46a89c + uri: huggingface://mudler/timestral-cpp-gguf/28M/timestral-28m-f32.gguf + - filename: LICENSE-timestral-28m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/28M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-28m-f32.gguf + variants: + - model: timestral-28m-f16 +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: timestral-28m-f16 + description: | + Timestral 28M (28 million parameters) time-series forecaster by Mistral AI, + half precision (F16) build, about half the size of the F32 build. Served by the + native timestral.cpp backend (C++/ggml, CPU only) through POST /v1/forecast. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-28m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-28M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + files: + - filename: timestral-28m-f16.gguf + sha256: 64b5b836c54c2d3a83b428e6b9e44e2bbf668c60297245c97a0df160d2c5366d + uri: huggingface://mudler/timestral-cpp-gguf/28M/timestral-28m-f16.gguf + - filename: LICENSE-timestral-28m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/28M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-28m-f16.gguf + variants: [] +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: timestral-123m + description: | + Timestral 123M (123 million parameters) time-series forecaster by Mistral AI, + served by the native timestral.cpp backend (C++/ggml, CPU only). Given one or + more numeric series, with optional covariates, it returns quantile forecasts + through POST /v1/forecast. Full precision (F32). The CPU engine is slower than + the PyTorch reference at long context lengths. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-123m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-123M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + files: + - filename: timestral-123m-f32.gguf + sha256: 90a2cad5bff9ac6ef695689b03d459df1cdfb5bb140755da7b99d01cfa0cbeb4 + uri: huggingface://mudler/timestral-cpp-gguf/123M/timestral-123m-f32.gguf + - filename: LICENSE-timestral-123m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/123M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-123m-f32.gguf + variants: + - model: timestral-123m-f16 +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: timestral-123m-f16 + description: | + Timestral 123M (123 million parameters) time-series forecaster by Mistral AI, + half precision (F16) build, about half the size of the F32 build. Served by the + native timestral.cpp backend (C++/ggml, CPU only) through POST /v1/forecast. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-123m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-123M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + files: + - filename: timestral-123m-f16.gguf + sha256: cf3e7eb0ab9581407d9edc2f4b2d1b8dda8c3a39674dbf3b6967971f906c54b0 + uri: huggingface://mudler/timestral-cpp-gguf/123M/timestral-123m-f16.gguf + - filename: LICENSE-timestral-123m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/123M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-123m-f16.gguf + variants: [] +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: timestral-209m + description: | + Timestral 209M (209 million parameters) time-series forecaster by Mistral AI, + served by the native timestral.cpp backend (C++/ggml, CPU only). Given one or + more numeric series, with optional covariates, it returns quantile forecasts + through POST /v1/forecast. Full precision (F32). The CPU engine is slower than + the PyTorch reference at long context lengths. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-209m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-209M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + files: + - filename: timestral-209m-f32.gguf + sha256: bf7605545245a2dd4d32f7ac2f7bbe5c2a12d8f47370814a254862150255b8e8 + uri: huggingface://mudler/timestral-cpp-gguf/209M/timestral-209m-f32.gguf + - filename: LICENSE-timestral-209m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/209M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-209m-f32.gguf + variants: + - model: timestral-209m-f16 +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: timestral-209m-f16 + description: | + Timestral 209M (209 million parameters) time-series forecaster by Mistral AI, + half precision (F16) build, about half the size of the F32 build. Served by the + native timestral.cpp backend (C++/ggml, CPU only) through POST /v1/forecast. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-209m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-209M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + files: + - filename: timestral-209m-f16.gguf + sha256: 24242f55b330a3794649700f6fadd7e59de320aabb42fe377d8df4ab811fb94f + uri: huggingface://mudler/timestral-cpp-gguf/209M/timestral-209m-f16.gguf + - filename: LICENSE-timestral-209m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/209M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-209m-f16.gguf + variants: [] - url: github:mudler/LocalAI/gallery/virtual.yaml@master name: rfdetr-cpp-base description: | diff --git a/gallery/models/index.yaml b/gallery/models/index.yaml index 232e04cef..4ff2c82ef 100644 --- a/gallery/models/index.yaml +++ b/gallery/models/index.yaml @@ -37719,6 +37719,390 @@ parameters: model: depth-anything2-metric-vkitti-large-q4_k.gguf variants: [] +- kind: model + id: timestral-5m + name: timestral-5m + version: 0.0.0-gallery.a144cd3eed19 + description: | + Timestral 5M (5 million parameters) time-series forecaster by Mistral AI, + served by the native timestral.cpp backend (C++/ggml, CPU only). Given one or + more numeric series, with optional covariates, it returns quantile forecasts + through POST /v1/forecast. Full precision (F32). The CPU engine is slower than + the PyTorch reference at long context lengths. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-5m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-5M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: timestral-5m-f32.gguf + sha256: c320276d8db95fb72fe5f4e043147e29142a1bf197c1b45daf6736780f6736ed + uri: huggingface://mudler/timestral-cpp-gguf/5M/timestral-5m-f32.gguf + - filename: LICENSE-timestral-5m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/5M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-5m-f32.gguf + # size is left unset: the memory each build needs has not been measured yet. + variants: + - model: timestral-5m-f16 +- kind: model + id: timestral-5m-f16 + name: timestral-5m-f16 + version: 0.0.0-gallery.9c5bcf33c573 + description: | + Timestral 5M (5 million parameters) time-series forecaster by Mistral AI, + half precision (F16) build, about half the size of the F32 build. Served by the + native timestral.cpp backend (C++/ggml, CPU only) through POST /v1/forecast. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-5m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-5M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: timestral-5m-f16.gguf + sha256: 933e2e518300e97e7799deb1920dd4e539da89a90c743b663cda772ee79d4a38 + uri: huggingface://mudler/timestral-cpp-gguf/5M/timestral-5m-f16.gguf + - filename: LICENSE-timestral-5m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/5M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-5m-f16.gguf + variants: [] +- kind: model + id: timestral-28m + name: timestral-28m + version: 0.0.0-gallery.25b334874a14 + description: | + Timestral 28M (28 million parameters) time-series forecaster by Mistral AI, + served by the native timestral.cpp backend (C++/ggml, CPU only). Given one or + more numeric series, with optional covariates, it returns quantile forecasts + through POST /v1/forecast. Full precision (F32). The CPU engine is slower than + the PyTorch reference at long context lengths. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-28m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-28M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: timestral-28m-f32.gguf + sha256: fbb5a8dcb8b224a45b149dc802428beca20504e71f585d942b0530f67f46a89c + uri: huggingface://mudler/timestral-cpp-gguf/28M/timestral-28m-f32.gguf + - filename: LICENSE-timestral-28m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/28M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-28m-f32.gguf + # size is left unset: the memory each build needs has not been measured yet. + variants: + - model: timestral-28m-f16 +- kind: model + id: timestral-28m-f16 + name: timestral-28m-f16 + version: 0.0.0-gallery.d4ece96e4a5e + description: | + Timestral 28M (28 million parameters) time-series forecaster by Mistral AI, + half precision (F16) build, about half the size of the F32 build. Served by the + native timestral.cpp backend (C++/ggml, CPU only) through POST /v1/forecast. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-28m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-28M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: timestral-28m-f16.gguf + sha256: 64b5b836c54c2d3a83b428e6b9e44e2bbf668c60297245c97a0df160d2c5366d + uri: huggingface://mudler/timestral-cpp-gguf/28M/timestral-28m-f16.gguf + - filename: LICENSE-timestral-28m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/28M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-28m-f16.gguf + variants: [] +- kind: model + id: timestral-123m + name: timestral-123m + version: 0.0.0-gallery.cecc4077197c + description: | + Timestral 123M (123 million parameters) time-series forecaster by Mistral AI, + served by the native timestral.cpp backend (C++/ggml, CPU only). Given one or + more numeric series, with optional covariates, it returns quantile forecasts + through POST /v1/forecast. Full precision (F32). The CPU engine is slower than + the PyTorch reference at long context lengths. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-123m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-123M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: timestral-123m-f32.gguf + sha256: 90a2cad5bff9ac6ef695689b03d459df1cdfb5bb140755da7b99d01cfa0cbeb4 + uri: huggingface://mudler/timestral-cpp-gguf/123M/timestral-123m-f32.gguf + - filename: LICENSE-timestral-123m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/123M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-123m-f32.gguf + # size is left unset: the memory each build needs has not been measured yet. + variants: + - model: timestral-123m-f16 +- kind: model + id: timestral-123m-f16 + name: timestral-123m-f16 + version: 0.0.0-gallery.48d7c5fc448f + description: | + Timestral 123M (123 million parameters) time-series forecaster by Mistral AI, + half precision (F16) build, about half the size of the F32 build. Served by the + native timestral.cpp backend (C++/ggml, CPU only) through POST /v1/forecast. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-123m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-123M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: timestral-123m-f16.gguf + sha256: cf3e7eb0ab9581407d9edc2f4b2d1b8dda8c3a39674dbf3b6967971f906c54b0 + uri: huggingface://mudler/timestral-cpp-gguf/123M/timestral-123m-f16.gguf + - filename: LICENSE-timestral-123m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/123M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-123m-f16.gguf + variants: [] +- kind: model + id: timestral-209m + name: timestral-209m + version: 0.0.0-gallery.33c776bf9d5d + description: | + Timestral 209M (209 million parameters) time-series forecaster by Mistral AI, + served by the native timestral.cpp backend (C++/ggml, CPU only). Given one or + more numeric series, with optional covariates, it returns quantile forecasts + through POST /v1/forecast. Full precision (F32). The CPU engine is slower than + the PyTorch reference at long context lengths. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-209m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-209M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: timestral-209m-f32.gguf + sha256: bf7605545245a2dd4d32f7ac2f7bbe5c2a12d8f47370814a254862150255b8e8 + uri: huggingface://mudler/timestral-cpp-gguf/209M/timestral-209m-f32.gguf + - filename: LICENSE-timestral-209m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/209M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-209m-f32.gguf + # size is left unset: the memory each build needs has not been measured yet. + variants: + - model: timestral-209m-f16 +- kind: model + id: timestral-209m-f16 + name: timestral-209m-f16 + version: 0.0.0-gallery.5ceb3fb10ee8 + description: | + Timestral 209M (209 million parameters) time-series forecaster by Mistral AI, + half precision (F16) build, about half the size of the F32 build. Served by the + native timestral.cpp backend (C++/ggml, CPU only) through POST /v1/forecast. + + LICENSE WARNING: the weights are under Mistral's Modified MIT licence. You may + not use them if the global consolidated monthly revenue of your company (or + your employer) exceeded USD 20 million in the preceding month, unless you + get a commercial licence from Mistral AI. The restriction also covers + derivatives, so it applies to these GGUF conversions. The licence text is + installed next to the weights as LICENSE-timestral-209m.txt. + license: mistral-modified-mit + urls: + - https://github.com/mudler/timestral.cpp + - https://huggingface.co/mistralai/Timestral-209M + - https://huggingface.co/mudler/timestral-cpp-gguf + tags: + - forecasting + - time-series + - timestral + - native + - cpp + - cpu + - revenue-restricted-license + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: timestral-209m-f16.gguf + sha256: 24242f55b330a3794649700f6fadd7e59de320aabb42fe377d8df4ab811fb94f + uri: huggingface://mudler/timestral-cpp-gguf/209M/timestral-209m-f16.gguf + - filename: LICENSE-timestral-209m.txt + sha256: 648194727930df730890e9f9affdbcc5c930d406d6829559b25ca3d5a973fc8c + uri: huggingface://mudler/timestral-cpp-gguf/209M/LICENSE + overrides: + backend: timestral-cpp + parameters: + model: timestral-209m-f16.gguf + variants: [] - kind: model id: rfdetr-cpp-base name: rfdetr-cpp-base diff --git a/pkg/grpc/backend.go b/pkg/grpc/backend.go index 7ffa8384b..1bca9bc40 100644 --- a/pkg/grpc/backend.go +++ b/pkg/grpc/backend.go @@ -83,6 +83,7 @@ type InferenceBackend interface { AudioTranscriptionStream(ctx context.Context, in *pb.TranscriptRequest, f func(chunk *pb.TranscriptStreamResponse), opts ...grpc.CallOption) error Detect(ctx context.Context, in *pb.DetectOptions, opts ...grpc.CallOption) (*pb.DetectResponse, error) Depth(ctx context.Context, in *pb.DepthRequest, opts ...grpc.CallOption) (*pb.DepthResponse, error) + Forecast(ctx context.Context, in *pb.ForecastRequest, opts ...grpc.CallOption) (*pb.ForecastResponse, error) FaceVerify(ctx context.Context, in *pb.FaceVerifyRequest, opts ...grpc.CallOption) (*pb.FaceVerifyResponse, error) FaceAnalyze(ctx context.Context, in *pb.FaceAnalyzeRequest, opts ...grpc.CallOption) (*pb.FaceAnalyzeResponse, error) VoiceVerify(ctx context.Context, in *pb.VoiceVerifyRequest, opts ...grpc.CallOption) (*pb.VoiceVerifyResponse, error) diff --git a/pkg/grpc/base/base.go b/pkg/grpc/base/base.go index 2c8f52329..82f32a59d 100644 --- a/pkg/grpc/base/base.go +++ b/pkg/grpc/base/base.go @@ -99,6 +99,10 @@ func (llm *Base) Depth(*pb.DepthRequest) (pb.DepthResponse, error) { return pb.DepthResponse{}, fmt.Errorf("unimplemented") } +func (llm *Base) Forecast(*pb.ForecastRequest) (pb.ForecastResponse, error) { + return pb.ForecastResponse{}, fmt.Errorf("unimplemented") +} + func (llm *Base) FaceVerify(*pb.FaceVerifyRequest) (pb.FaceVerifyResponse, error) { return pb.FaceVerifyResponse{}, fmt.Errorf("unimplemented") } diff --git a/pkg/grpc/client.go b/pkg/grpc/client.go index f73da7382..fa4f63dcb 100644 --- a/pkg/grpc/client.go +++ b/pkg/grpc/client.go @@ -735,6 +735,23 @@ func (c *Client) Depth(ctx context.Context, in *pb.DepthRequest, opts ...grpc.Ca return client.Depth(ctx, in, opts...) } +func (c *Client) Forecast(ctx context.Context, in *pb.ForecastRequest, opts ...grpc.CallOption) (*pb.ForecastResponse, error) { + if !c.parallel { + c.opMutex.Lock() + defer c.opMutex.Unlock() + } + c.setBusy(true) + defer c.setBusy(false) + defer c.wdMark()() + conn, err := c.dial() + if err != nil { + return nil, err + } + defer func() { _ = conn.Close() }() + client := pb.NewBackendClient(conn) + return client.Forecast(ctx, in, opts...) +} + func (c *Client) FaceVerify(ctx context.Context, in *pb.FaceVerifyRequest, opts ...grpc.CallOption) (*pb.FaceVerifyResponse, error) { if !c.parallel { c.opMutex.Lock() diff --git a/pkg/grpc/embed.go b/pkg/grpc/embed.go index fab97e4f7..6d735aef1 100644 --- a/pkg/grpc/embed.go +++ b/pkg/grpc/embed.go @@ -87,6 +87,10 @@ func (e *embedBackend) Depth(ctx context.Context, in *pb.DepthRequest, opts ...g return e.s.Depth(ctx, in) } +func (e *embedBackend) Forecast(ctx context.Context, in *pb.ForecastRequest, opts ...grpc.CallOption) (*pb.ForecastResponse, error) { + return e.s.Forecast(ctx, in) +} + func (e *embedBackend) FaceVerify(ctx context.Context, in *pb.FaceVerifyRequest, opts ...grpc.CallOption) (*pb.FaceVerifyResponse, error) { return e.s.FaceVerify(ctx, in) } diff --git a/pkg/grpc/interface.go b/pkg/grpc/interface.go index 0e15734b4..f29f0e541 100644 --- a/pkg/grpc/interface.go +++ b/pkg/grpc/interface.go @@ -30,6 +30,7 @@ type AIModel interface { Animate3D(*pb.Animate3DRequest) error Detect(*pb.DetectOptions) (pb.DetectResponse, error) Depth(*pb.DepthRequest) (pb.DepthResponse, error) + Forecast(*pb.ForecastRequest) (pb.ForecastResponse, error) FaceVerify(*pb.FaceVerifyRequest) (pb.FaceVerifyResponse, error) FaceAnalyze(*pb.FaceAnalyzeRequest) (pb.FaceAnalyzeResponse, error) VoiceVerify(*pb.VoiceVerifyRequest) (pb.VoiceVerifyResponse, error) diff --git a/pkg/grpc/model_identity_modalities_test.go b/pkg/grpc/model_identity_modalities_test.go index a166c77bf..62d10a03b 100644 --- a/pkg/grpc/model_identity_modalities_test.go +++ b/pkg/grpc/model_identity_modalities_test.go @@ -62,6 +62,11 @@ func (b *modalityBackend) Depth(*pb.DepthRequest) (pb.DepthResponse, error) { return pb.DepthResponse{}, nil } +func (b *modalityBackend) Forecast(*pb.ForecastRequest) (pb.ForecastResponse, error) { + b.served++ + return pb.ForecastResponse{}, nil +} + func (b *modalityBackend) FaceVerify(*pb.FaceVerifyRequest) (pb.FaceVerifyResponse, error) { b.served++ return pb.FaceVerifyResponse{}, nil @@ -133,6 +138,7 @@ func callAllModalities(c Backend, identity string) map[string]error { errs["AudioTranscriptionStream"] = c.AudioTranscriptionStream(ctx, &pb.TranscriptRequest{ModelIdentity: identity}, func(*pb.TranscriptStreamResponse) {}) _, errs["Detect"] = c.Detect(ctx, &pb.DetectOptions{ModelIdentity: identity}) _, errs["Depth"] = c.Depth(ctx, &pb.DepthRequest{ModelIdentity: identity}) + _, errs["Forecast"] = c.Forecast(ctx, &pb.ForecastRequest{ModelIdentity: identity}) _, errs["FaceVerify"] = c.FaceVerify(ctx, &pb.FaceVerifyRequest{ModelIdentity: identity}) _, errs["FaceAnalyze"] = c.FaceAnalyze(ctx, &pb.FaceAnalyzeRequest{ModelIdentity: identity}) _, errs["VoiceVerify"] = c.VoiceVerify(ctx, &pb.VoiceVerifyRequest{ModelIdentity: identity}) diff --git a/pkg/grpc/server.go b/pkg/grpc/server.go index 4608ae748..0b06c54b6 100644 --- a/pkg/grpc/server.go +++ b/pkg/grpc/server.go @@ -345,6 +345,21 @@ func (s *server) Depth(ctx context.Context, in *pb.DepthRequest) (*pb.DepthRespo return &res, nil } +func (s *server) Forecast(ctx context.Context, in *pb.ForecastRequest) (*pb.ForecastResponse, error) { + if err := s.checkModelIdentity(in); err != nil { + return nil, err + } + if s.llm.Locking() { + s.llm.Lock() + defer s.llm.Unlock() + } + res, err := s.llm.Forecast(in) + if err != nil { + return nil, err + } + return &res, nil +} + func (s *server) FaceVerify(ctx context.Context, in *pb.FaceVerifyRequest) (*pb.FaceVerifyResponse, error) { if err := s.checkModelIdentity(in); err != nil { return nil, err diff --git a/pkg/model/connection_evicting_client.go b/pkg/model/connection_evicting_client.go index 92b4d6441..d2c11ae02 100644 --- a/pkg/model/connection_evicting_client.go +++ b/pkg/model/connection_evicting_client.go @@ -127,6 +127,12 @@ func (c *ConnectionEvictingClient) Depth(ctx context.Context, in *pb.DepthReques return result, err } +func (c *ConnectionEvictingClient) Forecast(ctx context.Context, in *pb.ForecastRequest, opts ...ggrpc.CallOption) (*pb.ForecastResponse, error) { + result, err := c.Backend.Forecast(ctx, in, opts...) + c.checkErr(err) + return result, err +} + func (c *ConnectionEvictingClient) Rerank(ctx context.Context, in *pb.RerankRequest, opts ...ggrpc.CallOption) (*pb.RerankResult, error) { result, err := c.Backend.Rerank(ctx, in, opts...) c.checkErr(err) diff --git a/scripts/lib/backend-filter.mjs b/scripts/lib/backend-filter.mjs index 069433d30..5511e1bc7 100644 --- a/scripts/lib/backend-filter.mjs +++ b/scripts/lib/backend-filter.mjs @@ -19,6 +19,13 @@ export function inferBackendPath(item) { if (item.backend === "parakeet-cpp") { return `backend/go/parakeet-cpp/`; } + // timestral-cpp is a Go backend (Dockerfile.golang) wrapping the timestral.cpp + // ggml port via purego, living in backend/go/timestral-cpp/. Same + // explicit-branch rationale as parakeet-cpp: the generic golang fallthrough + // would also resolve it, but this documents the mapping. + if (item.backend === "timestral-cpp") { + return `backend/go/timestral-cpp/`; + } // ced is a Go backend (Dockerfile.golang) wrapping the ced.cpp ggml port via // purego, living in backend/go/ced/. Same explicit-branch rationale as // parakeet-cpp above: the generic golang fallthrough would also resolve it, diff --git a/swagger/docs.go b/swagger/docs.go index 56abb07f3..c682fdcb0 100644 --- a/swagger/docs.go +++ b/swagger/docs.go @@ -7060,6 +7060,33 @@ const docTemplate = `{ } } }, + "/v1/forecast": { + "post": { + "tags": [ + "forecast" + ], + "summary": "Forecasts numeric time series as quantile trajectories.", + "parameters": [ + { + "description": "query params", + "name": "request", + "in": "body", + "required": true, + "schema": { + "$ref": "#/definitions/schema.ForecastRequest" + } + } + ], + "responses": { + "200": { + "description": "Response", + "schema": { + "$ref": "#/definitions/schema.ForecastResponse" + } + } + } + } + }, "/v1/images/generations": { "post": { "tags": [ @@ -12225,6 +12252,170 @@ const docTemplate = `{ } } }, + "schema.ForecastRequest": { + "type": "object", + "properties": { + "clip_nonnegative": { + "description": "null = model default", + "type": "boolean" + }, + "max_context": { + "description": "cap on history used; 0 = model default", + "type": "integer" + }, + "model": { + "type": "string" + }, + "prediction_length": { + "description": "steps to forecast", + "type": "integer" + }, + "quantiles": { + "description": "levels to return; default 0.1, 0.5, 0.9", + "type": "array", + "items": { + "type": "number" + } + }, + "series": { + "type": "array", + "items": { + "$ref": "#/definitions/schema.ForecastSeries" + } + }, + "symmetric_averaging": { + "description": "null = model default", + "type": "boolean" + } + } + }, + "schema.ForecastResponse": { + "type": "object", + "properties": { + "data": { + "type": "array", + "items": { + "$ref": "#/definitions/schema.ForecastSeriesResult" + } + }, + "model": { + "type": "string" + }, + "object": { + "type": "string" + }, + "prediction_length": { + "type": "integer" + }, + "quantile_levels": { + "type": "array", + "items": { + "type": "number" + } + } + } + }, + "schema.ForecastSeries": { + "type": "object", + "properties": { + "id": { + "description": "echoed back in the matching result", + "type": "string" + }, + "known_categorical": { + "description": "KnownCategorical is [C][T+H]: integer category codes for the future too.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + }, + "known_covariates": { + "description": "KnownCovariates is [C][T+H]: numeric covariates known for the future too.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + }, + "past_categorical": { + "description": "PastCategorical is [C][T]: integer category codes for the past.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + }, + "past_covariates": { + "description": "PastCovariates is [C][T]: numeric covariates known only for the past.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + }, + "target": { + "description": "[V][T] or [T], null = missing", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + } + } + }, + "schema.ForecastSeriesResult": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "index": { + "description": "position of the series in the request", + "type": "integer" + }, + "variates": { + "type": "array", + "items": { + "$ref": "#/definitions/schema.ForecastVariate" + } + } + } + }, + "schema.ForecastVariate": { + "type": "object", + "properties": { + "index": { + "type": "integer" + }, + "values": { + "description": "Values is [Q][H]: Values[q][h] is quantile_levels[q] at step h.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float32" + } + } + } + } + }, "schema.FunctionCall": { "type": "object", "properties": { diff --git a/swagger/swagger.json b/swagger/swagger.json index b7313248c..d2665ae6d 100644 --- a/swagger/swagger.json +++ b/swagger/swagger.json @@ -7057,6 +7057,33 @@ } } }, + "/v1/forecast": { + "post": { + "tags": [ + "forecast" + ], + "summary": "Forecasts numeric time series as quantile trajectories.", + "parameters": [ + { + "description": "query params", + "name": "request", + "in": "body", + "required": true, + "schema": { + "$ref": "#/definitions/schema.ForecastRequest" + } + } + ], + "responses": { + "200": { + "description": "Response", + "schema": { + "$ref": "#/definitions/schema.ForecastResponse" + } + } + } + } + }, "/v1/images/generations": { "post": { "tags": [ @@ -12222,6 +12249,170 @@ } } }, + "schema.ForecastRequest": { + "type": "object", + "properties": { + "clip_nonnegative": { + "description": "null = model default", + "type": "boolean" + }, + "max_context": { + "description": "cap on history used; 0 = model default", + "type": "integer" + }, + "model": { + "type": "string" + }, + "prediction_length": { + "description": "steps to forecast", + "type": "integer" + }, + "quantiles": { + "description": "levels to return; default 0.1, 0.5, 0.9", + "type": "array", + "items": { + "type": "number" + } + }, + "series": { + "type": "array", + "items": { + "$ref": "#/definitions/schema.ForecastSeries" + } + }, + "symmetric_averaging": { + "description": "null = model default", + "type": "boolean" + } + } + }, + "schema.ForecastResponse": { + "type": "object", + "properties": { + "data": { + "type": "array", + "items": { + "$ref": "#/definitions/schema.ForecastSeriesResult" + } + }, + "model": { + "type": "string" + }, + "object": { + "type": "string" + }, + "prediction_length": { + "type": "integer" + }, + "quantile_levels": { + "type": "array", + "items": { + "type": "number" + } + } + } + }, + "schema.ForecastSeries": { + "type": "object", + "properties": { + "id": { + "description": "echoed back in the matching result", + "type": "string" + }, + "known_categorical": { + "description": "KnownCategorical is [C][T+H]: integer category codes for the future too.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + }, + "known_covariates": { + "description": "KnownCovariates is [C][T+H]: numeric covariates known for the future too.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + }, + "past_categorical": { + "description": "PastCategorical is [C][T]: integer category codes for the past.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + }, + "past_covariates": { + "description": "PastCovariates is [C][T]: numeric covariates known only for the past.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + }, + "target": { + "description": "[V][T] or [T], null = missing", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float64" + } + } + } + } + }, + "schema.ForecastSeriesResult": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "index": { + "description": "position of the series in the request", + "type": "integer" + }, + "variates": { + "type": "array", + "items": { + "$ref": "#/definitions/schema.ForecastVariate" + } + } + } + }, + "schema.ForecastVariate": { + "type": "object", + "properties": { + "index": { + "type": "integer" + }, + "values": { + "description": "Values is [Q][H]: Values[q][h] is quantile_levels[q] at step h.", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number", + "format": "float32" + } + } + } + } + }, "schema.FunctionCall": { "type": "object", "properties": { diff --git a/swagger/swagger.yaml b/swagger/swagger.yaml index c75e7169c..77b4c6541 100644 --- a/swagger/swagger.yaml +++ b/swagger/swagger.yaml @@ -3156,6 +3156,124 @@ definitions: "y": type: number type: object + schema.ForecastRequest: + properties: + clip_nonnegative: + description: null = model default + type: boolean + max_context: + description: cap on history used; 0 = model default + type: integer + model: + type: string + prediction_length: + description: steps to forecast + type: integer + quantiles: + description: levels to return; default 0.1, 0.5, 0.9 + items: + type: number + type: array + series: + items: + $ref: '#/definitions/schema.ForecastSeries' + type: array + symmetric_averaging: + description: null = model default + type: boolean + type: object + schema.ForecastResponse: + properties: + data: + items: + $ref: '#/definitions/schema.ForecastSeriesResult' + type: array + model: + type: string + object: + type: string + prediction_length: + type: integer + quantile_levels: + items: + type: number + type: array + type: object + schema.ForecastSeries: + properties: + id: + description: echoed back in the matching result + type: string + known_categorical: + description: 'KnownCategorical is [C][T+H]: integer category codes for the + future too.' + items: + items: + format: float64 + type: number + type: array + type: array + known_covariates: + description: 'KnownCovariates is [C][T+H]: numeric covariates known for the + future too.' + items: + items: + format: float64 + type: number + type: array + type: array + past_categorical: + description: 'PastCategorical is [C][T]: integer category codes for the past.' + items: + items: + format: float64 + type: number + type: array + type: array + past_covariates: + description: 'PastCovariates is [C][T]: numeric covariates known only for + the past.' + items: + items: + format: float64 + type: number + type: array + type: array + target: + description: '[V][T] or [T], null = missing' + items: + items: + format: float64 + type: number + type: array + type: array + type: object + schema.ForecastSeriesResult: + properties: + id: + type: string + index: + description: position of the series in the request + type: integer + variates: + items: + $ref: '#/definitions/schema.ForecastVariate' + type: array + type: object + schema.ForecastVariate: + properties: + index: + type: integer + values: + description: 'Values is [Q][H]: Values[q][h] is quantile_levels[q] at step + h.' + items: + items: + format: float32 + type: number + type: array + type: array + type: object schema.FunctionCall: properties: arguments: @@ -10604,6 +10722,23 @@ paths: summary: Verify that two images depict the same person. tags: - face-recognition + /v1/forecast: + post: + parameters: + - description: query params + in: body + name: request + required: true + schema: + $ref: '#/definitions/schema.ForecastRequest' + responses: + "200": + description: Response + schema: + $ref: '#/definitions/schema.ForecastResponse' + summary: Forecasts numeric time series as quantile trajectories. + tags: + - forecast /v1/images/generations: post: parameters: