* fix(sglang): keep nvidia-modelopt on a stable release
Every cublas sglang image currently fails to build:
Failed to build `nvidia-modelopt==0.46.0rc0`
Call to `wheel_stub.buildapi.build_wheel` failed
ModuleNotFoundError: No module named 'wheel_stub'
sglang[all] pulls nvidia-modelopt in through its `diffusion` extra with no
version bound of its own, and install.sh adds a GLOBAL --prerelease=allow so
that flash-attn-4, which only ships 4.0.0b* wheels, can resolve. Unbounded plus
prereleases-allowed picks 0.46.0rc0, whose build backend imports wheel_stub
without declaring it in build-system.requires. EXTRA_PIP_INSTALL_FLAGS also
starts with --no-build-isolation, so nothing installs wheel_stub and the build
dies. Latest stable is 0.45.0 and resolves cleanly.
Bounding this one package rather than dropping the global flag, because the
flag is load-bearing for flash-attn-4 and this is the narrower change with the
smaller blast radius. Raise the bound when 0.46.0 final ships.
This is invisible on master because the backend build is path-filtered: sglang
is only rebuilt when sglang changes. It surfaces on any PR touching a shared
build input such as backend/backend.proto, which rebuilds the whole matrix.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(nemo): build the darwin venv on Python 3.12
The darwin nemo image fails to build:
ModuleNotFoundError: No module named 'maturin'
nemo_toolkit pulls in text2num, a Rust extension built with maturin, whose
macOS arm64 wheels start at cp311: 3.0.2 publishes cp311, cp312, cp313 and
cp314 and no cp310. libbackend.sh defaults PYTHON_VERSION to 3.10, so pip finds
no wheel, falls back to the sdist, and dies in the PEP 517 hook because
EXTRA_PIP_INSTALL_FLAGS carries --no-build-isolation and nothing installs the
build backend. Taking the prebuilt wheel avoids the source build entirely, so
the runner needs no Rust toolchain.
Darwin only, deliberately: the Linux profiles resolve a cp310 manylinux wheel
for the same package and have no reason to move. The override is set after
libbackend.sh is sourced and before installRequirements, the same shape
sglang's install.sh already uses for its l4t13 profile.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(nemo): pin the darwin portable-Python patch level too
The 3.12 bump alone traded one failure for another:
curl: (56) The requested URL returned error: 404
make[1]: *** [nemo-asr] Error 56
libbackend builds the portable-Python URL from
cpython-${PYTHON_VERSION}.${PYTHON_PATCH}+${PY_STANDALONE_TAG}, and
PYTHON_PATCH defaults to 18 because the default interpreter is 3.10.18. Setting
only PYTHON_VERSION asked for a 3.12.18 that was never released.
Patch 11, not the 12 that sglang/install.sh pairs with 3.12 for l4t13: at the
20250818 tag python-build-standalone published 3.12.12 for linux aarch64 but
not for aarch64-apple-darwin, where 3.12.11 is the newest. Both URLs were
checked against the release assets rather than assumed to match.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* feat(nemo): enable word-level timestamps for ASR models
The nemo backend ignored timestamp_granularities and always returned a
single segment with start=0 end=0, making word-level timestamps
impossible to obtain even though the NeMo models (parakeet-tdt, etc.)
fully support them.
Changes:
- Add _get_stride_seconds() to compute frame duration from the model's
preprocessor window_stride and encoder subsampling_factor.
- Add _build_segments_with_words() that extracts word offsets from the
NeMo Hypothesis.timestamp dict and converts frame indices to
nanosecond timestamps.
- Support 'word' granularity (one segment per word) and 'segment'
granularity (merge at time-gap boundaries using a dynamic threshold).
- Populate TranscriptSegment.words with TranscriptWord entries so
callers get both segment-level and word-level timing.
- Only request timestamps from NeMo when the caller actually asks for
them (timestamp_granularities is non-empty), keeping the fast path
unchanged for callers that don't need timestamps.
Tested with nvidia/parakeet-tdt-0.6b-v3 on the JFK "ask not" clip:
curl -X POST /v1/audio/transcriptions \
-F file=@jfk.wav -F model=nemo-parakeet-tdt-0.6b \
-F 'timestamp_granularities[]=word' -F response_format=verbose_json
→ each word has correct start/end times in seconds.
Signed-off-by: fqscfqj <fqscfqj@outlook.com>
* fix(nemo): address Copilot review feedback
- Narrow exception handling in _get_stride_seconds to catch only
AttributeError, KeyError, TypeError instead of bare Exception, and
emit a warning when falling back to the hardcoded stride.
- Remove explicit return_hypotheses=False when timestamps are requested;
timestamps=True already forces NeMo to return Hypothesis objects.
- Add a warning when NeMo does not return Hypothesis objects despite
timestamps being requested.
Signed-off-by: fqscfqj <fqscfqj@outlook.com>
---------
Signed-off-by: fqscfqj <fqscfqj@outlook.com>
Pin texterrors==1.1.6 before nemo_toolkit[asr] in requirements-cublas13.txt.
The texterrors package (a NeMo transitive dependency) contains a compiled
C++ extension (texterrors_align.so) that may be built from source during
OCI image creation. When built on systems with GCC 14+ (e.g. Ubuntu 24.04),
the resulting binary requires GLIBCXX_3.4.32, which is not available in
the default LocalAI container (Ubuntu 22.04, GLIBCXX up to 3.4.30).
Pinning to 1.1.6 (the latest release) ensures:
- Reproducible builds across environments
- pip resolves the pre-built manylinux2014 wheel (needs only GLIBCXX_3.4.11)
instead of potentially building from source with a newer toolchain
Fixes#10056
Signed-off-by: 番茄摔成番茄酱 <fqscfqj@outlook.com>
* fix(nemo): extract Hypothesis.text for TDT/RNNT ASR models
CTC models (e.g. Whisper) return List[str] from transcribe(), but
TDT/RNNT models (e.g. parakeet-tdt-0.6b-v3) return List[Hypothesis]
where the decoded text lives in the Hypothesis.text attribute.
Previously, results[0] was assigned directly to the protobuf string
field, causing silent empty output for non-CTC models.
Now checks the return type and extracts .text from Hypothesis objects,
with a safe fallback via getattr().
* refactor: simplify Hypothesis text extraction per Copilot review
Use single getattr() call instead of hasattr() + double access,
and return empty string for unknown types instead of str(result)
to avoid leaking internal repr to clients.
* feat: add distributed mode (experimental)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix data races, mutexes, transactions
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix events and tool stream in agent chat
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* use ginkgo
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(cron): compute correctly time boundaries avoiding re-triggering
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* enhancements, refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not flood of healthy checks
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not list obvious backends as text backends
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* tests fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Drop redundant healthcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* enhancements, refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
