Compare commits

..
Author SHA1 Message Date
Evan efa0da0912 wah 2026-04-22 12:51:10 +01:00
Evan Quiney df332035ef swap camelcasemodels for frozenmodels globally (#1957) 2026-04-22 11:49:25 +00:00
ciaranbor af673845d3 Ignore HF remote repo changes (temporary fix) (#1958)
## Motivation

Fixes #1918. Downloaded model status reverts from "completed" to
"pending" during each download scan. Reproduced with `zai-org/GLM-5.1`.

## Changes

- `coordinator.py`: In the periodic rescan, don't downgrade
already-completed models; fall back to `resolve_existing_model()`
(safetensors weight check) when per-file size check reports incomplete
- New `test_download_status_not_lost.py`: 3 regression tests

## Why It Works

The rescan compares local file sizes against HF's `main` revision. When
HF updates text files (README, jinja, etc.), remote sizes change but
local files still match the old revision — causing a false "incomplete".
The fix uses the safetensors weight check as ground truth instead.

Long-term: pin the downloaded revision SHA rather than always checking
against `main`.

## Test Plan

### Manual Testing

- Mac Studio M3 Ultra with GLM-5.1 downloaded (natural reproduction of
the issue)
- Confirmed GLM-5.1 stays `DownloadCompleted` through multiple rescan
cycles

### Automated Testing

- 3 new tests: completed-not-downgraded, fallback-to-resolve,
genuinely-incomplete-stays-pending
2026-04-22 11:11:01 +01:00
ciaranbor 49670c8624 Handle missing total_size in safetensors index files (#1956)
## Motivation

Image models fail to load after a mid-download instance deletion and
recreation. The system skips the download and crashes with
`FileNotFoundError: No safetensors files found in .../vae`.

## Changes

- Make `ModelSafetensorsIndexMetadata.total_size` optional (`PositiveInt
| None = None`)
- Add null guard in `fetch_safetensors_size`
- Add regression test

## Why It Works

Exolabs quantized image models have safetensors index files with mflux
metadata (`quantization_level`, `mflux_version`) but no `total_size`.
The required `PositiveInt` field caused Pydantic validation to fail,
which was silently swallowed by `except Exception: continue` in
`_scan_model_directory`. This skipped all weight map checks, making
incomplete models appear complete.

## Test Plan

### Manual Testing

- Hardware: Mac Studio
- Before: `CreateRunner → LoadModel` (crash). After: `CreateRunner →
DownloadModel` (correct).

### Automated Testing

- `test_safetensors_index.py`: 3 cases covering missing, valid, and null
metadata
2026-04-21 16:39:14 +01:00
rltakashige fcc3718efb Add sampling defaults (#1947)
## Motivation

Model quality issues

### Manual Testing
TODO
2026-04-21 06:45:33 +00:00
rltakashige 8ccfd7fcb6 Fix some misc build issues (#1948)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-21 07:41:07 +01:00
Evan Quiney 7b416155de bump uv lock for linux builds (#1942)
followup from a rebase issue in #1874
2026-04-20 11:41:40 +00:00
Evan Quiney 93a24748e6 mlx cuda 13 (dgx spark) support (#1874) 2026-04-20 11:49:12 +01:00
Evan Quiney e32829e51d chore: bump versions in line with release (#1941) 2026-04-20 09:39:28 +00:00
rltakashige 09e894dd52 Fix vision models on M5 Pro/Max MacBooks (#1927)
## Motivation

Vision models don't understand images on M5 series MacBooks. The
upstream NAX addmm fix (https://github.com/ml-explore/mlx/pull/3422)
fixes this.

## Why It Works
Same conclusion I came to when I was debugging the issue on an M5 Max.
It works after this fix.

## Test Plan

### Manual Testing
Works for Qwen3.5 27B
2026-04-19 12:55:20 +01:00
rltakashige bf8aacfd41 Improve build CI (#1920)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-17 20:55:15 +00:00
af9e847edb fix: force gc + clear_cache after KV prefix cache eviction (#1832)
## Summary
- After `KVPrefixCache` evicts LRU entries, the MLX Metal buffers stay
allocated until Python's GC runs
- This leaks ~3-4 GB between long-context requests, reducing the
effective context ceiling for back-to-back requests
- Adding `gc.collect()` + `mx.clear_cache()` after eviction frees Metal
buffers promptly

## Test plan
- [x] Measured on 2-node PP cluster with Qwen3.5-397B-A17B-4bit at 63K
context
- [x] Before: 108.88 GB retained after eviction (3.78 GB above baseline)
- [x] After: 105.48 GB retained after eviction (0.38 GB above baseline —
draft model KV + minor overhead)
- [x] `gc.collect()` adds ~2-3ms latency, runs once per eviction cycle
(not per token)
- [ ] Verify with `uv run pytest`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Adam Durham <adam@example.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
2026-04-17 17:57:32 +00:00
mlpy0 01598960bd Add model card for Qwen3.6-35B-A3B-8bit (#1917)
Adds the 8bit variant missing from #1907 — the safetensors index is now
live on HF.

- `mlx-community/Qwen3.6-35B-A3B-8bit` (~35 GB)

Architectural fields match the existing 4bit/5bit/bf16 cards.
`storage_size.in_bytes` is taken from `metadata.total_size` of the
upstream `model.safetensors.index.json`.
2026-04-17 10:06:23 +00:00
63b8e64715 Add model cards for Qwen3.6-35B-A3B variants (#1907)
## Motivation

`mlx-community` has just published the new **Qwen3.6-35B-A3B**
multimodal MoE family on HuggingFace. Without static model cards exo
doesn't surface these models in the dashboard picker or match its
placement / prefill logic, so users can't one-click launch them. This PR
adds cards for the three quants whose safetensors indexes are already
live on HF (4bit / 5bit / bf16).

## Changes

Three new TOML files in `resources/inference_model_cards/`:

- `mlx-community--Qwen3.6-35B-A3B-4bit.toml` (~19 GB)
- `mlx-community--Qwen3.6-35B-A3B-5bit.toml` (~23 GB)
- `mlx-community--Qwen3.6-35B-A3B-bf16.toml` (~65 GB)

All three share the same architectural fields (`n_layers = 40`,
`hidden_size = 2048`, `num_key_value_heads = 2`, `context_length =
262144`, capabilities `text, thinking, thinking_toggle, vision`,
`base_model = "Qwen3.6 35B A3B"`) — only `model_id`, `quantization`, and
`storage_size.in_bytes` differ between variants.

## Why It Works

- Qwen3.6-35B-A3B reuses the `qwen3_5_moe` architecture
(`Qwen3_5MoeForConditionalGeneration`) — the same one already wired into
exo's MLX runner at `src/exo/worker/engines/mlx/auto_parallel.py:47` via
`Qwen3_5MoeModel`. The architectural fields are taken verbatim from the
HF `config.json.text_config` and match the existing `Qwen3.5-35B-A3B-*`
cards.
- Storage sizes are the exact `metadata.total_size` read from each
variant's `model.safetensors.index.json` on HF, so download progress and
cluster-memory-fit checks are accurate.
- Vision support is flagged in `capabilities`; the `[vision]` block is
auto-detected by `ModelCard._autodetect_vision` from the upstream
`config.json`, so no hand-written vision config is required.
- The card loader (`_refresh_card_cache` in
`src/exo/shared/models/model_cards.py`) globs every `.toml` in
`resources/inference_model_cards/` on startup, so nothing else needs to
change — the `/models` endpoint and the dashboard picker pick them up
automatically.

The `mxfp4` / `mxfp8` / `nvfp4` variants are still uploading upstream
(index JSONs currently 404) and can be added in a follow-up PR once HF
completes.

## Test Plan

### Manual Testing

Hardware: MacBook Pro M4 Max, 48 GB unified memory.

- Built the dashboard, ran `uv run exo`, waited for the API to come up
on `http://localhost:52415`.
- `curl -s http://localhost:52415/models` returns the three new model
ids (`mlx-community/Qwen3.6-35B-A3B-{4bit,5bit,bf16}`) alongside
existing models.
- Opened the dashboard, clicked SELECT MODEL, typed "Qwen3.6" into the
search box. A single **"Qwen3.6 35B A3B"** group appears showing `3
variants (19GB-65GB)`. Expanding it lists the `4bit` / `5bit` / `bf16`
quants with sizes `19GB` / `23GB` / `65GB`, exactly as expected:

![Qwen3.6 35B A3B in model
picker](https://gist.githubusercontent.com/AlexCheema/68c2c02da9450b44968e6b0e0b1d255e/raw/127119f70382353c65a847188e5a2c9013db68d2/qwen36-picker.png)

- Programmatically loaded each TOML via `ModelCard.load_from_path(...)`
and confirmed the parsed fields (layers / hidden / KV heads / context /
quant / base_model / caps / bytes) match what's written in the files.

### Automated Testing

No code paths were touched — these are pure TOML data files that plug
into the existing model-card loader. The existing pytest suite covers
TOML parsing and card serving; adding new TOMLs doesn't require new test
scaffolding. `uv run ruff check` and `nix fmt` are clean.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Ryuichi Leo Takashige <rl.takashige@gmail.com>
2026-04-16 23:25:26 +01:00
rltakashige 28c797846a Update mlx and mlx lm to latest (#1906)
Just bumping to the very latest upstream versions.
2026-04-16 10:59:33 +00:00
rltakashige 058bb08261 Allow copying on dashboard even on HTTP (#1902)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-15 23:30:12 +01:00
rltakashige 3eead80238 Better environment variables in MacOS app (#1901)
## Motivation

Closes #1858 

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-15 20:14:52 +00:00
rltakashige 87329c80ef Add usage stats to tool calls and handle multiple tool calls correctly (#1899)
## Motivation

Tool calls are usually not end tokens, so they didn't have usage stats.
2026-04-15 19:40:00 +01:00
rltakashige 8cdc833892 Drain tokens silently skipped in thinking parsing (#1898)
## Motivation
Closes #1882
2026-04-15 14:23:07 +00:00
rltakashige 2cd66ae4cf Fix out of order event idx causing fatal crashes (#1894)
## Motivation

<img width="828" height="373" alt="Screenshot 2026-04-14 at 22 56 52"
src="https://github.com/user-attachments/assets/f8f48c1d-68c5-4acc-a6de-9d180672da9d"
/>

if is_new_master=True, _elect_loop creates a new EventRouter before the
worker has receivers. Then, event router runs _run_ext_in and
buf.drain_indexed() will pick off events, even though
self.internal_outbound is not populated fully.

Finally, when the worker does try requesting events, the next event it
receives is not the first event, meaning the worker crashes.

## Changes

Start the event router after all the receivers are registered

## Why It Works

self.internal_outbound is populated before the loop begins.

## Test Plan

### Manual Testing
No more crashes observed in testing (it's actually quite easy to
reproduce the issue if you have one node with this fix but the other
node on main).

I'm convinced this is a fix, at least.
2026-04-15 08:46:02 +00:00
rltakashige 2ecefa0cfe Fix Qwen3-VL and autodetect vision config (#1893)
## Motivation

Qwen3 VL TP doesn't work atm, and vision is not behaving.

## Test Plan

### Manual Testing
Works now.
2026-04-14 23:05:55 +01:00
rltakashige b8eaf707a8 Add gemma 4 tensor parallelism (#1891) 2026-04-14 20:31:59 +01:00
rltakashige 8d81811b89 Try harder to clean up processes nicely (#1889)
## Motivation

Model loading is actually quite reliable now. No need to kill if you
have a slow SSD or it's a massive model; the user can shut the instance
down if necessary.

This was a major cause of signal=9 issues although not the only one (can
happen during inference too?).
The reason signal=9 is so bad is that RDMA will no longer work until
restart if this ever happens.

## Changes

- no more model load timeout
- no more crazy sigkills
- try harder to clean up processes on model shutdown

## Test Plan

### Manual Testing
Tested with some RDMA instances
2026-04-14 16:37:49 +01:00
rltakashige f2709dcde6 Add prefix cache flag to exo bench (#1888)
## Motivation
For using Exo-Bench extensively, there are many cases that we could use
prefix caching to speed up the benchmarks, especially when the focus is
on the token generation.

At the same time, it's very clear that prefix caching decode tokens is
not very useful in most current scenarios. Surprisingly, even for
non-thinking models, the chat template means that a continued
conversation will be formatted such that the existing cache is not
effective.

We already (slightly accidentally) do this for the batch generator - we
should do it for the sequential generator too.

## Changes

We can now speed up exo bench by having a use prefix caching flag. Of
course, for most accurate pp results, it is better to not have it, but
this speeds up tg and large benchmarking significantly.
Updated methodology to match

## Test Plan

### Manual Testing
Tested on many configurations that the difference in results is
negligible, even with multiple --pp options.
2026-04-14 11:12:58 +01:00
ciaranbor 77ffe039b3 Complete responses api usage response field (#1885)
## Motivation

The Responses API usage response was missing `input_tokens_details` and
`output_tokens_details`. The chat completions API already reports these.

## Changes

- Added `InputTokensDetails` (`cached_tokens`) and `OutputTokensDetails`
(`reasoning_tokens`) to `ResponseUsage`
- Extracted shared `_build_response_usage()` helper for both streaming
and non-streaming paths

## Test Plan

### Manual Testing

4-node cluster, `Qwen3-30B-A3B-4bit` — verified both detail objects
present with correct values in streaming and non-streaming responses.

### Automated Testing

13 tests in `test_openai_responses_api.py`.
2026-04-13 17:38:33 +00:00
rltakashige 3f0df404a5 Reduce memory consumption by adding Flash Attention to Qwen3.5 and Gemma 4, and fix RotatingKVCache prefix cache memory leak (#1886)
## Motivation

Part 1 of many memory improvements.

## Changes
As written in the title

## Test Plan

### Manual Testing
Gemma 4 26B cache reduced from 54GB -> 10GB per 100k tokens, Qwen3.5 35B
A3B cache reduced from 21GB every 100000 tokens to 7GB.
2026-04-13 18:32:17 +01:00
634 changed files with 5319 additions and 30357 deletions

No files matched your search

+2
View File
@@ -32,6 +32,7 @@ jobs:
SPARKLE_ED25519_PRIVATE: ${{ secrets.SPARKLE_ED25519_PRIVATE }}
SPARKLE_S3_BUCKET: ${{ secrets.SPARKLE_S3_BUCKET }}
SPARKLE_S3_PREFIX: ${{ secrets.SPARKLE_S3_PREFIX }}
EXO_BUG_REPORT_PRESIGNED_URL_ENDPOINT: ${{ secrets.EXO_BUG_REPORT_PRESIGNED_URL_ENDPOINT }}
AWS_REGION: ${{ secrets.AWS_REGION }}
EXO_BUILD_NUMBER: ${{ github.run_number }}
EXO_LIBP2P_NAMESPACE: ${{ github.ref_name }}
@@ -346,6 +347,7 @@ jobs:
EXO_BUILD_COMMIT="$GITHUB_SHA" \
SPARKLE_FEED_URL="$SPARKLE_FEED_URL" \
SPARKLE_ED25519_PUBLIC="$SPARKLE_ED25519_PUBLIC" \
EXO_BUG_REPORT_PRESIGNED_URL_ENDPOINT="$EXO_BUG_REPORT_PRESIGNED_URL_ENDPOINT" \
CODE_SIGNING_IDENTITY="$SIGNING_IDENTITY" \
CODE_SIGN_INJECT_BASE_ENTITLEMENTS=YES
mkdir -p ../../output
+1 -2
View File
@@ -18,6 +18,7 @@ digest.txt
app/EXO/build/
dist/
# rust
target/
**/*.rs.bk
@@ -39,5 +40,3 @@ bench/**/*.json
tmp/models
/build/exo
/.claude/skills
/.claude
/.codex
-3
View File
@@ -4,7 +4,4 @@
<option name="sdkName" value="Python 3.13 (exo)" />
</component>
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.13 (exo)" project-jdk-type="Python SDK" />
<component name="RuffConfiguration">
<option name="enabled" value="true" />
</component>
</project>
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loaded 100 of 634 files, more files were not shown because too many files have changed in this diff. Show more