Compare commits

..
Author SHA1 Message Date
Evan efa0da0912 wah 2026-04-22 12:51:10 +01:00
Evan Quiney df332035ef swap camelcasemodels for frozenmodels globally (#1957) 2026-04-22 11:49:25 +00:00
ciaranbor af673845d3 Ignore HF remote repo changes (temporary fix) (#1958)
## Motivation

Fixes #1918. Downloaded model status reverts from "completed" to
"pending" during each download scan. Reproduced with `zai-org/GLM-5.1`.

## Changes

- `coordinator.py`: In the periodic rescan, don't downgrade
already-completed models; fall back to `resolve_existing_model()`
(safetensors weight check) when per-file size check reports incomplete
- New `test_download_status_not_lost.py`: 3 regression tests

## Why It Works

The rescan compares local file sizes against HF's `main` revision. When
HF updates text files (README, jinja, etc.), remote sizes change but
local files still match the old revision — causing a false "incomplete".
The fix uses the safetensors weight check as ground truth instead.

Long-term: pin the downloaded revision SHA rather than always checking
against `main`.

## Test Plan

### Manual Testing

- Mac Studio M3 Ultra with GLM-5.1 downloaded (natural reproduction of
the issue)
- Confirmed GLM-5.1 stays `DownloadCompleted` through multiple rescan
cycles

### Automated Testing

- 3 new tests: completed-not-downgraded, fallback-to-resolve,
genuinely-incomplete-stays-pending
2026-04-22 11:11:01 +01:00
ciaranbor 49670c8624 Handle missing total_size in safetensors index files (#1956)
## Motivation

Image models fail to load after a mid-download instance deletion and
recreation. The system skips the download and crashes with
`FileNotFoundError: No safetensors files found in .../vae`.

## Changes

- Make `ModelSafetensorsIndexMetadata.total_size` optional (`PositiveInt
| None = None`)
- Add null guard in `fetch_safetensors_size`
- Add regression test

## Why It Works

Exolabs quantized image models have safetensors index files with mflux
metadata (`quantization_level`, `mflux_version`) but no `total_size`.
The required `PositiveInt` field caused Pydantic validation to fail,
which was silently swallowed by `except Exception: continue` in
`_scan_model_directory`. This skipped all weight map checks, making
incomplete models appear complete.

## Test Plan

### Manual Testing

- Hardware: Mac Studio
- Before: `CreateRunner → LoadModel` (crash). After: `CreateRunner →
DownloadModel` (correct).

### Automated Testing

- `test_safetensors_index.py`: 3 cases covering missing, valid, and null
metadata
2026-04-21 16:39:14 +01:00
rltakashige fcc3718efb Add sampling defaults (#1947)
## Motivation

Model quality issues

### Manual Testing
TODO
2026-04-21 06:45:33 +00:00
rltakashige 8ccfd7fcb6 Fix some misc build issues (#1948)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-21 07:41:07 +01:00
Evan Quiney 7b416155de bump uv lock for linux builds (#1942)
followup from a rebase issue in #1874
2026-04-20 11:41:40 +00:00
Evan Quiney 93a24748e6 mlx cuda 13 (dgx spark) support (#1874) 2026-04-20 11:49:12 +01:00
Evan Quiney e32829e51d chore: bump versions in line with release (#1941) 2026-04-20 09:39:28 +00:00
rltakashige 09e894dd52 Fix vision models on M5 Pro/Max MacBooks (#1927)
## Motivation

Vision models don't understand images on M5 series MacBooks. The
upstream NAX addmm fix (https://github.com/ml-explore/mlx/pull/3422)
fixes this.

## Why It Works
Same conclusion I came to when I was debugging the issue on an M5 Max.
It works after this fix.

## Test Plan

### Manual Testing
Works for Qwen3.5 27B
2026-04-19 12:55:20 +01:00
rltakashige bf8aacfd41 Improve build CI (#1920)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-17 20:55:15 +00:00
af9e847edb fix: force gc + clear_cache after KV prefix cache eviction (#1832)
## Summary
- After `KVPrefixCache` evicts LRU entries, the MLX Metal buffers stay
allocated until Python's GC runs
- This leaks ~3-4 GB between long-context requests, reducing the
effective context ceiling for back-to-back requests
- Adding `gc.collect()` + `mx.clear_cache()` after eviction frees Metal
buffers promptly

## Test plan
- [x] Measured on 2-node PP cluster with Qwen3.5-397B-A17B-4bit at 63K
context
- [x] Before: 108.88 GB retained after eviction (3.78 GB above baseline)
- [x] After: 105.48 GB retained after eviction (0.38 GB above baseline —
draft model KV + minor overhead)
- [x] `gc.collect()` adds ~2-3ms latency, runs once per eviction cycle
(not per token)
- [ ] Verify with `uv run pytest`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Adam Durham <adam@example.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
2026-04-17 17:57:32 +00:00
mlpy0 01598960bd Add model card for Qwen3.6-35B-A3B-8bit (#1917)
Adds the 8bit variant missing from #1907 — the safetensors index is now
live on HF.

- `mlx-community/Qwen3.6-35B-A3B-8bit` (~35 GB)

Architectural fields match the existing 4bit/5bit/bf16 cards.
`storage_size.in_bytes` is taken from `metadata.total_size` of the
upstream `model.safetensors.index.json`.
2026-04-17 10:06:23 +00:00
63b8e64715 Add model cards for Qwen3.6-35B-A3B variants (#1907)
## Motivation

`mlx-community` has just published the new **Qwen3.6-35B-A3B**
multimodal MoE family on HuggingFace. Without static model cards exo
doesn't surface these models in the dashboard picker or match its
placement / prefill logic, so users can't one-click launch them. This PR
adds cards for the three quants whose safetensors indexes are already
live on HF (4bit / 5bit / bf16).

## Changes

Three new TOML files in `resources/inference_model_cards/`:

- `mlx-community--Qwen3.6-35B-A3B-4bit.toml` (~19 GB)
- `mlx-community--Qwen3.6-35B-A3B-5bit.toml` (~23 GB)
- `mlx-community--Qwen3.6-35B-A3B-bf16.toml` (~65 GB)

All three share the same architectural fields (`n_layers = 40`,
`hidden_size = 2048`, `num_key_value_heads = 2`, `context_length =
262144`, capabilities `text, thinking, thinking_toggle, vision`,
`base_model = "Qwen3.6 35B A3B"`) — only `model_id`, `quantization`, and
`storage_size.in_bytes` differ between variants.

## Why It Works

- Qwen3.6-35B-A3B reuses the `qwen3_5_moe` architecture
(`Qwen3_5MoeForConditionalGeneration`) — the same one already wired into
exo's MLX runner at `src/exo/worker/engines/mlx/auto_parallel.py:47` via
`Qwen3_5MoeModel`. The architectural fields are taken verbatim from the
HF `config.json.text_config` and match the existing `Qwen3.5-35B-A3B-*`
cards.
- Storage sizes are the exact `metadata.total_size` read from each
variant's `model.safetensors.index.json` on HF, so download progress and
cluster-memory-fit checks are accurate.
- Vision support is flagged in `capabilities`; the `[vision]` block is
auto-detected by `ModelCard._autodetect_vision` from the upstream
`config.json`, so no hand-written vision config is required.
- The card loader (`_refresh_card_cache` in
`src/exo/shared/models/model_cards.py`) globs every `.toml` in
`resources/inference_model_cards/` on startup, so nothing else needs to
change — the `/models` endpoint and the dashboard picker pick them up
automatically.

The `mxfp4` / `mxfp8` / `nvfp4` variants are still uploading upstream
(index JSONs currently 404) and can be added in a follow-up PR once HF
completes.

## Test Plan

### Manual Testing

Hardware: MacBook Pro M4 Max, 48 GB unified memory.

- Built the dashboard, ran `uv run exo`, waited for the API to come up
on `http://localhost:52415`.
- `curl -s http://localhost:52415/models` returns the three new model
ids (`mlx-community/Qwen3.6-35B-A3B-{4bit,5bit,bf16}`) alongside
existing models.
- Opened the dashboard, clicked SELECT MODEL, typed "Qwen3.6" into the
search box. A single **"Qwen3.6 35B A3B"** group appears showing `3
variants (19GB-65GB)`. Expanding it lists the `4bit` / `5bit` / `bf16`
quants with sizes `19GB` / `23GB` / `65GB`, exactly as expected:

![Qwen3.6 35B A3B in model
picker](https://gist.githubusercontent.com/AlexCheema/68c2c02da9450b44968e6b0e0b1d255e/raw/127119f70382353c65a847188e5a2c9013db68d2/qwen36-picker.png)

- Programmatically loaded each TOML via `ModelCard.load_from_path(...)`
and confirmed the parsed fields (layers / hidden / KV heads / context /
quant / base_model / caps / bytes) match what's written in the files.

### Automated Testing

No code paths were touched — these are pure TOML data files that plug
into the existing model-card loader. The existing pytest suite covers
TOML parsing and card serving; adding new TOMLs doesn't require new test
scaffolding. `uv run ruff check` and `nix fmt` are clean.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Ryuichi Leo Takashige <rl.takashige@gmail.com>
2026-04-16 23:25:26 +01:00
rltakashige 28c797846a Update mlx and mlx lm to latest (#1906)
Just bumping to the very latest upstream versions.
2026-04-16 10:59:33 +00:00
rltakashige 058bb08261 Allow copying on dashboard even on HTTP (#1902)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-15 23:30:12 +01:00
rltakashige 3eead80238 Better environment variables in MacOS app (#1901)
## Motivation

Closes #1858 

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-15 20:14:52 +00:00
rltakashige 87329c80ef Add usage stats to tool calls and handle multiple tool calls correctly (#1899)
## Motivation

Tool calls are usually not end tokens, so they didn't have usage stats.
2026-04-15 19:40:00 +01:00
rltakashige 8cdc833892 Drain tokens silently skipped in thinking parsing (#1898)
## Motivation
Closes #1882
2026-04-15 14:23:07 +00:00
rltakashige 2cd66ae4cf Fix out of order event idx causing fatal crashes (#1894)
## Motivation

<img width="828" height="373" alt="Screenshot 2026-04-14 at 22 56 52"
src="https://github.com/user-attachments/assets/f8f48c1d-68c5-4acc-a6de-9d180672da9d"
/>

if is_new_master=True, _elect_loop creates a new EventRouter before the
worker has receivers. Then, event router runs _run_ext_in and
buf.drain_indexed() will pick off events, even though
self.internal_outbound is not populated fully.

Finally, when the worker does try requesting events, the next event it
receives is not the first event, meaning the worker crashes.

## Changes

Start the event router after all the receivers are registered

## Why It Works

self.internal_outbound is populated before the loop begins.

## Test Plan

### Manual Testing
No more crashes observed in testing (it's actually quite easy to
reproduce the issue if you have one node with this fix but the other
node on main).

I'm convinced this is a fix, at least.
2026-04-15 08:46:02 +00:00
rltakashige 2ecefa0cfe Fix Qwen3-VL and autodetect vision config (#1893)
## Motivation

Qwen3 VL TP doesn't work atm, and vision is not behaving.

## Test Plan

### Manual Testing
Works now.
2026-04-14 23:05:55 +01:00
rltakashige b8eaf707a8 Add gemma 4 tensor parallelism (#1891) 2026-04-14 20:31:59 +01:00
rltakashige 8d81811b89 Try harder to clean up processes nicely (#1889)
## Motivation

Model loading is actually quite reliable now. No need to kill if you
have a slow SSD or it's a massive model; the user can shut the instance
down if necessary.

This was a major cause of signal=9 issues although not the only one (can
happen during inference too?).
The reason signal=9 is so bad is that RDMA will no longer work until
restart if this ever happens.

## Changes

- no more model load timeout
- no more crazy sigkills
- try harder to clean up processes on model shutdown

## Test Plan

### Manual Testing
Tested with some RDMA instances
2026-04-14 16:37:49 +01:00
rltakashige f2709dcde6 Add prefix cache flag to exo bench (#1888)
## Motivation
For using Exo-Bench extensively, there are many cases that we could use
prefix caching to speed up the benchmarks, especially when the focus is
on the token generation.

At the same time, it's very clear that prefix caching decode tokens is
not very useful in most current scenarios. Surprisingly, even for
non-thinking models, the chat template means that a continued
conversation will be formatted such that the existing cache is not
effective.

We already (slightly accidentally) do this for the batch generator - we
should do it for the sequential generator too.

## Changes

We can now speed up exo bench by having a use prefix caching flag. Of
course, for most accurate pp results, it is better to not have it, but
this speeds up tg and large benchmarking significantly.
Updated methodology to match

## Test Plan

### Manual Testing
Tested on many configurations that the difference in results is
negligible, even with multiple --pp options.
2026-04-14 11:12:58 +01:00
ciaranbor 77ffe039b3 Complete responses api usage response field (#1885)
## Motivation

The Responses API usage response was missing `input_tokens_details` and
`output_tokens_details`. The chat completions API already reports these.

## Changes

- Added `InputTokensDetails` (`cached_tokens`) and `OutputTokensDetails`
(`reasoning_tokens`) to `ResponseUsage`
- Extracted shared `_build_response_usage()` helper for both streaming
and non-streaming paths

## Test Plan

### Manual Testing

4-node cluster, `Qwen3-30B-A3B-4bit` — verified both detail objects
present with correct values in streaming and non-streaming responses.

### Automated Testing

13 tests in `test_openai_responses_api.py`.
2026-04-13 17:38:33 +00:00
rltakashige 3f0df404a5 Reduce memory consumption by adding Flash Attention to Qwen3.5 and Gemma 4, and fix RotatingKVCache prefix cache memory leak (#1886)
## Motivation

Part 1 of many memory improvements.

## Changes
As written in the title

## Test Plan

### Manual Testing
Gemma 4 26B cache reduced from 54GB -> 10GB per 100k tokens, Qwen3.5 35B
A3B cache reduced from 21GB every 100000 tokens to 7GB.
2026-04-13 18:32:17 +01:00
Evan Quiney 9b381f7bfe bump and simplify flake (#1866)
seems like stablepkgs swiftfmt works now! also bump macmon to 0.7
2026-04-13 15:45:17 +00:00
Alex Cheema d2f67b5d10 dashboard: group Gemma under Google with proper logo (#1883)
## Motivation

In the dashboard model picker sidebar, the Gemma 4 models were showing
up under a "Gemma" family with the generic fallback tick/checkmark icon
(the default case in `FamilyLogos.svelte`), since no dedicated logo
branch existed for `family === "gemma"`. Every other vendor (Meta,
NVIDIA, OpenAI, DeepSeek, Qwen, …) has its own brand mark.

Gemma is Google's model family, so it should live under a **Google**
bucket that future Google-authored models can join, and it should render
with a proper Google logo in the same style as its neighbors.

## Changes

- `dashboard/src/lib/components/FamilyLogos.svelte`: added a `family ===
"google"` branch rendering a monochrome Google "G" as a single `<path>`
inside the shared `24×24` viewBox with `fill="currentColor"`, matching
the other vendor logos.
- `dashboard/src/lib/components/FamilySidebar.svelte`: added `google:
"Google"` to the `familyNames` display map.
- `dashboard/src/lib/components/ModelPickerModal.svelte`: inserted
`"google"` into the `familyOrder` array (next to `"llama"`) so the
vendor has a deterministic sort position.
- `resources/inference_model_cards/mlx-community--gemma-4-*.toml` (16
files): changed `family = "gemma"` → `family = "google"`. `base_model =
"Gemma 4 …"` is unchanged, so the model titles still read "Gemma".

## Why It Works

The sidebar builds its family list from whatever values appear in
`model.family` across the loaded model cards (`ModelPickerModal.svelte`
`uniqueFamilies`). Renaming the family string on the 16 Gemma cards from
`"gemma"` to `"google"` collapses them into a single "Google" bucket,
and the new logo branch + display-name map entry gives that bucket a
real brand mark and label. All other logos share the same `w-6 h-6 /
viewBox="0 0 24 24" / fill="currentColor"` shape, so inheriting
`text-exo-yellow` / `text-white/50` just works.

## Test Plan

### Manual Testing
<!-- Hardware: MacBook Pro M3 Max -->
- `cd dashboard && npm install && npm run build` — dashboard builds
cleanly.
- `uv run exo`, opened `http://localhost:52415`, clicked **SELECT
MODEL**:
- sidebar shows a **Google** entry with a monochrome Google "G" logo in
the same style as Meta / NVIDIA / etc.
  - old "Gemma" entry with the generic tick is gone.
- clicking **Google** filters to the Gemma 4 variants (e2b / e4b / 26B
A4B / 31B).
- hover/selected color states switch between `text-white/50` and
`text-exo-yellow` correctly.

### Automated Testing
- No new tests — this is a cosmetic grouping/logo change. Existing
dashboard build verifies the Svelte + TS compiles.
2026-04-13 14:08:15 +00:00
8973503322 fix: use configured api_port for IP connectivity probes (#1877)
## Motivation

Fixes #1861

When `--api-port` is set to a non-default value (e.g., `--api-port
55555`), the IP connectivity discovery system still probes peers on the
hardcoded default port 52415. Since the API is not listening on 52415,
all reachability checks fail, the topology reports zero reachable nodes,
and the dashboard shows "No valid configurations for current settings."

## Changes

Thread the configured `api_port` from `Args` through `Worker` into the
reachability probe functions:

- `net_profile.py`: `check_reachability()` and `check_reachable()`
accept an `api_port` parameter (default 52415 for backward
compatibility)
- `worker/main.py`: `Worker` stores `api_port` and passes it to
`check_reachable()`, uses it in `Multiaddr` construction and the mDNS
connection filter
- `main.py`: passes `args.api_port` to the `Worker` constructor

## Why It Works

The `/node_id` endpoint used by reachability probes is served by the
FastAPI app, which binds to `args.api_port`. The probes must use the
same port the API is actually listening on. Before this fix, the port
was hardcoded in three places in `net_profile.py` and `worker/main.py`;
now it uses the value from the CLI flag.

## Test Plan

### Manual Testing
<!-- Hardware: not available for multi-node testing -->
- Verified ruff passes on all changed files
- Code inspection: traced `api_port` flow from `Args.parse()` →
`Node.create()` → `Worker.__init__()` → `_poll_connection_updates()` →
`check_reachable()` → `check_reachability()` → HTTP probe URL

### Automated Testing
- No existing automated tests cover the reachability probe code path
- The new `api_port` parameter defaults to `52415`, so all existing
behavior is preserved when `--api-port` is not specified

---------

Co-authored-by: lawrence3699 <lawrence3699@users.noreply.github.com>
Co-authored-by: Evan <evanev7@gmail.com>
2026-04-13 14:01:42 +00:00
Alex CheemaandClaude Opus 4.6 eb9228615f models: add MiniMax M2.7 cards (#1884)
## Motivation

The mlx-community [MiniMax-M2.7
collection](https://huggingface.co/collections/mlx-community/minimax-m27)
landed but exo didn't have model cards for any of the variants yet, so
they weren't selectable from the dashboard model picker. Adding cards
also makes them discoverable under the existing MiniMax family entry.

## Changes

Added 6 new model cards in `resources/inference_model_cards/`, one per
quant of MiniMax M2.7:

- `mlx-community--MiniMax-M2.7.toml` (bf16, full precision — 457 GB)
- `mlx-community--MiniMax-M2.7-4bit.toml` (128 GB)
- `mlx-community--MiniMax-M2.7-4bit-mxfp4.toml` (121 GB)
- `mlx-community--MiniMax-M2.7-5bit.toml` (157 GB)
- `mlx-community--MiniMax-M2.7-6bit.toml` (185 GB)
- `mlx-community--MiniMax-M2.7-8bit.toml` (243 GB)

All six use `family = "minimax"` and share `base_model = "MiniMax M2.7"`
so they collapse into a single group in the picker with the existing
MiniMax logo. Architecture fields (`n_layers = 62`, `hidden_size =
3072`, `num_key_value_heads = 8`, `context_length = 196608`) were read
from each repo's `config.json`; `storage_size.in_bytes` was summed from
the HF tree API per repo.

`capabilities = ["text", "thinking"]` follows the existing MiniMax M2.5
cards — the chat template always emits `<think>` tags (no toggle),
matching M2.5 behavior.

## Why It Works

Model cards in `resources/inference_model_cards/` are auto-loaded by
`src/exo/shared/models/model_cards.py::get_model_cards`. The dashboard
picker groups by `base_model` and filters by `family`, so sharing both
across all six variants gives a single "MiniMax M2.7" group under the
MiniMax sidebar entry, with the quant variants exposed as selectable
sub-options.

## Test Plan

### Manual Testing
<!-- Hardware: MacBook Pro M3 Max -->
- Ran `uv run python -c "…await get_model_cards()…"` and confirmed all 6
new cards load with `family=minimax`, `base_model="MiniMax M2.7"`, and
correct quant + byte sizes.
- `cd dashboard && npm run build` then `uv run exo`, opened the model
picker → **MiniMax** family → **MiniMax M2.7** group shows all six quant
variants.

### Automated Testing
- No new automated tests — these are data files validated by the
existing Pydantic `ModelCard` schema at load time.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 14:47:09 +01:00
MikkoParkkola 4b13735ea3 build: remove pyinstaller temp artifacts (#1868)
removes PyInstaller `build/` leftovers after `just package`
2026-04-11 11:46:03 +00:00
rltakashigeandEvan 196543ce69 Add Gemma 4 + VLM fixes + thinking parsing updates (#1851)
## Motivation
Add support for Gemma 4, including VLM!

## Changes

- Add auto parallel strategies and model cards for Gemma 4
- Normalise Gemma 4's special Vision Transformer handling to be in line
with the rest of our vision processors.
- Also adds reprs to messages and b64 hashes to prevent log spam.

## Test Plan

### Manual Testing
Tested manually on 4bit E2B and 8bit 26B

### Automated Testing
Model onboarding shows small logit diffs.

---------

Co-authored-by: Evan <evanev7@gmail.com>
2026-04-11 12:29:33 +01:00
ciaranbor 6172617b00 add env override to macos app (#1869)
## Motivation

- Let users pass/override arbitrary exo env vars from the macOS app
without a code change.

## Changes

- `ExoProcessController.swift`: `CustomEnvironmentVariable` struct +
`@Published` list persisted to `UserDefaults`, injected into the child
process after built-ins.
- `SettingsView.swift`: new **Environment** tab with add/remove rows,
trim + dedup on save, and a POSIX name validator with a warning badge.

## Why It Works

- Custom vars applied last in `makeEnvironment`, so overriding a
built-in works with no special-casing.

## Test Plan

### Manual Testing

- Set `EXO_LIBP2P_NAMESPACE` via the new UI; confirmed override in
`~/.exo/exo_log/exo.log`.
2026-04-10 17:39:55 +01:00
ciaranbor 93a980a61e just package first builds dashboard (#1867)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-10 16:25:30 +00:00
ciaranbor 2962ebee60 Fix pdf inputs on Safari (#1865)
## Motivation

PDF attachments weren't working on Safari

## Changes

Create async readable stream if none exists

## Why It Works
pdfjs-dist requires an async readable stream internally

## Test Plan

### Manual Testing
pdf attachments now work on Safari, still work on Firefox
2026-04-10 14:59:34 +00:00
rltakashigeandEvan abd75ae06c Truncate long logs with repr (#1854)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->

---------

Co-authored-by: Evan <evanev7@gmail.com>
2026-04-10 15:53:48 +01:00
kaiisfree ee2e505b3c fix: handle BrokenResourceError in download progress callback (#1846)
## Summary
- Wraps progress callback `send()` in try/except to gracefully handle
`BrokenResourceError` when the memory stream is closed
- Prevents unhandled `ExceptionGroup` from crashing the process when the
download consumer disconnects during transfer

## Root Cause
The download progress callback sends updates through an anyio memory
object stream. When the receiving end closes (e.g., client disconnect,
timeout, or task cancellation), `send()` raises `BrokenResourceError`.
Inside an anyio `TaskGroup`, this unhandled exception becomes an
`ExceptionGroup` that propagates up and crashes the coordinator.

## Fix
Catch `BrokenResourceError` (and `ClosedResourceError` for completeness)
in the progress callback and handle gracefully — the download continues
but progress updates are silently dropped for disconnected consumers.

Fixes #1844

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-04-10 11:43:42 +00:00
Evan Quiney f2e6b1ef76 prevent some crash loops (#1827)
extension to #1763 that prevents crash looping in some common scenarios.
2026-04-09 11:34:35 +00:00
ciaranbor e2e17eafb7 Fix reasoning_tokens counting for multi-token thinking tag models (#1848)
## Motivation

`reasoning_tokens` is always 0 in usage stats, even when thinking
content streams correctly via `reasoning_content` SSE deltas. The MLX
generators had their own thinking detection comparing individual
detokenized tokens against think tags — this never fires for models
where tags span multiple tokens (e.g. gpt-oss-120b) or are already in
the prompt.

## Changes

- Removed broken per-token thinking detection from `batch_generate.py`
and `generate.py`
- Added `_count_reasoning_tokens` wrapper in `model_output_parsers.py`
that counts `is_thinking=True` responses and patches the total into
Usage on the final response
- Wired it as the outermost stage of `apply_all_parsers`, so it works
regardless of which parser sets `is_thinking`
- Added 3 tests covering `parse_thinking_models` and `parse_gpt_oss`
paths

## Why It Works

The parser pipeline already correctly sets `is_thinking` on each
response. Counting at the output of `apply_all_parsers` means one
counting point that works for all model types, replacing the duplicate
broken logic in two generators.

## Test Plan

### Manual Testing

- 4-node cluster, `mlx-community/gpt-oss-120b-MXFP4-Q8`
- Main branch: `reasoning_tokens: 0` — fix branch: `reasoning_tokens:
25`

### Automated Testing

- 3 new tests: explicit think tags, `starts_in_thinking=True`, and
gpt-oss Harmony analysis channel
2026-04-09 12:29:35 +01:00
ciaranbor b12cd1b186 Cancel SSE keep-alive when instance is deleted (#1828)
## Motivation

When a model instance is deleted (e.g. node disconnect, manual
teardown), any in-flight SSE streaming connections for that instance
hang indefinitely. The API never closes the response stream, so clients
block forever waiting for more chunks.

## Changes

- Listen for `InstanceDeleted` events in the API event loop
- Add `_close_streams_for_instance()` to find and close any active
text/image generation queues tied to tasks on the deleted instance
- Add unit tests covering text gen, image gen, and
unrelated-instance-not-closed scenarios

## Why It Works

When an instance is deleted, we iterate `state.tasks` to find commands
running on that instance, then close and remove their send-side queue
handles. This causes the SSE generator to terminate, unblocking the
client.

## Test Plan

### Manual Testing
- This was causing issues for me on another branch (integration tests).
Including this fix solved the issue

### Automated Testing
- `test_instance_deleted_stream_cleanup.py`: 3 tests covering text gen
cleanup, image gen cleanup, and ensuring unrelated streams are not
affected
2026-04-08 16:14:28 +01:00
mlpy0 62570227ff Catch ClosedResourceError when forwarding chunks to client queues (#1856)
While stress-testing inference with rapid client cancels mid-stream, I
hit a reproducible crash where the entire exo process exits.

When a client cancels a streaming chat completion partway through, its
receive stream gets closed cleanly via its context manager. The producer
in `API._apply_state` then calls `queue.send(event.chunk)`, which raises
`anyio.ClosedResourceError` rather than `BrokenResourceError`. The
existing handler only catches `BrokenResourceError`, so the exception
propagates through the API task group, kills the Node task group, and
the process exits with `EXO Shutdown complete`.

Trace from one of the crashes:

```
File "exo/api/main.py", line 1818, in _apply_state
    await queue.send(event.chunk)
File "anyio/streams/memory.py", line 212, in send_nowait
    raise ClosedResourceError
anyio.ClosedResourceError
```

The fix is to catch `ClosedResourceError` alongside
`BrokenResourceError` in both queue handlers (text and image), so the
dead queue gets dropped and `_apply_state` keeps running for other
in-flight requests.
2026-04-08 08:50:04 +00:00
Alex CheemaandClaude Opus 4.6 645bc20950 Add Fast Synch Enabled toggle to macOS app settings (#1852)
## Motivation

The exo backend already supports `--fast-synch` / `--no-fast-synch` CLI
flags and the `EXO_FAST_SYNCH` environment variable, but there was no
way to toggle this from the macOS app UI. Users who want fast CPU-to-GPU
synchronization for RDMA with Tensor Parallelism had to use CLI flags.

## Changes

- **ExoProcessController.swift**: Added `fastSynchEnabled`
UserDefaults-backed property and pass `EXO_FAST_SYNCH=on` to the exo
process environment when enabled.
- **SettingsView.swift**: Added a "Performance" section to the Advanced
tab with a "Fast Synch Enabled" toggle, an info icon (ⓘ) tooltip
explaining the feature and trade-offs, and a "Save & Restart" button.

## Why It Works

Follows the exact same pattern as the existing `offlineMode` and
`enableImageModels` settings — UserDefaults persistence, `@Published`
property with `didSet`, environment variable passthrough in
`makeEnvironment()`, and pending state with Save & Restart in the
settings UI. The `EXO_FAST_SYNCH=on` value matches what the Python
backend already reads in `main.py`.

## Test Plan

### Manual Testing
<!-- Hardware: macOS app -->
- Open Settings → Advanced tab → verify "Performance" section with "Fast
Synch Enabled" toggle appears
- Hover the ⓘ icon → verify tooltip explains the feature and GPU lock
trade-off
- Toggle on → click "Save & Restart" → verify process restarts with
`EXO_FAST_SYNCH=on` in env
- Close and reopen Settings → verify the toggle state persists
- Verify "Save & Restart" button is disabled when no changes are pending

### Automated Testing
- Existing settings patterns are well-established; no new automated
tests needed for this UI toggle

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 01:04:42 +00:00
rltakashige 5757c27dd5 Add download utility script (#1855)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-04-08 00:58:39 +00:00
Andrei Cravtov fd5b23281c Workspace tweaks (#1849)
## Changes

Mostly chore changes around vscode and jetbrains workspace settings, and
some basedpyright settings tweaks, to allow direnv to work and nixd
autocomplete with flake parts to work
2026-04-07 17:26:29 +00:00
rltakashige 43b3df45fb Fix BatchGenerator in line with upstream refactor (and prevent Qwen3.5 memory leak) (#1835)
## Motivation

MLX LM has had a massive refactor to their BatchGenerator recently.
Since we'd like new features from MLX LM such as Gemma 4, we need to
update the code to handle this.

Additionally this fixes a significant memory leak in GatedDeltaNet (the
difference is quite substantial, up to 1GB every 1000 tokens, explaining
several memory issues users were facing with Qwen3.5 models)

## Testing
Before
<img width="3146" height="884" alt="image"
src="https://github.com/user-attachments/assets/5af0f55a-393c-4a32-9eed-ae43f1611af4"
/>


After (no memory leak, as one of the changes upstream)
<img width="3190" height="892" alt="image"
src="https://github.com/user-attachments/assets/f0bd128d-fd48-40d4-9bbd-50a564beab14"
/>
2026-04-07 11:50:12 +00:00
mlpy0 24420eb10a Fix reasoning_tokens always reported as 0 for thinking models (#1836)
When `enable_thinking` is set, chat templates (Qwen3, DeepSeek, etc.)
append `<think>` to the prompt. The model starts generating thinking
content directly without emitting a `<think>` token in the output
stream.

Both generators initialized `in_thinking = False` and only set it to
`True` on seeing a `<think>` token in output. Since that token was part
of the prompt, the flag never flipped and `reasoning_tokens` stayed at 0
in the usage response.

Fix: initialize `in_thinking` from `detect_thinking_prompt_suffix()`,
which already exists and is used by `model_output_parsers` for routing
thinking content correctly.
2026-04-05 00:05:18 +00:00
rltakashige 59669c1168 Tighten EXO bench concurrency numbers and explain methodology (#1811)
## Motivation

The timings in the batch generator are a little optimistic; a minor
change is needed to make them more correct.

## Changes

Include the time spent in the API in the generation tps and make sure to
send all requests simultaneously
2026-04-05 00:57:52 +01:00
ciaranbor 1d2ce464dc Allow pausing and deleting active downloads (#1829)
## Motivation

No way to pause active downloads or delete partial/failed downloads from
the dashboard.

## Changes

- **Backend:** Added `POST /download/cancel` endpoint with
`CancelDownloadParams`/`CancelDownloadResponse` types. Wires into the
existing `CancelDownload` command + coordinator handler.
- **Dashboard store:** Added `cancelDownload(nodeId, modelId)` function.
- **Dashboard UI:**
  - Pause + delete buttons on active (downloading) cells
  - Delete button on paused/pending and failed cells
- Extracted duplicated SVG icons into `{#snippet}` blocks (`trashIcon`,
`downloadIcon`, `pauseIcon`, `deleteButton`)
- **Tests:** 3 coordinator-level tests for cancel: active download →
pending, nonexistent → no-op, cancel then resume.

## Why It Works

`CancelDownload` command and coordinator handler already existed — just
needed an HTTP endpoint and dashboard wiring. Delete endpoint already
supported all download states.

## Test Plan

### Manual Testing

Started a model download, paused it. Deleted some paused downloads.
Deleted some ongoing downloads.

### Automated Testing

- `test_cancel_active_download_transitions_to_pending` — cancels
in-progress download, asserts `DownloadPending` event and cleanup
- `test_cancel_nonexistent_download_is_noop` — no events emitted
- `test_cancel_then_resume_download` — restart after cancel works
2026-04-02 15:56:33 +01:00
ciaranborandEvan eb6ae9fd3c Prevent failed instance retries (#1763)
## Motivation

Currently, when a runner fails, the master retries the instance. Most of
the time, this causes a loop over failure. Retries need backoff and a
cap.

## Changes

- src/exo/worker/main.py: Before creating a runner, check an exponential
backoff timer per instance. After EXO_MAX_INSTANCE_RETRIES failures,
send DeleteInstance to permanently remove the instance. Record attempts
on Shutdown; reset on InstanceDeleted.
- src/exo/utils/keyed_backoff.py: Add attempts() method to query retry
count
- src/exo/shared/constants.py: Add EXO_MAX_INSTANCE_RETRIES = 3.

## Why It Works

The worker gates CreateRunner tasks behind a KeyedBackoff, adding
exponential delay (2s base, 30s cap) between retries. After 3 failures
the worker sends DeleteInstance, stopping retries entirely. The backoff
resets when the instance is deleted, so a fresh placement starts clean.

---------

Co-authored-by: Evan <evanev7@gmail.com>
2026-04-01 21:03:34 +01:00
rltakashige 4688adb5d2 Support PDFs in dashboard (#1822)
Like ChatGPT does, we now send both the extracted text and the image of
each PDF page.
2026-03-31 18:25:40 +01:00
rltakashige d9ed943034 Fix Nemotron cache leak upstream (#1819)
## Motivation
Nemotron Cascade and Nano failing at long decodes.

## Changes

Fixed upstream, just change pyproject and uv lock here.


## Test Plan
### Automated Testing
Tested with a reproduce script upstream
2026-03-30 16:53:21 +00:00
rltakashige c6815bfdce Only update KV prefix cache on a good cache hit (#1817)
## Motivation

Addresses #1816 

## Changes

Update on min prefix cache > min_prefix_hit_length **and** hit ratio >
_MIN_PREFIX_HIT_RATIO_TO_UPDATE
min_prefix_hit_length = max(1000, system prompt length) -> system
prompts must match exactly.

## Test Plan

### Manual Testing
Test on OpenCode and Claude Code
2026-03-30 15:04:38 +01:00
rltakashige 39c39e8199 Integrations helpers (#1810)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-03-30 14:28:41 +01:00
rltakashige e5cb7b80d0 Add SSE-keepalive to not time out on long prefill on clients (#1803)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-03-30 12:18:38 +01:00
rltakashigeandDavid Hind 635801d515 Add multimodality! (#1802)
## Motivation

Images!

TODO (in a future PR): Add audio and video support.

## Test Plan

### Manual Testing
<img width="2652" height="1900" alt="image"
src="https://github.com/user-attachments/assets/7d3a7137-542f-4f94-9193-2c73b7c4a5ec"
/>

<img width="2770" height="1956" alt="image"
src="https://github.com/user-attachments/assets/e3c3a096-8029-4409-97a6-aca31a9a3f24"
/>
<img width="2738" height="1768" alt="image"
src="https://github.com/user-attachments/assets/d70ea37f-cd1d-4a4c-ad08-3beb9fafa380"
/>

(And batching also works)

---------

Co-authored-by: David Hind <davehind@yahoo.co.uk>
2026-03-30 11:52:19 +01:00
rltakashige 2efbb8ab4f Improve exo harness with path state (#1815)
<img width="3224" height="1476" alt="image"
src="https://github.com/user-attachments/assets/d90a7d8a-9fe5-43a1-a715-1ef7ecc15422"
/>
2026-03-30 11:20:46 +01:00
Evan Quiney c6c5a3e73c feat: /state/paths (#1796)
adds a path option to the /state endpoint, allowing you to query
subfields of state without grabbing the whole blob

## test plan
poking around in the api
2026-03-30 10:10:00 +00:00
ArvidSU 10ef7ec9e8 feat: add Firefox AI sidebar (?q=) support to dashboard (#1814)
This PR builds on https://github.com/exo-explore/exo/pull/1677 to enable
custom prompts sent from Firefox `browser.ml.chat` to EXO dashboard
using URL parameters in sidebar for summary and other browser
interactions. See "Summarize page" example below.

## Summary
- Parse `?q=<encoded prompt>` URL parameter on page load and auto-submit
it as a chat message
- Clean up the URL with `history.replaceState` to prevent re-submission
on refresh
- Defer auto-send until both cluster state and model list are loaded so
model auto-selection works correctly

## Context
Firefox's built-in AI sidebar (`about:config: browser.ml.chat.enabled`)
integrates with chat providers by appending the user's prompt as
`?q=<URL-encoded prompt>`. Previously the exo dashboard ignored this
parameter. Users can now configure `http://localhost:52415` as a Firefox
AI chatbot provider.

See: https://support.mozilla.org/en-US/kb/ai-chatbot

## Technical notes
- Frontend-only change in `dashboard/src/routes/+page.svelte`
- Uses a Svelte `$effect` that reacts to `pendingFirefoxQuery`, `data`
(cluster state), and `models.length` — fires exactly once when all three
are ready
- If no model is selected, `handleAutoSend` auto-picks the best
available model; if no model fits memory, a toast is shown
- If a model is selected but not running, the message is queued until
the model loads

## Testing
```
http://localhost:52415/?q=Hello+world
http://localhost:52415/?q=Summarize+this+page%3A+%5Bpage+title%5D+%5Bpage+url%5D
```

<img width="2056" height="1329" alt="image"
src="https://github.com/user-attachments/assets/74463eb4-ca1a-400d-806a-c19ba93147b9"
/>
2026-03-30 11:02:35 +01:00
Evan Quiney 1e51dc89b0 chore: bump exo-version with release version (#1807)
our pyproject.toml version was 0.3.68 - update to .69 in line with
release!!
2026-03-27 11:47:13 +00:00
Alex CheemaandClaude Opus 4.6 5327bdde84 Fix custom model add requiring two attempts + enlarge sidebar buttons (#1805)
## Motivation

Adding a custom model from the Hub tab shows "Added" toast but the model
doesn't appear in the All tab. You have to add it a second time for it
to work. Also, the "All" button in the model picker sidebar is too small
to read comfortably.

## Changes

**Race condition fix (`src/exo/api/main.py`):**
- Call `add_to_card_cache(card)` directly in `add_custom_model()` after
sending the `ForwarderCommand`, before the API response returns

**Sidebar sizing
(`dashboard/src/lib/components/FamilySidebar.svelte`):**
- Increased sidebar min-width from 72/64px to 80/72px
- Increased "All" icon from `w-5 h-5` to `w-6 h-6`
- Increased all sidebar labels from 9px to 11px

## Why It Works

`POST /models/add` sends a `ForwarderCommand(AddCustomModelCard)` and
returns immediately. The frontend then calls `GET /models` which reads
from `_card_cache`. But the cache was only updated by the worker event
handler after the event round-trips through the master — a race the
frontend almost always loses. By updating the cache directly in the API
handler, `GET /models` immediately reflects the new model. The worker's
later `add_to_card_cache` call is idempotent (dict key assignment).

## Test Plan

### Manual Testing
<!-- Hardware: any Mac -->
- Open model picker → Hub tab → add a custom model → verify it appears
in All tab on the first attempt
- Verify sidebar "All" button and other labels are visually larger and
readable

### Automated Testing
- `uv run basedpyright` passes with 0 errors
- `uv run ruff check` passes

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 17:57:00 -07:00
ciaranbor 15f1b61f4c Rework model storage directory management (for external storage) (#1765)
## Motivation

Replace confusing EXO_MODELS_DIR/EXO_MODELS_PATH with clearer
multi-directory support, enabling automatic download spillover across
volumes.

## Changes

- EXO_MODELS_DIRS: colon-separated writable dirs (default always
prepended, first with enough space wins)
- EXO_MODELS_READ_ONLY_DIRS: colon-separated read-only dirs (protected
from deletion)
- select_download_dir(): picks writable dir by free space
- resolve_existing_model(): unified lookup across all dirs
- is_read_only_model_dir(): path-based read-only detection instead of
hardcoded flag
- Updated coordinator, worker, model cards, tests

## Why It Works

Default dir always included so zero-config behavior is unchanged. Disk
space checked at download time for automatic spillover. Read-only status
derived from path, not hardcoded.

## Test Plan

### Manual Testing

- No env vars set → identical behavior
- EXO_MODELS_DIRS=/Volumes/SSD/models → downloads to external storage
- EXO_MODELS_READ_ONLY_DIRS=/mnt/nfs → models found, deletion blocked

### Automated Testing

- 4 new tests in test_xdg_paths.py (prepend, default-only, overlap,
empty read-only)
- Existing tests updated to patch new constants
2026-03-26 17:46:46 +00:00
Michael HarriganandEvan 9034300163 [Fix] Node hang on reelection (#1801)
## Motivation

During master reelection, `_elect_loop` called `worker.shutdown()` (fire
& forget) then immediately created and started a new Worker.

This caused the old runner subprocess's Metal/GPU teardown to race with
the new worker's startup, resulting in `IOConnectUnmapMemory failed:
kr=0xe00002bc` errors and a full node hang requiring `^C`. Same issue
existed for `DownloadCoordinator`.

## Changes

- Added `anyio.Event`-based `_stopped` signal to `Worker` and
`DownloadCoordinator`, set at the end of their `run()` finally blocks
- Added `wait_stopped()` async method to both classes
- Updated `_elect_loop` to `await wait_stopped()` after calling
`shutdown()` on the old Worker and DownloadCoordinator before creating
replacements

## Why It Works

The old Worker's task group contains the RunnerSupervisor tasks, whose
finally blocks join the runner subprocess (with 5s timeout + SIGTERM +
SIGKILL escalation). By awaiting `wait_stopped()`, we guarantee the old
runner process has fully exited — including GPU memory cleanup — before
a new Worker can start and potentially access the GPU. This eliminates
the race without changing the shutdown mechanics themselves.

## Test Plan

### Manual Testing
Hardware: M4 Pro Mac Mini 24GB + M3 Ultra Mac Studio 96GB, connected via
Thunderbolt

**Repro steps:**
1. Start exo on two nodes with a model sharded across both (e.g.
`Josiefied-Qwen3-14B-abliterated-v3-4bit`)
2. Wait for "runner ready" on both
3. `kill -9` the master node
4. Observe the surviving node's re-election behavior

**Before fix (original crash):**
```
[ 11:02:39.0896AM ] Runner supervisor shutting down
[ 11:02:39.0905AM ] bye from the runner
[ 11:02:39.1052AM ] Stopping Worker
IOConnectUnmapMemory failed: kr=0xe00002bc
IOConnectUnmapMemory failed: kr=0xe00002bc
IOConnectUnmapMemory failed: kr=0xe00002bc
IOConnectUnmapMemory failed: kr=0xe00002bc
^C[ 11:03:45 ] ← hung for over a minute, required manual kill
```

**After fix (clean re-election):**
```
[ 12:15:22.4703PM ] runner loaded
[ 12:15:24.1672PM ] runner ready
[ 12:15:33.5393PM ] Waiting for other campaign to finish
[ 12:15:36.5409PM ] Node elected Master
[ 12:15:36.5413PM ] Unpausing API
```
No `IOConnectUnmapMemory` errors, no hang, no `^C` needed.

### Automated Testing
- No existing tests cover the `_elect_loop` re-election path; this is an
integration-level flow requiring a live router/election/worker stack
- All existing tests pass (307/308, 1 pre-existing Rust binding failure)
- basedpyright: 0 errors, ruff: all checks passed

---------

Co-authored-by: Evan <evanev7@gmail.com>
2026-03-26 17:28:47 +00:00
rltakashige 1d1dfaa1f3 Don't download original/ and metal/ folders from HF (#1800)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-03-26 13:36:07 +00:00
Evan Quiney 7625213df0 fix: enable macmon if preflight fails (#1799)
missed in #1747, issue #1798.

### the issue

we didn't set the memory poll rate after failling the macmon preflight,
only after failing the followups - as we never ran macmon if preflight
failed, we never hit the followup errors etc.

### testing

requires testing on an m5 pro, but the core issue is solved.
2026-03-26 11:35:22 +00:00
Alex CheemaandClaude Opus 4.6 f318f9ea14 Fix macOS build bundling wrong macmon binary (#1797)
## Motivation

PR #1747 fixed macmon support for M5 Pro/Max by pinning the
`swiftraccoon/macmon` fork in `flake.nix`. This works when running from
source (via Nix) but the distributed macOS `.app` build was still broken
on M5 Pro/Max because it was bundling the wrong macmon.

The error on M5 Pro/Max:
```
macmon preflight failed with return code -6: thread 'main' panicked at src/sources.rs:394:41
```

## Changes

- Removed `macmon` from `brew install` in `build-app.yml` — this was
installing the upstream `vladkens/macmon` which doesn't support M5
Pro/Max
- Added a new step that resolves the pinned macmon fork from the Nix dev
shell (same `swiftraccoon/macmon` at rev `9154d23` already defined in
`flake.nix`) and adds it to `$GITHUB_PATH`
- Added a safety `brew uninstall macmon` to ensure no Homebrew macmon
can shadow the pinned version

## Why It Works

PyInstaller bundles macmon via `shutil.which("macmon")`. Previously this
found the Homebrew (upstream) binary. Now it finds the Nix-overlayed
fork that has M5 Pro/Max support, because `$GITHUB_PATH` prepends the
Nix store path before the PyInstaller step runs.

## Test Plan

### Manual Testing
<!-- Hardware: M5 Pro -->
- Trigger a macOS build and verify the bundled macmon is the pinned fork
- Run the built `.app` on M5 Pro/Max and confirm macmon preflight
succeeds

### Automated Testing
- Existing CI build workflow will validate that the macmon binary is
found and bundled correctly

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 09:47:48 +00:00
ciaranbor 30fd5aa1cc Prefer higher % downloaded nodes for API placement previews (#1795)
Follow up to https://github.com/exo-explore/exo/pull/1767

Same thing for placement previews through API
2026-03-25 17:26:31 +00:00
ciaranbor 6de14cfedb Support image generation cancellation (#1774)
## Motivation

Support cancelling image generation, similar to existing support for
cancelling text generation

## Changes

- Dashboard (app.svelte.ts): Wire up AbortController for both
generateImage and editImage API calls. On abort, show "Cancelled"
instead of an error. Clean up the controller in finally.
- Pipeline runner (pipeline/runner.py): Introduce a cancel_checker
callback and NaN-sentinel cancellation protocol for distributed
diffusion:
  - _check_cancellation() - only rank 0 polls the cancel callback
- _send() - replaces data with NaN sentinels when cancelling, so
downstream ranks detect cancellation via _recv_and_check()
  - _recv() / _recv_like() wrappers that eval and check for NaN sentinel
  - After cancellation, drains any pending ring recv to prevent deadlock
  - Skips partial image yields and final decode when cancelled
- Image runner (runner/image_models/runner.py): Deduplicate the
ImageGeneration and ImageEdits match arms into a shared
_run_image_task() method. Thread a cancel_checker closure (backed by the
existing cancel_receiver + cancelled_tasks set) into generate_image().
- Plumbing (distributed_model.py, generate.py): Pass cancel_checker
through the call chain.

## Why It Works

- Rank 0 is the only node that knows about task-level cancellation. When
it detects cancellation, it sends NaN tensors instead of real data.
Higher-order ranks detect the NaN sentinel on recv, set their own
_cancelling flag, and propagate NaN forward
- A drain step after the loop prevents the deadlock case where the last
rank already sent patches that the first would never consume.
- For single-node mode, the loop simply breaks immediately on
cancellation.

## Test Plan

### Automated Testing

New tests in src/exo/worker/tests/unittests/test_image
2026-03-25 16:56:04 +00:00
fc1ae90111 fix: DeepSeek V3.2 warmup crash and tool calling + add catalog cards (#1769)
## Summary

DeepSeek V3.2 (`DeepseekV32ForCausalLM`) is already supported by exo's
inference engine (architecture whitelisted in `model_cards.py`, DSML
encoding added in #1548), but **doesn't work out of the box** due to two
bugs:

### Bug 1: `warmup_inference` passes empty model ID

`warmup_inference()` in `generate.py` accepts `model_id: ModelId` as a
parameter but creates `TextGenerationTaskParams(model=ModelId(""), ...)`
instead of using it. Since `_needs_dsml_encoding()` checks
`"deepseek-v3.2" in task_params.model.lower()`, the empty string never
matches → falls back to `tokenizer.apply_chat_template()` →
**ValueError** because V3.2 has no Jinja chat template.

**Fix:** `model=ModelId("")` → `model=model_id` (one line).

### Bug 2: `_needs_dsml_encoding` limited to tool calling

`_needs_dsml_encoding()` returns `True` only when `task_params.tools` is
present or tool messages exist in `chat_template_messages`. For warmup
and regular chat requests without tools → `return False` → Jinja
fallback → **ValueError**.

Unlike V3.1 (which has a `.jinja` chat template file that transformers
picks up automatically), V3.2 **has no Jinja template at all** — it uses
Python-based DSML encoding for all message types.

**Fix:** For V3.2, always return `True` — DSML encoding handles all
message types.

### Catalog cards

Added inference model cards for:
- `mlx-community/DeepSeek-V3.2-8bit`
- `mlx-community/DeepSeek-V3.2-4bit`

Parameters taken from model `config.json` on HuggingFace, storage sizes
from HF API. Capabilities include `thinking_toggle` (related: #1456).

## Notes

- The model ID string matching approach (`"deepseek-v3.2" in
model.lower()`) is acknowledged tech debt — see #1371 for the planned
architecture-based approach.

## Test plan

- [x] Start exo with DeepSeek V3.2 model → warmup should complete
without crash
- [x] Send a regular chat message (no tools) → should get a response
- [x] Send a chat message with tools → should work as before
- [x] V3.2 cards should appear in the dashboard model catalog

---------

Co-authored-by: user <user@m1.note>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
Co-authored-by: Evan <evanev7@gmail.com>
2026-03-25 16:20:35 +00:00
rltakashige 565ed41c13 Fix occasional warmup bugs by using mlx_generate (#1794)
## Motivation

Warmup occasionally had issues; e.g. #1748 and #1793 because we were
using a standard stream_generate, all of which are issues that are
resolved in the wrapper function mlx_generate.
2026-03-25 15:53:54 +00:00
DeepZimaandEvan 2da740c387 Feat/static peer discovery (#1690)
**Enabling peers to be discovered in environments where mDNS is
unavailable (SSH sessions, headless servers, Docker).**

## Motivation
Exo discovers peers exclusively via mDNS, which works great on a local
network but breaks once you move beyond a single L2 broadcast domain:

- SSH sessions on macOS — TCC blocks mDNS multicast from non-GUI
sessions (#1488)
- Headless servers/rack machines — #1682 ("DGX Spark does not find other
nodes")
- Docker Compose — mDNS is often unavailable across container networks;
e.g. #1462 (E2E test framework) needs an alternative

Related works: 
#1488 (working implementation made by @AlexCheema and closed because SSH
had a GUI workaround),
#1023 (Headscale WAN then closed due to merge conflicts), 
#1656 (discovery cleanup, open). 

This PR introduces an optional bootstrap mechanism for peer discovery
while leaving the existing mDNS behavior unchanged.

## Changes
Adds two new CLI flags:

- `--bootstrap-peers` (env: `EXO_BOOTSTRAP_PEERS`) — comma-separated
libp2p multiaddrs to dial on startup and retry periodically
- `--libp2p-port` — fixed TCP port for libp2p to listen on (default:
OS-assigned). Required when bootstrap peers, so other nodes know which
port to dial.

8 files: 
- `rust/networking/src/discovery.rs`: Store bootstrap addrs, dial in
existing retry loop
- `rust/networking/src/swarm.rs`: Thread `bootstrap_peers` parameter to
`Behaviour`
- `rust/networking/examples/chatroom.rs`: Updated call site for new
create_swarm signature
- `rust/networking/tests/bootstrap_peers.rs`: Integration tests
- `rust/exo_pyo3_bindings/src/networking.rs`: Accept optional
`bootstrap_peers` in PyO3 constructor
- `rust/exo_pyo3_bindings/exo_pyo3_bindings.pyi` : Update type stub 
- `src/exo/routing/router.py`: Pass peers to `NetworkingHandle` 
- `src/exo/main.py` : `--bootstrap-peers` CLI arg +
`EXO_BOOTSTRAP_PEERS` env var

## Why It Works

Bootstrap peers are dialed in the existing retry loop — the same path
taken by peers when mDNS-discovered. The swarm handles connection, Noise
handshake, and gossipsub mesh joining from there.

PeerId is intentionally not required in the multiaddr, the Noise
handshake discovers it.

Docker Compose example:

```yaml
services:
  exo-1:
    environment:
      EXO_BOOTSTRAP_PEERS: "/ip4/exo-2/tcp/30000"
  exo-2:
    environment:
      EXO_BOOTSTRAP_PEERS: "/ip4/exo-1/tcp/30000"
```

## Test Plan

### Manual Testing
<details>
<summary>Docker Compose config</summary>

```
services:
  exo-node1:
    build:
      context: .
      dockerfile: Dockerfile.bootstrap-test
    container_name: exo-bootstrap-node1
    hostname: exo-node1
    command: ["-q", "--libp2p-port", "30000", "--bootstrap-peers", "/ip4/172.30.20.3/tcp/30000"]
    environment:
      - EXO_LIBP2P_NAMESPACE=bootstrap-test
    ports:
      - "52415:52415"
    networks:
      bootstrap-net:
        ipv4_address: 172.30.20.2
    deploy:
      resources:
        limits:
          memory: 4g

  exo-node2:
    build:
      context: .
      dockerfile: Dockerfile.bootstrap-test
    container_name: exo-bootstrap-node2
    hostname: exo-node2
    command: ["-q", "--libp2p-port", "30000", "--bootstrap-peers", "/ip4/172.30.20.2/tcp/30000"]
    environment:
      - EXO_LIBP2P_NAMESPACE=bootstrap-test
    ports:
      - "52416:52415"
    networks:
      bootstrap-net:
        ipv4_address: 172.30.20.3
    deploy:
      resources:
        limits:
          memory: 4g

networks:
  bootstrap-net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.30.20.0/24
```
</details> 

Two containers on a bridge network (`172.30.20.0/24`), fixed IPs,
`--libp2p-port 30000`, cross-referencing `--bootstrap-peers`.

Both nodes found each other and established a connection then ran the
election protocol.

### Automated Testing

4 Rust integration tests in `rust/networking/tests/bootstrap_peers.rs`
(`cargo test -p networking`):

| Test | What it verifies | Result |
|------|-----------------|--------|
| `two_nodes_connect_via_bootstrap_peers` | Node B discovers Node A via
bootstrap addr (real TCP connection) | PASS |
| `create_swarm_with_empty_bootstrap_peers` | Backward compatibility —
no bootstrap peers works | PASS |
| `create_swarm_ignores_invalid_bootstrap_addrs` | Invalid multiaddrs
silently filtered | PASS |
| `create_swarm_with_fixed_port` | `listen_port` parameter works | PASS
|

All 4 pass. The connection test takes ~6s

---------

Signed-off-by: DeepZima <deepzima@outlook.com>
Co-authored-by: Evan <evanev7@gmail.com>
2026-03-25 10:55:12 +00:00
rltakashige 7117d748ec Update dependencies including mlx 0.31.2 (#1789)
Update mlx fork to 0.31.2 and mflux to 0.17.2
2026-03-25 06:03:19 +00:00
Alex CheemaandClaude Opus 4.6 178c617bbb Rename Nemotron to NVIDIA in model picker with logo (#1790)
## Motivation

The Nemotron model family in the model picker sidebar was displaying as
"Nemotron" with a generic checkmark icon. Since these models are
NVIDIA's Nemotron models, the category should be branded as "NVIDIA"
with the official NVIDIA logo, consistent with how other families are
branded (e.g., "llama" → "Meta", "gpt-oss" → "OpenAI").

## Changes

- **FamilySidebar.svelte**: Added `nemotron: "NVIDIA"` to the
`familyNames` mapping so the sidebar displays "NVIDIA" instead of
"Nemotron"
- **FamilyLogos.svelte**: Added the NVIDIA "eye" logo as an inline SVG
for the `nemotron` family, matching the `viewBox="0 0 24 24"` /
`fill="currentColor"` pattern used by all other brand logos
- **ModelPickerModal.svelte**: Added `"nemotron"` to the `familyOrder`
array so NVIDIA appears in a consistent position in the sidebar

## Why It Works

The model picker derives categories from the `family` field in TOML
model cards. Nemotron models already have `family = "nemotron"`, but the
three UI components (display name, logo, sort order) lacked explicit
entries for it, causing fallback behavior (auto-capitalized name,
checkmark icon, alphabetical sorting). Adding explicit entries for all
three aligns NVIDIA with the existing brand pattern.

## Test Plan

### Manual Testing
<!-- Hardware: N/A - dashboard UI change only -->
- Built dashboard successfully (`npm run build`)
- Verified the NVIDIA logo renders in the sidebar alongside existing
brand logos

### Automated Testing
- No test changes needed — this is a purely cosmetic dashboard change

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 01:56:59 +00:00
rltakashige 7277c90389 Fix enable thinking (#1786)
## Motivation

Minor regression from #1746 

## Why It Works

Pass thinking properly

## Test Plan

### Manual Testing
Works, it thinks
2026-03-24 16:26:25 -07:00
Evan QuineyandAlex Cheema 7ee88c1f05 override macmon in flake (#1747)
updates macmon to an upstream fork that fixes m5 max issues.
might see if the upstream version gets merged before we release.

---------

Co-authored-by: Alex Cheema <alexcheema123@gmail.com>
2026-03-24 17:30:19 +00:00
Evan Quiney 509533d49e send error finish reason on failing to parse a tool call (#1785)
a simplification of #1757 which is now stale
2026-03-24 17:21:39 +00:00
rltakashige b6240a97e8 Prevent Qwen3.5 looping by using mlx lm fork (#1784)
## Motivation

Move back to an MLX LM to improve the Qwen 3.5 experience. 

## Test Plan

### Manual Testing
Seems to loop less from testing, no speed regressions.
2026-03-24 17:16:41 +00:00
rltakashige 6cdfbb7e8b Add HF_ENDPOINT in the app settings (#1783)
## Motivation
Some users (primarily in China) are unable to access HuggingFace.co. HF
supports the HF_ENDPOINT env variable, and we also support it, but there
is no way to easily do that from the app currently.

## Test Plan

### Manual Testing

hf-mirror works
empty field works
google.com endpoint fails
2026-03-24 17:05:49 +00:00
Evan Quiney fac6832e5f fix warmup consistency for slow machines (#1748)
a fix from pr #1643 which is now stale - should make prefill more
consistent on very slow machines

## testing
qwen-3.5-35b-a3b loads normally
gpt-oss-120b-mxfp4-q8 loads normally
2026-03-24 16:45:55 +00:00
rltakashige 7df3774ca2 Improve batch performance and stats reporting (#1777)
## Motivation

Batch generation reports incorrect statistics, as mlx lm never clears
the original stats, meaning they get polluted over time.
The dashboard also seems considerably slower than bench statistics.
We also have a large discrepancy between B=1 batch generating and
mlx_generate.
Extracting logprobs is massively expensive, causing up to a 25% slowdown
compared to pure batching.
```
[ 12:02:01.1240AM | INFO    ] step overhead: 3.49ms (next=12.49ms total=15.99ms)
[ 12:02:02.1600AM | INFO    ] step overhead: 3.23ms (next=13.01ms total=16.24ms)
[ 12:02:03.2228AM | INFO    ] step overhead: 3.28ms (next=13.38ms total=16.66ms)
[ 12:02:04.2798AM | INFO    ] step overhead: 3.25ms (next=12.84ms total=16.10ms)
[ 12:02:05.3152AM | INFO    ] step overhead: 3.18ms (next=12.61ms total=15.79ms)
[ 12:02:06.3522AM | INFO    ] step overhead: 3.41ms (next=12.83ms total=16.25ms)
[ 12:02:07.3987AM | INFO    ] step overhead: 3.38ms (next=13.14ms total=16.52ms)
[ 12:02:08.4537AM | INFO    ] step overhead: 1.84ms (next=19.44ms total=21.28ms)
```

## Changes

1. Report stats ourselves instead of using mlx lm's stats for batch
generation (they use perf_counter anyway).
2. Adjust exo bench to match
3. Improve logprobs extraction speed by 10x, improving tps for dashboard
& any requests for logprobs
4. Use an SSE comment to align the speed to the real numbers at the end
of generation
5. Patch mlx for several optimizations given our assumptions and use
cases (e.g. use vllm style RoPE).
6. Switch MLX LM version to latest main, including support for Nemotron
Super and some Qwen3.5 fixes.

## Why It Works
1. Exo bench no longer reports polluted stats
2. Exo bench now handles the reported per-request stats rather than the
aggregate stats
3. The decode speed now jumps back to a real number at the end of the
generation
4. Large batch speedup for rotating KV cache models + 1:1 matching cache
with vllm

## Test Plan

### Manual Testing
Needs testing on OpenCode and CC
Needs eval testing

### Automated Testing
Only going to show the performance optimization difference after the
accurate reporting:

**GPT OSS 20B MXFP4 Q8 (large change)**
Before:
<img width="2466" height="1534" alt="image"
src="https://github.com/user-attachments/assets/88b50637-fca2-4db4-9413-b9eee6e2057e"
/>
<img width="2410" height="1240" alt="image"
src="https://github.com/user-attachments/assets/21e5c76a-2f5f-44d2-8953-121b3ebdbd68"
/>


After:
<img width="2476" height="1472" alt="image"
src="https://github.com/user-attachments/assets/fec5cfbd-fff8-430a-b12e-a329410107a2"
/>
<img width="2454" height="1236" alt="image"
src="https://github.com/user-attachments/assets/0400344b-a4a6-42c0-a9dd-4ee91ade714a"
/>



**Qwen 3.5 35B A3B 8bit (No change)**
Before:
<img width="2414" height="1396" alt="image"
src="https://github.com/user-attachments/assets/e75f0b38-df5d-49fd-ab90-bc1667d981b3"
/>


After:
<img width="2346" height="1234" alt="image"
src="https://github.com/user-attachments/assets/eabfb59c-851f-4d88-b927-e1e699a75cc6"
/>


**Llama 3.2 1B Instruct 4bit (small change)**
Before:
<img width="2516" height="1220" alt="image"
src="https://github.com/user-attachments/assets/c2873655-acff-4536-8263-fb8aea33db80"
/>

After:
<img width="2566" height="1370" alt="image"
src="https://github.com/user-attachments/assets/15f95c75-1c2f-4474-85a2-88c4d0a32543"
/>
2026-03-24 14:03:03 +00:00
ciaranbor 248919c2a8 Fix first start in offline mode crash (#1782)
## Motivation

Running exo in offline mode on a machine where a model has never been
downloaded causes a crash

## Changes

- `download_shard` now catches `FileNotFoundError` from
`fetch_file_list_with_cache` and returns a `not_started` progress
instead of propagating the exception

## Why It Works

A status query should never crash its caller. By returning
`not_started`, the coordinator's existing offline guard (`if
self.offline:` at line 198) is reached and emits a graceful
`DownloadFailed` event. This also eliminates the warning spam on startup
where the status iterator catches the same exception for every
predefined model.

## Test Plan

### Manual Testing
- Start exo with `--offline` on a machine with no cached models, request
a model via the API — should get a graceful failure instead of a crash
2026-03-24 13:58:23 +00:00
ciaranbor 49951e1b1a Sync custom model cards across nodes (#1768)
## Motivation

Custom model cards were only saved locally on the node that handled the
API request.

## Changes

- Added AddCustomModelCard and DeleteCustomModelCard commands
- Added CustomModelCardAdded and CustomModelCardDeleted events
- Added custom_model_cards field to cluster State
- Master handles new commands by emitting corresponding events
- Workers persist model cards to disk and update the in-memory cache on
event receipt
- Separated fetch_from_hf (pure fetch) from disk persistence (now
handled by event-sourcing layer)
- Exposed add_to_card_cache() helper for the worker to update the cache

## Why It Works

- Follows the existing event-sourcing pattern: API → Command → Master →
Event → all Workers
- Every node applies the same events, so custom model cards are
consistent across the cluster

## Test Plan

### Manual Testing

Add/delete a custom model via the API on one node, verify it
appears/disappears on all nodes

### Automated Testing

Existing tests cover apply() logic; new event types follow the same
discriminated-union pattern
2026-03-24 12:51:36 +00:00
ciaranbor e06e70a835 Prefer higher model download % for placement (#1767)
## Motivation

When placing a model instance across the cluster, the master previously
only considered available RAM. This meant it could pick a node that
hasn't downloaded the model yet, even when another node already has it
(or is further along in downloading it).

## Changes

- Added download_status parameter to place_instance() in placement.py
- Added _get_node_download_fraction() to compute 0.0–1.0 download
progress per node/model
- Added _cycle_download_score() to sum download fractions across a
cycle's nodes
- Cycle selection now uses a (download_score, available_ram) tuple key —
download progress is the primary sort, RAM is the tiebreaker
- Passed self.state.downloads into place_instance() from master/main.py

## Why It Works

Python's tuple comparison gives download progress strict priority over
RAM, so a node with the model already downloaded will always be
preferred over one with more free RAM but no download.

## Test Plan

### Automated Testing

3 new tests cover: completed download preferred, higher partial progress
preferred, failed download not preferred over no-download node
2026-03-24 12:11:56 +00:00
vskiwianduser e9fdd8d4af improve logging: add dates to verbose stderr, match file log level to verbosity (#1772)
## Summary

- **Add ISO date to verbose stderr format**: when running with `-v`, the
stderr timestamp changes from `HH:mm:ss.SSS` to `YYYY-MM-DD
HH:mm:ss.SSS`, matching the file log format. This makes it possible to
correlate entries across days when stderr is captured by launchd,
systemd, Docker, or file redirection.
- **File log respects verbosity**: `exo.log` now uses DEBUG level when
`-v` is passed, so the persistent log has the same detail as stderr.
Previously it was always INFO regardless of verbosity.
- **Startup banner with PID**: adds a visual separator and process ID to
the startup message, making it easy to identify session boundaries in
long-running logs (e.g. `grep "Starting EXO"`).

The non-verbose (default) stderr format is unchanged — end-user terminal
experience is not affected.

## Motivation

When exo runs as a service (launchd, systemd, Docker), stderr is
typically captured to a file. Without calendar dates in the timestamp,
it is impossible to tell which day a log entry belongs to. This caused
misidentification of log entries during a multi-day RDMA debugging
session on a 4-node cluster.

The file log (`exo.log`) already had dates and rotation, but only
captured INFO level — missing the DEBUG output needed for postmortem
analysis.

## Test plan

- [x] `basedpyright` — 0 errors, 0 warnings, 0 notes
- [x] `ruff check` — all checks passed
- [x] `pytest` — 249 passed, 1 skipped

Co-authored-by: user <user@m1.note>
2026-03-23 16:06:26 +00:00
Evan Quiney 07598a3af1 teeny refactor (#1753)
api.py keeps growing. it's not tied to the master, so should have it's
own top level folder.

## testing
ci, pytest
2026-03-19 15:57:50 +00:00
ciaranbor 63f57fc193 Ciaran/dashboard download bug (#1755)
## Motivation

Download progress in the dashboard was broken: mainly treating all
download statuses as ongoing

## Changes

- Backend (apply.py): Deduplicate download progress events by model_id
instead of full shard_metadata, preventing duplicate entries per node
- Dashboard (+page.svelte): Extract shared collectDownloadStatus()
helper that both getModelDownloadStatus and getInstanceDownloadStatus
use, eliminating ~100 lines of duplicated logic. Adds proper handling
for
DownloadCompleted/DownloadFailed events, uses a Map to deduplicate
per-node entries, and introduces a typed NodeDownloadStatus with
explicit status states (downloading/completed/partial/pending)
- ModelCard: Replace single aggregate progress bar with per-node
download bars, each color-coded by status. Instance preview now scopes
download status to participating nodes only

## Why It Works

- Deduplicating by model_id in apply.py ensures each node has exactly
one download entry per model
- The perNodeMap in the frontend keeps only the latest event per node,
preventing duplicate bars
- Handling DownloadCompleted allows the UI to show finished downloads
instead of dropping them
- Scoping instance previews to assigned nodes avoids showing irrelevant
download progress
2026-03-19 14:47:45 +00:00
650 changed files with 16057 additions and 11794 deletions

No files matched your search

-20
View File
@@ -1,20 +0,0 @@
from enum import Enum
class HarmonyEncodingName(Enum):
HARMONY_GPT_OSS = ...
class HarmonyEncoding: ...
class HarmonyError(Exception): ...
class Role(Enum):
ASSISTANT = ...
class StreamableParser:
last_content_delta: str
current_channel: str | None
current_recipient: str | None
def __init__(self, encoding: HarmonyEncoding, role: Role = ...) -> None: ...
def process(self, token_id: int) -> None: ...
def load_harmony_encoding(name: HarmonyEncodingName) -> HarmonyEncoding: ...
-17
View File
@@ -1,17 +0,0 @@
class NvmlMemoryInfo:
used: int
total: int
free: int
class NvmlUtilizationRates:
gpu: int
memory: int
def nvmlInit() -> None: ...
def nvmlShutdown() -> None: ...
def nvmlDeviceGetCount() -> int: ...
def nvmlDeviceGetHandleByIndex(index: int) -> object: ...
def nvmlDeviceGetUtilizationRates(handle: object) -> NvmlUtilizationRates: ...
def nvmlDeviceGetTemperature(handle: object, sensor_type: int) -> int: ...
def nvmlDeviceGetPowerUsage(handle: object) -> int: ...
def nvmlDeviceGetMemoryInfo(handle: object) -> NvmlMemoryInfo: ...
-61
View File
@@ -1,61 +0,0 @@
from typing import Any, Sequence
from torch import backends as backends
from torch import cuda as cuda
from torch import distributed as distributed
__version__: str
class version:
cuda: str
class dtype: ...
bfloat16: dtype
float16: dtype
float32: dtype
int8: dtype
int32: dtype
int64: dtype
long: dtype
float8_e4m3fn: dtype
class Tensor:
shape: Sequence[int]
dtype: dtype
def __getitem__(self, key: Any) -> Tensor: ...
def __setitem__(self, key: Any, value: Any) -> None: ...
def to(self, *args: Any, **kwargs: Any) -> Tensor: ...
def cpu(self) -> Tensor: ...
def detach(self) -> Tensor: ...
def clone(self) -> Tensor: ...
def flatten(self, start_dim: int = 0, end_dim: int = -1) -> Tensor: ...
def view(self, *shape: Any) -> Tensor: ...
def squeeze(self, dim: int = ...) -> Tensor: ...
def unsqueeze(self, dim: int) -> Tensor: ...
def permute(self, *dims: int) -> Tensor: ...
def float(self) -> Tensor: ...
def numpy(self) -> Any: ...
def numel(self) -> int: ...
def nelement(self) -> int: ...
@property
def is_cuda(self) -> bool: ...
@property
def device(self) -> device: ...
def __len__(self) -> int: ...
def data_ptr(self) -> int: ...
def tolist(self) -> Any: ...
def abs(self) -> Tensor: ...
def max(self) -> Tensor: ...
def mean(self) -> Tensor: ...
def sum(self, dim: int = ...) -> Tensor: ...
def item(self) -> float: ...
def tensor(data: Any, dtype: dtype | None = None, device: Any = None) -> Tensor: ...
def zeros(*size: Any, dtype: dtype | None = None, device: Any = None) -> Tensor: ...
def empty(*size: Any, dtype: dtype | None = None, device: Any = None) -> Tensor: ...
def from_numpy(ndarray: Any) -> Tensor: ...
def inference_mode() -> Any: ...
class device:
def __init__(self, type: str, index: int = ...) -> None: ...
@@ -1 +0,0 @@
from torch.backends import cuda as cuda
@@ -1 +0,0 @@
def is_built() -> bool: ...
-10
View File
@@ -1,10 +0,0 @@
class _DeviceProperties:
total_memory: int
def is_available() -> bool: ...
def get_device_name(device: int) -> str: ...
def get_device_properties(device: int) -> _DeviceProperties: ...
def empty_cache() -> None: ...
def mem_get_info() -> tuple[int, int]: ...
def synchronize() -> None: ...
def max_memory_allocated() -> int: ...
@@ -1,2 +0,0 @@
def is_initialized() -> bool: ...
def destroy_process_group() -> None: ...
-1
View File
@@ -1 +0,0 @@
__version__: str
-2
View File
@@ -1,2 +0,0 @@
class ModelConfig:
max_model_len: int
-18
View File
@@ -1,18 +0,0 @@
from dataclasses import dataclass
@dataclass
class EngineArgs:
model: str = ...
served_model_name: str | list[str] | None = ...
tokenizer: str | None = ...
trust_remote_code: bool = ...
dtype: str = ...
seed: int = ...
max_model_len: int | None = ...
gpu_memory_utilization: float = ...
enforce_eager: bool = ...
tensor_parallel_size: int = ...
pipeline_parallel_size: int = ...
quantization: str | None = ...
load_format: str = ...
enable_sleep_mode: bool = ...
-17
View File
@@ -1,17 +0,0 @@
class CompletionOutput:
index: int
text: str
token_ids: list[int]
cumulative_logprob: float | None
logprobs: object | None
finish_reason: str | None
stop_reason: int | str | None
def finished(self) -> bool: ...
class RequestOutput:
request_id: str
prompt: str | None
prompt_token_ids: list[int] | None
outputs: list[CompletionOutput]
finished: bool
-11
View File
@@ -1,11 +0,0 @@
class SamplingParams:
n: int
temperature: float
top_p: float
top_k: int
min_p: float
seed: int | None
stop: str | list[str] | None
max_tokens: int | None
logprobs: int | None
repetition_penalty: float
@@ -1,3 +0,0 @@
from vllm.tokenizers.protocol import TokenizerLike
__all__ = ["TokenizerLike"]
@@ -1,15 +0,0 @@
from typing import Protocol
class TokenizerLike(Protocol):
@property
def eos_token_id(self) -> int: ...
@property
def vocab_size(self) -> int: ...
def encode(self, text: str, add_special_tokens: bool = ...) -> list[int]: ...
def decode(self, ids: list[int] | int, skip_special_tokens: bool = ...) -> str: ...
def apply_chat_template(
self,
messages: list[dict[str, str]],
tools: list[dict[str, object]] | None = ...,
**kwargs: object,
) -> str | list[int]: ...
-1
View File
@@ -1 +0,0 @@
-1
View File
@@ -1 +0,0 @@
@@ -1,24 +0,0 @@
from collections.abc import Sequence
from vllm.v1.core.kv_cache_utils import BlockPool, KVCacheBlock
from vllm.v1.kv_cache_interface import KVCacheConfig
class KVCacheBlocks:
blocks: tuple[Sequence[KVCacheBlock], ...]
def __init__(self, blocks: tuple[Sequence[KVCacheBlock], ...]) -> None: ...
def get_block_ids(self) -> tuple[list[int], ...]: ...
class KVCacheManager:
block_pool: BlockPool
kv_cache_config: KVCacheConfig
enable_caching: bool
num_kv_cache_groups: int
coordinator: object
def __init__(self, *args: object, **kwargs: object) -> None: ...
def allocate_slots(
self, request: object, num_new_tokens: int, *args: object, **kwargs: object
) -> KVCacheBlocks | None: ...
def get_computed_blocks(self, request: object) -> tuple[KVCacheBlocks, int]: ...
def create_kv_cache_blocks(
self, blocks: tuple[list[KVCacheBlock], ...]
) -> KVCacheBlocks: ...
@@ -1,16 +0,0 @@
class KVCacheBlock:
block_id: int
ref_cnt: int
def __init__(self, block_id: int) -> None: ...
class FreeKVCacheBlockQueue:
def append_n(self, blocks: list[KVCacheBlock]) -> None: ...
def popleft_n(self, n: int) -> list[KVCacheBlock]: ...
class BlockPool:
blocks: list[KVCacheBlock]
free_block_queue: FreeKVCacheBlockQueue
num_gpu_blocks: int
enable_caching: bool
def get_num_free_blocks(self) -> int: ...
def get_new_blocks(self, num_blocks: int) -> list[KVCacheBlock]: ...
Whitespace-only changes.
@@ -1,22 +0,0 @@
from vllm.config import ModelConfig
from vllm.engine.arg_utils import EngineArgs
from vllm.outputs import RequestOutput
from vllm.sampling_params import SamplingParams
from vllm.tokenizers import TokenizerLike
class LLMEngine:
tokenizer: TokenizerLike | None
model_config: ModelConfig
@classmethod
def from_engine_args(cls, engine_args: EngineArgs) -> LLMEngine: ...
def add_request(
self,
request_id: str,
prompt: str,
params: SamplingParams,
arrival_time: float | None = ...,
) -> None: ...
def step(self) -> list[RequestOutput]: ...
def has_unfinished_requests(self) -> bool: ...
def get_tokenizer(self) -> TokenizerLike: ...
@@ -1,23 +0,0 @@
from dataclasses import dataclass
@dataclass
class KVCacheSpec:
block_size: int
num_kv_heads: int
head_size: int
@dataclass
class KVCacheGroupSpec:
layer_names: list[str]
kv_cache_spec: KVCacheSpec
@dataclass
class KVCacheTensorSpec:
shared_by: list[str]
size: int
@dataclass
class KVCacheConfig:
num_blocks: int
kv_cache_groups: list[KVCacheGroupSpec]
kv_cache_tensors: list[KVCacheTensorSpec]
-6
View File
@@ -1,6 +0,0 @@
class Request:
request_id: str
prompt_token_ids: list[int] | None
num_prompt_tokens: int
num_computed_tokens: int
num_tokens: int
@@ -1 +0,0 @@
@@ -1,24 +0,0 @@
import torch
class _CompilationConfig:
static_forward_context: dict[str, object]
class _ModelConfig:
hf_config: object
class GPUModelRunner:
kv_caches: list[torch.Tensor]
compilation_config: _CompilationConfig
model_config: _ModelConfig | None
def _allocate_kv_cache_tensors(
self, kv_cache_config: object
) -> dict[str, torch.Tensor]: ...
def initialize_kv_cache_tensors(
self, kv_cache_config: object, kernel_block_sizes: list[int]
) -> dict[str, torch.Tensor]: ...
def _reshape_kv_cache_tensors(
self,
kv_cache_config: object,
raw_tensors: dict[str, torch.Tensor],
kernel_block_sizes: list[int],
) -> dict[str, torch.Tensor]: ...
@@ -1,6 +0,0 @@
from vllm.v1.worker.gpu_model_runner import GPUModelRunner
class Worker:
model_runner: GPUModelRunner
def determine_available_memory(self) -> int: ...
def initialize_from_config(self, kv_cache_config: object) -> None: ...
-1
View File
@@ -1 +0,0 @@
def extract_layer_index(layer_name: str, num_attn_module: int) -> int: ...
+7
View File
@@ -1 +1,8 @@
use flake
# creates .venv if doesn't exist and loads its environment
export VIRTUAL_ENV=".venv"
if ! [ -d "./$VIRTUAL_ENV" ]; then
uv venv
fi
layout python
+124 -2
View File
@@ -159,7 +159,7 @@ jobs:
fi
- name: Install Homebrew packages
run: brew install just awscli macmon
run: brew install just awscli
- name: Install UV
uses: astral-sh/setup-uv@v6
@@ -239,10 +239,92 @@ jobs:
# Export keychain path for other steps
echo "BUILD_KEYCHAIN_PATH=$KEYCHAIN_PATH" >> $GITHUB_ENV
# ============================================================
# Pre-flight credential / profile validation
# Runs BEFORE the ~16 min build so auth/expiry failures surface in <1 min.
# ============================================================
- name: Validate Apple notarization credentials
env:
APPLE_NOTARIZATION_USERNAME: ${{ secrets.APPLE_NOTARIZATION_USERNAME }}
APPLE_NOTARIZATION_PASSWORD: ${{ secrets.APPLE_NOTARIZATION_PASSWORD }}
APPLE_NOTARIZATION_TEAM: ${{ secrets.APPLE_NOTARIZATION_TEAM }}
run: |
# All-or-nothing: either all three creds are set, or none are.
CRED_COUNT=0
for v in "$APPLE_NOTARIZATION_USERNAME" "$APPLE_NOTARIZATION_PASSWORD" "$APPLE_NOTARIZATION_TEAM"; do
[[ -n "$v" ]] && CRED_COUNT=$((CRED_COUNT + 1))
done
if [[ "$CRED_COUNT" -eq 0 ]]; then
echo "No notarization credentials configured — skipping notarization for this build."
exit 0
fi
if [[ "$CRED_COUNT" -ne 3 ]]; then
echo "ERROR: partial notarization credentials set ($CRED_COUNT/3). Aborting before build."
exit 1
fi
# Cheap, ~5s, auth-only call. Fails instantly with a clear message if
# the app-specific password is stale, wrong team-id, etc.
echo "Verifying Apple notarization credentials via notarytool history..."
if ! xcrun notarytool history \
--apple-id "$APPLE_NOTARIZATION_USERNAME" \
--password "$APPLE_NOTARIZATION_PASSWORD" \
--team-id "$APPLE_NOTARIZATION_TEAM" >/dev/null; then
echo "ERROR: notarytool rejected the provided credentials. Fix before rerunning."
echo "Common causes: app-specific password expired/revoked, wrong team-id,"
echo "Apple ID not on the team, or 2FA not configured for this Apple ID."
exit 1
fi
echo "Apple notarization credentials OK."
- name: Validate provisioning profile expiry
run: |
PROFILE="$HOME/Library/Developer/Xcode/UserData/Provisioning Profiles/EXO.provisionprofile"
if [[ ! -f "$PROFILE" ]]; then
echo "ERROR: provisioning profile not found at $PROFILE"
exit 1
fi
EXPIRY=$(security cms -D -i "$PROFILE" | plutil -extract ExpirationDate raw -o - - 2>/dev/null || true)
if [[ -z "$EXPIRY" ]]; then
echo "WARNING: could not read ExpirationDate from provisioning profile; skipping expiry check."
exit 0
fi
# Try a couple of known plutil date formats. If none parse, skip the check rather
# than risk a false-positive "expired" block on a format we didn't anticipate.
EXPIRY_EPOCH=""
for fmt in "%Y-%m-%dT%H:%M:%SZ" "%Y-%m-%d %H:%M:%S %z" "%Y-%m-%d %H:%M:%S +0000"; do
if parsed=$(date -j -f "$fmt" "$EXPIRY" +%s 2>/dev/null); then
EXPIRY_EPOCH="$parsed"
break
fi
done
if [[ -z "$EXPIRY_EPOCH" ]]; then
echo "WARNING: could not parse ExpirationDate '$EXPIRY'; skipping expiry check."
exit 0
fi
NOW_EPOCH=$(date +%s)
if [[ "$EXPIRY_EPOCH" -le "$NOW_EPOCH" ]]; then
echo "ERROR: provisioning profile expired on $EXPIRY. Regenerate it before rerunning."
exit 1
fi
DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
echo "Provisioning profile valid until $EXPIRY ($DAYS_LEFT days remaining)."
if [[ "$DAYS_LEFT" -lt 14 ]]; then
echo "WARNING: profile expires in under 14 days — regenerate soon."
fi
# ============================================================
# Build the bundle
# ============================================================
- name: Add pinned macmon to PATH
run: |
MACMON_DIR=$(nix develop --command sh -c 'dirname $(which macmon)')
echo "Using macmon from: $MACMON_DIR"
echo "$MACMON_DIR" >> $GITHUB_PATH
# Remove any Homebrew macmon so PyInstaller can't accidentally pick it up
brew uninstall macmon 2>/dev/null || true
- name: Build PyInstaller bundle
run: uv run pyinstaller packaging/pyinstaller/exo.spec
@@ -298,11 +380,41 @@ jobs:
APPLE_NOTARIZATION_PASSWORD: ${{ secrets.APPLE_NOTARIZATION_PASSWORD }}
APPLE_NOTARIZATION_TEAM: ${{ secrets.APPLE_NOTARIZATION_TEAM }}
run: |
set -o pipefail
cd output
security unlock-keychain -p "$MACOS_CERTIFICATE_PASSWORD" "$BUILD_KEYCHAIN_PATH"
SIGNING_IDENTITY=$(security find-identity -v -p codesigning "$BUILD_KEYCHAIN_PATH" | awk -F '"' '{print $2}')
# Fail fast if notarization creds are partial. All-or-nothing.
CRED_COUNT=0
for v in "$APPLE_NOTARIZATION_USERNAME" "$APPLE_NOTARIZATION_PASSWORD" "$APPLE_NOTARIZATION_TEAM"; do
[[ -n "$v" ]] && CRED_COUNT=$((CRED_COUNT + 1))
done
if [[ "$CRED_COUNT" -ne 0 && "$CRED_COUNT" -ne 3 ]]; then
echo "ERROR: partial Apple notarization credentials set ($CRED_COUNT/3). Aborting."
exit 1
fi
/usr/bin/codesign --deep --force --timestamp --options runtime \
--sign "$SIGNING_IDENTITY" EXO.app
# Pre-flight: verify the signed app BEFORE building DMG and submitting to Apple.
# If this fails, notarization will fail too — cheap way to fail in seconds, not 15 minutes.
echo "===== codesign --verify EXO.app ====="
if ! /usr/bin/codesign --verify --deep --strict --verbose=2 EXO.app; then
echo "ERROR: EXO.app failed codesign verification. Dumping signing status of every executable:"
find EXO.app -type f \( -perm -111 -o -name "*.dylib" -o -name "*.so" -o -name "*.framework" \) -print0 |
while IFS= read -r -d '' f; do
printf -- '--- %s\n' "$f"
/usr/bin/codesign -dv --verbose=2 "$f" 2>&1 | sed 's/^/ /' || true
done
exit 1
fi
# Gatekeeper assessment. A failure here strongly predicts notarization rejection.
echo "===== spctl assessment (predicts notarization outcome) ====="
/usr/bin/spctl -a -vvv -t install EXO.app || echo "WARNING: spctl assessment failed — notarization is likely to fail too."
mkdir -p dmg-root
cp -R EXO.app dmg-root/
ln -s /Applications dmg-root/Applications
@@ -310,12 +422,22 @@ jobs:
hdiutil create -volname "EXO" -srcfolder dmg-root -ov -format UDZO "$DMG_NAME"
/usr/bin/codesign --force --timestamp --options runtime \
--sign "$SIGNING_IDENTITY" "$DMG_NAME"
echo "===== codesign --verify DMG ====="
if ! /usr/bin/codesign --verify --verbose=2 "$DMG_NAME"; then
echo "ERROR: DMG failed codesign verification."
exit 1
fi
if [[ -n "$APPLE_NOTARIZATION_USERNAME" ]]; then
echo "===== notarytool submit ====="
# `|| true` so set -e doesn't abort before we can echo output / fetch the log.
# We rely on the parsed STATUS below to decide pass/fail.
SUBMISSION_OUTPUT=$(xcrun notarytool submit "$DMG_NAME" \
--apple-id "$APPLE_NOTARIZATION_USERNAME" \
--password "$APPLE_NOTARIZATION_PASSWORD" \
--team-id "$APPLE_NOTARIZATION_TEAM" \
--wait --timeout 15m 2>&1)
--wait --timeout 15m 2>&1) || true
echo "$SUBMISSION_OUTPUT"
SUBMISSION_ID=$(echo "$SUBMISSION_OUTPUT" | awk 'tolower($1)=="id:" && $2 ~ /^[0-9a-fA-F-]+$/ {print $2; exit}')
-3
View File
@@ -91,9 +91,6 @@ jobs:
nix build .#metal-toolchain
fi
# Build mlx (depends on metal-toolchain)
nix build .#mlx
- name: Build all Nix outputs
run: |
nix flake show --json | jq -r '
+2
View File
@@ -38,3 +38,5 @@ bench/**/*.json
# tmp
tmp/models
/build/exo
/.claude/skills
-31
View File
@@ -1,31 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="EMPTY_MODULE" version="4">
<component name="FacetManager">
<facet type="Python" name="Python facet">
<configuration sdkName="Python 3.13 virtualenv at ~/Desktop/exo/.venv" />
</facet>
</component>
<component name="Go" enabled="true" />
<component name="NewModuleRootManager">
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/scripts/src" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/src" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/rust/exo_pyo3_bindings/src" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/rust/exo_pyo3_bindings/tests" isTestSource="true" />
<sourceFolder url="file://$MODULE_DIR$/rust/util/src" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/rust/networking/examples" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/rust/networking/src" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/rust/networking/tests" isTestSource="true" />
<sourceFolder url="file://$MODULE_DIR$/rust/system_custodian/src" isTestSource="false" />
<excludeFolder url="file://$MODULE_DIR$/.venv" />
<excludeFolder url="file://$MODULE_DIR$/.direnv" />
<excludeFolder url="file://$MODULE_DIR$/build" />
<excludeFolder url="file://$MODULE_DIR$/dist" />
<excludeFolder url="file://$MODULE_DIR$/.go_cache" />
<excludeFolder url="file://$MODULE_DIR$/rust/target" />
</content>
<orderEntry type="jdk" jdkName="Python 3.13 (exo)" jdkType="Python SDK" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" name="Python 3.13 virtualenv at ~/Desktop/exo/.venv interpreter library" level="application" />
</component>
</module>
+1 -1
View File
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ExternalDependencies">
<plugin id="systems.fehn.intellijdirenv" />
<plugin id="al.aoli.intellijdirenv" />
</component>
</project>
-14
View File
@@ -1,14 +0,0 @@
<component name="InspectionProjectProfileManager">
<profile version="1.0">
<option name="myName" value="Project Default" />
<inspection_tool class="PyCompatibilityInspection" enabled="true" level="WARNING" enabled_by_default="true">
<option name="ourVersions">
<value>
<list size="1">
<item index="0" class="java.lang.String" itemvalue="3.14" />
</list>
</value>
</option>
</inspection_tool>
</profile>
</component>
-3
View File
@@ -4,7 +4,4 @@
<option name="sdkName" value="Python 3.13 (exo)" />
</component>
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.13 (exo)" project-jdk-type="Python SDK" />
<component name="PythonCompatibilityInspectionAdvertiser">
<option name="version" value="3" />
</component>
</project>
File renamed without changes.
File renamed without changes.
@@ -2,11 +2,10 @@
This type stub file was generated by pyright.
"""
from typing import Protocol
import mlx.core as mx
import PIL.Image
import tqdm
from typing import Protocol
from mflux.models.common.config.config import Config
class BeforeLoopCallback(Protocol):
@@ -3,7 +3,6 @@ This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
from mflux.callbacks.callback import (
AfterLoopCallback,
BeforeLoopCallback,
@@ -2,11 +2,10 @@
This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
import mlx.core as mx
import PIL.Image
import tqdm
from typing import TYPE_CHECKING
from mflux.callbacks.callback_registry import CallbackRegistry
from mflux.models.common.config.config import Config
File renamed without changes.
File renamed without changes.
@@ -2,12 +2,11 @@
This type stub file was generated by pyright.
"""
import mlx.core as mx
from pathlib import Path
from typing import Any
import mlx.core as mx
from mflux.models.common.config.model_config import ModelConfig
from tqdm import tqdm
from mflux.models.common.config.model_config import ModelConfig
logger = ...
@@ -2,11 +2,10 @@
This type stub file was generated by pyright.
"""
import mlx.core as mx
from functools import lru_cache
from typing import Literal
import mlx.core as mx
class ModelConfig:
precision: mx.Dtype = ...
def __init__(
@@ -2,10 +2,10 @@
This type stub file was generated by pyright.
"""
import mlx.core as mx
from pathlib import Path
from typing import TYPE_CHECKING, TypeAlias
import mlx.core as mx
from mlx import nn
from mflux.models.common.vae.tiling_config import TilingConfig
from mflux.models.fibo.latent_creator.fibo_latent_creator import FiboLatentCreator
from mflux.models.flux.latent_creator.flux_latent_creator import FluxLatentCreator
@@ -13,7 +13,6 @@ from mflux.models.qwen.latent_creator.qwen_latent_creator import QwenLatentCreat
from mflux.models.z_image.latent_creator.z_image_latent_creator import (
ZImageLatentCreator,
)
from mlx import nn
if TYPE_CHECKING:
LatentCreatorType: TypeAlias = type[
@@ -2,8 +2,8 @@
This type stub file was generated by pyright.
"""
from mflux.models.common.lora.layer.linear_lora_layer import LoRALinear
from mlx import nn
from mflux.models.common.lora.layer.linear_lora_layer import LoRALinear
class FusedLoRALinear(nn.Module):
def __init__(
@@ -2,11 +2,10 @@
This type stub file was generated by pyright.
"""
from collections.abc import Callable
from dataclasses import dataclass
import mlx.core as mx
import mlx.nn as nn
from collections.abc import Callable
from dataclasses import dataclass
from mflux.models.common.lora.mapping.lora_mapping import LoRATarget
@dataclass
@@ -2,12 +2,11 @@
This type stub file was generated by pyright.
"""
import mlx.core as mx
from collections.abc import Callable
from dataclasses import dataclass
from typing import List, Protocol
import mlx.core as mx
@dataclass
class LoRATarget:
model_path: str
@@ -36,3 +36,4 @@ class Rule(NamedTuple):
name: str
check: str
action: QuantizationAction | PathAction | LoraAction | ConfigAction
...
@@ -3,7 +3,6 @@ This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
from mflux.models.common.config.model_config import ModelConfig
if TYPE_CHECKING: ...
@@ -2,9 +2,8 @@
This type stub file was generated by pyright.
"""
from abc import ABC, abstractmethod
import mlx.core as mx
from abc import ABC, abstractmethod
class BaseScheduler(ABC):
@property
@@ -2,9 +2,8 @@
This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
import mlx.core as mx
from typing import TYPE_CHECKING
from mflux.models.common.config.config import Config
from mflux.models.common.schedulers.base_scheduler import BaseScheduler
@@ -2,9 +2,8 @@
This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
import mlx.core as mx
from typing import TYPE_CHECKING
from mflux.models.common.config.config import Config
from mflux.models.common.schedulers.base_scheduler import BaseScheduler
@@ -2,9 +2,8 @@
This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
import mlx.core as mx
from typing import TYPE_CHECKING
from mflux.models.common.config.config import Config
from mflux.models.common.schedulers.base_scheduler import BaseScheduler
@@ -4,10 +4,9 @@ This type stub file was generated by pyright.
from abc import ABC, abstractmethod
from typing import Protocol, runtime_checkable
from mflux.models.common.tokenizer.tokenizer_output import TokenizerOutput
from PIL import Image
from transformers import PreTrainedTokenizer
from mflux.models.common.tokenizer.tokenizer_output import TokenizerOutput
"""
This type stub file was generated by pyright.
@@ -3,7 +3,6 @@ This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
from mflux.models.common.tokenizer.tokenizer import BaseTokenizer
from mflux.models.common.weights.loading.weight_definition import TokenizerDefinition
@@ -2,9 +2,8 @@
This type stub file was generated by pyright.
"""
from dataclasses import dataclass
import mlx.core as mx
from dataclasses import dataclass
"""
This type stub file was generated by pyright.
@@ -2,9 +2,8 @@
This type stub file was generated by pyright.
"""
from typing import Callable
import mlx.core as mx
from typing import Callable
class VAETiler:
@staticmethod
@@ -3,8 +3,8 @@ This type stub file was generated by pyright.
"""
import mlx.core as mx
from mflux.models.common.vae.tiling_config import TilingConfig
from mlx import nn
from mflux.models.common.vae.tiling_config import TilingConfig
class VAEUtil:
@staticmethod
@@ -2,9 +2,8 @@
This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
import mlx.nn as nn
from typing import TYPE_CHECKING
from mflux.models.common.weights.loading.loaded_weights import LoadedWeights
from mflux.models.common.weights.loading.weight_definition import (
ComponentDefinition,
@@ -2,12 +2,11 @@
This type stub file was generated by pyright.
"""
from dataclasses import dataclass
from typing import TYPE_CHECKING, Callable, List, TypeAlias
import mlx.core as mx
from mflux.models.common.tokenizer.tokenizer import BaseTokenizer
from dataclasses import dataclass
from typing import Callable, List, TYPE_CHECKING, TypeAlias
from mflux.models.common.weights.mapping.weight_mapping import WeightTarget
from mflux.models.common.tokenizer.tokenizer import BaseTokenizer
from mflux.models.depth_pro.weights.depth_pro_weight_definition import (
DepthProWeightDefinition,
)
@@ -3,7 +3,6 @@ This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING
from mflux.models.common.weights.loading.loaded_weights import LoadedWeights
from mflux.models.common.weights.loading.weight_definition import (
ComponentDefinition,
@@ -2,9 +2,8 @@
This type stub file was generated by pyright.
"""
from typing import Dict, List, Optional
import mlx.core as mx
from typing import Dict, List, Optional
from mflux.models.common.weights.mapping.weight_mapping import WeightTarget
class WeightMapper:
@@ -2,11 +2,10 @@
This type stub file was generated by pyright.
"""
import mlx.core as mx
from dataclasses import dataclass
from typing import Callable, List, Optional, Protocol
import mlx.core as mx
"""
This type stub file was generated by pyright.
"""
@@ -2,8 +2,7 @@
This type stub file was generated by pyright.
"""
from typing import TYPE_CHECKING, Any
from typing import Any, TYPE_CHECKING
from mflux.models.common.weights.loading.weight_definition import WeightDefinitionType
if TYPE_CHECKING: ...
@@ -2,10 +2,9 @@
This type stub file was generated by pyright.
"""
import mlx.core as mx
from dataclasses import dataclass
from pathlib import Path
import mlx.core as mx
from PIL import Image
@dataclass
@@ -14,6 +13,7 @@ class DepthResult:
depth_array: mx.array
min_depth: float
max_depth: float
...
class DepthPro:
def __init__(self, quantize: int | None = ...) -> None: ...
Loaded 100 of 650 files, more files were not shown because too many files have changed in this diff. Show more