Compare commits

...
Author SHA1 Message Date
Andrei Cravtov 11f9d000e5 Merge branch 'main' into andrei/force_oom 2026-05-15 13:53:11 +01:00
Andrei Cravtov 551e94ed38 fin 2026-05-15 13:52:53 +01:00
rltakashigeandEvan bc6661e6aa Add node backends to model cards (#2071)
Co-authored-by: Evan <evanev7@gmail.com>
2026-05-15 12:52:12 +00:00
Andrei Cravtov b0834c8a2e ff 2026-05-15 13:46:26 +01:00
Andrei Cravtov d545a4ea25 fff 2026-05-15 13:42:02 +01:00
Andrei CravtovandEvan Quiney 14aab35688 Runner error handling (#2093)
# Runner error handling

## Motivation

Runner failures were mostly surfaced as plain shutdown messages, which
made root cause hard to spot from API errors or runner status.

This adds a MVP path for preserving runner crash context and attaching
known stderr diagnostics to failure reports.

## Changes

- Added `RunnerTerminationError` for Python exceptions raised inside
runner bootstrap
- Changed runner bootstrap to send `Event | RunnerTerminationError` over
the private runner channel
- Moved public `RunnerFailed` emission back into supervisor
- Added stderr-only `RunnerDiagnosticCollector`
- + Added known diagnostics for Metal GPU timeout, ring socket receive
errno, and ring transport abort
- Added diagnostics to `RunnerFailed` and `ErrorChunk`
- Tweaked async process termination to join briefly before
terminate/kill
- Updated tests/fixtures for new failure payload shape
- Added Ruff VS Code formatter settings

## Why It Works

Runner child now reports raw-ish failure context to supervisor instead
of publishing failed status directly.

Supervisor still owns process lifecycle, exit code/signal handling,
in-flight task error chunks, and final runner status. Stderr diagnostics
stay best effort and only known root-cause variants are surfaced.

## Test Plan

### Manual Testing

Hardware: remote runner logs from e16/e11/e4/e2

What you did:
- inspected live runner stderr logs
- used observed Metal GPU timeout and ring socket errors as initial
diagnostic targets

### Automated Testing

- `nix flake check`
- supervisor test covers error chunk + failed status emission
- plan lifecycle test updated for failed runner diagnostics
- type/lint checks cover new runner channel union

---------

Co-authored-by: Evan Quiney <evanev7@gmail.com>
2026-05-15 12:40:59 +00:00
Andrei Cravtov 0e7721ba0e ffff 2026-05-15 13:28:08 +01:00
Andrei Cravtov abb4402036 fff 2026-05-15 13:26:13 +01:00
Andrei Cravtov e8b9c808a6 ff 2026-05-15 13:23:11 +01:00
Andrei Cravtov cf00cd83ed fff 2026-05-15 13:20:13 +01:00
Andrei Cravtov 41ff366b1d fff 2026-05-15 13:01:55 +01:00
Andrei Cravtov 303600616a ff 2026-05-15 12:45:12 +01:00
Andrei Cravtov 51d9912fcf ff 2026-05-15 12:37:04 +01:00
Andrei Cravtov d5da78b39b ff 2026-05-15 12:32:16 +01:00
Andrei Cravtov 88027475a5 ff 2026-05-15 12:30:50 +01:00
Andrei Cravtov 423dc407f8 ff 2026-05-15 12:24:42 +01:00
Andrei Cravtov f62b3b09d0 ff 2026-05-15 11:58:47 +01:00
Andrei Cravtov 94424d92c3 ff 2026-05-15 11:56:39 +01:00
Andrei Cravtov e029a1ea65 print 2026-05-15 11:51:13 +01:00
Andrei Cravtov 49e5350b17 ff 2026-05-15 11:47:56 +01:00
Andrei Cravtov feaf05f82d ooom 2 2026-05-15 11:43:08 +01:00
Andrei Cravtov 280031d431 Use MemoryUsage for MLX OOM fallback 2026-05-15 11:05:54 +01:00
Andrei Cravtov 58551f2c32 nix-direnv 2026-05-15 10:57:00 +01:00
Andrei Cravtov cecc37085a Move one-shot macmon sampling helper 2026-05-14 20:25:34 +01:00
Andrei Cravtov a73be3c2bb Scope MLX OOM ceil division helper 2026-05-14 20:13:28 +01:00
Andrei Cravtov acec94316d Inline one-off MLX OOM helpers 2026-05-14 20:05:33 +01:00
Andrei Cravtov 5c7068387d yes 2026-05-14 19:57:53 +01:00
Andrei Cravtov d62b333405 Fix MLX utils type ignores 2026-05-14 19:51:31 +01:00
Andrei Cravtov eeedc6c17a Add adaptive MLX force OOM helper 2026-05-14 19:49:40 +01:00
Andrei Cravtov 2d11b6520f yes 2026-05-14 19:45:07 +01:00
Andrei Cravtov 42b26af63b yes 2026-05-14 17:56:56 +01:00
Andrei Cravtov 1649708ccb yes 2026-05-14 17:55:51 +01:00
Andrei Cravtov fbe962b08e Merge branch 'main' into andrei/error-handling 2026-05-14 17:44:53 +01:00
Andrei Cravtov 38258fb5f3 Merge branch 'main' into andrei/error-handling
# Conflicts:
#	uv.lock
2026-05-14 17:33:33 +01:00
Heidar 88d46d46fd fix: omit null delta fields in streaming chat completions (issue #2082) (#2092)
## Motivation

Streaming /v1/chat/completions responses emitted null for tool_calls,
function_call, name, and tool_call_id in every delta chunk. The OpenAI
streaming spec marks these fields as non-nullable — they must either
carry a
  real value or be absent entirely. Spec-correct clients doing
delta.get("tool_calls", []) receive None and crash with 'NoneType'
object is
  not iterable.

Root cause: the streaming serialisation path called model_dump_json()
without
exclude_none=True, while the request-parsing path already used it
correctly.
Three call sites in chat_completions.py and two in responses.py were
affected.

## Testing

Before — every delta carries explicit nulls:

  $ curl -sN -X POST http://localhost:52415/v1/chat/completions \
    -H 'Content-Type: application/json' \
-d
'{"model":"mlx-community/Qwen3.5-2B-MLX-8bit","messages":[{"role":"user","
  content":"hi"}],"max_tokens":3,"stream":true}' \
    | grep "^data: "
data:
{"id":"7c4dae10-...","choices":[{"index":0,"delta":{"role":"assistant","c

ontent":null,"reasoning_content":"Okay","name":null,"tool_calls":null,"tool_cal

l_id":null,"function_call":null},"logprobs":null,"finish_reason":null,"usage":n
  ull}],"usage":null,"service_tier":null}
data:
{"id":"7c4dae10-...","choices":[{"index":0,"delta":{"role":"assistant","c

ontent":null,"reasoning_content":",","name":null,"tool_calls":null,"tool_call_i

d":null,"function_call":null},"logprobs":null,"finish_reason":null,"usage":null
  }],"usage":null,"service_tier":null}
data:
{"id":"7c4dae10-...","choices":[{"index":0,"delta":{"role":"assistant","c
ontent":"
the","reasoning_content":null,"name":null,"tool_calls":null,"tool_cal

l_id":null,"function_call":null},"logprobs":null,"finish_reason":"length","usag
  e":{"prompt_tokens":11,...}}],"usage":null,"service_tier":null}
  data: [DONE]

  After — only populated fields are emitted:
data:
{"id":"demo","object":"chat.completion","created":...,"model":"mlx-commun

ity/Qwen3.5-2B-MLX-8bit","choices":[{"index":0,"delta":{"role":"assistant","rea
  soning_content":"Okay"}}]}
data:
{"id":"demo","object":"chat.completion","created":...,"model":"mlx-commun

ity/Qwen3.5-2B-MLX-8bit","choices":[{"index":0,"delta":{"role":"assistant","rea
  soning_content":","}}]}
data:
{"id":"demo","object":"chat.completion","created":...,"model":"mlx-commun

ity/Qwen3.5-2B-MLX-8bit","choices":[{"index":0,"delta":{"role":"assistant","con
tent":"
the"},"finish_reason":"length"}],"usage":{"prompt_tokens":11,"completio
  n_tokens":3,"total_tokens":14,...}}
  data: [DONE]
2026-05-14 16:32:54 +00:00
Andrei Cravtov 86398a2bd1 ffff 2026-05-14 17:31:47 +01:00
Andrei Cravtov bab245759d fff 2026-05-14 17:26:24 +01:00
Andrei Cravtov 2f506167e0 fff 2026-05-14 17:24:20 +01:00
HeidarandClaude Opus 4.7 e8ec8d5010 fix ollama API compatibility for VS Code Copilot (#2091)
Ollama adapter fixes for VS Code Copilot (#2042):

  - /api/version: bare semver "1.0.0" - Copilot parseInts each segment.
- /api/show: populate model_info + capabilities - Copilot crashes on
null model_info and filters by `tools`.
- Add POST /ollama/v1/chat/completions - ollama serves the OpenAI-compat
route here, BYOK clients 405 without it.


Before:
<img width="1380" height="144" alt="image"
src="https://github.com/user-attachments/assets/99d5464f-187d-4432-9a31-8229c55aa209"
/>

After:
<img width="1362" height="181" alt="image"
src="https://github.com/user-attachments/assets/361dc006-d8df-435f-8d8b-4fa4f44a8c23"
/>
<img width="279" height="909" alt="image"
src="https://github.com/user-attachments/assets/4621aba7-bd57-4762-8568-34a3383a6025"
/>

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-14 16:12:58 +00:00
Andrei Cravtov 88d4c37bfa fff 2026-05-14 17:12:41 +01:00
Andrei Cravtov 9ec09ec59d ff 2026-05-14 17:07:46 +01:00
Heidar 1fd15d59fc create directory on startup (#2089)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

When you first run `uv run exo` you get an error like :

`FileNotFoundError: [Errno 2] No such file or directory:
'/Users/heidar/.exo/models'`

Manually tested on Macbook Pro M1 32GB

Fixes issue - https://github.com/exo-explore/exo/issues/2090
2026-05-14 16:03:51 +00:00
Andrei Cravtov cd22516186 ff 2026-05-14 16:54:37 +01:00
Andrei Cravtov ddc6e95e68 fff 2026-05-14 16:51:20 +01:00
Andrei Cravtov feef63b98c fff 2026-05-14 16:50:39 +01:00
Andrei Cravtov 3611c33b61 ff 2026-05-14 16:06:11 +01:00
Andrei Cravtov 547b58bf48 ff 2026-05-14 15:51:02 +01:00
Andrei Cravtov 36b1b82b48 ff 2026-05-14 15:49:32 +01:00
Andrei Cravtov b1d09b026d ff 2026-05-14 15:29:40 +01:00
Andrei Cravtov faafc6f4ad yes 2026-05-14 15:17:34 +01:00
Andrei Cravtov 2969b7d4c6 yes 2026-05-14 15:16:48 +01:00
Andrei Cravtov 6c13b4550f yes 2026-05-14 15:11:17 +01:00
Andrei Cravtov 3fc56b2da5 yes 2026-05-14 15:08:58 +01:00
Andrei Cravtov 9c4cc084db yes 2026-05-14 12:36:14 +01:00
Andrei Cravtov 49d7555ea6 yes 2026-05-14 12:31:49 +01:00
Andrei Cravtov 32155bdf00 yes 2026-05-14 12:31:10 +01:00
Andrei Cravtov 163ef39bbb yes 2026-05-14 12:29:34 +01:00
Andrei Cravtov 9636e86855 yes 2026-05-14 12:25:10 +01:00
Andrei Cravtov 4ea2532dd6 yes 2026-05-14 12:14:09 +01:00
Andrei Cravtov 1d17f4de18 yes 2026-05-14 12:09:34 +01:00
Andrei Cravtov 1c7f514ca3 yes 2026-05-14 12:04:43 +01:00
Andrei Cravtov c3aa1f47fc yes 2026-05-14 11:56:00 +01:00
Andrei Cravtov 9686c02251 yes 2026-05-14 11:50:21 +01:00
Andrei Cravtov 2f9ba7d1de yes 2026-05-14 11:14:39 +01:00
Evan Quiney 4466cd5323 use custom mlx sources for linux (#2087)
switch to hosting mlx sources on github & cachix instead of using a
broken version of mlx. closes #2043.
2026-05-13 10:45:11 +01:00
Andrei Cravtov ab172322c5 Merge branch 'main' into andrei/error-handling 2026-05-12 11:49:17 +01:00
Andrei Cravtov ed2d10bdc6 Redirect runner stdout/stderr to file logs (#2084)
## Motivation

We want to use log mining tools like
[Drain3](https://github.com/logpai/Drain3) to get standardized error
formats, but for that we should record runner stdout/stderr in a massive
append-only log to gather training data for such tools. Also useful for
future opt-in telemetry.

## Changes

The stdout/stderr from runner now splits into 3 tasks: 
1) raw write to dedicated runner logs 
2) sanitized line-by-line logging with log-guru 
3) stub for further error-processing (i.e. turning lines into errors)

### Manual Testing
Works on 4x mac mini clusted connected as TB4 ring.
2026-05-12 11:48:08 +01:00
Andrei Cravtov 278bea11ad fmt 2026-05-12 11:40:07 +01:00
Andrei Cravtov dbd5ca00eb fmt 2026-05-11 20:04:58 +01:00
Andrei Cravtov 197cbf4f9d fmt 2026-05-11 19:44:38 +01:00
Andrei Cravtov d58ab113d1 fmt 2026-05-11 19:32:47 +01:00
Andrei Cravtov 9c7908f247 integrated it 2026-05-11 19:32:25 +01:00
Andrei Cravtov ad2c1e7c66 initial changes 2026-05-11 19:19:47 +01:00
Andrei Cravtov 87c72fc1fd Fixes issue #2068 (#2083)
## Motivation

To fix https://github.com/exo-explore/exo/issues/2068

## Changes

Adds queue shutdown logic & hard-timeouts for closing server.

## Why It Works

Prevents API from hanging more than 5 seconds.
2026-05-11 12:15:22 +00:00
Evan Quiney b76bc30107 bump rust versions (#2081) 2026-05-10 17:11:46 +00:00
08ffa5f637 Map GLM 4.7 stop tokens to GLM 4 IDs (#2061)
## Motivation

GLM 4.7 reuses the GLM 4 chat-template tokenizer, but the model card and
EOS-detection path didn't have an explicit mapping for it, so
OpenAI-compatible clients didn't see a clean stop and the runner emitted
follow-on role turns (e.g. \`<|user|>\` continuations after
\`<|assistant|>\`'s output).

## Changes

\`src/exo/worker/engines/mlx/utils_mlx.py\` — add the GLM 4 stop-token
IDs as the EOS set when the loaded model's tokenizer matches GLM 4 / 4.7
chat templates.

## Why It Works

The GLM 4 tokenizer's \`<|user|>\`, \`<|observation|>\`, and
\`<|endoftext|>\` IDs are stable across the GLM 4 / 4.7 line; treating
any of them as EOS lets the runner stop at the assistant turn boundary
the same way it stops at \`</s>\` for Llama-style models. No
prompt-template changes — only the stop set widens.

## Test Plan

### Automated Testing

New unit test
\`src/exo/worker/tests/unittests/test_mlx/test_eos_token_ids.py\`
covering: GLM 4 / 4.7 path returns the expected stop ID set; non-GLM
path returns the standard EOS only.

\`\`\`
src/exo/worker/tests/unittests/test_mlx/test_eos_token_ids.py ..
=== 2 passed in 0.01s ===
\`\`\`

\`uv run basedpyright\` and \`uv run ruff check\` both clean.

### Manual Testing

Hardware: 4-node Apple Silicon cluster, M5 Max master.

- Loaded \`mlx-community/GLM-4.7-Air-mlx-4bit\`, ran chat completion via
\`/v1/chat/completions\`. Before this fix the assistant turn ran on into
a synthetic \`<|user|>\` continuation; after the fix the response stops
cleanly at the assistant boundary.

---------

Co-authored-by: jw-wcv <101585096+jw-wcv@users.noreply.github.com>
Co-authored-by: Evan Quiney <evanev7@gmail.com>
2026-05-10 17:02:22 +00:00
Andrei Cravtov 45df74ba98 Andrei/mp capture stdio (#2056)
## Motivation

Process-isolated runner crashes and C-extension failures can write
directly to fd-level stdout/stderr, bypassing Python/loguru. We need to
capture that output per runner process without polluting the main
process or other workers, and without breaking operation when the parent
stdio is detached.

## Changes

- Added `AsyncProcess`, a spawn-only multiprocessing wrapper that
redirects child stdout/stderr to pipes and exposes them as in-memory
`Receiver[bytes]`s
- Replaced runner-supervisor's raw `multiprocessing.Process` usage with
`AsyncProcess`
- Added `--no-stdio`, redirecting stdin/stdout/stderr to `/dev/null`
after logging is configured
- Disabled verbose MLX
- Added tests covering stdio capture, child crashes, repeated bad
children, SIGTERM/SIGKILL shutdown escalation, stdio detachment, and
spawning captured children from a stdio-detached parent

## Why It Works

The parent can redirect its own stdio fds to `/dev/null`, while
`AsyncProcess` installs fresh pipe fds over fd 1 and 2 inside each
spawned child. That keeps stdio-detached parents quiet while preserving
per-runner stdout/stderr capture. Runner shutdown is still bounded:
SIGTERM grace first, then SIGKILL escalation if needed.

Next direction: the runner supervisor currently drains captured output
and logs it as stdout/debug and stderr/warning. This should be split
into more useful process-isolated error reporting instead of just log
forwarding (regex match on errors to obtain "reason" string, best
effort).

## Test Plan

### Manual Testing

Ran on 4 Mac Minis in a Thunderbolt 4 ring, can see that runner's
stdout/stderr contents are being captured.

### Automated Testing

- Added async-process tests for fd-level stdout/stderr capture, Python
traceback capture, bounded-buffer output, child `exit`/abort, parent
stdio preservation, fd leak checks, spawn-context mp channels, and
SIGTERM/SIGKILL shutdown behavior
- Added stdio-detach tests proving stdio detaches to `/dev/null`, a
stdio-detached parent can still spawn and capture a child, and the same
stdio-detached parent can spawn/capture multiple children sequentially
- Updated runner-supervisor tests for the new `AsyncProcess.exitcode`
path
2026-05-09 22:45:14 +01:00
Kerollos Magdy ce37bdceb6 fix: Create directory for PID file if it doesn't exist (#2075)
Ensure the directory for the PID file exists before creating it.

## Motivation

Fixes https://github.com/exo-explore/exo/issues/2074

## Changes

<!-- Describe what you changed in detail -->

## Why It Works

<!-- Explain why your approach solves the problem -->

## Test Plan

### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->

### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
2026-05-09 12:10:22 +00:00
Andrei Cravtov e5a1e5dadb Create PID file locking for EXO (#2072)
## Motivation

EXO should be PID file locked, to prevent duplicate processes from
clobbering the log, right now this isn't the case.

## Changes

I added a wrapper around a Rust PID file lock library, and used it to
implement PID locking for EXO, with the PID file being in exo cache
directory.

## Test Plan

### Manual Testing
Tested on e11, trying to spawn duplicate EXO processes prevented.
2026-05-08 18:50:18 +01:00
ciaranbor fa57131374 Integration tests infra (#1995)
## Motivation

No automated integration tests exist for exo. Manual testing against
real hardware clusters is slow and error-prone. We need a pytest
framework that deploys clusters via `eco`, runs inference scenarios, and
tears down cleanly.

## Changes

- **`tools/src/exo_tools/`** — New workspace member shared by bench,
eval, and tests:
- `client.py` — `ExoClient` HTTP client (extracted from
`bench/harness.py`)
- `harness.py` — instance lifecycle helpers (placement, wait-for-ready,
etc.)
- `cluster.py` — `EcoSession` for eco cluster lifecycle
(deploy/stop/start/release/logs/exec) with unique `USER=<prefix>-<uuid>`
per session and atexit/signal cleanup
- **`tests/integration/`** — 17 pytest tests across 5 files:
- `test_1node.py` — place, chat, multi-turn, delete, state/models
endpoints, cluster snapshot, download-from-scratch
- `test_2node.py` — parametrized tensor/jaccl + pipeline/ring inference
and multi-turn
- `test_4node.py` — parametrized 4-node pipeline/ring inference, cluster
state
- `test_resilience.py` — full disconnect/reconnect cycle (2-node →
disconnect → 1-node → reconnect → 2-node)
- `test_dashboard.py` — Playwright: dashboard loads, shows node info,
chat flow
- `helpers.py` — placement/inference helpers, re-exports from
`exo_tools`
- `conftest.py` — session-scoped cluster fixtures with constraint-based
eco reservations; `--hosts` override; `EXO_REF` env var for CI
deployments from a GitHub branch
- **`bench/`** — Updated imports from `exo_tools.client` /
`exo_tools.harness`
- **`pyproject.toml`** — Added `tools` workspace member, `playwright`
dev dep, `--ignore=tests/integration`

## Why It Works

Tests use `eco` for cluster lifecycle and `ExoClient` for API
interactions — same tools humans use. Session-scoped fixtures deploy
once per file. Unique eco users prevent test runs from interfering with
each other or manual usage.

## Test Plan

### Automated Testing

- `uv run pytest tests/integration/ -v -s` — full suite (~4-5 min, 17/17
passing)
- `uv run pytest tests/integration/ -v -s --hosts s4,s9,s10,s22` — pin
specific hosts
- `EXO_REF=main uv run pytest tests/integration/ -v` — deploy from a
GitHub branch (CI)
- `uv run pytest` — confirms integration tests are excluded from default
runs
2026-05-08 17:15:08 +01:00
Alex Cheema 414132ae9c Use time-weighted power sampling (#2038)
## Why

The power sampler currently averages sampled wattage values
arithmetically. That can be materially wrong when sample intervals are
uneven: a short high-power spike gets the same weight as a long steady
interval. Energy should be computed by integrating power over time, and
average power should be derived from energy / elapsed time.

## How

- Store each power sample with its relative timestamp.
- Anchor the first sample at `t=0` and take a final sample at `elapsed`
when producing results.
- Integrate per-node power using the trapezoidal rule.
- Sum node energy for total cluster energy, then derive total average
system power from total energy / elapsed.
- Add focused unit tests for uneven sample intervals and the
single-sample fallback.

## Tests

- `uv run pytest src/exo/utils/tests/test_power_sampler.py`
- `uv run basedpyright`
- `uv run ruff check src/exo/utils/power_sampler.py
src/exo/utils/tests/test_power_sampler.py`
- `nix fmt`
2026-05-07 10:42:14 +00:00
512 changed files with 7845 additions and 897 deletions

No files matched your search

View File
File renamed without changes.
+5
View File
@@ -1,3 +1,8 @@
# installs nix-direnv if doesn't exist (speeds up evaluation)
if ! has nix_direnv_version || ! nix_direnv_version 3.1.1; then
source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/3.1.1/direnvrc" "sha256-p+fzQdrms/hDa7g+soShAybJNo4bN4SIAeSfqNKgD5I="
fi
use flake
# creates .venv if doesn't exist and loads its environment
+2 -1
View File
@@ -18,7 +18,6 @@ digest.txt
app/EXO/build/
dist/
# rust
target/
**/*.rs.bk
@@ -40,3 +39,5 @@ bench/**/*.json
tmp/models
/build/exo
/.claude/skills
/.claude
/.codex
+3
View File
@@ -4,4 +4,7 @@
<option name="sdkName" value="Python 3.13 (exo)" />
</component>
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.13 (exo)" project-jdk-type="Python SDK" />
<component name="RuffConfiguration">
<option name="enabled" value="true" />
</component>
</project>
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loaded 100 of 512 files, more files were not shown because too many files have changed in this diff. Show more