Compare commits

...
Author SHA1 Message Date
Andrei Cravtov 7a8153ecf3 perf(dataplane): reduce TCP hot-path overhead 2026-05-27 12:43:30 +01:00
Andrei Cravtov 6464bed05b perf(dataplane): tune forced TCP batches
Default forced TCP to 65535-byte TUN MTU and use 256 KiB TCP read/write batches so max-size frames do not require multiple TCP reads.
2026-05-27 11:58:56 +01:00
Andrei Cravtov 34a879ff54 fix(dataplane): use one macOS TCP listener 2026-05-26 23:07:31 +01:00
Andrei Cravtov 80e5da948e fix(dataplane): harden forced TCP path
Reject inbound TCP streams unless they match a link-local Babel neighbour, avoid receive-side frame allocation, and split queued vs written TCP counters. Forced TCP defaults to a 9000-byte TUN MTU for Mac throughput sweeps.
2026-05-26 21:54:32 +01:00
Andrei Cravtov 76f69a80a4 Fix macOS TCP dataplane listeners 2026-05-26 20:16:22 +01:00
Andrei Cravtov bd3151a4ff Add forced TCP dataplane transport 2026-05-26 19:41:32 +01:00
Andrei Cravtov 0b7a3ad377 receiver side batching (different socket type) 2026-05-19 22:32:23 +01:00
Andrei Cravtov 3cbb575824 try to cut costs 2026-05-19 21:57:54 +01:00
Andrei Cravtov 2f03858832 en0 en1 infinite cost 2026-05-19 21:11:27 +01:00
Andrei Cravtov b02cf2cbb5 added support for naive scoring 2026-05-19 21:02:17 +01:00
Andrei Cravtov 5a158a51e4 added support for scoring 2026-05-19 20:47:06 +01:00
Andrei Cravtov 5bf8f62f9f added iperf3 2026-05-19 20:05:41 +01:00
Andrei Cravtov af0b6e17d1 remove first interface requirement 2026-05-19 19:54:08 +01:00
Andrei Cravtov 82adc5d9ba it builds 2026-05-19 19:09:24 +01:00
Andrei Cravtov 597475294f no longer need optional build flags (may need to update hash) 2026-05-15 20:42:51 +01:00
Andrei Cravtov 49845f287d Merge branch 'main' into babbler
# Conflicts:
#	Cargo.lock
#	flake.nix
#	python/parts.nix
2026-05-15 20:36:31 +01:00
Evan Quiney 90f24bef30 fix model cards not validating properly after #2071 (#2096) 2026-05-15 15:17:35 +00:00
Andrei Cravtov 5097b2665d Tweaked workspace settings (#2095)
workspace settings
2026-05-15 13:04:50 +00:00
rltakashigeandEvan bc6661e6aa Add node backends to model cards (#2071)
Co-authored-by: Evan <evanev7@gmail.com>
2026-05-15 12:52:12 +00:00
Andrei CravtovandEvan Quiney 14aab35688 Runner error handling (#2093)
# Runner error handling

## Motivation

Runner failures were mostly surfaced as plain shutdown messages, which
made root cause hard to spot from API errors or runner status.

This adds a MVP path for preserving runner crash context and attaching
known stderr diagnostics to failure reports.

## Changes

- Added `RunnerTerminationError` for Python exceptions raised inside
runner bootstrap
- Changed runner bootstrap to send `Event | RunnerTerminationError` over
the private runner channel
- Moved public `RunnerFailed` emission back into supervisor
- Added stderr-only `RunnerDiagnosticCollector`
- + Added known diagnostics for Metal GPU timeout, ring socket receive
errno, and ring transport abort
- Added diagnostics to `RunnerFailed` and `ErrorChunk`
- Tweaked async process termination to join briefly before
terminate/kill
- Updated tests/fixtures for new failure payload shape
- Added Ruff VS Code formatter settings

## Why It Works

Runner child now reports raw-ish failure context to supervisor instead
of publishing failed status directly.

Supervisor still owns process lifecycle, exit code/signal handling,
in-flight task error chunks, and final runner status. Stderr diagnostics
stay best effort and only known root-cause variants are surfaced.

## Test Plan

### Manual Testing

Hardware: remote runner logs from e16/e11/e4/e2

What you did:
- inspected live runner stderr logs
- used observed Metal GPU timeout and ring socket errors as initial
diagnostic targets

### Automated Testing

- `nix flake check`
- supervisor test covers error chunk + failed status emission
- plan lifecycle test updated for failed runner diagnostics
- type/lint checks cover new runner channel union

---------

Co-authored-by: Evan Quiney <evanev7@gmail.com>
2026-05-15 12:40:59 +00:00
Heidar 88d46d46fd fix: omit null delta fields in streaming chat completions (issue #2082) (#2092)
## Motivation

Streaming /v1/chat/completions responses emitted null for tool_calls,
function_call, name, and tool_call_id in every delta chunk. The OpenAI
streaming spec marks these fields as non-nullable — they must either
carry a
  real value or be absent entirely. Spec-correct clients doing
delta.get("tool_calls", []) receive None and crash with 'NoneType'
object is
  not iterable.

Root cause: the streaming serialisation path called model_dump_json()
without
exclude_none=True, while the request-parsing path already used it
correctly.
Three call sites in chat_completions.py and two in responses.py were
affected.

## Testing

Before — every delta carries explicit nulls:

  $ curl -sN -X POST http://localhost:52415/v1/chat/completions \
    -H 'Content-Type: application/json' \
-d
'{"model":"mlx-community/Qwen3.5-2B-MLX-8bit","messages":[{"role":"user","
  content":"hi"}],"max_tokens":3,"stream":true}' \
    | grep "^data: "
data:
{"id":"7c4dae10-...","choices":[{"index":0,"delta":{"role":"assistant","c

ontent":null,"reasoning_content":"Okay","name":null,"tool_calls":null,"tool_cal

l_id":null,"function_call":null},"logprobs":null,"finish_reason":null,"usage":n
  ull}],"usage":null,"service_tier":null}
data:
{"id":"7c4dae10-...","choices":[{"index":0,"delta":{"role":"assistant","c

ontent":null,"reasoning_content":",","name":null,"tool_calls":null,"tool_call_i

d":null,"function_call":null},"logprobs":null,"finish_reason":null,"usage":null
  }],"usage":null,"service_tier":null}
data:
{"id":"7c4dae10-...","choices":[{"index":0,"delta":{"role":"assistant","c
ontent":"
the","reasoning_content":null,"name":null,"tool_calls":null,"tool_cal

l_id":null,"function_call":null},"logprobs":null,"finish_reason":"length","usag
  e":{"prompt_tokens":11,...}}],"usage":null,"service_tier":null}
  data: [DONE]

  After — only populated fields are emitted:
data:
{"id":"demo","object":"chat.completion","created":...,"model":"mlx-commun

ity/Qwen3.5-2B-MLX-8bit","choices":[{"index":0,"delta":{"role":"assistant","rea
  soning_content":"Okay"}}]}
data:
{"id":"demo","object":"chat.completion","created":...,"model":"mlx-commun

ity/Qwen3.5-2B-MLX-8bit","choices":[{"index":0,"delta":{"role":"assistant","rea
  soning_content":","}}]}
data:
{"id":"demo","object":"chat.completion","created":...,"model":"mlx-commun

ity/Qwen3.5-2B-MLX-8bit","choices":[{"index":0,"delta":{"role":"assistant","con
tent":"
the"},"finish_reason":"length"}],"usage":{"prompt_tokens":11,"completio
  n_tokens":3,"total_tokens":14,...}}
  data: [DONE]
2026-05-14 16:32:54 +00:00
HeidarandClaude Opus 4.7 e8ec8d5010 fix ollama API compatibility for VS Code Copilot (#2091)
Ollama adapter fixes for VS Code Copilot (#2042):

  - /api/version: bare semver "1.0.0" - Copilot parseInts each segment.
- /api/show: populate model_info + capabilities - Copilot crashes on
null model_info and filters by `tools`.
- Add POST /ollama/v1/chat/completions - ollama serves the OpenAI-compat
route here, BYOK clients 405 without it.


Before:
<img width="1380" height="144" alt="image"
src="https://github.com/user-attachments/assets/99d5464f-187d-4432-9a31-8229c55aa209"
/>

After:
<img width="1362" height="181" alt="image"
src="https://github.com/user-attachments/assets/361dc006-d8df-435f-8d8b-4fa4f44a8c23"
/>
<img width="279" height="909" alt="image"
src="https://github.com/user-attachments/assets/4621aba7-bd57-4762-8568-34a3383a6025"
/>

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-14 16:12:58 +00:00
Heidar 1fd15d59fc create directory on startup (#2089)
## Motivation

<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->

When you first run `uv run exo` you get an error like :

`FileNotFoundError: [Errno 2] No such file or directory:
'/Users/heidar/.exo/models'`

Manually tested on Macbook Pro M1 32GB

Fixes issue - https://github.com/exo-explore/exo/issues/2090
2026-05-14 16:03:51 +00:00
Evan Quiney 4466cd5323 use custom mlx sources for linux (#2087)
switch to hosting mlx sources on github & cachix instead of using a
broken version of mlx. closes #2043.
2026-05-13 10:45:11 +01:00
Andrei Cravtov ed2d10bdc6 Redirect runner stdout/stderr to file logs (#2084)
## Motivation

We want to use log mining tools like
[Drain3](https://github.com/logpai/Drain3) to get standardized error
formats, but for that we should record runner stdout/stderr in a massive
append-only log to gather training data for such tools. Also useful for
future opt-in telemetry.

## Changes

The stdout/stderr from runner now splits into 3 tasks: 
1) raw write to dedicated runner logs 
2) sanitized line-by-line logging with log-guru 
3) stub for further error-processing (i.e. turning lines into errors)

### Manual Testing
Works on 4x mac mini clusted connected as TB4 ring.
2026-05-12 11:48:08 +01:00
Andrei Cravtov 87c72fc1fd Fixes issue #2068 (#2083)
## Motivation

To fix https://github.com/exo-explore/exo/issues/2068

## Changes

Adds queue shutdown logic & hard-timeouts for closing server.

## Why It Works

Prevents API from hanging more than 5 seconds.
2026-05-11 12:15:22 +00:00
Evan Quiney b76bc30107 bump rust versions (#2081) 2026-05-10 17:11:46 +00:00
08ffa5f637 Map GLM 4.7 stop tokens to GLM 4 IDs (#2061)
## Motivation

GLM 4.7 reuses the GLM 4 chat-template tokenizer, but the model card and
EOS-detection path didn't have an explicit mapping for it, so
OpenAI-compatible clients didn't see a clean stop and the runner emitted
follow-on role turns (e.g. \`<|user|>\` continuations after
\`<|assistant|>\`'s output).

## Changes

\`src/exo/worker/engines/mlx/utils_mlx.py\` — add the GLM 4 stop-token
IDs as the EOS set when the loaded model's tokenizer matches GLM 4 / 4.7
chat templates.

## Why It Works

The GLM 4 tokenizer's \`<|user|>\`, \`<|observation|>\`, and
\`<|endoftext|>\` IDs are stable across the GLM 4 / 4.7 line; treating
any of them as EOS lets the runner stop at the assistant turn boundary
the same way it stops at \`</s>\` for Llama-style models. No
prompt-template changes — only the stop set widens.

## Test Plan

### Automated Testing

New unit test
\`src/exo/worker/tests/unittests/test_mlx/test_eos_token_ids.py\`
covering: GLM 4 / 4.7 path returns the expected stop ID set; non-GLM
path returns the standard EOS only.

\`\`\`
src/exo/worker/tests/unittests/test_mlx/test_eos_token_ids.py ..
=== 2 passed in 0.01s ===
\`\`\`

\`uv run basedpyright\` and \`uv run ruff check\` both clean.

### Manual Testing

Hardware: 4-node Apple Silicon cluster, M5 Max master.

- Loaded \`mlx-community/GLM-4.7-Air-mlx-4bit\`, ran chat completion via
\`/v1/chat/completions\`. Before this fix the assistant turn ran on into
a synthetic \`<|user|>\` continuation; after the fix the response stops
cleanly at the assistant boundary.

---------

Co-authored-by: jw-wcv <101585096+jw-wcv@users.noreply.github.com>
Co-authored-by: Evan Quiney <evanev7@gmail.com>
2026-05-10 17:02:22 +00:00
Andrei Cravtov 58ed37e652 Enable nix uio for PBProbe timestamps 2026-04-24 21:11:18 +01:00
Andrei Cravtov 9173f22e0c Use kernel timestamps for Apple PBProbe receive 2026-04-24 21:10:01 +01:00
Andrei Cravtov d60ef66375 Fix Apple PBProbe recv build 2026-04-24 21:06:09 +01:00
Andrei Cravtov c62d723d4e Use minimal Apple recvmsg_x for PBProbe 2026-04-24 21:05:11 +01:00
Andrei Cravtov 5e3ab72a1c Use connected sendmsg_x for Apple PBProbe 2026-04-24 21:03:10 +01:00
Andrei Cravtov 5492943151 Avoid per-packet PBProbe timeout reset 2026-04-24 20:59:16 +01:00
Andrei Cravtov d62d96770a Batch PBProbe bulk receives 2026-04-24 20:55:45 +01:00
Andrei Cravtov adf490683a Use connected PBProbe sender on Apple 2026-04-24 20:49:52 +01:00
Andrei Cravtov ae1bdeee62 Batch PBProbe bulk sends 2026-04-24 20:41:24 +01:00
Andrei Cravtov a9fcdd0302 Reject incomplete PBProbe bulks 2026-04-24 20:31:25 +01:00
Andrei Cravtov 891d1e125e Simplify babel bootstrap receive 2026-04-24 20:25:06 +01:00
Andrei Cravtov 43e6b36802 Reuse standard timing and iteration helpers 2026-04-24 20:18:38 +01:00
Andrei Cravtov 337f8f83dd Use standard helpers in profiling tools 2026-04-24 20:04:22 +01:00
Andrei Cravtov 4806710001 Log PBProbe server issue rate 2026-04-24 19:52:15 +01:00
Andrei Cravtov a830095af2 Keep PBProbe sender timing nonzero 2026-04-24 19:40:29 +01:00
Andrei Cravtov 5a51779dc0 Inline PBProbe sender timing 2026-04-24 19:37:02 +01:00
Andrei Cravtov 6585ae3209 Report PBProbe sender issue summary 2026-04-24 19:28:49 +01:00
Andrei Cravtov 3205cb02c2 Add PBProbe sender diagnostics 2026-04-24 19:19:31 +01:00
Andrei Cravtov 70440748f3 profiling 2026-04-24 18:52:44 +01:00
Andrei Cravtov 1e01124f82 profiling 2026-04-24 18:36:18 +01:00
Andrei Cravtov da848d9c21 make babblerd tun io nonblocking 2026-04-24 16:39:12 +01:00
Andrei Cravtov 0cfc424f8a optimize babblerd dataplane hot path 2026-04-24 16:27:16 +01:00
Andrei Cravtov b0f508ac74 fix babbler ula prefix 2026-04-23 19:13:16 +01:00
Andrei Cravtov 2e17eb03f7 retry skipped dataplane sockets 2026-04-23 18:41:04 +01:00
Andrei Cravtov 4efc883fad dedup unchanged fib snapshots 2026-04-23 18:13:14 +01:00
Andrei Cravtov 79a5a84bb5 instrument dataplane backpressure 2026-04-23 18:08:06 +01:00
Andrei Cravtov f0140a816d trim dataplane logs for live testing 2026-04-23 16:06:54 +01:00
Andrei Cravtov df2760b6c0 admit dataplane interfaces from babel neighbours 2026-04-23 15:42:58 +01:00
Andrei Cravtov 4aac2c97f3 update babblerd notes after one-hop bringup 2026-04-23 15:19:26 +01:00
Andrei Cravtov e1e4643e37 use tun-rs packet io in dataplane 2026-04-23 15:15:45 +01:00
Andrei Cravtov a6760fa9c0 instrument dataplane hot path 2026-04-23 15:03:41 +01:00
Andrei Cravtov 2d86256699 share tun device with dataplane 2026-04-23 14:58:13 +01:00
Andrei Cravtov ef523966a7 pass first interface to babeld 2026-04-23 14:49:31 +01:00
Andrei Cravtov 0e14989408 add interface allowlist for lab bringup 2026-04-23 14:46:09 +01:00
Andrei Cravtov 9dc1cf78fc enable nix fs support 2026-04-23 14:41:28 +01:00
Andrei Cravtov 1b4d7e6c68 fix tun clone on macos 2026-04-23 14:38:48 +01:00
Andrei Cravtov 3c2e1ee3f9 harden dataplane error handling 2026-04-23 14:19:09 +01:00
Andrei Cravtov 5c3f1dc21d fix more bugs before live test 2026-04-23 14:10:14 +01:00
Andrei Cravtov 6f530e47a6 databplane and fib bugs 2026-04-23 13:42:45 +01:00
Andrei Cravtov ffc7d67816 fix some things 2026-04-22 21:14:34 +01:00
Andrei Cravtov 3bf44983b4 add dataplane 2026-04-22 20:26:50 +01:00
Andrei Cravtov 1c9f0f15ab remove stale code, document shortcuts 2026-04-22 19:48:25 +01:00
Andrei Cravtov 01c603eb91 ensure tunnel device is configured correctly 2026-04-22 18:30:57 +01:00
Andrei Cravtov af872f0693 fix architecture notes 2026-04-22 15:53:02 +01:00
Andrei Cravtov 9b55968141 added dummy client to keep connection alive 2026-04-22 15:21:21 +01:00
Andrei Cravtov 9d0b0aa51a fix bug 2026-04-22 14:07:04 +01:00
Andrei Cravtov 84e8ebc9b8 moved out logic from main 2026-04-22 12:48:03 +01:00
Andrei Cravtov d3558f7aa9 added identity file 2026-04-21 18:07:29 +01:00
Andrei Cravtov 89eca49a7c clean config 2026-04-21 17:28:27 +01:00
Andrei Cravtov 019093f00b abnormal termination elimiated 2026-04-21 14:46:58 +01:00
Andrei Cravtov cf6947f577 moved towards monitor 2026-04-21 14:45:32 +01:00
Andrei Cravtov 87d0e2f3e8 fixed some runtime bugs 2026-04-21 14:31:41 +01:00
Andrei Cravtov c18a95ca97 watch value 2026-04-20 19:14:28 +01:00
Andrei Cravtov a5d2e23d44 reorganized the code to collapse into single runtime.rs concept 2026-04-20 18:59:03 +01:00
Andrei Cravtov 1ea79b1a78 state folder 2026-04-20 18:53:04 +01:00
Andrei Cravtov 3c77d40538 Box<str> 2026-04-20 18:23:42 +01:00
Andrei Cravtov 0c70a725e9 split out into commands and session and process 2026-04-20 18:21:02 +01:00
Andrei Cravtov dda297c268 docs 2026-04-20 17:39:35 +01:00
Andrei Cravtov cb060dc532 docs 2026-04-20 17:15:19 +01:00
Andrei Cravtov daff8d2bb4 moved around some things 2026-04-20 17:12:16 +01:00
Andrei Cravtov 82fe6ef741 moved around some things 2026-04-20 17:03:56 +01:00
Andrei Cravtov 2c3ba858d3 revert back to dump-logic 2026-04-20 15:17:16 +01:00
Andrei Cravtov 731d4a4acb revert back to dump-logic 2026-04-20 15:12:27 +01:00
Andrei Cravtov a5c01b6681 revert back to dump-logic 2026-04-20 15:04:24 +01:00
Andrei Cravtov 9a30017969 revert back to dump-logic 2026-04-20 13:51:29 +01:00
Andrei Cravtov 1a73cfa457 parsing docs 2026-04-20 13:46:40 +01:00
Andrei Cravtov 0766265076 remove boilerplate 2026-04-20 13:30:45 +01:00
Andrei Cravtov 54fab3f9dd add dependencies to remove reinventing the wheels 2026-04-20 13:23:52 +01:00
Andrei Cravtov 099fcdd60a run format + added initial parser 2026-04-20 13:10:04 +01:00
Andrei Cravtov ca49944f80 build new babeld fork 2026-04-20 12:55:01 +01:00
Andrei Cravtov 9083258d2f ripped out the lo-based logic and instead use utun 2026-04-17 20:30:43 +01:00
Andrei Cravtov 1afe56504d test if applications themselves can run this 2026-04-17 17:48:23 +01:00
Andrei Cravtov 4d0d1987fa filter 2026-04-17 15:06:39 +01:00
Andrei Cravtov 271f6755fd loopback only 2026-04-17 12:57:49 +01:00
Andrei Cravtov 04614b734c hot fix 2026-04-16 17:47:56 +01:00
Andrei Cravtov 69b9d2e5d6 whoops 2026-04-15 18:12:33 +01:00
Andrei Cravtov 1f9098cf86 minimal fix 2026-04-15 18:03:16 +01:00
Evan 34a0e35d93 flush old addrs 2026-04-13 16:59:51 +01:00
Evan 98c44406cd just /64s i guess 2026-04-13 16:59:51 +01:00
Evan 03873a916c remove iface flushing 2026-04-13 16:59:51 +01:00
Evan 9b78aac613 correctness 2026-04-13 16:59:51 +01:00
Evan 35afc3c160 add leo's bandwidth test 2026-04-13 16:59:51 +01:00
Evan 80bf99b8ca NO BAD lo 2026-04-13 16:59:51 +01:00
Evan f4a311a9ad trying lo again 2026-04-13 16:59:50 +01:00
Evan 94aebaf246 ok didnt work lets try /112 2026-04-13 16:59:50 +01:00
Evan 732d318d7f lets try just one thing per thing 2026-04-13 16:59:50 +01:00
Evan 342bd995ad cleanup 2026-04-13 16:59:50 +01:00
Evan f8c180c14c move private directories 2026-04-13 16:59:50 +01:00
Evan dd209fc733 error handling 2026-04-13 16:59:50 +01:00
Evan b424e4a9a1 uniform node ids 2026-04-13 16:59:50 +01:00
Evan e1e4b8ffb9 oops 2026-04-13 16:59:50 +01:00
Evan 8d369f6381 handle receiver lagged better 2026-04-13 16:59:50 +01:00
Evan 027b0d5153 more logs 2026-04-13 16:59:50 +01:00
Evan 6e9653a294 clippy 2026-04-13 16:59:50 +01:00
Evan 5c40d47be1 some safety 2026-04-13 16:59:50 +01:00
Evan 73a50e4d44 babble babble 2026-04-13 16:59:50 +01:00
516 changed files with 20229 additions and 2168 deletions

No files matched your search

+1 -1
View File
@@ -18,7 +18,6 @@ digest.txt
app/EXO/build/
dist/
# rust
target/
**/*.rs.bk
@@ -41,3 +40,4 @@ tmp/models
/build/exo
/.claude/skills
/.claude
/.codex
+3
View File
@@ -4,4 +4,7 @@
<option name="sdkName" value="Python 3.13 (exo)" />
</component>
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.13 (exo)" project-jdk-type="Python SDK" />
<component name="RuffConfiguration">
<option name="enabled" value="true" />
</component>
</project>
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loaded 100 of 516 files, more files were not shown because too many files have changed in this diff. Show more