mirror of
https://github.com/exo-explore/exo.git
synced 2026-09-23 03:15:16 -04:00
63b8e64715
## Motivation `mlx-community` has just published the new **Qwen3.6-35B-A3B** multimodal MoE family on HuggingFace. Without static model cards exo doesn't surface these models in the dashboard picker or match its placement / prefill logic, so users can't one-click launch them. This PR adds cards for the three quants whose safetensors indexes are already live on HF (4bit / 5bit / bf16). ## Changes Three new TOML files in `resources/inference_model_cards/`: - `mlx-community--Qwen3.6-35B-A3B-4bit.toml` (~19 GB) - `mlx-community--Qwen3.6-35B-A3B-5bit.toml` (~23 GB) - `mlx-community--Qwen3.6-35B-A3B-bf16.toml` (~65 GB) All three share the same architectural fields (`n_layers = 40`, `hidden_size = 2048`, `num_key_value_heads = 2`, `context_length = 262144`, capabilities `text, thinking, thinking_toggle, vision`, `base_model = "Qwen3.6 35B A3B"`) — only `model_id`, `quantization`, and `storage_size.in_bytes` differ between variants. ## Why It Works - Qwen3.6-35B-A3B reuses the `qwen3_5_moe` architecture (`Qwen3_5MoeForConditionalGeneration`) — the same one already wired into exo's MLX runner at `src/exo/worker/engines/mlx/auto_parallel.py:47` via `Qwen3_5MoeModel`. The architectural fields are taken verbatim from the HF `config.json.text_config` and match the existing `Qwen3.5-35B-A3B-*` cards. - Storage sizes are the exact `metadata.total_size` read from each variant's `model.safetensors.index.json` on HF, so download progress and cluster-memory-fit checks are accurate. - Vision support is flagged in `capabilities`; the `[vision]` block is auto-detected by `ModelCard._autodetect_vision` from the upstream `config.json`, so no hand-written vision config is required. - The card loader (`_refresh_card_cache` in `src/exo/shared/models/model_cards.py`) globs every `.toml` in `resources/inference_model_cards/` on startup, so nothing else needs to change — the `/models` endpoint and the dashboard picker pick them up automatically. The `mxfp4` / `mxfp8` / `nvfp4` variants are still uploading upstream (index JSONs currently 404) and can be added in a follow-up PR once HF completes. ## Test Plan ### Manual Testing Hardware: MacBook Pro M4 Max, 48 GB unified memory. - Built the dashboard, ran `uv run exo`, waited for the API to come up on `http://localhost:52415`. - `curl -s http://localhost:52415/models` returns the three new model ids (`mlx-community/Qwen3.6-35B-A3B-{4bit,5bit,bf16}`) alongside existing models. - Opened the dashboard, clicked SELECT MODEL, typed "Qwen3.6" into the search box. A single **"Qwen3.6 35B A3B"** group appears showing `3 variants (19GB-65GB)`. Expanding it lists the `4bit` / `5bit` / `bf16` quants with sizes `19GB` / `23GB` / `65GB`, exactly as expected:  - Programmatically loaded each TOML via `ModelCard.load_from_path(...)` and confirmed the parsed fields (layers / hidden / KV heads / context / quant / base_model / caps / bytes) match what's written in the files. ### Automated Testing No code paths were touched — these are pure TOML data files that plug into the existing model-card loader. The existing pytest suite covers TOML parsing and card serving; adding new TOMLs doesn't require new test scaffolding. `uv run ruff check` and `nix fmt` are clean. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Ryuichi Leo Takashige <rl.takashige@gmail.com>
