exo/TODO.md

1. Currently EXO just doesn't start cleanly a lot of the time. I see two kinds of issues: b. EXO starts but then after creating an instance that instance never loads (either gets stuck in Loading of Inactive).
2. Currently a lot of requests from the API are timing out, but we still process those requests internally. If an API request times out, we should cancel all corresponding tasks to that API request (why process a request with nobody listening).
3. I'd like to see profiled network latency / bandwidth.
4. I'd like to see how much bandwidth each link is using.
5. We should handle the case where one machine doesn't have the model downloaded and then other machines are waiting on it. In this case we get loads of timeout errors because the others are waiting for the one that needs to download the model.
6. Solve the problem of in continuous batching when a new prompt comes in, it will block decode of the current batch until the prefill is complete.
7. We want people to be able to copy models over to a new device without ever connecting EXO to the internet. Right now EXO require internet connection once to cache some files to check if a download is complete. Instead, we should simply check if there is a non-empty model folder locally with no .partial files. This indicates it's a fully downloaded model that can be loaded.
8. More granular control over how to deploy instances.
9. Nix is great but installing it is a pain and we have ended up in a lot of cases having PATH issues or installation issues. For example, after rebooting mike it seemed to no longer have a nix installation and needed reinstalling. It has a bunch of broken symlinks left over from nix that caused ssh to fail, making it even harder to debug. We need consistent environments (perhaps MDM) so we can guarantee nix is installed properly on each machine.
10. Memory pressure instead of memory used.
11. Show the type of each connection (TB5, Ethernet, etc.) in the UI. Refer to old exo: 56f783b38d/exo/helpers.py (L251)
12. Prioritise certain connection types (or by latency). TB5 > Ethernet > WiFi. Refer to old exo: 56f783b38d/exo/helpers.py (L251)
13. Dynamically switch to higher priority connection when it becomes available. Probably bring back InstanceReplacedAtomically.
14. Faster model loads by streaming model from other devices in cluster.
15. Add support for specifying the type of network connection to use in a test. Depends on 15/16.
16. Fix mx.distributed.Group typing.
17. Add chat completion cancellations (e.g OpenWebUI has something for cancelling an ongoing request).
18. Make two separate things: tensor or pipeline, and ring or ibv.

Potential refactors:

1. Make ForwarderEvent typed
2. Topology can be simplified
3. Get rid of InstanceReplacedAtomically