diff --git a/.github/assets/main-banner.png b/.github/assets/main-banner.png deleted file mode 100644 index 2f823849..00000000 Binary files a/.github/assets/main-banner.png and /dev/null differ diff --git a/.github/assets/output.gif b/.github/assets/output.gif deleted file mode 100644 index 0e3b390a..00000000 Binary files a/.github/assets/output.gif and /dev/null differ diff --git a/README.md b/README.md index 1e5c2798..0ba1cfd5 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,3 @@ -![Banner for OpenLLM](/.github/assets/main-banner.png) - - -

🦾 OpenLLM: Self-Hosting LLMs Made Easy

@@ -15,859 +11,200 @@
-## 📖 Introduction +## Introduction -OpenLLM helps developers **run any open-source LLMs**, such as Llama 2 and Mistral, as **OpenAI-compatible API endpoints**, locally and in the cloud, optimized for serving throughput and production deployment. +OpenLLM lets developers run any **open-source LLMs** as **OpenAI-compatible API** endpoints with **a single command**. -- 🚂 Support a wide range of open-source LLMs including LLMs fine-tuned with your own data -- ⛓️ OpenAI compatible API endpoints for seamless transition from your LLM app to open-source LLMs -- 🔥 State-of-the-art serving and inference performance -- 🎯 Simplified cloud deployment via [BentoML](https://www.bentoml.com) +- 🔬 Build for fast and production usages +- 🚂 Support a wide range of open-source LLMs including llama3, qwen2, gemma, etc and fine-tuned or quantized versions +- ⛓️ OpenAI-compatible API, easy to integrate with any openai dependent Apps +- 💬 Built-in ChatGPT like UI to interact with the LLM directly +- 🔥 Accelerated LLM decoding with state-of-the-art inference backends +- 🌥️ Optionally deploy to K8S or BentoCloud for an enterprise-grade LLM API service - +## Get started -![Gif showing OpenLLM Intro](/.github/assets/output.gif) - -
- -## 💾 TL/DR - -For starter, we provide two ways to quickly try out OpenLLM: - -### Jupyter Notebooks - -Try this [OpenLLM tutorial in Google Colab: Serving Phi 3 with OpenLLM](https://colab.research.google.com/github/bentoml/OpenLLM/blob/main/examples/llama2.ipynb). - -## 🏃 Get started - -The following provides instructions for how to get started with OpenLLM locally. - -### Prerequisites - -You have installed Python 3.9 (or later) and `pip`. We highly recommend using a [Virtual Environment](https://docs.python.org/3/library/venv.html) to prevent package conflicts. - -### Install OpenLLM - -Install OpenLLM by using `pip` as follows: +Install OpenLLM with pip. OpenLLM currently supports Python 3.9 and above: ```bash pip install openllm ``` -To verify the installation, run: +One can then run: ```bash -$ openllm -h +openllm hello ``` -### Start a LLM server +to explore models interactively. It will guide you to run LLMs either locally or in the cloud. -OpenLLM allows you to quickly spin up an LLM server using `openllm start`. For example, to start a [Phi-3](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) server, run the following: +https://github.com/bentoml/OpenLLM/assets/5886138/2aa5f9e4-859c-4be8-91b3-91d8a810ef73 + +## Supported models + +OpenLLM supports a variety of state-of-the-art LLMs. Here are some of the models supported by OpenLLM, each listed with a commonly used model size. + +| Model | Parameters | Quantinize | Required VRAM | Start a Server | +| ------- | ---------- | ---------- | ------------- | ------------------------------- | +| Llama 3 | 8B | - | 24G | `openllm serve llama3:8b` | +| Llama 3 | 8B | AWQ 4bit | 12G | `openllm serve llama3:8b-4bit` | +| Llama 3 | 70B | AWQ 4bit | 80G | `openllm serve llama3:70b-4bit` | +| Llama 2 | 7B | - | 16G | `openllm serve llama2:7b` | +| Llama 2 | 7B | AWQ 4bit | 12G | `openllm serve llama2:7b-4bit` | +| Mistral | 7B | - | 24G | `openllm serve mistral:7b` | +| Qwen2 | 1.5B | - | 12G | `openllm serve qwen2:1.5b` | +| Gemma | 7B | - | 24G | `openllm serve gemma:7b` | +| Phi3 | 3.8B | - | 12G | `openllm serve phi3:3.8b` | + +... + +For the full model list, see the [OpenLLM models repository](https://github.com/bentoml/openllm-models). + +## Start an LLM server + +To start an LLM server locally, use the `openllm serve` command and specify the model version. ```bash -openllm start microsoft/Phi-3-mini-4k-instruct --trust-remote-code +openllm serve llama3:8b ``` -To interact with the server, you can visit the web UI at [http://0.0.0.0:3000/](http://0.0.0.0:3000/) or send a request using `curl`. You can also use OpenLLM’s built-in Python client to interact with the server: +The server will be accessible at [http://localhost:3000](http://localhost:3000/), providing OpenAI-compatible APIs for interaction. You can call the endpoints with different frameworks and tools that support OpenAI-compatible APIs. Typically, you may need to specify the following: -```python -import openllm +- **The API host address**: By default, the LLM is hosted at [http://localhost:3000](http://localhost:3000/). +- **The model name:** The name can be different depending on the tool you use. +- **The API key**: The API key used for client authentication. This is optional. -client = openllm.HTTPClient('http://localhost:3000') -client.generate('Explain to me the difference between "further" and "farther"') -``` - -OpenLLM seamlessly supports many models and their variants. You can specify different variants of the model to be served. For example: - -```bash -openllm start -- -``` - -## 🧩 Supported models - -OpenLLM currently supports the following models. By default, OpenLLM doesn't include dependencies to run all models. The extra model-specific dependencies can be installed with the instructions below. - - -
- -Baichuan - -### Quickstart - -Run the following command to quickly spin up a Baichuan server: - -```bash -openllm start baichuan-inc/baichuan-7b --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Baichuan variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=baichuan) to see more Baichuan-compatible models. - -### Supported models - -You can specify any of the following Baichuan models via `openllm start`: - -- [baichuan-inc/baichuan2-7b-base](https://huggingface.co/baichuan-inc/baichuan2-7b-base) -- [baichuan-inc/baichuan2-7b-chat](https://huggingface.co/baichuan-inc/baichuan2-7b-chat) -- [baichuan-inc/baichuan2-13b-base](https://huggingface.co/baichuan-inc/baichuan2-13b-base) -- [baichuan-inc/baichuan2-13b-chat](https://huggingface.co/baichuan-inc/baichuan2-13b-chat) - -
+Here are some examples:
-ChatGLM - -### Quickstart - -Run the following command to quickly spin up a ChatGLM server: - -```bash -openllm start thudm/chatglm-6b --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: +OpenAI Python client ```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any ChatGLM variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=chatglm) to see more ChatGLM-compatible models. - -### Supported models - -You can specify any of the following ChatGLM models via `openllm start`: - -- [thudm/chatglm-6b](https://huggingface.co/thudm/chatglm-6b) -- [thudm/chatglm-6b-int8](https://huggingface.co/thudm/chatglm-6b-int8) -- [thudm/chatglm-6b-int4](https://huggingface.co/thudm/chatglm-6b-int4) -- [thudm/chatglm2-6b](https://huggingface.co/thudm/chatglm2-6b) -- [thudm/chatglm2-6b-int4](https://huggingface.co/thudm/chatglm2-6b-int4) -- [thudm/chatglm3-6b](https://huggingface.co/thudm/chatglm3-6b) - -
- -
- -Cohere - -### Quickstart - -Run the following command to quickly spin up a Cohere server: - -```bash -openllm start CohereForAI/c4ai-command-r-plus --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Cohere variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=commandr) to see more Cohere-compatible models. - -### Supported models - -You can specify any of the following Cohere models via `openllm start`: - -- [CohereForAI/c4ai-command-r-plus](https://huggingface.co/CohereForAI/c4ai-command-r-plus) -- [CohereForAI/c4ai-command-r-v01](https://huggingface.co/CohereForAI/c4ai-command-r-v01) - -
- -
- -Dbrx - -### Quickstart - -Run the following command to quickly spin up a Dbrx server: - -```bash -openllm start databricks/dbrx-instruct --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Dbrx variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=dbrx) to see more Dbrx-compatible models. - -### Supported models - -You can specify any of the following Dbrx models via `openllm start`: - -- [databricks/dbrx-instruct](https://huggingface.co/databricks/dbrx-instruct) -- [databricks/dbrx-base](https://huggingface.co/databricks/dbrx-base) - -
- -
- -DollyV2 - -### Quickstart - -Run the following command to quickly spin up a DollyV2 server: - -```bash -openllm start databricks/dolly-v2-3b --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any DollyV2 variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=dolly_v2) to see more DollyV2-compatible models. - -### Supported models - -You can specify any of the following DollyV2 models via `openllm start`: - -- [databricks/dolly-v2-3b](https://huggingface.co/databricks/dolly-v2-3b) -- [databricks/dolly-v2-7b](https://huggingface.co/databricks/dolly-v2-7b) -- [databricks/dolly-v2-12b](https://huggingface.co/databricks/dolly-v2-12b) - -
- -
- -Falcon - -### Quickstart - -Run the following command to quickly spin up a Falcon server: - -```bash -openllm start tiiuae/falcon-7b --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Falcon variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=falcon) to see more Falcon-compatible models. - -### Supported models - -You can specify any of the following Falcon models via `openllm start`: - -- [tiiuae/falcon-7b](https://huggingface.co/tiiuae/falcon-7b) -- [tiiuae/falcon-40b](https://huggingface.co/tiiuae/falcon-40b) -- [tiiuae/falcon-7b-instruct](https://huggingface.co/tiiuae/falcon-7b-instruct) -- [tiiuae/falcon-40b-instruct](https://huggingface.co/tiiuae/falcon-40b-instruct) - -
- -
- -Gemma - -### Quickstart - -Run the following command to quickly spin up a Gemma server: - -```bash -openllm start google/gemma-7b --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Gemma variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=gemma) to see more Gemma-compatible models. - -### Supported models - -You can specify any of the following Gemma models via `openllm start`: - -- [google/gemma-7b](https://huggingface.co/google/gemma-7b) -- [google/gemma-7b-it](https://huggingface.co/google/gemma-7b-it) -- [google/gemma-2b](https://huggingface.co/google/gemma-2b) -- [google/gemma-2b-it](https://huggingface.co/google/gemma-2b-it) - -
- -
- -GPTNeoX - -### Quickstart - -Run the following command to quickly spin up a GPTNeoX server: - -```bash -openllm start eleutherai/gpt-neox-20b --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any GPTNeoX variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=gpt_neox) to see more GPTNeoX-compatible models. - -### Supported models - -You can specify any of the following GPTNeoX models via `openllm start`: - -- [eleutherai/gpt-neox-20b](https://huggingface.co/eleutherai/gpt-neox-20b) - -
- -
- -Llama - -### Quickstart - -Run the following command to quickly spin up a Llama server: - -```bash -openllm start NousResearch/llama-2-7b-hf --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Llama variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=llama) to see more Llama-compatible models. - -### Supported models - -You can specify any of the following Llama models via `openllm start`: - -- [meta-llama/Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) -- [meta-llama/Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) -- [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) -- [meta-llama/Llama-2-70b-hf](https://huggingface.co/meta-llama/Llama-2-70b-hf) -- [meta-llama/Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf) -- [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) -- [NousResearch/llama-2-70b-chat-hf](https://huggingface.co/NousResearch/llama-2-70b-chat-hf) -- [NousResearch/llama-2-13b-chat-hf](https://huggingface.co/NousResearch/llama-2-13b-chat-hf) -- [NousResearch/llama-2-7b-chat-hf](https://huggingface.co/NousResearch/llama-2-7b-chat-hf) -- [NousResearch/llama-2-70b-hf](https://huggingface.co/NousResearch/llama-2-70b-hf) -- [NousResearch/llama-2-13b-hf](https://huggingface.co/NousResearch/llama-2-13b-hf) -- [NousResearch/llama-2-7b-hf](https://huggingface.co/NousResearch/llama-2-7b-hf) - -
- -
- -Mistral - -### Quickstart - -Run the following command to quickly spin up a Mistral server: - -```bash -openllm start mistralai/Mistral-7B-Instruct-v0.1 --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Mistral variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=mistral) to see more Mistral-compatible models. - -### Supported models - -You can specify any of the following Mistral models via `openllm start`: - -- [HuggingFaceH4/zephyr-7b-alpha](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha) -- [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) -- [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) -- [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) -- [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) - -
- -
- -Mixtral - -### Quickstart - -Run the following command to quickly spin up a Mixtral server: - -```bash -openllm start mistralai/Mixtral-8x7B-Instruct-v0.1 --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Mixtral variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=mixtral) to see more Mixtral-compatible models. - -### Supported models - -You can specify any of the following Mixtral models via `openllm start`: - -- [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) -- [mistralai/Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) - -
- -
- -MPT - -### Quickstart - -Run the following command to quickly spin up a MPT server: - -```bash -openllm start mosaicml/mpt-7b-instruct --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any MPT variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=mpt) to see more MPT-compatible models. - -### Supported models - -You can specify any of the following MPT models via `openllm start`: - -- [mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b) -- [mosaicml/mpt-7b-instruct](https://huggingface.co/mosaicml/mpt-7b-instruct) -- [mosaicml/mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat) -- [mosaicml/mpt-7b-storywriter](https://huggingface.co/mosaicml/mpt-7b-storywriter) -- [mosaicml/mpt-30b](https://huggingface.co/mosaicml/mpt-30b) -- [mosaicml/mpt-30b-instruct](https://huggingface.co/mosaicml/mpt-30b-instruct) -- [mosaicml/mpt-30b-chat](https://huggingface.co/mosaicml/mpt-30b-chat) - -
- -
- -OPT - -### Quickstart - -Run the following command to quickly spin up a OPT server: - -```bash -openllm start facebook/opt-1.3b -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any OPT variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=opt) to see more OPT-compatible models. - -### Supported models - -You can specify any of the following OPT models via `openllm start`: - -- [facebook/opt-125m](https://huggingface.co/facebook/opt-125m) -- [facebook/opt-350m](https://huggingface.co/facebook/opt-350m) -- [facebook/opt-1.3b](https://huggingface.co/facebook/opt-1.3b) -- [facebook/opt-2.7b](https://huggingface.co/facebook/opt-2.7b) -- [facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b) -- [facebook/opt-66b](https://huggingface.co/facebook/opt-66b) - -
- -
- -Phi - -### Quickstart - -Run the following command to quickly spin up a Phi server: - -```bash -openllm start microsoft/Phi-3-mini-4k-instruct --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Phi variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=phi) to see more Phi-compatible models. - -### Supported models - -You can specify any of the following Phi models via `openllm start`: - -- [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) -- [microsoft/Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) -- [microsoft/Phi-3-small-8k-instruct](https://huggingface.co/microsoft/Phi-3-small-8k-instruct) -- [microsoft/Phi-3-small-128k-instruct](https://huggingface.co/microsoft/Phi-3-small-128k-instruct) -- [microsoft/Phi-3-medium-4k-instruct](https://huggingface.co/microsoft/Phi-3-medium-4k-instruct) -- [microsoft/Phi-3-medium-128k-instruct](https://huggingface.co/microsoft/Phi-3-medium-128k-instruct) - -
- -
- -Qwen - -### Quickstart - -Run the following command to quickly spin up a Qwen server: - -```bash -openllm start qwen/Qwen-7B-Chat --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Qwen variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=qwen) to see more Qwen-compatible models. - -### Supported models - -You can specify any of the following Qwen models via `openllm start`: - -- [qwen/Qwen-7B-Chat](https://huggingface.co/qwen/Qwen-7B-Chat) -- [qwen/Qwen-7B-Chat-Int8](https://huggingface.co/qwen/Qwen-7B-Chat-Int8) -- [qwen/Qwen-7B-Chat-Int4](https://huggingface.co/qwen/Qwen-7B-Chat-Int4) -- [qwen/Qwen-14B-Chat](https://huggingface.co/qwen/Qwen-14B-Chat) -- [qwen/Qwen-14B-Chat-Int8](https://huggingface.co/qwen/Qwen-14B-Chat-Int8) -- [qwen/Qwen-14B-Chat-Int4](https://huggingface.co/qwen/Qwen-14B-Chat-Int4) - -
- -
- -StableLM - -### Quickstart - -Run the following command to quickly spin up a StableLM server: - -```bash -openllm start stabilityai/stablelm-tuned-alpha-3b --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any StableLM variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=stablelm) to see more StableLM-compatible models. - -### Supported models - -You can specify any of the following StableLM models via `openllm start`: - -- [stabilityai/stablelm-tuned-alpha-3b](https://huggingface.co/stabilityai/stablelm-tuned-alpha-3b) -- [stabilityai/stablelm-tuned-alpha-7b](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b) -- [stabilityai/stablelm-base-alpha-3b](https://huggingface.co/stabilityai/stablelm-base-alpha-3b) -- [stabilityai/stablelm-base-alpha-7b](https://huggingface.co/stabilityai/stablelm-base-alpha-7b) - -
- -
- -StarCoder - -### Quickstart - -Run the following command to quickly spin up a StarCoder server: - -```bash -openllm start bigcode/starcoder --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any StarCoder variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=starcoder) to see more StarCoder-compatible models. - -### Supported models - -You can specify any of the following StarCoder models via `openllm start`: - -- [bigcode/starcoder](https://huggingface.co/bigcode/starcoder) -- [bigcode/starcoderbase](https://huggingface.co/bigcode/starcoderbase) - -
- -
- -Yi - -### Quickstart - -Run the following command to quickly spin up a Yi server: - -```bash -openllm start 01-ai/Yi-6B --trust-remote-code -``` - -You can run the following code in a different terminal to interact with the server: - -```python -import openllm_client -client = openllm_client.HTTPClient('http://localhost:3000') -client.generate('What are large language models?') -``` - -> **Note:** Any Yi variants can be deployed with OpenLLM. Visit the [HuggingFace Model Hub](https://huggingface.co/models?sort=trending&search=yi) to see more Yi-compatible models. - -### Supported models - -You can specify any of the following Yi models via `openllm start`: - -- [01-ai/Yi-6B](https://huggingface.co/01-ai/Yi-6B) -- [01-ai/Yi-34B](https://huggingface.co/01-ai/Yi-34B) -- [01-ai/Yi-6B-200K](https://huggingface.co/01-ai/Yi-6B-200K) -- [01-ai/Yi-34B-200K](https://huggingface.co/01-ai/Yi-34B-200K) - -
- - - -More models will be integrated with OpenLLM and we welcome your contributions if you want to incorporate your custom LLMs into the ecosystem. Check out [Adding a New Model Guide](https://github.com/bentoml/OpenLLM/blob/main/ADDING_NEW_MODEL.md) to learn more. - -## 📐 Quantization - -Quantization is a technique to reduce the storage and computation requirements for machine learning models, particularly during inference. By approximating floating-point numbers as integers (quantized values), quantization allows for faster computations, reduced memory footprint, and can make it feasible to deploy large models on resource-constrained devices. - -OpenLLM supports the following quantization techniques - -- [AWQ: Activation-aware Weight Quantization](https://arxiv.org/abs/2306.00978). -- [GPTQ: Accurate Post-Training Quantization](https://arxiv.org/abs/2210.17323). -- [SqueezeLLM: Dense-and-Sparse Quantization](https://arxiv.org/abs/2306.07629). - -> [!NOTE] -> Make sure to use pre-quantized models weights when using with `openllm start`. - -## ⚙️ Integrations - -OpenLLM is not just a standalone product; it's a building block designed to -integrate with other powerful tools easily. We currently offer integration with -[OpenAI's Compatible Endpoints](https://platform.openai.com/docs/api-reference/completions/object), -[LlamaIndex](https://www.llamaindex.ai/), -[LangChain](https://github.com/hwchase17/langchain). - -### OpenAI Compatible Endpoints - -OpenLLM Server can be used as a drop-in replacement for OpenAI's API. Simply -specify the base_url to `llm-endpoint/v1` and you are good to go: - -```python -import openai - -client = openai.OpenAI(base_url='http://localhost:3000/v1', api_key='na') # Here the server is running on 0.0.0.0:3000 - -completions = client.chat.completions.create( - prompt='Write me a tag line for an ice cream shop.', model=model, max_tokens=64, stream=stream +from openai import OpenAI + +client = OpenAI(base_url='http://localhost:3000/v1', api_key='na') + +# Use the following func to get the available models +# model_list = client.models.list() +# print(model_list) + +chat_completion = client.chat.completions.create( + model="meta-llama/Meta-Llama-3-8B-Instruct", + messages=[ + { + "role": "user", + "content": "Explain superconductors like I'm five years old" + } + ], + stream=True, ) +for chunk in chat_completion: + print(chunk.choices[0].delta.content or "", end="") ``` -The compatible endpoints supports `/chat/completions`, and `/models` + -> [!NOTE] -> You can find out OpenAI example clients under the -> [examples](https://github.com/bentoml/OpenLLM/tree/main/examples) folder. +
-### [LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/llm/openllm/) +LangChain -You can use `llama_index.llms.openllm.OpenLLMAPI` to interact with a LLM running server: +
-```python -from llama_index.llms.openllm import OpenLLMAPI +
+ +LlamaIndex + +
+ +## Chat UI + +OpenLLM provides a chat user interface (UI) at the `/chat` endpoint for an LLM server. You can visit the chat UI at http://localhost:3000/chat and start different conversations with the model. + +openllm_ui + +## Chat with a model in the CLI + +To start a chat conversation in the CLI, use the `openllm run` command and specify the model version. + +```bash +openllm run llama3:8b ``` -> [!NOTE] -> All synchronous and asynchronous API from `llama_index.llms.OpenLLMAPI` are supported. -> Make sure to install `llama-index-integrations-llm-openllm` to use the supported class. +## Model repository -### [LangChain](https://python.langchain.com/docs/integrations/llms/openllm/) +A model repository in OpenLLM represents a catalog of available LLMs that you can run. OpenLLM provides a default model repository that includes the latest open-source LLMs like Llama 3, Mistral, and Qwen2, hosted at [this GitHub repository](https://github.com/bentoml/openllm-models). To see all available models from the default and any added repository, use: -Spin up an OpenLLM server, and connect to it by specifying its URL: - -```python -from langchain.llms import OpenLLMAPI - -llm = OpenLLMAPI(server_url='http://44.23.123.1:3000') -llm.invoke('What is the difference between a duck and a goose? And why there are so many Goose in Canada?') - -# streaming -for it in llm.stream('What is the difference between a duck and a goose? And why there are so many Goose in Canada?'): - print(it, flush=True, end='') - -# async context -await llm.ainvoke('What is the difference between a duck and a goose? And why there are so many Goose in Canada?') - -# async streaming -async for it in llm.astream('What is the difference between a duck and a goose? And why there are so many Goose in Canada?'): - print(it, flush=True, end='') +```bash +openllm model list ``` - +To ensure your local list of models is synchronized with the latest updates from all connected repositories, run: -## 🚀 Deploying models to production - -There are several ways to deploy your LLMs: - -### 🐳 Docker container - -1. **Building a Bento**: With OpenLLM, you can easily build a Bento for a - specific model, like `mistralai/Mistral-7B-Instruct-v0.1`, using the `build` command.: - - ```bash - openllm build mistralai/Mistral-7B-Instruct-v0.1 - ``` - - A - [Bento](https://docs.bentoml.com/en/latest/concepts/bento.html#what-is-a-bento), - in BentoML, is the unit of distribution. It packages your program's source - code, models, files, artefacts, and dependencies. - -2. **Containerize your Bento** - - ```bash - bentoml containerize - ``` - - This generates a OCI-compatible docker image that can be deployed anywhere - docker runs. For best scalability and reliability of your LLM service in - production, we recommend deploy with BentoCloud。 - -### ☁️ BentoCloud - -Deploy OpenLLM with [BentoCloud](https://www.bentoml.com/), the inference platform -for fast moving AI teams. - -1. **Create a BentoCloud account:** [sign up here](https://bentoml.com/) - -2. **Log into your BentoCloud account:** - - ```bash - bentoml cloud login --api-token --endpoint - ``` - -> [!NOTE] -> Replace `` and `` with your -> specific API token and the BentoCloud endpoint respectively. - -3. **Bulding a Bento**: With OpenLLM, you can easily build a Bento for a - specific model, such as `mistralai/Mistral-7B-Instruct-v0.1`: - - ```bash - openllm build mistralai/Mistral-7B-Instruct-v0.1 - ``` - -4. **Pushing a Bento**: Push your freshly-built Bento service to BentoCloud via - the `push` command: - - ```bash - bentoml push - ``` - -5. **Deploying a Bento**: Deploy your LLMs to BentoCloud with a single - `bentoml deployment create` command following the - [deployment instructions](https://docs.bentoml.com/en/latest/reference/cli.html#bentoml-deployment-create). - -## 👥 Community - -Engage with like-minded individuals passionate about LLMs, AI, and more on our -[Discord](https://l.bentoml.com/join-openllm-discord)! - -OpenLLM is actively maintained by the BentoML team. Feel free to reach out and -join us in our pursuit to make LLMs more accessible and easy to use 👉 -[Join our Slack community!](https://l.bentoml.com/join-slack) - -## 🎁 Contributing - -We welcome contributions! If you're interested in enhancing OpenLLM's -capabilities or have any questions, don't hesitate to reach out in our -[discord channel](https://l.bentoml.com/join-openllm-discord). - -Checkout our -[Developer Guide](https://github.com/bentoml/OpenLLM/blob/main/DEVELOPMENT.md) -if you wish to contribute to OpenLLM's codebase. - -## 📔 Citation - -If you use OpenLLM in your research, we provide a [citation](./CITATION.cff) to -use: - -```bibtex -@software{Pham_OpenLLM_Operating_LLMs_2023, -author = {Pham, Aaron and Yang, Chaoyu and Sheng, Sean and Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost}, -license = {Apache-2.0}, -month = jun, -title = {{OpenLLM: Operating LLMs in production}}, -url = {https://github.com/bentoml/OpenLLM}, -year = {2023} -} +```bash +openllm repo update ``` - +To review a model’s information, run: + +```bash +openllm model get llama3:8b +``` + +### Add a model to the default model repository + +You can contribute to the default model repository by adding new models that others can use. This involves creating and submitting a Bento of the LLM. For more information, check out this [example pull request](https://github.com/bentoml/openllm-models/pull/1). + +### Set up a custom repository + +You can add your own repository to OpenLLM with custom models. To do so, follow the format in the default OpenLLM model repository with a `bentos` directory to store custom LLMs. You need to [build your Bentos with BentoML](https://docs.bentoml.com/en/latest/guides/build-options.html) and submit them to your model repository. + +First, prepare your custom models in a `bentos` directory following the guidelines provided by [BentoML to build Bentos](https://docs.bentoml.com/en/latest/guides/build-options.html). Check out the [default model repository](https://github.com/bentoml/openllm-repo) for an example and read the [Developer Guide](https://github.com/bentoml/OpenLLM/blob/main/DEVELOPMENT.md) for details. + +Then, register your custom model repository with OpenLLM: + +```bash +openllm repo add +``` + +**Note**: Currently, OpenLLM only supports adding public repositories. + +## Deploy to BentoCloud + +OpenLLM supports LLM cloud deployment via BentoML, the unified model serving framework, and BentoCloud, an AI inference platform for enterprise AI teams. BentoCloud provides fully-managed infrastructure optimized for LLM inference with autoscaling, model orchestration, observability, and many more, allowing you to run any AI model in the cloud. + +[Sign up for BentoCloud](https://www.bentoml.com/) for free and [log in](https://docs.bentoml.com/en/latest/bentocloud/how-tos/manage-access-token.html). Then, run `openllm deploy` to deploy a model to BentoCloud: + +```bash +openllm deploy llama3:8b +``` + +Once the deployment is complete, you can run model inference on the BentoCloud console: + +bentocloud_ui + +## Community + +OpenLLM is actively maintained by the BentoML team. Feel free to reach out and join us in our pursuit to make LLMs more accessible and easy to use 👉 [Join our Slack community!](https://l.bentoml.com/join-slack) + +Also, engage with like-minded individuals passionate about LLMs, AI, and more on the [OpenLLM Discord](https://l.bentoml.com/join-openllm-discord)! + +## Contributing + +As an open-source project, we welcome contributions of all kinds, such as new features, bug fixes, and documentation. Here are some of the ways to contribute: + +- Repost a bug by [creating a GitHub issue](https://github.com/bentoml/OpenLLM/issues/new/choose). +- [Submit a pull request](https://github.com/bentoml/OpenLLM/compare) or help review other developers’ [pull requests](https://github.com/bentoml/OpenLLM/pulls). +- Add an LLM to the OpenLLM default model repository so that other users can run your model. See the [pull request template](https://github.com/bentoml/openllm-models/pull/1). +- Check out the [Developer Guide](https://github.com/bentoml/OpenLLM/blob/main/DEVELOPMENT.md) to learn more. + +## Acknowledgements + +This project uses the following open-source projects: + +- [bentoml/bentoml](https://github.com/bentoml/bentoml) for production level model serving +- [blrchen/chatgpt-lite](https://github.com/blrchen/chatgpt-lite) for a fancy Web Chat UI +- [chujiezheng/chat_templates](https://github.com/chujiezheng/chat_templates) +- [astral-sh/uv](https://github.com/astral-sh/uv) for blazing fast model requirements installing + +We are grateful to the developers and contributors of these projects for their hard work and dedication. diff --git a/pyproject.toml b/pyproject.toml index b1238a99..1f5c7ca9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -82,11 +82,7 @@ Twitter = "https://twitter.com/bentomlai" src-dir = "src/openllm" [build-system] -requires = [ - "hatchling==1.25.0", - "hatch-vcs==0.4.0", - "hatch-fancy-pypi-readme==24.1.0", -] +requires = ["hatchling==1.25.0", "hatch-vcs==0.4.0"] build-backend = 'hatchling.build' [tool.hatch.version] @@ -112,25 +108,3 @@ only-include = ["src/openllm"] sources = ["src"] [tool.hatch.build.targets.sdist] exclude = ["/.git_archival.txt", "/.python-version-default"] - -[tool.hatch.metadata.hooks.fancy-pypi-readme] -content-type = "text/markdown" -# PyPI doesn't support the tag. -[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]] -text = """ -

- - Banner for OpenLLM - -

-""" -[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]] -end-before = "\n" -path = "README.md" -start-after = "\n" -[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]] -text = """ -

- Gif showing OpenLLM Intro -

-"""