mirror/OpenLLM
1
0
Fork 0
mirror of https://github.com/bentoml/OpenLLM.git synced 2026-04-18 22:18:34 -04:00
Code Issues Packages Projects Releases Wiki Activity

chore(deps): bump vllm to 0.2.7 (#837)

Browse Source 
* chore(deps): bump vllm to 0.2.7

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update changelog

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
This commit is contained in:
Aaron Pham
2024-01-08 14:41:58 -05:00
committed by GitHub
parent 07d15ed217
commit 79da419d87
5 changed files with 5 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
openllm-python/README.md generated
View File

@@ -1445,7 +1445,7 @@ openllm start squeeze-ai-lab/sq-llama-2-7b-w4-s0 --quantize squeezellm --seriali
```
> [!IMPORTANT]
> Since both `squeezellm` and `awq` are weight-aware quantization methods, meaning the quantization is done during training, all pre-trained weights needs to get quantized before inference time. Make sure to fine compatible weights on HuggingFace Hub for your model of choice.
> Since both `squeezellm` and `awq` are weight-aware quantization methods, meaning the quantization is done during training, all pre-trained weights needs to get quantized before inference time. Make sure to find compatible weights on HuggingFace Hub for your model of choice.
## 🛠️ Serving fine-tuning layers

2
openllm-python/pyproject.toml
View File

@@ -119,7 +119,7 @@ openai = ["openai[datalib]>=1", "tiktoken"]
playground = ["jupyter", "notebook", "ipython", "jupytext", "nbformat"]
qwen = ["cpm-kernels", "tiktoken"]
starcoder = ["bitsandbytes"]
vllm = ["vllm==0.2.6", "ray==2.6.0"]
vllm = ["vllm==0.2.7", "ray==2.6.0"]
[tool.hatch.version]
fallback-version = "0.0.0"