mirror/OpenLLM
1
0
Fork 0
mirror of https://github.com/bentoml/OpenLLM.git synced 2026-01-18 12:29:38 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,094 Commits 4 Branches 194 Tags
ea02aaaa230b9d4b770561d72ea1f3703eac8289
Commit Graph

140 Commits

Author SHA1 Message Date
xianxian.zhang
ea02aaaa23 fix: correct OPENLLM_DEV_BUILD check (#653) 2023-11-14 22:21:37 -05:00
Aaron Pham
6a6d689a77 feat: Yi models (#651) 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-14 21:55:24 -05:00
Aaron Pham
b4b70e2f20 fix(cli): update context name parsing correctly (#652) 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-14 21:53:56 -05:00
Aaron
9eddae83a6 infra: update cohere client 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-14 01:52:38 -05:00
Aaron Pham
31a799ff61 refactor: use DEBUG env-var instead of OPENLLMDEVDEBUG (#647) 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-14 01:39:58 -05:00
Aaron Pham
145aafd386 infra: prepare for release 0.4.6 [generated] [skip ci] 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-14 06:18:14 +00:00
Aaron Pham
00d6016bcb chore(openapi): unify inject param (#645) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-14 01:16:20 -05:00
Aaron Pham
b0ab8ccdf6 experimental: Cohere compatible endpoints. (#644) 
* feat: add generate endpoint

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update generation

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* fix(cohere): generate endpoints

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: --wip--

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* feat: update testing clients and chat implementation

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: disable schemas for easter eggs

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-14 01:07:43 -05:00
Aaron Pham
0bf6ec7537 fix(dependencies): lock build < 1 for now (#643) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-14 00:36:08 -05:00
Aaron Pham
b30a412398 fix(cli): set default dtype to auto infer (#642) 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 23:05:27 -05:00
Aaron Pham
99a5d26527 fix(service): to yield out correct JSON objects (#640) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 22:41:52 -05:00
Aaron Pham
2d428f12da fix(cpu): more verbose definition for dtype casting (#639) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 20:40:50 -05:00
Aaron Pham
b20c7d1c1d fix(generation): compatibility dtype with CPU (#638) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 20:32:07 -05:00
Zhao Shenyang
ae69524749 doc: update adding new model guide (#637) 
* update

* Update openllm-python/ADDING_NEW_MODEL.md

Co-authored-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
Signed-off-by: Zhao Shenyang <dev@zsy.im>

* Update openllm-python/ADDING_NEW_MODEL.md

Co-authored-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
Signed-off-by: Zhao Shenyang <dev@zsy.im>

* Update openllm-python/ADDING_NEW_MODEL.md

Co-authored-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
Signed-off-by: Zhao Shenyang <dev@zsy.im>

* move ADDING_NEW_MODEL.md to git root directory

---------

Signed-off-by: Zhao Shenyang <dev@zsy.im>
Co-authored-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 18:30:44 -05:00
Aaron Pham
a6387d1d15 chore: cleanup unused code path (#633) 
we now rely on tokenizer.chat_templates to format prompts correctly

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 17:23:07 -05:00
Aaron Pham
a0d74017cd infra: prepare for release 0.4.5 [generated] [skip ci] 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 18:49:31 +00:00
Aaron Pham
d358e68539 fix(torch_dtype): load eagerly (#631) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 13:48:04 -05:00
pre-commit-ci[bot]
52367d1e8b ci: pre-commit autoupdate [pre-commit.ci] (#629) 
* ci: pre-commit autoupdate [pre-commit.ci]

updates:
- [github.com/pre-commit/mirrors-prettier: v3.0.3 → v3.1.0](https://github.com/pre-commit/mirrors-prettier/compare/v3.0.3...v3.1.0)

* ci: auto fixes from pre-commit.ci

For more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-11-13 13:07:53 -05:00
Aaron Pham
852cd863a9 fix(cli): make sure to pass the dtype to subprocess service (#628) 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 05:32:17 -05:00
Aaron Pham
099c0dc31b feat(cli): --dtype arguments (#627) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-13 05:25:50 -05:00
Aaron Pham
22eaaf3ce1 feat(vllm): support passing specific dtype (#626) 
* feat(vllm): support passing specific dtype

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

* fix: correctly cached the item

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

* ci: auto fixes from pre-commit.ci

For more information, see https://pre-commit.ci

---------

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-11-13 05:08:33 -05:00
Aaron Pham
126e6c9d63 fix(ruff): correct consistency between isort and formatter (#624) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 21:12:50 -05:00
Aaron Pham
de04de7136 fix(sdk): make sure build to quiet out stdout (#622) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 18:59:48 -05:00
Aaron Pham
e667dac82f chore(cli): always show available models (#621) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 18:36:19 -05:00
Aaron Pham
c50a7db80d fix(cli): correct set working_dir (#620) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 18:34:11 -05:00
Aaron Pham
e0632a85ed refactor(cli): move out to its own packages (#619) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 18:25:44 -05:00
Aaron Pham
b1f3a7297b infra: prepare for release 0.4.4 [generated] [skip ci] 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 22:39:52 +00:00
Aaron Pham
c3416c0afd feat(llm): update warning envvar and add embedded mode (#618) 
* chore: unify warning envvar and update type inference

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore; update documentation about embedded

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 17:39:06 -05:00
Aaron Pham
7e1fb35a71 chore(llm): expose quantise and lazy load heavy imports (#617) 
* chore(llm): expose quantise and lazy load heavy imports

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: move transformers to TYPE_CHECKING block

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 14:55:37 -05:00
Aaron Pham
f3b16a4db0 infra: prepare for release 0.4.3 [generated] [skip ci] 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 07:39:56 +00:00
Aaron Pham
fad4186dbc feat(server): helpers endpoints for conversation format (#613) 
* feat: add support for helpers conversation conversion endpoint

also correct schema generation for openllm client

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update clients to reuse `openllm-core` logics

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: add changelog

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 01:02:27 -05:00
Aaron Pham
4d4df66188 infra: prepare for release 0.4.2 [generated] [skip ci] 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-12 03:38:52 +00:00
Aaron Pham
7438005c04 refactor(config): simplify configuration and update start CLI output (#611) 
* chore(config): simplify configuration and update start CLI output
handling

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: remove state and message sent after server lifecycle

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update color stream and refactor reusable logic

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update documentations and mypy

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-11 22:36:10 -05:00
Aaron Pham
c41828f68f feat(client): support authentication token and shim implementation (#605) 
* chore: synch generate_iterator to be the same as server

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* --wip--

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* wip

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* feat: cleanup shim implementation

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* ci: auto fixes from pre-commit.ci

For more information, see https://pre-commit.ci

* chore: fix pre-commit

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update changelog

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update check with tuple

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-11-10 17:44:31 -05:00
Aaron Pham
f89bec261c fix: correct importmodules locally (#601) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-10 03:32:12 -05:00
Aaron Pham
fa2038f4e2 fix: loading correct local models (#599) 
* fix(model): loading local correctly

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

* chore: update repr and correct bentomodel processor

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* ci: auto fixes from pre-commit.ci

For more information, see https://pre-commit.ci

* chore: cleanup transformers implementation

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* fix: ruff to ignore I001 on all stubs

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-11-10 02:36:12 -05:00
Aaron Pham
5e45245457 package: add openllm core dependencies to labels (#600) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-10 02:33:55 -05:00
Aaron Pham
665a41940e revert: configuration not to dump flatten (#597) 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-09 14:48:23 -05:00
Aaron Pham
d60f2fb909 infra: remove tsconfig (#595) 
* infra: remove tsconfig

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* ci: auto fixes from pre-commit.ci

For more information, see https://pre-commit.ci

* chore: filter only ec python and jsx

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update pnpm lock

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: run vendor

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: ignore blame

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: ignore on CI

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-11-09 13:06:31 -05:00
Aaron Pham
ac377fe490 infra: using ruff formatter (#594) 
Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-09 12:44:05 -05:00
Aaron Pham
b8a2e8cf91 refactor(cli): cleanup API (#592) 
* chore: remove unused imports

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* refactor(cli): update to only need model_id

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* feat: `openllm start model-id`

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: add changelog

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update changelog notice

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update correct config and running tools

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: update backward compat options and treat JSON outputs
corespondingly

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-09 11:40:17 -05:00
Aaron Pham
0d88370127 infra: prepare for release 0.4.1 [generated] [skip ci] 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-08 13:24:46 +00:00
Aaron Pham
e87830ef0a container: update tracing dependencies (#591) 
* chore: update build message

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: add tracing dependencies to container

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-08 08:08:40 -05:00
Aaron Pham
0ea025da5a fix(cli): append model-id instruction to build (#590) 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-08 07:44:36 -05:00
Aaron Pham
47107727b3 feat(vllm): squeezellm (#588) 
* feat(vllm): squeezellm

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

* fix: correct import_model with awq and gatekeep squeezellm for PyTorch

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-08 07:21:27 -05:00
Aaron Pham
ff8b6377c8 fix(awq): correct awq detection for support (#586) 
* fix(awq): correct detection for awq

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

* chore: update base docker to work

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

* chore: disable awq on pytorch for now

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

* ci: auto fixes from pre-commit.ci

For more information, see https://pre-commit.ci

---------

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-11-08 06:57:11 -05:00
Aaron Pham
387637405d fix(gptq): update config fields (#585) 
Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-08 05:20:30 -05:00
Aaron Pham
85a7243ac3 fix: device imports using strategies (#584) 
* fix: device imports using strategies

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

* chore: support trust_remote_code for vLLM runners

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
 2023-11-08 05:10:50 -05:00
Aaron Pham
ea42108e45 chore(service): cleanup API (#579) 
* chore(service): cleanup API

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* chore: running tools

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

* fix: tests import

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

---------

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-08 02:53:08 -05:00
Aaron Pham
7398ae0486 refactor(strategies): move logics into openllm-python (#578) 
fix(strategies): move to openllm

Strategies shouldn't be a part of openllm-core

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
 2023-11-08 02:23:08 -05:00