mirror/OpenLLM
1
0
Fork 0
mirror of https://github.com/bentoml/OpenLLM.git synced 2026-04-21 15:39:36 -04:00
Code Issues Packages Projects Releases Wiki Activity

chore(runner): yield the outputs directly (#573)

Browse Source 
update openai client examples to >1

Signed-off-by: Aaron Pham <29749331+aarnphm@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Aaron Pham
2023-11-07 22:34:11 -05:00
committed by GitHub
parent b3c4e204b2
commit cfd09bfc47
18 changed files with 311 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
openllm-python/src/openllm/entrypoints/openai.py
View File

@@ -128,7 +128,7 @@ async def create_chat_completions(req: Request, llm: openllm.LLM[M, T]) -> Respo
model_name, request_id = request.model, gen_random_uuid('chatcmpl')
created_time = int(time.monotonic())
prompt = llm.tokenizer.apply_chat_template(request.messages, tokenize=False)
prompt = llm.tokenizer.apply_chat_template(request.messages, tokenize=False, add_generation_prompt=llm.config['add_generation_prompt'])
logger.debug('Prompt: %r', prompt)
config = llm.config.with_openai_request(request)