FEAT: support guided decoding for vllm async engine

[wxiwnd]

Support Guided Decoding for vllm async engine
waiting for vllm release, a version bump is needed.

#1562
vllm-project/vllm#8252

[qinxuye]

Which version is required?

[wxiwnd]

Which version is required?

latest version after 0.6.2, waiting for vllm to release new version

[qinxuye]

vllm has release v0.6.3, is this PR ready to work?

[wxiwnd]

I will do the test.

…

________________________________ 寄件者: Xuye Qin ***@***.***> 寄件日期: 星期四, 10月 17, 2024 5:38:13 下午 收件者: xorbitsai/inference ***@***.***> 副本: wxiwnd ***@***.***>; Author ***@***.***> 主旨: Re: [xorbitsai/inference] feat: support guided decoding for vllm async engine (PR #2391) vllm has release v0.6.3, is this PR ready to work? — Reply to this email directly, view it on GitHub<#2391 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJSDNXWKTHLIQE35VTQLT23Z36AQDAVCNFSM6AAAAABPJCVJQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJZGA2DOMZWGE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[wxiwnd]

vllm has release v0.6.3, is this PR ready to work?

Works on my machine now
Also This PR implemented response_format call, like:

curl --location --request POST 'http://ip:9997/v1/chat/completions' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Content-Type: application/json' \
--data '{
    "model": "qwen1.5-32b-chat-int4",
    "messages": [
        {
            "role": "user",
            "content": "“give me a recipe in json format"
        }
    ],
    "temperature": 0,
    "max_tokens": 1000,
    "stream": true,
    "response_format": {"type": "json_object"}
}'

[qinxuye]

vllm has release v0.6.3, is this PR ready to work?

Works on my machine now Also This PR implemented response_format call, like:

curl --location --request POST 'http://ip:9997/v1/chat/completions' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Content-Type: application/json' \
--data '{
    "model": "qwen1.5-32b-chat-int4",
    "messages": [
        {
            "role": "user",
            "content": "“give me a recipe in json format"
        }
    ],
    "temperature": 0,
    "max_tokens": 1000,
    "stream": true,
    "response_format": {"type": "json_object"}
}'

Can you confirm there is no exception if the vllm is an old version?

[wxiwnd]

vllm has release v0.6.3, is this PR ready to work?

Works on my machine now Also This PR implemented response_format call, like:

curl --location --request POST 'http://ip:9997/v1/chat/completions' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Content-Type: application/json' \
--data '{
    "model": "qwen1.5-32b-chat-int4",
    "messages": [
        {
            "role": "user",
            "content": "“give me a recipe in json format"
        }
    ],
    "temperature": 0,
    "max_tokens": 1000,
    "stream": true,
    "response_format": {"type": "json_object"}
}'

Can you confirm there is no exception if the vllm is an old version?

It now works properly even if vllm version < 0.6.3
All the guided encoding parameters will be ignored if vllm version is under 0.6.3

[wxiwnd]

This feature has been tested on my machine and appears to be functioning properly. @qinxuye

[qinxuye]

This feature has been tested on my machine and appears to be functioning properly. @qinxuye

Thanks, hope we can merge this before tomorrow before new version released.

[qinxuye]

LGTM

[qinxuye]

Thanks, I think we can port the ability of guide decoding for other engines later.

[wxiwnd]

Thanks, I think we can port the ability of guide decoding for other engines later.

That's OK, we can use outlines directly for other engines.