Support Running Both qwen2vl (VLLM) and showui2b on GPU Server via API in OOTB

[DongyoungKim2]

Description:

I am using the software ootb, which currently runs qwen2vl on an API and showui2b on the local machine's GPU. However, I would like to run both showui2b and qwen2vl on a GPU server for improved performance and scalability.

Current Setup:

• qwen2vl: Runs on the API and is powered by VLLM, which is OpenAI API compatible.
• showui2b: Currently runs on a local machine’s GPU.

Desired Setup:

• Both qwen2vl and showui2b should be run on the GPU server via API calls in ootb, instead of relying on the local machine's GPU for showui2b.

Challenges:

• qwen2vl runs on VLLM, which is OpenAI API compatible, and could potentially be added to OOTB for the desired setup.
• However, showui2b requires the local GPU for rendering, and there's no clear way to use a GPU server with OOTB to handle showui2b via API.

Proposed Solution:

1. Integrate a GPU server-based API for showui2b within the OOTB system.
2. Allow OOTB to call both qwen2vl (via VLLM API) and showui2b (via GPU server API) seamlessly.
3. Ensure that the server is capable of handling the GPU-intensive operations for both services.

Additional Context:

• Integration with VLLM for qwen2vl API calls has been confirmed to be possible.
• A similar API or service configuration for showui2b is needed to make this setup work.

[h-siyuan]

Thank you for your detailed feedback! We’ll definitely take it into consideration. This feature is already on our future update list, stay tuned:)