Issue with Integration Test Using subprocess and bentoml.SyncHTTPClient

[takhyun12]

Hello! I'm using bentoml and greatly appreciate the excellent serving framework you've created.

I'm writing an integration test code following the official guide, but I've encountered a problem.

I want to start the bentoml server via subprocess and request images using bentoml.SyncHTTPClient.
However, I'm getting the following error:

FAILED tests/test_integration.py::test_inpaint_service_integration - bentoml.exceptions.BentoMLException: Failed to fetch schema from http://localhost:8100/schema.json

Based on the test results, it appears that the component loading and the TorchScript JIT compile warm-up process are functioning correctly:

----------------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------------
UserWarning: Specified provider 'OpenVINOExecutionProvider' is not in available provider names.Available providers: 'TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider'
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1719550013.872491   13835 task_runner.cc:85] GPU suport is not available: INTERNAL: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:77) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c
Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Loading pipeline components...:  14%|█▍        | 1/7 [00:00<00:02,  2.61it/s]W0000 00:00:1719550014.473112   14099 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
UserWarning: MessageFactory class is deprecated. Please use GetMessageClass() instead of MessageFactory.GetPrototype. MessageFactory class will be removed after 2024.
Loading pipeline components...: 100%|██████████| 7/7 [00:05<00:00,  1.23it/s]6,  2.63s/it]
[face_parser] Warming up with dummy images: 100%|██████████| 30/30 [00:17<00:00,  1.72it/s]
Warming up with asset images:   2%|▏         | 1/50 [00:03<02:44,  3.36s/it]UserWarning: operator() profile_node %385 : int[] = prim::profile_ivalue(%383)
 does not have profile information (Triggered internally at ../third_party/nvfuser/csrc/graph_fuser.cpp:104.)
Warming up with asset images: 100%|██████████| 50/50 [00:42<00:00,  1.17it/s]

It seems the /readyz endpoint is functioning without issues, but the error occurs at /schema.json. However, even when running the server normally using bentoml serve, http://localhost:8100/schema.json does not seem to exist.

My API uses Multipart(data=File()) from bentoml.io, and I couldn't find appropriate guides for this setup either.
Additionally, I was unable to find any relevant information about this schema issue online.

Could you please provide guidance on how to resolve this issue?

(...)
2024-06-28T04:51:43+0000 [INFO] [runner:skin_retoucher_runner:1] 127.0.0.1:55854 (scheme=http,method=GET,path=/readyz,type=,length=) (status=200,type=text/plain; charset=utf-8,length=1) 0.220ms (trace=524bd5092294a3c28a1b4de864e2d23a,span=6874a13e48158846,sampled=0,service.name=skin_retoucher_runner)
2024-06-28T04:51:43+0000 [INFO] [api_server:inpaint_service:3] 127.0.0.1:52784 (scheme=http,method=GET,path=/readyz,type=,length=) (status=200,type=text/plain; charset=utf-8,length=1) 248.525ms (trace=524bd5092294a3c28a1b4de864e2d23a,span=cc2fe7d0d36a8d1a,sampled=0,service.name=inpaint_service)
2024-06-28T04:51:43+0000 [INFO] [api_server:inpaint_service:4] 127.0.0.1:44950 (scheme=http,method=GET,path=/schema.json,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 3.587ms (trace=175a16c7ede10f45c72ee2698cac0978,span=d5ab2d88ebeee1d0,sampled=0,service.name=inpaint_service)

FAILED tests/test_integration.py::test_inpaint_service_integration - bentoml.exceptions.BentoMLException: Failed to fetch schema from http://localhost:8100/schema.json

test_integration.py:

def _load_image_as_bytes(image_path: str) -> (io.BytesIO, tuple):
    image = load_image(file_path=image_path)
    image = resize_image_multiples_of_eight(input_image=image, max_size=1024)
    image_io = io.BytesIO()
    image.save(image_io, format="PNG")
    image_io.seek(0)
    return image_io, image.size


def test_inpaint_service_integration():
    image_path = os.path.join("assets", "00_original.png")
    image_bytes, image_size = _load_image_as_bytes(image_path)

    URL: str = "http://localhost:8100"
    COMMAND: str = "BENTOML_CONFIG=services/inpaint/configuration_local.yml bentoml serve inpaint_service:latest --port 8100"
    with subprocess.Popen(COMMAND, shell=True) as server_proc:
        try:
            client = bentoml.SyncHTTPClient(URL, server_ready_timeout=120)
            if client.is_ready():
                response = client.api(data=image_bytes)
                assert response.status_code == 200, f"API request failed with status code: {response.status_code}"

                output_image: Image.Image = Image.open(io.BytesIO(response.content))
                assert output_image.mode == "RGBA", "The output image must be in RGBA mode."
                assert output_image.size == image_size, "The output image must have the same dimensions as the input image."
                assert output_image.getchannel("A") is not None, "The output image must have an alpha channel."

                rgb_image: Image.Image = output_image.convert("RGB")
                alpha_image: Image.Image = output_image.getchannel("A")
                assert rgb_image is not None, "The RGB channels must be present."
                assert alpha_image is not None, "The alpha channel must be present."

        finally:
            server_proc.terminate()

service.py:

runner_a = bentoml.Runner(RunnableA, name="sample")
(...)
runner_e = bentoml.Runner(RunnableE, name="sample")

service = bentoml.Service(
    name="inpaint_service", runners=[runner_a, (...), runner_e]
)

REQUEST_COUNT = Counter("request_count", "Total number of requests")
REQUEST_LATENCY = Histogram("request_latency_seconds", "Request latency in seconds")

@service.api(
    input=Multipart(data=File()),
    output=Image(mime_type="image/png"),
    route="/inpaint/v1/wrinkles",
)
async def api(data: BytesIO):
    REQUEST_COUNT.inc()
    with REQUEST_LATENCY.time():
        image = PILImage.open(BytesIO(data.read())).convert("RGB")
        image = exif_transpose(image=image)

        (...)

[aarnphm]

I would recommend controlling the lifecycle of the subprocess manually, instead of using a context manager, additionally after the server start, you can try to sleep for a certain amount of time to avoid race condition.

server = subprocess.Popen(COMMAND)

time.sleep(30)

Since the client does support as a context manager, you can do this

try:
  with bentoml.SyncHTTPClient(...) as client:
    ...
finally:
  server.terminate()

Hope this helps.