Lose /invocations path when using envrironment tarball from model-settings.json

[ylambrus]

Hi,

Is it expected that we lose the /invocations path ( mlflow backward compatible inference path) when using a tarball environment per model, eg when using model-settings.json ?

Note that when using global custom env tarball, it's working fine and invocations path is still present.

To reproduce :

Build a dummy iris model as follows ( or any model that can rely on mlflow ) :

import mlflow
import mlflow.sklearn
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
import pandas as pd
from mlflow.models.signature import infer_signature

data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2, random_state=42)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

input_example = pd.DataFrame(X_train[:1], columns=data.feature_names)
signature = infer_signature(X_train, model.predict(X_train))

with mlflow.start_run() as run:
    mlflow.sklearn.log_model(
        model, 
        "iris_model", 
        signature=signature, 
        input_example=input_example
    )
    model_uri = f"runs:/{run.info.run_id}/iris_model"

print(f"Model saved in MLflow at: {model_uri}")

Then build a conda tarball to match dependencies for this model :

conda env create -f conda.yaml
conda activate mlflow-env
conda install -c conda-forge conda-pack -y
conda pack -n mlflow-env -o environment.tar.gz

Then put your tarball inside your model artefact folder as usual and be sure to have your model-settings.json as follows :

{
    "name": "iris-model",
    "implementation": "mlserver_mlflow.MLflowRuntime",
    "parameters": {
        "uri": ".",
        "environment_tarball": "environment.tar.gz",
        "version": "v1"
    }
}

Then run mlserver ( here we tried with 1.5.0 and 1.6.1 as well):

mlserver start ./mlruns

2024-11-14 14:57:47,635 [mlserver] WARNING - Model name 'iris-model' is different than model's folder name 'iris_model'.
2024-11-14 14:57:47,662 [mlserver.parallel] DEBUG - Starting response processing loop...
2024-11-14 14:57:47,663 [mlserver.rest] INFO - HTTP server running on http://0.0.0.0:8080
INFO:     Started server process [34785]
INFO:     Waiting for application startup.
2024-11-14 14:57:47,676 [mlserver.metrics] INFO - Metrics server running on http://0.0.0.0:8082
2024-11-14 14:57:47,676 [mlserver.metrics] INFO - Prometheus scraping endpoint can be accessed on http://0.0.0.0:8082/metrics
INFO:     Started server process [34785]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
2024-11-14 14:57:47,678 [mlserver.grpc] INFO - gRPC server running on http://0.0.0.0:8081
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     Uvicorn running on http://0.0.0.0:8082 (Press CTRL+C to quit)
2024-11-14 14:57:47,777 [mlserver] INFO - Extracting environment tarball from /Users/yannick.lambruschi/repos/MLServer/mlruns/0/720aee73574a49418d4ae70011aee331/artifacts/iris_model/environment_iris.tar.gz...
2024-11-14 14:57:58,639 [mlserver.parallel][iris-model:v1] DEBUG - Starting response processing loop...

2024-11-14 14:58:22,056 [mlserver][iris-model:v1] INFO - Loaded model 'iris-model' successfully.
2024-11-14 14:58:22,057 [mlserver][iris-model:v1] INFO - Loaded model 'iris-model' successfully.

Now when querying /invocations you'll face a 404 :

curl -X POST "http://localhost:8080/invocations" -H "Content-Type: application/json" -d '{"inputs": [[5.1, 3.5, 1.4, 0.2]]}'

{"detail":"Not Found"}%

==> mlserver logs

INFO:     127.0.0.1:59245 - "POST /invocations HTTP/1.1" 404 Not Found

[sakoush]

Thanks @ylambrus for raising this issue. We will have to look more into why this happens when an environment tarball is passed. This issue exists when parallel workers are used though. Alternatively, the following should disable parallel workers and expose /invocations:

MLSERVER_PARALLEL_WORKERS=0 mlserver start ./mlruns

We also welcome contributions if you want to fix it in the case of parallel workers.

[ylambruschi]

Thx @sakoush for the infos. Though I need to point out that if you disable parallel works by doing :

export MLSERVER_PARALLEL_WORKERS=0

For some reasons it's actually cancelling the custom tarball extraction and indeed let the /invocations path work again
it's just to clarify, I was not sure if you were saying that it could solve part of the issue by setting workers to 0.

[sakoush]

@ylambruschi you are correct (sorry for the confusion). For a custom environment defined at the model level, mlserver spins off a new set of processes (based on the number of parallel workers) with this custom environment and therefore by definition it requires parallel workers to be enabled.

I suspect that in the case of a new environment that is different from the parent process the custom handlers (eg /invocations) are not registered properly, but we will have to look more into it to confirm exactly what's happening.

[ylambrus]

I think I understand the issue
It's because mlserver rather spawn an abstract MLModel instance (instead of concrete implementation as mlflow one) here :

MLServer/mlserver/parallel/registry.py

Line 198 in c5cc74a

# Otherwise, return a dummy model for now and wait for the load_model

For some reason, when instantiating the ParallelModel and trigger the custom handlers, it still keep the instance of this abstract model.

I'll try to do a PR in the coming weeks to contribute and try to help on this =)