Check the model status before traffic gets redirected to the pods

[bdattoma]

/kind feature

Describe the solution you'd like
As of now, while the model is being loaded into memory, user queries may still reach a pod which may not be ready to answer (since the model is not loaded). The larger the model the larger the time to load it the larger the "downtime"

Anything else you would like to add:
This could be useful in different tasks like model/runtime canary rollout, replicas auto-scaling, etc

[danielezonca]

I think this ticket should be moved to upstream kserve repo ( https://github.com/kserve/kserve/issues ).
We can keep this as "clone" to simplify the tracking but the solution should be implemented upstream

[bdattoma]

upstream issue: kserve#3113

[bdattoma]

A such feature is already implemented upstream: https://kserve.github.io/website/0.10/modelserving/data_plane/v2_protocol/#healthreadinessliveness-probes