Support for BYTES / string tensor types

[njhill]

For BYTES I think the spec is currently ambiguous (to me at least). For example if strings are used then how should the encoding be determined. Also it's not clear how the byte lengths should be encoded when flattening. One option is to say that they must all be the same length and add a dimension to the shape for this. If their lengths can vary however then something extra is needed, like a corresponding integer tensor to hold them.

We can look at MLServer's BYTES string tests and base64 tests for reference.

[adriangonz]

Hey @njhill ,

This was quite confusing for me as well when doing it for MLServer. That was until I noticed that, in protobuf, the repeated bytes type (used for BYTES contents) actually is just an array where each element can be a variable-length bytes array (so each individual element of the array, is itself a variable-length bytes array). Therefore, it's alright to consider each bytes array as an individual element, so the following input is valid:

input = {
	"datatype": "BYTES",
	"data": [b"foo", b"asdasdasd"],
	"shape": [2]
}

Not sure if that may make it more confusing though - happy to clarify if you've got any questions!

[njhill]

Thanks @adriangonz! yes I'd also noticed that more recently. Along with content_type parameters this will be enough to implement.

I think the problem still applies to the raw_input_contents / raw_output_contents fields though - it doesn't seem that there's a way to use those currently with BYTES tensors. I guess that optimization is more valuable for primitive types anyhow.

[adriangonz]

Good point @njhill.

We've recently added support for those in MLServer. What we found is that Triton packs BYTES tensors into a single bytes array, where each string is preceded by its length. That way you can accommodate variable-length strings. For example, the BYTES tensor [b"foo", b"asdasd"] would get packed as 3foo6asdasd.

You can see this implemented in MLServer in the mlserver.raw module (which is pretty similar to Triton's own implementation - also linked there in a comment):

https://github.com/SeldonIO/MLServer/blob/f0748c7a14e4476f75b6154099247dc2c9b6d43b/mlserver/raw.py#L39-L55
https://github.com/SeldonIO/MLServer/blob/f0748c7a14e4476f75b6154099247dc2c9b6d43b/mlserver/raw.py#L58-L70

[njhill]

Thanks @adriangonz! I wasn't aware of that.

[njhill]

Now addressed via #17 with the following limitations:

• Does not obey request-level content_type request parameters (has to be set at tensor level)
• Always encodes response tensor bytes as base64

[lizzzcai]

Hi @njhill , this is something we are confusing when comparing with the REST API in MLServer and the v2 spec, thanks for the fix.

want to confirm that with the above fix, the input accepts raw string but the output response will still be base64 encoded?

for example:

curl -i -X POST -H "Content-Type: application/json" "http://localhost:8008/v2/models/${MODEL_NAME}/infer" -d '{"inputs": [{ "name": "text", "shape": [2], "datatype": "BYTES", "data": ["I loved this food, it was very good", "I don't loved this food, it was not good"] }] }'

{"model_name":"custom-predictor__ksp-89efd40320","model_version":"v0.1.0","outputs":[{"name":"predictions","datatype":"BYTES","shape":[2],"parameters":{"content_type":{"ParameterChoice":{"StringParam":"str"}}},"data":["Y29tcGxpbWVudA==","Y29tcGxhaW50"]}]}%  # rather than ["compliment","complaint"]

and when will a new version of rest-proxy be released? thanks.

[njhill]

@lizzzcai yes that's exactly right. Just a small thing in your example though - the value of the returned content_type parameter will be "base64" rather than "str" ... reflecting the fact that the returned strings are b64 encoded.

[lizzzcai]

@lizzzcai yes that's exactly right. Just a small thing in your example though - the value of the returned content_type parameter will be "base64" rather than "str" ... reflecting the fact that the returned strings are b64 encoded.

Thanks @njhill for your reply. We have tested the latest release and it works as expected. However, for the output response, will str be supported in the near future?

[njhill]

@lizzzcai we can open a new issue for it to replace this one :)

A complication is knowing when to encode as utf8 vs base64 because you can't encode arbitrary bytes as utf8. We could detect whether the bytes are valid utf8 but this would incur a performance overhead (espec. for non-text data cases).

and when will a new version of rest-proxy be released? thanks.

We are in the process of doing a 0.10.0 release of all the KServe components right now.

[njhill]

This is complete apart from the issues mentioned above and @lizzzcai has now opened #23 to track those.