Example of using Ragged Batching with FasterTransformer / TRT-LLM for zero-padding BERT inference ("continuous batching")

[vadimkantorov]

There is the ragged batching feature in Triton: https://github.com/triton-inference-server/server/blob/main/docs/user_guide/ragged_batching.md

And it also appears that FasterTransformer supports zero-padding mode for BERT inference: https://github.com/NVIDIA/FasterTransformer/blob/main/docs/bert_guide.md#model-architecture based on initial impl in https://github.com/bytedance/effective_transformer

This is also maybe supported by ORT (as "Effective Transformer"): https://github.com/microsoft/onnxruntime/blob/e7987a6b0ba429c0bec248c4a471e1782da4be6c/onnxruntime/python/tools/transformers/notebooks/PyTorch_Bert-Squad_OnnxRuntime_GPU.ipynb

Seems not supported by TRT: NVIDIA/TensorRT#4234

I propose to have a complete example in Triton repo of BERT inference using ragged batching and zero-padding mode:

• with FasterTransformer backend
• with modern TRT-LLM if it supports this Effective Transformer flavor of BERT
• with vllm if it supports it: https://www.anyscale.com/blog/continuous-batching-llm-inference