How to run BERT model fine-tuned as Tensorflow checkpoint with FT backend?

[churinga]

Hi, we have a BERT model fine-tuned with Tensorflow, and saved as a Tensorflow checkpoint file. Is there any example or instructions on how to run this BERT model with the FT backend on the latest triton inference server?

Also, does the FT backend have all the optimizations described in https://developer.nvidia.com/blog/nlu-with-tensorrt-bert/ when used to power this BERT model? We followed the examples in https://github.com/NVIDIA/TensorRT/tree/release/5.1/demo/BERT/python on triton inference server 20.09, and were able to reproduce the optimized latency results, but if we upgrade to the newer triton server version with the FasterTransformer backend, should we expect similar (or even better) latency results?

Thanks!

[byshiue]

FT does not have converter to convert the TF checkpoint to triton. FT only provide the converter of HF bert now (at examples/pytorch/bert/utils/huggingface_bert_convert.py).

FT's BERT does not contains some optimization in TensorRT, like fusion of GEMM+GELU. So, there are small performance gap between FT and TRT on BERT model. For BERT inference, you can also ask how to run TRT BERT on triton directly, which should be the suggested solution on BERT model.

[churinga]

Thanks for the quick reply. Where should I ask about TRT BERT support for the latest version of TensorRT and triton inference server?

[byshiue]

You can try to ask in TensorRT repo or triton server repo.