error with T5 inference

[sivakura]

hello community,

trying T5 inference(https://developer.nvidia.com/blog/deploying-gpt-j-and-t5-with-fastertransformer-and-triton-inference-server/). getting below error on execution of this

import os
os.system("CUDA_VISIBLE_DEVICES=0 /opt/tritonserver/bin/tritonserver --log-warning false --model-repository=./triton-model-store/t5/ &")

I0517 04:22:59.609618 24895 libtorch.cc:1309] TRITONBACKEND_Initialize: pytorch
I0517 04:22:59.609747 24895 libtorch.cc:1319] Triton TRITONBACKEND API version: 1.8
I0517 04:22:59.609760 24895 libtorch.cc:1325] 'pytorch' TRITONBACKEND API version: 1.8
2023-05-17 04:22:59.844393: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-05-17 04:22:59.907030: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
I0517 04:22:59.907125 24895 tensorflow.cc:2176] TRITONBACKEND_Initialize: tensorflow
I0517 04:22:59.907157 24895 tensorflow.cc:2186] Triton TRITONBACKEND API version: 1.8
I0517 04:22:59.907170 24895 tensorflow.cc:2192] 'tensorflow' TRITONBACKEND API version: 1.8
I0517 04:22:59.907181 24895 tensorflow.cc:2216] backend configuration:
{}
I0517 04:22:59.909302 24895 onnxruntime.cc:2319] TRITONBACKEND_Initialize: onnxruntime
I0517 04:22:59.909336 24895 onnxruntime.cc:2329] Triton TRITONBACKEND API version: 1.8
I0517 04:22:59.909349 24895 onnxruntime.cc:2335] 'onnxruntime' TRITONBACKEND API version: 1.8
I0517 04:22:59.909358 24895 onnxruntime.cc:2365] backend configuration:
{}
I0517 04:22:59.935877 24895 openvino.cc:1207] TRITONBACKEND_Initialize: openvino
I0517 04:22:59.935910 24895 openvino.cc:1217] Triton TRITONBACKEND API version: 1.8
I0517 04:22:59.935923 24895 openvino.cc:1223] 'openvino' TRITONBACKEND API version: 1.8
I0517 04:23:00.184609 24895 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f275a000000' with size 268435456
I0517 04:23:00.185115 24895 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I0517 04:23:00.188362 24895 model_repository_manager.cc:997] loading: fastertransformer:1
I0517 04:23:00.411902 24895 libfastertransformer.cc:1226] TRITONBACKEND_Initialize: fastertransformer
I0517 04:23:00.411933 24895 libfastertransformer.cc:1236] Triton TRITONBACKEND API version: 1.8
I0517 04:23:00.411940 24895 libfastertransformer.cc:1242] 'fastertransformer' TRITONBACKEND API version: 1.8
I0517 04:23:00.411981 24895 libfastertransformer.cc:1274] TRITONBACKEND_ModelInitialize: fastertransformer (version 1)
I0517 04:23:00.415133 24895 libfastertransformer.cc:1320] TRITONBACKEND_ModelInstanceInitialize: fastertransformer_0 (device 0)
terminate called after throwing an instance of 'std::runtime_error'
what(): [FT][ERROR] shared_ft_model->getTensorParaSize() * shared_ft_model->getPipelineParaSize() == world_size Assertion fail: /workspace/build/fastertransformer_backend/src/libfastertransformer.cc:498

[01ed9ece5842:24895] *** Process received signal ***
[01ed9ece5842:24895] Signal: Aborted (6)
[01ed9ece5842:24895] Signal code: (-6)
[01ed9ece5842:24895] [ 0] /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f28b7a5d420]
[01ed9ece5842:24895] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f28b71e300b]
[01ed9ece5842:24895] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f28b71c2859]
[01ed9ece5842:24895] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e911)[0x7f28b759c911]
[01ed9ece5842:24895] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa38c)[0x7f28b75a838c]
[01ed9ece5842:24895] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3f7)[0x7f28b75a83f7]
[01ed9ece5842:24895] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa6a9)[0x7f28b75a86a9]
[01ed9ece5842:24895] [ 7] /opt/tritonserver/backends/fastertransformer/libtriton_fastertransformer.so(+0x2a660)[0x7f27e608b660]
[01ed9ece5842:24895] [ 8] /opt/tritonserver/backends/fastertransformer/libtriton_fastertransformer.so(+0x1e45f)[0x7f27e607f45f]
[01ed9ece5842:24895] [ 9] /opt/tritonserver/backends/fastertransformer/libtriton_fastertransformer.so(+0x1fa02)[0x7f27e6080a02]
[01ed9ece5842:24895] [10] /opt/tritonserver/backends/fastertransformer/libtriton_fastertransformer.so(TRITONBACKEND_ModelInstanceInitialize+0x38c)[0x7f27e60832fc]
[01ed9ece5842:24895] [11] /opt/tritonserver/bin/../lib/libtritonserver.so(+0x30fece)[0x7f28b7d8dece]
[01ed9ece5842:24895] [12] /opt/tritonserver/bin/../lib/libtritonserver.so(+0x311493)[0x7f28b7d8f493]
[01ed9ece5842:24895] [13] /opt/tritonserver/bin/../lib/libtritonserver.so(+0x309147)[0x7f28b7d87147]
[01ed9ece5842:24895] [14] /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18c37a)[0x7f28b7c0a37a]
[01ed9ece5842:24895] [15] /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19a351)[0x7f28b7c18351]
[01ed9ece5842:24895] [16] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4)[0x7f28b75d4de4]
[01ed9ece5842:24895] [17] /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f28b7a51609]
[01ed9ece5842:24895] [18] /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f28b72bf133]
[01ed9ece5842:24895] *** End of error message ***

Thanks in advance