TensorRT deployment Support, Pre-Processing alignment and more

[zhiqwang]

This release provides TensorRT deployment support both for Python and C++ interface and supports pre-processing alignment with the official yolov5.

Highlights

Use yolort to deploy YOLOv5 on TensorRT

Unlike other pipelines that deal with YOLOv5 on TensorRT, we embed the whole post-processing into the serialized engine with TensorRT's EfficientNMS_TRT plugin. This pipeline could simplify deployment and gain in speed due to TensorRT's efficient implementation of this plugin. Check out this tutorial detailing yolort's model conversion and use cases. Here is a Python pipeline to deploy YOLOv5 on TensorRT.

import torch
from yolort.runtime import PredictorTRT

# Load the serialized TensorRT engine
engine_path = "yolov5n6.engine"
device = torch.device("cuda")
y_runtime = PredictorTRT(engine_path, device=device)

# Perform inference on an image file
predictions = y_runtime.predict("bus.jpg")

• Support exporting yolort to TensorRT serialized engine (Add YOLOInference for downstream inference frameworks #238, Remove low scoring filter from LogitsDecoder #242, Add tensorrt export tools #243, Fix register_nms in YOLOGraphSurgeon #252, Refactor module structure for exporting TensorRT #254, Remove expired parameters in detect.py #263, Add unittests for YOLOGraphSurgeon #275, Use EfficientNMS_TRT plugin when exporting TensorRT #288, Add TensorRT installation instructions to tutorial #311, Refactor TensorRT engine export #312, Support exporting TensorRT serialized engine in CLI tools #326)
• Add TensorRT Python inference pipeline (Add TensorRT Python interface and CLI #251, Add TensorRT export and inference notebooks #253, Move LogitsDecoder into trt_helper.py and fix docstrings #256, Remove unused parameters in PredictorTRT #259, Fix TensorRT Python examples #283, Add ablation experiment results with EfficientNMS_TRT #290, Refactoring TensorRT Python interface #307, Add pre-processing in PredictorTRT pipeline #309)
• Add TensorRT C++ inference example ([WIP] Add TensorRT C++ interface for yolort #245, Complete TRT example! #257, Add Shiquan as co-authors for great contributions on TRT #258, Support yolov5x and yolov5x6 models #261, Fix tensorrt async bug #262, Add letterbox for TRT inference #305, Fix pre-processing in TensorRT C++ example #321, Support loading with TensorRT SerializedModel #323, Fix initializer error in TensorRT example #324)

Alignment with the official yolov5 Pre-Processing

Pre-processing misalignment has been an obstacle to getting yolort to land, and we've fixed that in this release.

• Fix the consistency of pre-processing with yolov5 (Fix the consistency of pre-processing with yolov5 #293)
• Support fixed shape inference in pre-processing (Support fixed shape inference in pre-processing #301)
• Fix pre-processing for TVM VM inference (Fix pre-processing for TVM VM inference #302)

Backwards Incompatible Changes

• Rename relaying to relay (Rename relaying to relay #313)
• Move Lightning wrapper into trainer (Move Lightning wrapper into trainer #314)
• Deprecate the methods run_on_image function in PredictorORT (Enhance PredictorORT to adapt to more scenarios #327)

Improvement

• Support loading the latest version of yolov5 (Fix inconsistency with the latest version of yolov5 #235)
• Formal documentation support (Fix building the docs #224, Add gh-pages deploy bot #225, Deploy docs with Material Sphinx Theme #292)
• Cleanup AnchorGenerator and PostProcess (Cleanup AnchorGenerator and PostProcess #203)
• Revert SPPF to SPP for downstream compatible (Revert SPPF to SPP #240)
• Support custom image size when exporting ONNX models (Set argument image_size configurable in export_model.py #250, Fix list expanding in export_model.py #264, Fix arguments in export_model.py #269)
• Upgrade with latest PyTorch version in GH Actions (Upgrade PyTorch to 1.10.1 in GH Actions #255)
• Update PyTorch to 1.10 in TVM relay VM tutorial (Update PyTorch to 1.10 in TVM Relay VM notebook #287)
• Enhance PredictorORT to adapt to more scenarios (Refactor ONNX export and tutorial #319, Support multiple batches for exporting ONNX with pre-processing #320, Enhance PredictorORT to adapt to more scenarios #327)
• Support publishing 🐍 to PyPI.org 📦 with GH Actions (Add PyPI publish GH Action #229, Fix package version in setuptools #241, Only publish distribution to Test PyPI #306, Fix version definition in setuptools #328, Specify version when publishing to PyPI #329)

Bugfixes

• Small fixes to CONTRIBUTING guide (Fix git branch in CONTRIBUTING.md #218)
• Fix type casting in ONNX Runtime C++ example (Fix the type casting in labels #221)
• Fix TorchScript exporting for custom checkpoints (Fix TorchScript exporting for custom checkpoints #267)
• Fix non-agnostic parameter in tutorial (Fix alignment notebook with yolov5  #284)
• Fix a bug and document RandomZoomOut (Fix a bug and document RandomZoomOut #289)
• Compatible with Windows for LibTorch deployment (Compatible with Windows for LibTorch deployment #303)

Documentation improvements

• Add comparison docs for yolov5 & yolort (Add comparison docs for yolov5 & yolort #300)
• Update Docs and Readme (Fix docstring for jit scripting #223, Update Contributing and badges #299, Update Docs and Readme #304, Add docstrings for input type in YOLOTransform #308, Fix docs and add docstrings for YOLOv5 #315)
• Add Fidan and Shiquan as co-authors for great contributions (Add Fidan as co-authors for great contributions on ORT #231, Add Shiquan as co-authors for great contributions on TRT #258)

Code Quality

• Update to .clang-format (Update to .clang-format #222)
• Enable pip cache dependencies in GH Actions (Enable pip cache dependencies in GH Actions #226)
• Move CONTRIBUTING and CODE_OF_CONDUCT into .github (Move CONTRIBUTING and CODE_OF_CONDUCT into .github #228)
• Cleanup unit-test (Cleanup unit-test #246)
• Fixed some spelling/typos (Fixed some spelling/typos #276)
• Set OpenCV as optional (Set opencv as optional #286)
• [pre-commit.ci] pre-commit autoupdate ([pre-commit.ci] pre-commit autoupdate #244, [pre-commit.ci] pre-commit autoupdate #249, [pre-commit.ci] pre-commit autoupdate #260, [pre-commit.ci] pre-commit autoupdate #270)
• Install yolort in GH Actions (Install yolort in GH Actions #294)
• Replaced to_tensor() with pil_to_tensor() + convert_image_dtype() (Replaced to_tensor() with pil_to_tensor() + convert_image_dtype() #298)
• Remove CLI tools for detection and training temporarily (Do not maintain CLI tools for detection and training temporarily #325)

Contributors

We're grateful for our community, which helps us improving yolort by submitting issues and PRs, and providing feedback and suggestions. The following persons have contributed patches for this release:

@datumbox @lzmisscc @mattpopovich @ncnnnnn @q3394101 @ShiquanYu @yang-gis @2428513107

Special thanks to @ShiquanYu for his great contributions on TensorRT C++ example!

---

This discussion was created from the release TensorRT deployment Support, Pre-Processing alignment and more.