Update on the development branch

[Shixiaowei02]

Hi,

The TensorRT-LLM team is pleased to announce that we have pushed an update to the development branch (and the Triton backend) this Aug 27, 2024.

This update includes:

• Model Support
• Features
  • Add redrafter support for curand and bfloat16
  • Introduce sparse mixer normalization mode for MOE models
  • Add support for QKV scaling in FP8 FMHA
• API
  • [BREAKING CHANGE] Update LLM.generate arguments to include PromptInputs and tqdm
• Bug fixes
  • Fix issue where temperature < 0.001 resulted in garbage output
  • Avoid unnecessary conversion of logging arguments
  • Fix transient workerPool hang
  • Fix default factory for LoraConfig (in python 3.11 and release 0.8.0: ValueError: mutable default <class 'datasets.utils.version.Version'> for field version is not allowed: use default_factory #1323)
• Memory optimization
• Benchmark
  • Add acceptance rate for gptManagerBenchmark
• Performance
• Infra
• Documentation

Thanks,
The TensorRT-LLM Engineering Team