decoder MMHA kernel support INT8 SCALE_Q_INSTEAD_OF_K and SCALE_P_INS…

[lishicheng1996]

Following the logic of MMHA_FP8_SCALE_Q_INSTEAD_OF_K and MMHA_FP8_SCALE_P_INSTEAD_OF_V, I implemented the INT8 version.

It is theoretically equivalent to the original compute logic without any numeric accuracy degradation.

I tested the speed on H20, A100, and 4090 GPUs. The results show that the average latency of the MMHA kernel decreased by about 5% to 8%, thanks to fewer FMUL instructions.

Nsight Compute Kernel Summary

Nsight Compute Instruction Statistics

[lishicheng1996]

@byshiue @Shixiaowei02 May you pls review and verify this PR~ Thank you very much!

[PerkzZheng]

@lishicheng1996 moving forward the INT8 scale is not enabled because we observed the accuracy drop because of that. Have you checked the accuracy after enabling that ? I will give it another try locally.

[lishicheng1996]

@lishicheng1996 moving forward the INT8 scale is not enabled because we observed the accuracy drop because of that. Have you checked the accuracy after enabling that ? I will give it another try locally.

Hi, thanks for your review~
I checked the MMLU accuracy score on Llama3.0-8B, with int8 smooth quant.
With moving forward the INT8 scale, I didn't see accuracy drop~

[PerkzZheng]

@lishicheng1996 moving forward the INT8 scale is not enabled because we observed the accuracy drop because of that. Have you checked the accuracy after enabling that ? I will give it another try locally.

Hi, thanks for your review~ I checked the MMLU accuracy score on Llama3.0-8B, with int8 smooth quant. With moving forward the INT8 scale, I didn't see accuracy drop~

Thanks. I will run more tests for different models locally, and merge that into the internal TRT-LLM, and release it in coming weeks if everything looks good. Thanks again.

[lishicheng1996]

@lishicheng1996 moving forward the INT8 scale is not enabled because we observed the accuracy drop because of that. Have you checked the accuracy after enabling that ? I will give it another try locally.

Hi, thanks for your review~ I checked the MMLU accuracy score on Llama3.0-8B, with int8 smooth quant. With moving forward the INT8 scale, I didn't see accuracy drop~

Thanks. I will run more tests for different models locally, and merge that into the internal TRT-LLM, and release it in coming weeks if everything looks good. Thanks again.

Hi, may I ask how's the accuracy in your local test? ^_^