Train imatrix with model weights in 64Bit Precision

[joseph777111]

This thought has been bugging me in the back of my mind: can we train imatrices with the model weights in 64bit precision? I know it's overkill, but is it possible? Training the imatrix with an F32 of the model yields superior results. In fact, on my M1 Mac, I can train the F32 imatrix with the --process-output flag set (for llama-matrix), and the model actually benefits from it. So, extrapolating from that, I imagine that training imatrices in 64bit would yield even better results, considering the fact that I run my GGUF IQuants in OF32.EF32.IQ8_0 (Output Tensor.Embedddings.QuantSize). So, I'm curious what an Imatrix and model computed and quantized respectively from 64Bit model weights would yield? Is this possible, or am I ranting like a mad man? 🤔

PS... Llama.cpp + Apple Metal + FA (Flash Attention) is AWESOME! ❤️

@ggerganov @bartowski1182 @ikawrakow

[bartowski1182]

Unless the model was trained and saved at 64 bits it provides no value, even going bf16 -> f32 has no value (other than allowing GPU offloading during calculation, until we get CUDA bf16 support)

[joseph777111]

I see... Okay, so no 64bit precision without the model weights trained in 64bit. Onto the next part of our discussion: I have tested the bf16 -> F32 idea and the bf16 -> F16. The F32 imatrix always outmatches the F16 in model coherence and creativity, in my testings. 🤔

[bartowski1182]

Sure, going to f32 is better than f16 since that conversion is lossless, where going to f16 will clamp some weights and round others, though neglibly at worst there is a non zero change

[joseph777111]

What system do you run quants and imatrix computations on? Do you use Vulkan? I've tried your imatrices, and they are better than the ones I can compute using RunPod's rentable GPU systems. I'm curious what your setup is.

[bartowski1182]

I run them on my epyc 7702 and 3090

[joseph777111]

Thanks, as always, Bartowski! 😁