GS/HW: Use proper alpha min max for palettes when looking up a source, adjust dither on blend mix.

GS/HW: Use proper alpha min max for palettes when looking up a source, adjust dither on blend mix. #8990