No brightness sync between MBP display brightness and 2 LG Ultrafine display's brightness

[logicito]

I have a MBP M1 with the light sensor active in settings, one LG Ultrafine 5K monitor and one LG Ultrafine 4K monitor. The 5K is connected to the second Thunderbolt port of the 4K, and the first Thunderbolt port of the 4K is connected to the MBP (daisy chain).

When the MBP goes into sleep mode, I press any key to wake it up, and all external monitors will be super bright, and will stay like that for a long time, I have to press the "lower the brightness" key in the Magic Keyboard (F1) to force them to sync to the correct brightness.

I am using the MBP light sensor mode, I have created the group and enable brightness sync. To test if things are configured correctly, I cover the camera/light sensor in the MBP and I can see how the brightness gets reduced across all monitors, then, when removed, brightness comes back, so it is working, but for some reason it doesn't when coming back from display off or sleep or both.

Do you have a suggestion on how to address this? This was my problem before I installed BetterDisplay, I thought it will solve it, maybe I was wrong, waiting for your feedback, thank you