Resolution maxes out at 1280x800 on 1440p Samsung monitor with a CalDigit USB-C Dock

[youbzh]

Trying to get BetterDummy to work on a 2560x1440 Samsung monitor, connected to a 2021 Macbook Pro M1 Pro, throught a USB-C dock with a DisplayPort output:
• the monitor can output 2560x1440 (non hidpi) when connected "normally" using MacOS
• when I set BetterDummy up, following the instructions, I get a strange behavior: my monitor shows black borders all around it, and the monitor OSD shows a 1280x800 resolution
• fiddling with the resolution settings, on the Dummy Monitor, I see that the UI scales but it stays within the black borders, and the interface gets downscaled to the 1280x800 resolution
• the only way I get it to work is using RDM on top of BetterDummy, and setting the monitor to 2560x1440 on RDM. However this is not a viable solution: the desktop shows 1080p (hidpi), but there are extra desktop pixels outside of the monitor's borders.

[waydabber]

Hi, what is the model number of dock exactly? Does this issue happen with direct connection? Thank you!

[youbzh]

Hi, so the dock is a Caldigit USB-C Dock (I haven't found a model number on their website, it's a legacy product). Unfortunately I can't test a direct connection as the monitor only has displayport inputs. I might have a HDMI to DisplayPort adapter around that I could test tomorrow, or maybe I'll try gettting an HDMI to Dual Link DVI cable if that can help.
Also for information, the monitor model is a Samsung S27A850D.
Thank you very much for your reply, I appreciate it!

[waydabber]

You could try an USB-C to DisplayPort cable as that gives you the most direct connection (USB-C can output DisplayPort Alt mode directly). If the Display OSD shows 1280x800 then it seems macOS thinks that this is the base resolution of the display and scales to this resolution. But it is unclear why this happens only with BetterDummy running - BD should not affect how a physical display is handled by the OS. However, the dock in question seems to be a first gen TB product with some proprietary built-in magic to allow both HDMI and DisplayPort outputs so that might somehow make things special?

[youbzh]

I don't have that type of cable handy, but I'll try some other connections to test things. Regarding how MacOS handles this display, when I first connected it without BD running, it did connect using 1280x800. I had to go to the System Prefs and hit the alt/option key to get the native resolution.
As for the dock, it's not even TB, it's just USB-C, but it's supposed to handle "DP alt mode". Also, I did manage to get my HiDPI resolution with my older mac, using a "display override" plist file.

[waydabber]

All right, then this is the problem. If you enable mirroring, macOS will revert to the default/base resolution and mirrors to that. There is no control to override it in System Preferences/Displays. I could programmatically try to override it in BD, but probably that's what you do with RDM as well (BD and RDM uses the same API to set resolutions).

[waydabber]

(I'll convert this into a discussion, we can continue it there).

[youbzh]

Thanks so much!
So maybe changing the default resolution from BD could be the solution. To my understanding, the issue I have with RDM is that it can't change the resolution for both monitors at the same time. I'm not at my office right now, but I could do more testing later today. Basically what happens when I use RDM is the real monitor gets set to 2560x1440, while the dummy monitor stays at 1920x1080 (or maybe even 1280x800), so I end up seeing a perfect HiDPI resolution, but the actual desktop is larger than what I see (it's a bit strange, as I can see the menu bar, but for instance, I can't access the dock at the bottom. I'll try to send a screenshot later.

[youbzh]

OK, so I've tried with a HDMI to DVI (dual link) cable, the behavior is the same, using the dock, or using the HDMI port that's on the computer.

[waydabber]

All right. The issue might be with the display's EDID then, it reports itself with a 1280x800 preferred resolution. The problem is that M1 macs tend to take it at face value and use this as a base resolution for scaling.

[youbzh]

RIght, I've been tempted to change the plist file, but with SIP, and the read only file system, I thought finding another solution would be safer/better. I might take that route if other options are too complicated.

[youbzh]

Just adding screenshots of RDM and the pref pane so that you see how it looks when I use RDM to set up the "right" resolutions. Basically that's when the screen looks right, but the dock gets hidden because it's out of bounds


.

[youbzh]

[youbzh]

Hello, I'm getting back to you, as you mentionned you could possibly change the "base resolution" for the monitor programmatically. I'm sure you have a lot of other things to do, but I'd be very interested if this was possible :-)

[youbzh]

Hi there, a quick update to say that I finally found a way to solve my problem. It involves using Better Dummy and SwitchResX. I basically have to set some "base" resolution for my monitor in SwitchResX (see attached screenshot), and then Better Dummy works like expected.
I figured it out when trying to set my external monitor as a mirror of my Macbook display. That's when it reverts to the ugly 1200x800 resolution. So it looks like there's a setting that forces the monitor to stay at it's native resolution in SwitchResX.

Thank you anyway, and I'll be happy to help if you need to do any testing or anything.

(the two checked settings are "startup resolution" and "base scaling resolution", one of them solved my issue)

[waydabber]

Yes, this is what I suspected, the display's edid reports a wrong base resolution which the display uses when doing scaling or mirroring. It's good to know that SwitchResX can fix that on M1 as well as on Intel.

[youbzh]

So I investigated further, and here is what needed to be added to the EDID property list to make my Monitor work:
</dict> <key>default-resolution</key> <data> AAAKAAAABaAAPAAA </data>
This goes into:
/Library/Displays/Contents/Resources/Overrides
with the proper Vendor subfolder (in my case DisplayVendorID-4c2d). Strangely the plist file is named DisplayProductID-83d without any extension.