WiX installer fails to load custom action DLL

[mtfriesen]

Hi, we recently upgraded from WiX 3 to WiX 5, and some Windows machines started failing early in the install process with the WiX 5 MSIs. This seems to be a problem loading a custom action for WixQueryOsWellKnownSID. Do you have any idea what could cause this, or how to debug further?

MSI (s) (88:64) [10:41:51:823]: PROPERTY CHANGE: Adding ACTION property. Its value is 'INSTALL'.
MSI (s) (88:64) [10:41:51:823]: Doing action: INSTALL
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2205 2:  3: ActionText 
Action start 10:41:51: INSTALL.
MSI (s) (88:64) [10:41:51:823]: Running ExecuteSequence
MSI (s) (88:64) [10:41:51:823]: Doing action: FindRelatedProducts
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2205 2:  3: ActionText 
Action start 10:41:51: FindRelatedProducts.
MSI (s) (88:64) [10:41:51:823]: Doing action: AppSearch
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2205 2:  3: ActionText 
Action ended 10:41:51: FindRelatedProducts. Return value 1.
Action start 10:41:51: AppSearch.
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2262 2: Signature 3: -2147287038 
MSI (s) (88:64) [10:41:51:823]: Doing action: Wix4QueryOsWellKnownSID_X64
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2205 2:  3: ActionText 
Action ended 10:41:51: AppSearch. Return value 1.
MSI (s) (88:30) [10:41:51:823]: Invoking remote custom action. DLL: C:\windows\Installer\MSIA9C2.tmp, Entrypoint: WixQueryOsWellKnownSID
MSI (s) (88:9C) [10:41:51:823]: Generating random cookie.
MSI (s) (88:9C) [10:41:51:838]: Created Custom Action Server with PID 15024 (0x3AB0).
MSI (s) (88:70) [10:41:51:854]: Running as a service.
MSI (s) (88:70) [10:41:51:869]: Hello, I'm your 64bit Impersonated custom action server.
Action start 10:41:51: Wix4QueryOsWellKnownSID_X64.
MSI (s) (88:64) [10:41:51:901]: Note: 1: 1723 2: Wix4QueryOsWellKnownSID_X64 3: WixQueryOsWellKnownSID 4: C:\windows\Installer\MSIA9C2.tmp 
MSI (s) (88:64) [10:41:51:901]: Note: 1: 2205 2:  3: Error 
MSI (s) (88:64) [10:41:51:901]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1723 
CustomAction Wix4QueryOsWellKnownSID_X64 returned actual error code 1157 (note this may not be 100% accurate if translation happened inside sandbox)
MSI (s) (88:64) [10:41:51:901]: Note: 1: 2205 2:  3: Error 
MSI (s) (88:64) [10:41:51:901]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709 
MSI (s) (88:64) [10:41:51:901]: Product: XDP for Windows -- Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor.  Action Wix4QueryOsWellKnownSID_X64, entry: WixQueryOsWellKnownSID, library: C:\windows\Installer\MSIA9C2.tmp 

Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor.  Action Wix4QueryOsWellKnownSID_X64, entry: WixQueryOsWellKnownSID, library: C:\windows\Installer\MSIA9C2.tmp 
Action ended 10:41:51: Wix4QueryOsWellKnownSID_X64. Return value 3.

[chrpai]

Do you have a repro repos?

[mtfriesen]

Unfortunately I do not have repro steps - this occurs only on one of our customer's systems. They are running Windows Server 2022, but my own installation of that OS does not repro.

Are there any more detailed logs we can collect from the repro system, or any hypotheses of potential root causes I could try to simulate on another system?