Test Suite execution in Windows containers: Has anyone tried it?

[bevanweiss]

Just wondering if anyone has tried to use Windows containers...
https://hub.docker.com/_/microsoft-windows-servercore

... and the new Visual Studio Docker integration for the MSI Test Suite execution.
https://learn.microsoft.com/en-au/visualstudio/test/remote-testing?view=vs-2022

[barnson]

I did some work on using Sandbox for runtime tests but haven't completed it. Sandbox has an advantage of not needing additional licensing.

[bevanweiss]

Awesome. I did see Rob's videos on it, but didn't actually try it myself. Have just done so, and it's great.
So I'll see what I can do with that tonight. I'm thinking if it can be a bit scripted, it might be possible to spin up a test running in it and have it nicely integrated with Visual Studio VSTest.

[bevanweiss]

Only drawback I can see with it right now is that it only supports 'same platform' execution (down to OS version), so it's not possible to test ARM64 on an AMD64 development environment, nor is it possible to test Windows 7 if running Windows 11 etc.
But I think for initial smoke tests it would be more than adequate.

[bevanweiss]

@barnson You might have some better scripts etc than what I've already put together around the Sandbox. But they might still be worth a look for you. They've certainly made my workflow a bit easier. wixtoolset/wix#533

I've also found that the Sandbox seems to get a different IP address each time it's launched. But the system name is a GUID that is fixed from what I've seen so far. So that can be used in Visual Studio debugger so you don't need to change IPs all the time.
Unfortunately I couldn't find a way to expose that GUID nicely in the menu batch file... %ComputerName% doesn't show enough characters for that to work.

[bevanweiss]

There's still one IntegrationMSI error that I'm getting with the main branch.
Where the Util User Restricted Domain isn't populating the log with the expected message.

It appears to be due to the WcaLog behaviour

Is it working fine for others?

[barnson]

I get that one too. It's due to logging setup that happens in the official build: https://github.com/wixtoolset/wix/blob/main/.github/workflows/build.yml#L44 Can you add those? (I'm actually surprised that's the only failure due to different logging...)

[barnson]

Confirmed that there are zero errors with the logging policy enabled.

[bevanweiss]

Done. Not sure I was just being a spud, but I could not get the sandbox to accept either "reg add" or "reg import".
But regedit itself appeared to accept the registry file to populate. /s then made it silent.
I've named it (the registry import file) sandbox_registry, without a .reg extension to avoid people running it inadvertently on the host.
We'd always call it explicitly from startup in the Sandbox, so the lack of 'valid extension' I think is better here.

I swapped out the %CD% variable in the github action command for just a fixed reference to C:, annoyingly registry file format only accepts EXPAND_SZ types in hexadecimal, so it's a bit less human readable in the import.

[bevanweiss]

One item that I've struggled to get working in the Sandbox is domain activity.
It appears that the Sandbox itself is not able to join a domain, so I've built up a VM environment with (VMware) snapshots for a Domain Controller and a Workstation. I've also added a new variant of the [RuntimeTest] called (unimaginatively) [RuntimeDomainTest] with its own Environment variable.
Would there be any merit in putting forward a PR for just this [RuntimeDomainTest] E2E functionality? It feels (to me) like it's probably not worth me doing that until I've also got a test (working) to back it up.

[barnson]

Makes sense to hold it until you've used it (validated it). It sounds good to have, though a bummer it won't be tested in PR builds.

[bevanweiss]

The Sandbox definitely has quite a few limitations that I'm banging into:

• Cannot join a domain
• Cannot install Optional Features (such as MSMQ / IIS)
• Cannot start COM+ System Application (the host for all things COM+)

Are the limitations documented anywhere?

[barnson]

Not that I know of. It's treated as a nifty thing for users to play in (e.g., no API).

[bevanweiss]

It looks like the COM+ situation is resolved in a Canary build, this is also the location to report any issues / feature requests for the Windows Sandbox feature. They seem reasonably responsive with it.
microsoft/Windows-Sandbox#47