Thoughts on Code Coverage?

[bevanweiss]

It appears that there's quite a few WiX Extension elements which don't currently get tested through changes to other aspects of the system.
It would be ideal if there was a way to readily identify where some of those 'black holes' of code might still be that don't currently get exercised through any of the existing Build / Runtime tests.

It would be ideal if there was a way to have Code Coverage across the WiX landscape.
I'm unsure if this has been looked into. Perhaps some low hanging fruit might just be around the C# front-end.
For the various Extension Custom Actions I can imagine that capturing the coverage data might be difficult, is it something that would have actual benefits that others can see? (or am I perhaps just being naive in thinking it might be possible)

[barnson]

Getting code coverage on the unit tests would probably be straightforward. It'd be harder on the runtime tests, which mostly exist so far only for Burn and new custom actions.

[robmen]

any chance someone has had success with clang-cl for the WiX build

I have very little appetite for managing another C/C++ toolchain.

[barnson]

VC++ has instrumentation: https://learn.microsoft.com/en-us/visualstudio/profiling/instrument-native-component-and-collect-timing-data?view=vs-2022 No idea how well it would work for CAs or elevated Burn.

[bevanweiss]

I'll see if I can get something working with it.
Any tips for cutting down on the WiX build time? I thought inc would do the job... but it doesn't seem to make a significant difference.

[barnson]

The build is mostly single-proc. We have some bits that are multi-proc using traversal projects (e.g., src\libs\dutil\dutil_t.proj). We need more of them. The last time I tried adding traversal projects to the ext tree, I ran into problems with the test projects, so I put it on the back burner. (We think we'll get help from https://devblogs.microsoft.com/dotnet/introducing-ms-test-runner/.)

[bevanweiss]

You wouldn't still happen to have any of the work that you did around the Ext tree would you?
If there was an example .proj for one of the extensions it would be useful to reference.

[robmen]

Also note that historically, the Standard CustomActions had no integration tests before WiX v4. In general, the test infrastructure was pretty underdeveloped in WiX v3. That improved in WiX v4 for the core and Burn, but time ran out for the CustomActions. So, it does not surprise me when we hit issues like wixtoolset/issues##8628 and #8638 and indicates there would be value in adding integration tests for the CustomActions.

Code coverage would be cool too, if not too onerous to deal with.

[bevanweiss]

I'm interested enough to put in some CA integration tests, and the framework around them... but the vast majority of extension CAs I just haven't used myself, nor even used the feature they target (like with MSMQ / COMPlus)... any clues for finding examples around for some of these CAs? I did try a github search on the ComPartition keyword, but it just came back with forks of the WiX repo... which doesn't currently contain any examples for this element.

My thoughts around some of the integration tests are that they could actually tick two boxes:

1. They would test functionality of WiX (i.e. for CI pipeline usage, or at least local change validations before raising PR)
2. They could also be referenced in the user documentation as 'examples', which is something that lots of commenters appear to be calling out for. I'm confident those same commenters will still complain about the 'quality' of such examples... but something is better than nothing.

[robmen]

COM+ is kinda' the worst case. It was contributed by someone who is no longer involved and is mightily complex to use (much less setup). It's also pretty dead technology from Microsoft. It's the perfect storm of maintenance headaches.

There's no great solution to this problem. One of the challenges with the WiX Toolset is that it essentially needs to span the entire technology space of Windows plus the software that runs on Windows. Obviously, we've not reached that point, which is probably good since we don't generally get help maintaining the breadth that we already have.

This situation is also why I'm ruminating over some telemetry designs for WiX v6 so we can start figuring out how much of the WiX Toolset is actively in-use.

In the end, I generally agree with your two bullet points. It's a large task that must be tackled one custom element/one CustomAction at a time.

[bevanweiss]

I think I've got something put together for the COM+ wxs (I found the FG wixv3 tutorial for it, unfortunately after I'd already put together something), just missing a good reference DLL (since this is just a simple smoke test I'm looking at, either .NET or native would suffice...).
Is there a good option for such a DLL (like one that already exists within the WiX project that I'm unaware of, or should I just build a trivial version for it (in which case I assume just the .DLL would want to go into the WiX repo, not the full project to create the .DLL itself).

[barnson]

None that I'm aware of. Maybe a sample someplace?