Moving to Files in WiX 5, "SourceDir" doesn't seem magic anymore

[BPodszun]

Disclaimer: I accept that I have trouble navigating the docs at times and don't do MSIs for a living.

I have a solution with a bunch of projects, one being the Installer and one - for the sake of this discussion - a windows service Foo.

\FooInstaller
-> WiX here
\Foo
-> Dotnet core windows service here
Foo.sln

In my installer project I have a file like this

<Include xmlns="http://wixtoolset.org/schemas/v4/wxs">
    <?ifndef $(env.CI)?>
    <?include SourcePaths.Local.wxi?>
    <?else?>
    <?if $(env.CI) != ""?>
    <?include SourcePaths.CI.wxi?>
    <?else?>
    <?include SourcePaths.Local.wxi?>
    <?endif?>
    <?endif?>
</Include>

which for Some Reasons™ defines a different set of paths for CI and local builds. It looks like this in my v4 project:

<Include xmlns="http://wixtoolset.org/schemas/v4/wxs">
    <?warning This is NOT a CI build (local?)?>
    <?define FooSource = "SourceDir"?> <!-- in the .Local variant. In the .CI variant it's an absolute path -->
</Include>

The component then is using the WiX4 style heat based harvesting to generate a huge file, with gazillion entries that look like

            <Component Id="cmpDufKoLsEXviYiwOW8uU0YqqkTrU" Directory="foodir" Guid="{BD9D7EF7-C97D-48DB-B0B6-1F5CBA3E2DE9}">
                <File Id="filPesrMHNYqq.3oDZxmNT0UmP9XUI" KeyPath="yes" Source="$(FooSource)\appsettings.json" />
            </Component>

.. and that works fine for CI and local builds. $(FooSource) is either a full path or it is 'SourceDir' which to me is undocumented black magic that just works and fishes the files from the project references and their output folders.

Now I wanted to cut down on these auto-generated files and use the new WiX 5 features. My naive attempt looked like this (relevant snippet):

<Fragment>
        <?include SourcePaths.wxi?>
        <ComponentGroup Id="foofiles">
            <Files Include="$(FooSource)\**">
                <!--
                Don't harvest the service because it needs manual authoring (below).
                -->
                <Exclude Files="$(FooSource)\foo-service.exe" />
            </Files>

which I was hoping to be a drop-in replacement for the working WiX 4 version. But unfortunately this warns me during build-time:

warning WIX8601: Missing directory for harvesting files: Could not find a part of the path 'D:\Foo\FooInstaller\SourceDir'

which is .. unsurprising. That takes SourceDir literally and that directory doesn't exist. Is there a way to make this work or am I stuck with the WiX4 way of manually generating my files with heat and then manually editing them (say to exclude the service etc etc)? Where did my SourceDir magic go?