Windows support

[OwenTrokeBillard]

Thank you for all your excellent work. Rust has a bright future with Enzyme for automatic differentiation.

Does Enzyme for Rust support Windows, or is Windows support planned?

[ZuseZ4]

Thanks for the nice words!
LLVM and Rust both support windows, so there is no technical reason on why Enzyme (or Rust-Emzyme)
would not support windows, except of possibly smaller issues like us having messed up some path handling,
did you encounter such bugs?
If yes please feel free to submit smaller bugfixes if you have the capacity, or just create issues.
I don't have a windows machine at hand, but once we got rust-enzyme cleaned up a bit more I will ask a friend or use the Rust CI to find possible windows bugs.

[OwenTrokeBillard]

I tried for several hours to build Rust with Enzyme on Windows. There are some linker errors that are beyond my comprehension. I am likely doing something incorrectly.

Errors

...
LINK : warning LNK4044: unrecognized option '/-gc-sections'; ignored
LLVMTableGenGlobalISel.lib(GIMatchDag.cpp.obj) : error LNK2001: unresolved external symbol __imp__CrtDbgReport
LLVMTableGenGlobalISel.lib(GIMatchTree.cpp.obj) : error LNK2001: unresolved external symbol __imp__CrtDbgReport
LLVMTableGenGlobalISel.lib(CodeExpander.cpp.obj) : error LNK2001: unresolved external symbol __imp__CrtDbgReport
LLVMSupport.lib(JSON.cpp.obj) : error LNK2001: unresolved external symbol __imp__CrtDbgReport
(many more)
...

Steps to reproduce

1. Install Python, CMake, and Ninja.
2. Install Windows Subsystem for Linux (WSL). The configure script requires bash.
3. Install the "C++ Clang tools for Windows" and "Windows 10 SDK" components in Visual Studio Community 2022.
4. Run the configure script from within WSL.

   mkdir build
   cd build
   ../configure --enable-llvm-enzyme --release-channel=nightly --enable-llvm-assertions --enable-clang --enable-lld --enable-option-checking --enable-ninja --disable-docs
   

   • The flag "--enable-llvm-link-shared" is omitted since it is not supported on Windows.
5. In PowerShell, configure CMake to use the Clang compiler.

   $env:CC="C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin\clang.exe"
   $env:CXX="C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin\clang++.exe"
   $env:RC="C:\Program Files (x86)\Windows Kits\10\bin\10.0.22000.0\x64\rc.exe"
   

   • The paths may be different for you.
6. Configure Clang to use the MSVC linker.

   $env:CFLAGS="-fuse-ld=link"
   $env:CXXFLAGS="-fuse-ld=link"
   

7. Run the build script.

   & python.exe ../x.py build --stage 1 library/std library/proc_macro library/test tools/rustdoc
   

Alternatives to MSVC

On Windows, it sounds like Rust uses LLVM for compilation and MSVC for linking. I tried to configure it as such in the steps above.

LLVM appears to come with its own linker, "lld-link", that aims to be a drop-in replacement for the MSVC linker on Windows. It also comes with its own resource compiler, "llvm-rc". I tried these in place of the MSVC variants to no avail.

All of this is beyond my familiarity. @ZuseZ4 Do any of these steps strike you as obviously incorrect?

[ZuseZ4]

Thanks for the list of things you tried, it looks reasonable to me. I know that someone in the Enzyme core repo added a few cmake changes to support static Enzyme compilation, but I am not sure how rust and windows are supposed to work here.
Unfortunately I will be traveling for the next week, so I can't promise to directly have a look at it, but afterwards I can ask to see if any of the real rustc devs has some insight here,
since we do hope to upstream this soonish.

[OwenTrokeBillard]

@ZuseZ4 I hope you enjoy your travels! Thanks for your quick responses on this issue.

Please let me know if there is anything I can do to help in the future (e.g., testing). I am also open to calling one-on-one.

[OwenTrokeBillard]

@ZuseZ4 Do you know why LLVM is configured to be linked statically (--enable-llvm-link-shared)? This option is not supported on Windows. I was able to build on Linux just fine without this option on your macro2 branch.

[ZuseZ4]

Just to be sure, you meant to write windows, not linux in your last sentence, right? Lorenz and I tried a lot of different settings in our different proof of concept implementations, I would just assume that it fixed an outdated bug and isn't needed then. I'll test it on the windows of a friend, great if the fix is that easy, thanks!

[OwenTrokeBillard]

I did mean Linux. I built on Linux without --enable-llvm-link-shared to make sure that flag is not required. I am still working on the Windows build.

[wsmoses]

I presume the shared library is a holdover from when enzyme was linked in via shared library. If it works without it the rust build state should be put back to its default

…

On Sat, Jan 6, 2024 at 7:36 PM Owen Troke-Billard ***@***.***> wrote: I did mean Linux. I built on Linux without --enable-llvm-link-shared to make sure that flag is not required. I am still working on the Windows build. — Reply to this email directly, view it on GitHub <#54 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJTUXBJDI2R7IMO7X5SCN3YNHUYHAVCNFSM6AAAAAA5HF6XTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZZHA4TIOJWHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[OwenTrokeBillard]

Enzyme is not configured to be built with Clang + MSVC. It can be built with Clang + MinGW (see EnzymeAD/Enzyme#1449), and I believe this is how Enzyme.jl works on Windows.

I see a few options for proceeding:

1. Make Enzyme compatible with Clang + MSVC.
2. Force Windows users to install MinGW, and build Enzyme with Clang + MinGW.
3. Use a precompiled Enzyme library instead of building from scratch.
4. Do not support Windows.

I am trying to get Enzyme to build with Clang + MSVC. Please let me know if this approach is not possible or desirable.

As an aside, please ignore my claim about --enable-llvm-link-shared being redundant. I tried another build without it since and it failed.

[ZuseZ4]

Rust does support Windows via MSVC, right? I guess in this case 1) is the best path forward, so it would be great if you find a way to get this to work. @wsmoses Do you know any blockers from the llvm side?

[wsmoses]

Looks like a CMake issue on windows. I don't have a windows machine to reproduce/debug, but I don't see any blockers besides someone finding a machine to test and fix cmake on.

[OwenTrokeBillard]

@wsmoses @ZuseZ4 Here is a log containing the errors that arise when attempting to build Rust with Enzyme on Windows, as you requested: log-2024-01-19.txt. No action is required on your part.

[KhaledHosseini]

@wsmoses @ZuseZ4 Here is a log containing the errors that arise when attempting to build Rust with Enzyme on Windows, as you requested: log-2024-01-19.txt. No action is required on your part.

Did you manage to solve the problem? the exact problem for me. I tried Visual Studio 2019 too, but the same result.
I removed the following line from "src/tools/enzyme/enzyme/CMakeLists.txt":

SET(CMAKE_CXX_FLAGS "-Wall -fno-rtti ${CMAKE_CXX_FLAGS} -Werror=unused-variable -Werror=dangling-else -Werror=unused-but-set-variable -Werror=return-type -Werror=nonnull -Werror=unused-result -Werror=reorder -Werror=switch")

and the 1st series of errors gone (those in your log file), but another error appeared:

fatal error C1189: #error:  The C++ Standard Library forbids macroizing the keyword "private". Enable warning C4005 to find the forbidden define.

I guess I need to add some keywords to the compiler like _ALLOW_KEYWORD_MACROS, but I do not know how.

[OwenTrokeBillard]

Did you manage to solve the problem?

Windows support is a work in progress. See EnzymeAD/Enzyme#1607. The main blocker is llvm/llvm-project#83052 AFAIK. @ZuseZ4 is that right?

[ZuseZ4]

The main blocker is llvm/llvm-project#83052 AFAIK. @ZuseZ4 is that right?

Yes, it's Enzyme doing some prohibited c++ hacks. Unfortunately the PR author got busy lately and someone raised some last minute concerns due to which the PR can't directly be merged. I had intentions to fix it at some point, but lately instead had to work on some other Enzyme internal bug. I therefore might merge a first Rust-AD version with Linux/Mac support only, just because it's a lot of effort to maintain this fork out-of-tree, and will then just update the Enzyme submodule, once either Billy, I, or someone else later had time to fix this.

[OwenTrokeBillard]

@ZuseZ4 I think merging Linux & Mac support first is an excellent idea. A significant benefit, on top of those you mentioned, is that Enzyme can begin gaining adoption sooner. Windows folks can also use WSL in the interim.

[ZuseZ4]

@OwenTrokeBillard Just to be sure, did you (or anyone else here) use WSL for Rust-Enzyme successfully in the past? I think it should work, but it would be nice to have some confirmation.

[OwenTrokeBillard]

Just to be sure, did you (or anyone else here) use WSL for Rust-Enzyme successfully?

No. And I since switched to Linux because the Rust compile times are significantly faster. Hopefully someone else can try.

[OwenTrokeBillard]

@KhaledHosseini Can you try building with WSL? That might solve your issue and give @ZuseZ4 some valuable feedback.

[wsmoses]

The problem is specifically the Microsoft c++ compiler iirc, it should be successful with say clang or cygwin on windows

…

On Sun, May 19, 2024 at 8:41 PM Owen Troke-Billard ***@***.***> wrote: Just to be sure, did you (or anyone else here) use WSL for Rust-Enzyme successfully? No. And I since switched to Linux because the Rust compile times are significantly faster. Hopefully someone else can try. — Reply to this email directly, view it on GitHub <#54 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJTUXCDE424KUQIMAUF7KLZDFIDRAVCNFSM6AAAAAA5HF6XTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGUZDAMRXGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[KhaledHosseini]

@KhaledHosseini Can you try building with WSL? That might solve your issue and give @ZuseZ4 some valuable feedback.

Sure. I'll try and tell you the results.

[KhaledHosseini]

The problem is specifically the Microsoft c++ compiler iirc, it should be successful with say clang or cygwin on windows
…
On Sun, May 19, 2024 at 8:41 PM Owen Troke-Billard @.> wrote: Just to be sure, did you (or anyone else here) use WSL for Rust-Enzyme successfully? No. And I since switched to Linux because the Rust compile times are significantly faster. Hopefully someone else can try. — Reply to this email directly, view it on GitHub <#54 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJTUXCDE424KUQIMAUF7KLZDFIDRAVCNFSM6AAAAAA5HF6XTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGUZDAMRXGM . You are receiving this because you were mentioned.Message ID: @.>

I tried MSYS2 without success. I'm not sure if I did all the steps right or even it has anything to do with Enzyme. Here are my exact step-by-step commands on MSYS2 terminal:

1- cd c://rs_enz
2- git clone --depth 1 https://github.com/EnzymeAD/rust.git
3- cd rust
4- python ./src/bootstrap/configure.py --enable-llvm-link-shared --enable-llvm-plugins --enable-llvm-enzyme --release-channel=nightly --enable-llvm-assertions --enable-clang --enable-lld --enable-option-checking --enable-ninja --disable-docs
5- python x.py build --stage 1 library

the error is

thread 'main' panicked at src\core\config\config.rs:1249:74:
called `Result::unwrap()` on an `Err` value: Os { code: 3, kind: NotFound, message: "The system cannot find the path specified." }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:03:36

the log file is included.
log.txt

EDIT: This error in line 1249 of bootstrap\src\core\config\config.rs has been fixed in the original Rust repo. Is it possible to sync the fork with the original Rust repo?

[OwenTrokeBillard]

@ZuseZ4 Is the master branch working right now?

I am able to build Rust + Enzyme in WSL but running the first forward mode example produces incorrect results:

thread 'main' panicked at src/main.rs:15:5:
assertion `left == right` failed
  left: 7.0
 right: 0.0

[ZuseZ4]

@ZuseZ4 Is the master branch working right now?

I am able to build Rust + Enzyme in WSL but running the first forward mode example produces incorrect results:

thread 'main' panicked at src/main.rs:15:5:
assertion `left == right` failed
  left: 7.0
 right: 0.0

That usually means you don't use fat-lto in your Cargo.toml, can you check that? I meant to abort compilation, but for some reason my check doesn't always trigger.
And no, it's too much work to continuously keep up to date with upstream. I need to fix a few more Enzyme issues till the 3rd and will then reimplement it on the latest master.

[OwenTrokeBillard]

Rust + Enzyme does work in WSL after enabling fat LTO. Amazing work @ZuseZ4 🎉