Compilation failures due to ambiguous dependencies for code blocks

[mauvealerts]

Some examples fail to compile because rustc can't resolve the extern/dependencies.

A small, concrete example (mdbook-keeper-tokio.zip):

• chapter_1.md contains

#[tokio::main]
async fn main() {}

• book.toml contains

[preprocessor.keeper]
after = ["links"]
manifest_dir = "."
externs = ["tokio"]

• Cargo.toml contains

[dependencies]
tokio = { version = "1.28" }

Initially, if we run mdbook serve the test fails because tokio didn't include the macro feature. So, we update Cargo.toml to

[dependencies]
tokio = { version = "1.28", features = ["full"] }

Now, if we mdbook serve we'll get a compiler error like the following:

error[E0464]: multiple candidates for `rlib` dependency `tokio` found
 --> /usr/local/google/home/mauve/temp/mdbook-keeper-tokio/doctest_cache/keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.rs:1:3
  |
1 | #[tokio::main]
  |   ^^^^^
  |
  = note: candidate #1: /usr/local/google/home/mauve/temp/mdbook-keeper-tokio/doctest_cache/target/debug/deps/libtokio-0c2612ec2ee67576.rlib
  = note: candidate #2: /usr/local/google/home/mauve/temp/mdbook-keeper-tokio/doctest_cache/target/debug/deps/libtokio-1262abbd0eb831c3.rlib

I've observed a couple seemingly-similar problems

• thiserror conflicts between the Cargo workspace for the book and the one in sysroot (due to using build-std at one point, I think???)
• tokio fails to link on Windows error: linking with link.exe failed: exit code: 1181 = note: LINK : fatal error LNK1181: cannot open input file 'windows.0.48.0.lib' (dump of paths omitted, full error below)

Full error for Tokio on Windows:

error: linking with `link.exe` failed: exit code: 1181
  |
  = note: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.36.32532\\bin\\HostX64\\x64\\link.exe" "/NOLOGO" "C:\\Users\\Mauve\\AppData\\Local\\Temp\\rustcXhui8e\\symbols.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.0.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.1.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.10.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.11.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.12.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.13.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.14.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.15.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.2.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.3.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.4.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.5.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.6.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.7.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.8.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.c0439f01-cgu.9.rcgu.o" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.38466xrtp5a5gy8q.rcgu.o" "/LIBPATH:C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target" "/LIBPATH:C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug/deps" "/LIBPATH:C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libtokio-bf6360dac18357e0.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libnum_cpus-1e033af2486169e1.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libsocket2-6fdecbf4161fc574.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libwinapi-40764d50e5ba5ab0.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libbytes-34003459513432c9.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libmio-1a53f60450b2b4c8.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libwindows_sys-fd79f9c7b17a9540.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libpin_project_lite-ae131f81c3df234f.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libparking_lot-02de01d13cdfd805.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libparking_lot_core-dda214187733c960.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libwindows_targets-49372df59ca57fe6.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libcfg_if-507736ef1a6c32e4.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libsmallvec-f1d9b8215b39a8fe.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\liblock_api-a690dc46067acd13.rlib" "C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\target\\debug\\deps\\libscopeguard-ac07cd769cedafdc.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-43ecf4bdbb5e1e31.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-03eee9fc7fe44114.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-f350829ea004292f.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_detect-52475675c4d47720.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-d28662d15d021313.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libminiz_oxide-f5a9ba230b395602.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libadler-adcc534ee710b11e.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-75e91ab7163af4fe.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-6c926cb6aa5ed58c.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-b2a86741ec07ae31.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liblibc-ecdbc3c93c11f60c.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-39d88c9804b36d9f.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-2e112df156ffddae.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-bd7106b7f2a04ee0.rlib" "C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-d62a26418f17519a.rlib" "advapi32.lib" "cfgmgr32.lib" "fwpuclnt.lib" "kernel32.lib" "ntdll.lib" "user32.lib" "ws2_32.lib" "ntdll.lib" "windows.0.48.0.lib" "windows.0.48.0.lib" "kernel32.lib" "advapi32.lib" "userenv.lib" "kernel32.lib" "ws2_32.lib" "bcrypt.lib" "msvcrt.lib" "legacy_stdio_definitions.lib" "/NXCOMPAT" "/LIBPATH:C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "/OUT:C:\\Users\\Mauve\\dev\\mdbook-keeper-tokio\\doctest_cache\\keeper_QUoPxyI0a7B8weTONdwg1ptcBNIdgd0obvEAPEKaels.exe" "/OPT:REF,NOICF" "/DEBUG" "/NATVIS:C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\Mauve\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note: LINK : fatal error LNK1181: cannot open input file 'windows.0.48.0.lib'


error: aborting due to previous error

[mauvealerts]

The example I gave with changing features at least has a workaround: you can delete the directory doctest_cache/target to remove the "leftover" copy of tokio. The other issues seem trickier...

I think the fundamental problem is that getting the rustc command-line arguments right is difficult in the general case. I don't currently understand the approach taken in run_tests.rs, but... here's some offhand ideas nonetheless:

• cargo rustc instead of rustc directly?
• cargo build --message-format=json to get information about the artifacts
• Cargo API somehow? (akin to cargo-script-mvs)

[tfpk]

Thanks for the ideas, I'll investigate this further.

[tfpk]

So I've looked into it, and I think using some of these features is just going to cause more headaches. I think my new plan works, but I'd love your thoughts. For each test:

• Create a new, fake Cargo.toml and Cargo.lock in a temp directory. Copy all the settings from the Cargo.toml we're given, except no [[bin]] or [lib]s.
• Create a fake [[bin]] or [lib] that we can then target.
• cargo build then cargo run the target as appropriate.

This should even mean we don't need any list of externs.

[Kayanski]

It's still an issue when you are working with rust-analyzer, is there a way to fix that and for the library to choose one library ?