Break up into multiple crates?

[infogulch]

scryer-prolog is currently 60k+ lines of rust, and alone takes 4+ minutes to compile on my machine. You can check compile times on your machine by running cargo build --release --timings which generates a nice .html file with a flow chart and details about building each crate.

Here are the timings from building on my machine with the above command after a cargo clean, first a full compile (5m6s) and then touching build/static_string_indexing.rs and doing an "incremental" build (4m9s): cargo-timings.zip

In addition, the peak memory usage while compiling the scryer-prolog crate is ~6GB, which I guess is 5x+ the memory usage of the next crate based on the chart I posted the other day. This may be the cause of some recent builds failing apparently due to out of memory errors.

I don't know if breaking the project up into more crates will help with memory usage. But breaking into multiple crates is the "standard line" when having trouble with compile times in rust projects, maybe it will help with memory usage too? 🤷

I'm not really familiar with the scryer-prolog rust codebase so I regret to admit that I'm not posting with a specific plan in mind, just a couple questions:

• Do we agree that it would be reasonable to try to split scryer-prolog up into more smaller crates?
• Are there any boundaries that would be good candidates for such a division?

[infogulch]

I tried using the mold linker which is known to improve compile times in some cases, but it didn't seem to help here.

[aarroyoc]

I think it's reasonable to split scryer-prolog into scryer-prolog and scryer-prolog-syscalls for system calls that are more unrelated to the core Prolog system (crypto, xml, net, ffi, ...). Right now, dispatch.rs can call any function, those functions don't need to be in system_calls.rs.

[infogulch]

Out of curiosity I ran scryer-prolog through codescene.io which is a static analysis tool that analyzes code for things like technical debt and code health, primarily based on version control history (which is somewhat unique among analyzers ime). As with any analyzer tool take its 'advice' with mountains of salt, but I found it amusing to poke around at the visualizations at least:

https://codescene.io/projects/45799/jobs/1744446/results/code/hotspots/system-map

https://codescene.io/projects/45799/jobs/1744446/results/code/hotspots/system-map#codehealth

[UWN]

How much would such a change accelerate the build? Probably not by much. Going stepwise back to identify the moment when an error has been introduced will still be forbiddingly costly. (Things somewhat improve by setting CARGO_INCREMENTAL=1 which should really be mentioned somewhere.)

Maybe instead, consider to build the executable with a unique suffix, like the one by git describe. And make a softlink from the generic scryer-prolog to the current. The executable of Scryer contains not only the actual binary but also the library files (or so I am told). In this manner it would be possible to rapidly try out previous versions.

[infogulch]

My goals with this suggestion are twofold:

1. Reduce compile times. This would be "nice to have" for fresh builds and all versions going forward, but not essential.
2. Hope to reduce memory usage. It would be a royal pain if scryer compilation used too much memory for the github runners and we were forced to run actions runners on our own hardware or pay for a vps or give up on it entirely. Note builds currently use 6GB, and runners only have 7GB.

Admittedly reducing memory usage is not a certain outcome of splitting into multiple crates, so who knows if its worth trying.

[bakaq]

I noticed you said "stepwise". It's way more efficient to do a binary search to search when a error was introduced, and that is exactly what git bisect does.

[triska]

git bisect is only useful if correctness is a monotonic property of the commits: It does not help, and may even waste time, if a mistake is introduced in a commit, is later overshadowed by other commits, and yet later appears again. It is really helpful to know the commit where a mistake can be observed for the very first time, since that is often a good chance to correct the actual problem. A chronological sequence of executables, one for each commit, lets us quickly find this commit.

[bakaq]

One idea to circumvent the low memory availability of the Github Runners is to use swapfiles to "increase RAM". Github Runners have 14GB of disk space (minus the space used by the system, Rust, and the build process itself) that we could potentially use in a swapfile. I found this action that proves this is possible to do, but it's not too hard to do it manually in the CI .yml file.

Reducing memory usage of the build would be much better of course, as this would not only benefit the runners but also contributors on lower end hardware. I have only 8GB of RAM for example. I have a humongous swap file (well, now it's 8GB, but I've made it over 30GB on one occasion) exactly for situations like this. Swap being orders of magnitude slower than RAM means that the builds are also very slow when they have to use it.

[infogulch]

Good idea. Using swap would be unfortunate and probably horribly slow, but better than not being able to compile. This will be a good plan b in case memory usage becomes a common problem.

[bakaq]

There's also this this action that also creates a swapfile in case it doesn't exist, which seems to be necessary in some big runners.

[infogulch]

I asked on the matrix rust channel and someone suggested running cargo llvm-lines https://github.com/dtolnay/cargo-llvm-lines

This tool measures the number and size of instantiations of each generic function in a program, indicating which parts of your code offer the highest leverage in improving compilation metrics.

I'm not sure what to do with it, but here's the output (click to expand):

  Lines                Copies             Function name
  -----                ------             -------------
  20394                755                (TOTAL)
   1708 (8.4%,  8.4%)   20 (2.6%,  2.6%)  <alloc::sync::Weak<T,A> as core::ops::drop::Drop>::drop
   1692 (8.3%, 16.7%)   36 (4.8%,  7.4%)  alloc::raw_vec::RawVec<T,A>::current_memory
    838 (4.1%, 20.8%)   33 (4.4%, 11.8%)  <alloc::vec::Vec<T,A> as core::ops::drop::Drop>::drop
    720 (3.5%, 24.3%)   36 (4.8%, 16.6%)  <alloc::raw_vec::RawVec<T,A> as core::ops::drop::Drop>::drop
    545 (2.7%, 27.0%)    5 (0.7%, 17.2%)  alloc::collections::vec_deque::VecDeque<T,A>::slice_ranges
    482 (2.4%, 29.3%)    2 (0.3%, 17.5%)  core::sync::atomic::atomic_compare_exchange
    460 (2.3%, 31.6%)    9 (1.2%, 18.7%)  <alloc::boxed::Box<T,A> as core::ops::drop::Drop>::drop
    374 (1.8%, 33.4%)    1 (0.1%, 18.8%)  std::thread::Builder::spawn_unchecked_
    360 (1.8%, 35.2%)    5 (0.7%, 19.5%)  alloc::collections::vec_deque::VecDeque<T,A>::as_mut_slices
    275 (1.3%, 36.5%)   19 (2.5%, 22.0%)  alloc::sync::Arc<T,A>::drop_slow
    272 (1.3%, 37.9%)   19 (2.5%, 24.5%)  <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
    253 (1.2%, 39.1%)    1 (0.1%, 24.6%)  std::io::error::repr_bitpacked::kind_from_prim
    249 (1.2%, 40.3%)    3 (0.4%, 25.0%)  core::sync::atomic::atomic_load
    225 (1.1%, 41.4%)    1 (0.1%, 25.2%)  alloc::alloc::Global::alloc_impl
    212 (1.0%, 42.5%)    5 (0.7%, 25.8%)  <alloc::collections::vec_deque::VecDeque<T,A> as core::ops::drop::Drop>::drop
    202 (1.0%, 43.5%)    2 (0.3%, 26.1%)  std::io::error::repr_bitpacked::decode_repr
    201 (1.0%, 44.5%)    3 (0.4%, 26.5%)  hashbrown::raw::RawIterRange<T>::next_impl
    183 (0.9%, 45.4%)    1 (0.1%, 26.6%)  core::char::methods::encode_utf8_raw
    172 (0.8%, 46.2%)    1 (0.1%, 26.8%)  core::ptr::drop_in_place<scryer_prolog::machine::machine_state::MachineState>
    169 (0.8%, 47.0%)    1 (0.1%, 26.9%)  hashbrown::raw::TableLayout::calculate_layout_for
    165 (0.8%, 47.8%)    1 (0.1%, 27.0%)  alloc::collections::btree::navigate::<impl alloc::collections::btree::node::Handle<alloc::collections::btree::node::NodeRef<alloc::collections::btree::node::marker::Dying,K,V,alloc::collections::btree::node::marker::Leaf>,alloc::collections::btree::node::marker::Edge>>::deallocating_next
    163 (0.8%, 48.6%)    1 (0.1%, 27.2%)  hashbrown::raw::inner::TableLayout::calculate_layout_for
    162 (0.8%, 49.4%)    3 (0.4%, 27.5%)  hashbrown::raw::RawTableInner::drop_elements
    154 (0.8%, 50.2%)    1 (0.1%, 27.7%)  ctrlc::platform::unix::init_os_handler
    149 (0.7%, 50.9%)    1 (0.1%, 27.8%)  std::io::Write::write_all
    145 (0.7%, 51.6%)    5 (0.7%, 28.5%)  alloc::collections::vec_deque::VecDeque<T,A>::to_physical_idx
    145 (0.7%, 52.3%)    1 (0.1%, 28.6%)  core::slice::index::range
    142 (0.7%, 53.0%)    1 (0.1%, 28.7%)  std::thread::Builder::spawn_unchecked_::{{closure}}
    132 (0.6%, 53.7%)    1 (0.1%, 28.9%)  alloc::raw_vec::RawVec<T,A>::allocate_in
    132 (0.6%, 54.3%)    1 (0.1%, 29.0%)  core::ptr::drop_in_place<scryer_prolog::machine::machine_indices::IndexStore>
    119 (0.6%, 54.9%)    3 (0.4%, 29.4%)  core::result::Result<T,E>::unwrap
    117 (0.6%, 55.5%)    3 (0.4%, 29.8%)  core::result::Result<T,E>::expect
    115 (0.6%, 56.1%)    1 (0.1%, 29.9%)  tokio::runtime::scheduler::multi_thread::queue::Local<T>::pop
    113 (0.6%, 56.6%)    1 (0.1%, 30.1%)  <alloc::collections::btree::map::BTreeMap<K,V,A> as core::iter::traits::collect::IntoIterator>::into_iter
    109 (0.5%, 57.1%)    1 (0.1%, 30.2%)  ctrlc::set_handler_inner
    108 (0.5%, 57.7%)    3 (0.4%, 30.6%)  hashbrown::raw::RawIterRange<T>::new
    108 (0.5%, 58.2%)    1 (0.1%, 30.7%)  ctrlc::init_and_set_handler
    101 (0.5%, 58.7%)    1 (0.1%, 30.9%)  <T as alloc::ffi::c_str::CString::new::SpecNewImpl>::spec_new_impl
    101 (0.5%, 59.2%)    1 (0.1%, 31.0%)  alloc::collections::btree::node::NodeRef<BorrowType,K,V,Type>::ascend
    101 (0.5%, 59.7%)    1 (0.1%, 31.1%)  nix::sys::signal::SigAction::new

[bakaq]

One tip I've seen some times is to pass the --jobs 1 (or another small number) flag to cargo build, so that it does less things at once, hopefully using less memory at any given time. This will obviously slow down compilation, and it's not obvious if it will actually reduce memory usage, as the memory usage could be all concentrated in one "job". I think this would be most impactful after breaking up the project into separate crates, in such a way that any of the crates separately don't use too much memory building.

[infogulch]

Yeah I tried -j 1 and it didn't work. Documented on another thread: #1880 (comment)

This is the memory usage over time for normal jobs and -j 1:

[bakaq]

I think a boundary that maybe is worth separating into a different crate is the Prolog libraries that are inlined into the executable. I don't think it would make much of a difference in memory usage though.

[infogulch]

I like that the libraries are embedded, it means an installation is just one file, and you can't get a desync between the engine and libraries.

[bakaq]

Putting them in a separate crate would still put them in the same file at the end of compilation. I also like that it is a self contained archive, but I feel there is a better way of doing this. Maybe have a look at how Zig does it?

[infogulch]

This thread contains some ideas for inspecting compile time memory usage:

How to debug rustc memory usage spikes

In particular, there are:

• /usr/bin/time -v cargo build --bin scryer-prolog --release
• valgrind --tool=massif rustc -- your_compile_options
• heaptrack rustc -- your_compile_options
• rustc -Z time-passes
• rustc -z self-profile
• cargo bloat

If I try any of these I'll post the results.

[infogulch]

This thread concluded with a way to debug rust memory usage spikes, though it doesn't have a "fix" per se.

https://users.rust-lang.org/t/how-to-debug-rustc-memory-usage-spikes/100750/15

[aarroyoc]

I tried to move some functions from system_calls.rs to another crate. It requires moving some definitions to a core/traits crate (at least the traits/structs) since cyclic dependencies are not allowed between crates, which will require more time to do if we agree it's worth doing it.