Use sccache to improve build time #81
Conversation
Populating the compilation cache would create a larget target folder (1.8 GB) in local-registry/ which would get copied over to the test image accidentally. That made the image balloon to over 3 GB.
|
I also want to note this increases build time for the image significantly. Precompiling all dependencies takes about as long as installing cargo-local-registry from source. I believe this should be fine assuming the caching works correctly. (which it maybe doesn't at the moment: #57) If the caching works, this should only need to rebuild mostly whenever the dependencies are actually updated (max 1x per week). |
|
Hm. So I took an axe to the list of supported crates, only leaving 35 (out of 180) which I think are a clear and significant added value for exercism students. That takes the size of the compilation cache down to 250 MB, which is not a linear improvement over the 420 MB. It's still a significant chunk. If we went the route of drastically removing supported crates, we would expect new crates to be added incrementally as we discover what else is useful to students. So the 250 MB would probably slowly creep upwards with time. I'm guesstimating ~300MB long term for the compilation cache. I'll investigate the actual speed improvement next. |
|
I'm somehow getting the exact same time for both an image with sccache and one without (~420ms for resistor-color with proc-macro dependencies). I must be doing something wrong, need more time to figure it out. |
|
So, I made the silly mistake of not cleaning up the target folder after a test run. With that fixed, the image with Now I have to figure out what makes a cache request hit or miss. |
|
My first guess was that stable and nightly compiler artifacts are incompatible. I'm still guessing that's true, using the nightly compiler to build the cache didn't fix the problem. Reading the documentation, it looks like a few other conditions must be met for the cache to be hit. Most of them apply to us I think, so I'll have to fix them before we actually hit the cache. |
For sccache to work, its cache must be populated with the same toolchain that will use it later. An additional docker build stage is a simple solution to make sure the build stage uses the same toolchain as the final stage.
by populating the cache in the same environment as the test runner will be executed in. Most importantly, this means the cache will be populated with the depencencies fetched from the previously generated local registry. This is important because sccache only works for dependencies if they are located at the same absolute path. Otherwise, cache requests to compile a solution are all misses, which even slows the test runner down. See sccache's documentation about known caveats: https://github.com/mozilla/sccache/tree/7074753bfc100ee3e22dc730cf71f3750fc42c1d#known-caveats
|
I have managed to get cache hits by making the environment where the cache is populated and where it is used mostly identical. Locally I'm getting 3.7 seconds without cache and 2.5 seconds with cache. That's finally a speedup, but it's not enough for the timeout of 2 seconds. Also, there might be more room for improvement, because I have also removed one more crate from the supported list, which cuts the cache down to 63 MB, which is much smaller then 250 or 400 MB. I was worried this endeavor might fail due to the image size cost, but 63 MB seems like a reasonable price to pay. CI won't pass yet because I haven't finalized the list of supported crates. |
|
@per-oestergaard Based on this discussion, I checked the size of the target directory when running a clean build in I kinda dropped off this PR because I didn't have any more ideas about how to turn |
closes #35
This is mostly a cherry-pick of #42, thanks @dhovart!
The compilation cache takes up 420 MB, taking the whole image from ~930 MB to ~1.4 GB. I will make an attempt to trim the list of available crates more aggressively (at a higher risk of breaking peoples solutions) to see what that does to the image size. It may well be the size is dominated by a couple crates and removing small obscure ones won't change much.
I also haven't measured the actual speed improvement yet. This is important data to make sure we're making a good trade off (image size vs testing speed).