What paths from out/ should one include between CI jobs?

[ayhon]

I'm setting up a set of GitLab CI jobs to run tests and assemble my project. These tasks are handled in separate jobs, however I would like the test job to be able to use the results from the build job. I tried asking GitLab to preserve the contents of the out/ directory between these jobs, but it gave me the following error:

WARNING: File ignored: out/mill-worker-MfYdUAhWymwPIq7yiMtfEgBbQzg=-1/mill-2464e2577abfd5cb4ed89a550d6709b7-io 
ERROR: Uploading artifacts as "archive" to coordinator... 413 Request Entity Too Large  id=44625 responseStatus=413 Request Entity Too Large status=413 token=glcbt-64

I guess that all of out/ is too much for this GitLab job and I need to prune some paths out. But I'm not sure which directories I should include and which I should leave out. My module is called server, so I guess out/server/ along with out/mill-build/, but anything else?

[lihaoyi]

TBH I would need more information to help here. What's the Gitlab upload limit? Can it be increased? What's the breakdown of files and subfolders and the size taken up by each one? Have you run ncdu to see where the space is going?

Without that information there's no way to guess what you would need to do. Everything in the out/ folder is used for something, so it's an empirical question what you can elide that has the highest ratio of bytes:buildtime

[ayhon]

The upload limit is 100MB. It could be increased, but I would imagine that as my project grows I may run into similar problems, so in a way I was wondering what kinds of directories would be the most impactful to keep, since I myself don't have much insight in how these directories inside of out/ are used.

For instance, my original question could be reformulated as, if I do mill server.assembly and mill server.test, which results in out/ are shared between both jobs?

My first guess is that this would be whatever the lowest common ancestors in the dependency graph of the nodes representing server.assembly and server.test are, but I was also wondering if I should also keep some other folders. (I'm also having an issue using the visualize command though, so I haven't been able to find these lowest common ancestors I was talking about. I suspect this may be due to my setup though, I'm on NixOS)

[lihaoyi]

For instance, my original question could be reformulated as, if I do mill server.assembly and mill server.test, which results in out/ are shared between both jobs?

The easiest way to find out is run plan on both, take the two output lists of tasks, and take their intersection. That's what would be shared

[ayhon]

But wouldn't some tasks in that list no longer be necessary, since they were only needed to compute tasks which are already in the list? That's what I tried to capture through "least common ancestor"

[lefou]

I think a good indicator could be to look at the runtime needed to produce the task results. E.g. compilation (out/*/compile.*) typically takes much longer than a path assembly of already cached jars.

You can find some numbers in oout/mill-profile.json and out/mill-chrome-profile.json. This pages might help as well: https://mill-build.org/mill/comparisons/why-mill.html#_chrome_profiles