Selective Execution based on input file and code changes (#4091)

Fixes #4024 and fixes
#3534

This PR adds support for selective execution based on build input
changes and build code changes. This allows us to consider the changes
to build inputs and build code and do a down-stream graph traversal to
select the potentially affected tasks, without needing to go through the
entire evaluation/cache-loading/etc. process to individually skip cached
tasks.

This is a major feature needed to support large monorepos, where you
want to only run the portion of your build/tests relevant to your code
change, but you want that selection of that portion to be done
automatically by your build tool

## Motivation

Selective execution differs from the normal "cache things and skip
cached tasks" workflow in a few ways:

1. We never need to actually run tasks in order to skip them
* e.g. a CI worker can run `selective.prepare` on `main`,
`selective.run` on the PR branch, and skip the un-affected tasks without
ever running them in the first place.
* Or if the tasks were run earlier on `main` on a different machine, we
do not need to do the plumbing necessary to move the caches onto the
right machine to be used

2. We can skip tasks entirely without going through the cache
load-and-validate process.
* In my experience with other build tools (e.g. Bazel), this can result
in significantly better performance for skipped tasks

4. We can skip the running of `Task.Command`s as well, which normally
never get cached


## Workflows

There are two workflows that this PR supports.


### Manual Task Selection

1. `./mill selective.prepare <selector>` saves an
`out/mill-selective-execution.json` file storing the current
`taskCodeSignatures` and `inputHashes` for all tasks and inputs upstream
of `<selector>`

2. `./mill selective.run <selector>` loads
`out/mill-selective-execution.json`, compares the previouos
`taskCodeSignatures` and `inputHashes` with their current values, and
then only executes (a) tasks upstream of `<selector>` and (b) downstream
of any tasks or inputs that changed since `selective.prepare` was run

This workflow is ideal for things like CI machines, where step (1) is
run on the target branch, (2) is run on the PR branch, and only the
tasks/tests/commands that can potentially be affected by the PR code
change will be run

The naming of `selective.prepare` and `selective.run` is a bit awkward
and can probably be improved.

### Watch And Re Run Selection

1. When you use `./mill -w <selector>`, we only re-run tasks in
`<selector>` if they are downstream of inputs whose values changed or
tasks whose code signature changed

This is a common use case when watching a large number of commands such
as `runBackground`, where you only want to re-run the commands related
to the build inputs or build code you changed.

-----

It's possible that there are other scenarios where selective execution
would be useful, but these were the ones I came up with for now


## Implementation

Implementation wise, we re-use of a lot of existing logic from
`EvaluatorCore`/`GroupEvaluator`/`MainModule`, extracted into
`CodeSigUtils.scala` and `SelectiveExecution.scala`.

The _"store `inputHashes`/`methodCodeHashSignatures` (grouped together
as `SelectiveExecution.Metadata`) from before, compare to value after,
traverse graph to find affected tasks"_ logic is relatively
straightforward, though the actual plumbing of the
`SelectiveExecution.Metadata` data to the code where it is used can be
tricky. We need to store it on disk in order to support `mill
--no-server`, so we serialize it to `out/mill-selective-execution.json`

The core logic is shared between the two workflows above, as is most of
the plumbing, although they hook into very different parts of the Mill
codebase

Covered by some basic integration tests for the various code paths
above, and one example test using 3 `JavaModule`s

## Limitations

Currently doesn't work with `-w show`, due to how `show` has two nested
evaluations that are hard to keep track of which one should be selective
and which one shouldn't. This is part of the general problem discussed
in #502 and I think we can
punt on a solution for now

Cannot yet be used on the com-lihaoyi/mill repo, due to the build graph
unnecessarily plumbing `millVersion` everywhere which invalidates
everything when the git dirty sha changes. Will clean that up in a
follow up

docs/modules/ROOT/pages/depth/large-builds.adoc

@@ -14,8 +14,11 @@ or resource usage, build files are incrementally re-compiled when modified, and
 lazily loaded and initialized only when needed. So you are encouraged to break up your project 
 into modules to manage the layering of your codebase or benefit from parallelism.
 
+== Selective Execution
 
 
+include::partial$example/depth/large/9-selective-execution.adoc[]
+
 == Multi-file Builds
 
 include::partial$example/depth/large/10-multi-file-builds.adoc[]

docs/modules/ROOT/pages/index.adoc

@@ -33,12 +33,12 @@ JVM build tools have a reputation for being sluggish and confusing. Mill tries t
 offer a better alternative, letting your build system take full advantage of the
 Java platform's performance and usability:
 
-* *Performance*: Mill's xref:fundamentals/tasks.adoc[build graph] automatically
+* *Performance*: Mill automatically
 xref:depth/evaluation-model.adoc#_caching_at_each_layer_of_the_evaluation_model[caches]
-and xref:cli/flags.adoc#_jobs_j[parallelizes] build
-tasks, keeping your workflows fast and responsive. Mill adds minimal overhead over
-the logic necessary to build your project, and avoids the long configuration
-times often seen in other build tools like Gradle or SBT
+and xref:cli/flags.adoc#_jobs_j[parallelizes] build tasks to keep local development fast,
+and avoids the long configuration times seen in other tools like Gradle or SBT.
+xref:depth/large-builds.adoc#_selective_execution[Selective execution] keeps
+CI validation times short by only running the tests necessary to validate a code change.
 
 * *Maintainability*: Mill's config and xref:javalib/intro.adoc#_custom_build_logic[custom logic]
 is written in xref:depth/why-scala.adoc[concise type-checked Scala code],

example/depth/large/9-selective-execution/bar/src/bar/Bar.java

@@ -0,0 +1,7 @@
+package bar;
+
+public class Bar {
+  public static String generateHtml(String text) {
+    return "<h1>" + text + "</h1>";
+  }
+}

example/depth/large/9-selective-execution/bar/test/src/bar/BarTests.java

@@ -0,0 +1,14 @@
+package bar;
+
+import static org.junit.Assert.assertEquals;
+
+import org.junit.Test;
+
+public class BarTests {
+
+  @Test
+  public void simple() {
+    String result = Bar.generateHtml("hello");
+    assertEquals("<h1>hello</h1>", result);
+  }
+}

example/depth/large/9-selective-execution/build.mill

@@ -0,0 +1,100 @@
+// Mill allows you to filter the tests and other tasks you execute by limiting them
+// to those affected by a code change. This is useful in managing large codebases where
+// running the entire test suite in CI is often very slow, so you only want to run the
+// tests or tasks that are affected by the changes you are making.
+//
+// This is done via the following commands:
+//
+// * `mill selective.prepare <selector>`: run on the codebase before the code change,
+// stores a snapshot of task inputs and implementations
+//
+// * `mill selective.run <selector>`: run on the codebase after the code change,
+// runs tasks in the given `<selector>` which are affected by the code changes
+// that have happen since `selective.prepare` was run
+//
+// * `mill selective.resolve <selector>`: a dry-run version of `selective.run`, prints
+// out the tasks in `<selector>` that are affected by the code changes and would have
+// run, without actually tunning them.
+//
+// For example, if you want to run all tests related to the code changes in a pull
+// request branch, you can do that as follows:
+//
+// ```bash
+// > git checkout main # start from the target branch of the PR
+//
+// > ./mill selective.prepare __.test
+//
+// > git checkout pull-request-branch # go to the pull request branch
+//
+// > ./mill selective.run __.test
+// ```
+//
+// The example below demonstrates selective test execution on a small 3-module Java build,
+// where `bar` depends on `foo` but `qux` is standalone:
+
+package build
+import mill._, javalib._
+
+trait MyModule extends JavaModule {
+  object test extends JavaTests with TestModule.Junit4
+}
+
+object foo extends MyModule {
+  def moduleDeps = Seq(bar)
+}
+
+object bar extends MyModule
+
+object qux extends MyModule
+
+// In this example, `qux.test` starts off failing with an error, while `foo.test` and
+// `bar.test` pass successfully. Normally, running `__.test` will run all three test
+// suites and show both successes and the one failure:
+
+/** Usage
+
+> mill __.test
+error: Test run foo.FooTests finished: 0 failed, 0 ignored, 1 total, ...
+Test run bar.BarTests finished: 0 failed, 0 ignored, 1 total, ...
+Test run qux.QuxTests finished: 1 failed, 0 ignored, 1 total, ...
+
+*/
+
+// However, this is not always what you want. For example:
+//
+// * If you are validating a pull request
+//   in CI that only touches `bar/`, you do not want the failure in `qux.test` to fail
+//   your tests, because you know that `qux.test` does not depend on `bar/` and thus the
+//   failure cannot be related to your changes.
+//
+// * Even if `qux.test` wasn't failing, running it on a pull request that changes `bar/` is
+//   wasteful, taking up compute resources to run tests that could not possibly be affected
+//   by the code change in question.
+//
+// To solve this, you can run `selective.prepare` before the code change, then `selective.run`
+// after the code change, to only run the tests downstream of the change (below, `foo.test` and `bar.test`):
+
+/** Usage
+
+> mill selective.prepare __.test
+
+> echo '//' >> bar/src/bar/Bar.java # emulate the code change
+
+> mill selective.resolve __.test # dry-run selective execution to show what would get run
+foo.test.test
+bar.test.test
+
+> mill selective.run __.test
+Test run foo.FooTests finished: 0 failed, 0 ignored, 1 total, ...
+Test run bar.BarTests finished: 0 failed, 0 ignored, 1 total, ...
+
+*/
+
+// Similarly, if we make a change `qux/`, using selective execution will only run tests
+// in `qux.test`, and skip those in `foo.test` and `bar.test`.
+// These examples all use `__.test` to selectively run tasks named `.test`, but you can
+// use selective execution on any subset of tasks by specifying them in the selector.
+//
+// Selective execution is very useful for larger codebases, where you are usually changing
+// only small parts of it, and thus only want to run the tests related to your changes.
+// This keeps CI times fast and prevents unrelated breakages from affecting your CI runs.

example/depth/large/9-selective-execution/foo/src/foo/Foo.java

@@ -0,0 +1,11 @@
+package foo;
+
+public class Foo {
+
+  public static final String VALUE = "hello";
+
+  public static void mainFunction(String fooText, String barText) {
+    System.out.println("Foo.value: " + Foo.VALUE);
+    System.out.println("Bar.value: " + bar.Bar.generateHtml(barText));
+  }
+}

example/depth/large/9-selective-execution/foo/test/src/bar/FooTests.java

@@ -0,0 +1,12 @@
+package foo;
+
+import static org.junit.Assert.assertEquals;
+
+import org.junit.Test;
+
+public class FooTests {
+  @Test
+  public void simple() {
+    assertEquals(Foo.VALUE, "hello");
+  }
+}

example/depth/large/9-selective-execution/qux/src/qux/Qux.java

@@ -0,0 +1,7 @@
+package qux;
+
+public class Qux {
+  public static String generateHtml(String text) {
+    return "<p>" + text + "</p>";
+  }
+}

example/depth/large/9-selective-execution/qux/test/src/qux/QuxTests.java

@@ -0,0 +1,14 @@
+package qux;
+
+import static org.junit.Assert.assertEquals;
+
+import org.junit.Test;
+
+public class QuxTests {
+
+  @Test
+  public void simple() {
+    String result = Qux.generateHtml("world");
+    assertEquals("<p>world!</p>", result);
+  }
+}

integration/ide/bsp-modules/src/BspModulesTests.scala

@@ -27,7 +27,8 @@ object BspModulesTests extends UtestIntegrationTestSuite {
           "HelloBsp.test",
           "proj1",
           "proj2",
-          "proj3"
+          "proj3",
+          "selective"
         ).sorted
         assert(readModules == expectedModules)
       }