From 24ba0d3d6e2fefce7211a5f970440ad4194540e7 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Wed, 11 Dec 2024 20:15:34 +0800 Subject: [PATCH] Selective Execution based on input file and code changes (#4091) Fixes https://github.com/com-lihaoyi/mill/issues/4024 and fixes https://github.com/com-lihaoyi/mill/issues/3534 This PR adds support for selective execution based on build input changes and build code changes. This allows us to consider the changes to build inputs and build code and do a down-stream graph traversal to select the potentially affected tasks, without needing to go through the entire evaluation/cache-loading/etc. process to individually skip cached tasks. This is a major feature needed to support large monorepos, where you want to only run the portion of your build/tests relevant to your code change, but you want that selection of that portion to be done automatically by your build tool ## Motivation Selective execution differs from the normal "cache things and skip cached tasks" workflow in a few ways: 1. We never need to actually run tasks in order to skip them * e.g. a CI worker can run `selective.prepare` on `main`, `selective.run` on the PR branch, and skip the un-affected tasks without ever running them in the first place. * Or if the tasks were run earlier on `main` on a different machine, we do not need to do the plumbing necessary to move the caches onto the right machine to be used 2. We can skip tasks entirely without going through the cache load-and-validate process. * In my experience with other build tools (e.g. Bazel), this can result in significantly better performance for skipped tasks 4. We can skip the running of `Task.Command`s as well, which normally never get cached ## Workflows There are two workflows that this PR supports. ### Manual Task Selection 1. `./mill selective.prepare ` saves an `out/mill-selective-execution.json` file storing the current `taskCodeSignatures` and `inputHashes` for all tasks and inputs upstream of `` 2. `./mill selective.run ` loads `out/mill-selective-execution.json`, compares the previouos `taskCodeSignatures` and `inputHashes` with their current values, and then only executes (a) tasks upstream of `` and (b) downstream of any tasks or inputs that changed since `selective.prepare` was run This workflow is ideal for things like CI machines, where step (1) is run on the target branch, (2) is run on the PR branch, and only the tasks/tests/commands that can potentially be affected by the PR code change will be run The naming of `selective.prepare` and `selective.run` is a bit awkward and can probably be improved. ### Watch And Re Run Selection 1. When you use `./mill -w `, we only re-run tasks in `` if they are downstream of inputs whose values changed or tasks whose code signature changed This is a common use case when watching a large number of commands such as `runBackground`, where you only want to re-run the commands related to the build inputs or build code you changed. ----- It's possible that there are other scenarios where selective execution would be useful, but these were the ones I came up with for now ## Implementation Implementation wise, we re-use of a lot of existing logic from `EvaluatorCore`/`GroupEvaluator`/`MainModule`, extracted into `CodeSigUtils.scala` and `SelectiveExecution.scala`. The _"store `inputHashes`/`methodCodeHashSignatures` (grouped together as `SelectiveExecution.Metadata`) from before, compare to value after, traverse graph to find affected tasks"_ logic is relatively straightforward, though the actual plumbing of the `SelectiveExecution.Metadata` data to the code where it is used can be tricky. We need to store it on disk in order to support `mill --no-server`, so we serialize it to `out/mill-selective-execution.json` The core logic is shared between the two workflows above, as is most of the plumbing, although they hook into very different parts of the Mill codebase Covered by some basic integration tests for the various code paths above, and one example test using 3 `JavaModule`s ## Limitations Currently doesn't work with `-w show`, due to how `show` has two nested evaluations that are hard to keep track of which one should be selective and which one shouldn't. This is part of the general problem discussed in https://github.com/com-lihaoyi/mill/issues/502 and I think we can punt on a solution for now Cannot yet be used on the com-lihaoyi/mill repo, due to the build graph unnecessarily plumbing `millVersion` everywhere which invalidates everything when the git dirty sha changes. Will clean that up in a follow up --- .../ROOT/pages/depth/large-builds.adoc | 3 + docs/modules/ROOT/pages/index.adoc | 10 +- .../bar/src/bar/Bar.java | 7 + .../bar/test/src/bar/BarTests.java | 14 + .../large/9-selective-execution/build.mill | 100 +++ .../foo/src/foo/Foo.java | 11 + .../foo/test/src/bar/FooTests.java | 12 + .../qux/src/qux/Qux.java | 7 + .../qux/test/src/qux/QuxTests.java | 14 + .../ide/bsp-modules/src/BspModulesTests.scala | 3 +- .../src/MultiLevelBuildTests.scala | 711 +++++++++--------- .../selective-execution/resources/bar/bar.txt | 1 + .../selective-execution/resources/build.mill | 26 + .../selective-execution/resources/foo/foo.txt | 1 + .../src/SelectiveExecutionTests.scala | 180 +++++ .../src/WatchSourceInputTests.scala | 2 +- .../client/src/mill/main/client/OutFiles.java | 7 + main/eval/src/mill/eval/CodeSigUtils.scala | 117 +++ main/eval/src/mill/eval/Evaluator.scala | 1 + main/eval/src/mill/eval/EvaluatorCore.scala | 42 +- main/eval/src/mill/eval/EvaluatorImpl.scala | 2 +- main/eval/src/mill/eval/GroupEvaluator.scala | 62 +- main/src/mill/main/MainModule.scala | 32 +- main/src/mill/main/RunScript.scala | 111 ++- main/src/mill/main/SelectiveExecution.scala | 150 ++++ .../mill/main/SelectiveExecutionModule.scala | 71 ++ .../src/mill/runner/MillBuildBootstrap.scala | 19 +- runner/src/mill/runner/MillMain.scala | 11 +- 28 files changed, 1219 insertions(+), 508 deletions(-) create mode 100644 example/depth/large/9-selective-execution/bar/src/bar/Bar.java create mode 100644 example/depth/large/9-selective-execution/bar/test/src/bar/BarTests.java create mode 100644 example/depth/large/9-selective-execution/build.mill create mode 100644 example/depth/large/9-selective-execution/foo/src/foo/Foo.java create mode 100644 example/depth/large/9-selective-execution/foo/test/src/bar/FooTests.java create mode 100644 example/depth/large/9-selective-execution/qux/src/qux/Qux.java create mode 100644 example/depth/large/9-selective-execution/qux/test/src/qux/QuxTests.java create mode 100644 integration/invalidation/selective-execution/resources/bar/bar.txt create mode 100644 integration/invalidation/selective-execution/resources/build.mill create mode 100644 integration/invalidation/selective-execution/resources/foo/foo.txt create mode 100644 integration/invalidation/selective-execution/src/SelectiveExecutionTests.scala create mode 100644 main/eval/src/mill/eval/CodeSigUtils.scala create mode 100644 main/src/mill/main/SelectiveExecution.scala create mode 100644 main/src/mill/main/SelectiveExecutionModule.scala diff --git a/docs/modules/ROOT/pages/depth/large-builds.adoc b/docs/modules/ROOT/pages/depth/large-builds.adoc index c4bb8ec30a3..bdfa62d5334 100644 --- a/docs/modules/ROOT/pages/depth/large-builds.adoc +++ b/docs/modules/ROOT/pages/depth/large-builds.adoc @@ -14,8 +14,11 @@ or resource usage, build files are incrementally re-compiled when modified, and lazily loaded and initialized only when needed. So you are encouraged to break up your project into modules to manage the layering of your codebase or benefit from parallelism. +== Selective Execution +include::partial$example/depth/large/9-selective-execution.adoc[] + == Multi-file Builds include::partial$example/depth/large/10-multi-file-builds.adoc[] diff --git a/docs/modules/ROOT/pages/index.adoc b/docs/modules/ROOT/pages/index.adoc index e03e2020467..2bec2abd99e 100644 --- a/docs/modules/ROOT/pages/index.adoc +++ b/docs/modules/ROOT/pages/index.adoc @@ -33,12 +33,12 @@ JVM build tools have a reputation for being sluggish and confusing. Mill tries t offer a better alternative, letting your build system take full advantage of the Java platform's performance and usability: -* *Performance*: Mill's xref:fundamentals/tasks.adoc[build graph] automatically +* *Performance*: Mill automatically xref:depth/evaluation-model.adoc#_caching_at_each_layer_of_the_evaluation_model[caches] -and xref:cli/flags.adoc#_jobs_j[parallelizes] build -tasks, keeping your workflows fast and responsive. Mill adds minimal overhead over -the logic necessary to build your project, and avoids the long configuration -times often seen in other build tools like Gradle or SBT +and xref:cli/flags.adoc#_jobs_j[parallelizes] build tasks to keep local development fast, +and avoids the long configuration times seen in other tools like Gradle or SBT. +xref:depth/large-builds.adoc#_selective_execution[Selective execution] keeps +CI validation times short by only running the tests necessary to validate a code change. * *Maintainability*: Mill's config and xref:javalib/intro.adoc#_custom_build_logic[custom logic] is written in xref:depth/why-scala.adoc[concise type-checked Scala code], diff --git a/example/depth/large/9-selective-execution/bar/src/bar/Bar.java b/example/depth/large/9-selective-execution/bar/src/bar/Bar.java new file mode 100644 index 00000000000..8c208da734c --- /dev/null +++ b/example/depth/large/9-selective-execution/bar/src/bar/Bar.java @@ -0,0 +1,7 @@ +package bar; + +public class Bar { + public static String generateHtml(String text) { + return "

" + text + "

"; + } +} diff --git a/example/depth/large/9-selective-execution/bar/test/src/bar/BarTests.java b/example/depth/large/9-selective-execution/bar/test/src/bar/BarTests.java new file mode 100644 index 00000000000..787d7931110 --- /dev/null +++ b/example/depth/large/9-selective-execution/bar/test/src/bar/BarTests.java @@ -0,0 +1,14 @@ +package bar; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +public class BarTests { + + @Test + public void simple() { + String result = Bar.generateHtml("hello"); + assertEquals("

hello

", result); + } +} diff --git a/example/depth/large/9-selective-execution/build.mill b/example/depth/large/9-selective-execution/build.mill new file mode 100644 index 00000000000..54653332a91 --- /dev/null +++ b/example/depth/large/9-selective-execution/build.mill @@ -0,0 +1,100 @@ +// Mill allows you to filter the tests and other tasks you execute by limiting them +// to those affected by a code change. This is useful in managing large codebases where +// running the entire test suite in CI is often very slow, so you only want to run the +// tests or tasks that are affected by the changes you are making. +// +// This is done via the following commands: +// +// * `mill selective.prepare `: run on the codebase before the code change, +// stores a snapshot of task inputs and implementations +// +// * `mill selective.run `: run on the codebase after the code change, +// runs tasks in the given `` which are affected by the code changes +// that have happen since `selective.prepare` was run +// +// * `mill selective.resolve `: a dry-run version of `selective.run`, prints +// out the tasks in `` that are affected by the code changes and would have +// run, without actually tunning them. +// +// For example, if you want to run all tests related to the code changes in a pull +// request branch, you can do that as follows: +// +// ```bash +// > git checkout main # start from the target branch of the PR +// +// > ./mill selective.prepare __.test +// +// > git checkout pull-request-branch # go to the pull request branch +// +// > ./mill selective.run __.test +// ``` +// +// The example below demonstrates selective test execution on a small 3-module Java build, +// where `bar` depends on `foo` but `qux` is standalone: + +package build +import mill._, javalib._ + +trait MyModule extends JavaModule { + object test extends JavaTests with TestModule.Junit4 +} + +object foo extends MyModule { + def moduleDeps = Seq(bar) +} + +object bar extends MyModule + +object qux extends MyModule + +// In this example, `qux.test` starts off failing with an error, while `foo.test` and +// `bar.test` pass successfully. Normally, running `__.test` will run all three test +// suites and show both successes and the one failure: + +/** Usage + +> mill __.test +error: Test run foo.FooTests finished: 0 failed, 0 ignored, 1 total, ... +Test run bar.BarTests finished: 0 failed, 0 ignored, 1 total, ... +Test run qux.QuxTests finished: 1 failed, 0 ignored, 1 total, ... + +*/ + +// However, this is not always what you want. For example: +// +// * If you are validating a pull request +// in CI that only touches `bar/`, you do not want the failure in `qux.test` to fail +// your tests, because you know that `qux.test` does not depend on `bar/` and thus the +// failure cannot be related to your changes. +// +// * Even if `qux.test` wasn't failing, running it on a pull request that changes `bar/` is +// wasteful, taking up compute resources to run tests that could not possibly be affected +// by the code change in question. +// +// To solve this, you can run `selective.prepare` before the code change, then `selective.run` +// after the code change, to only run the tests downstream of the change (below, `foo.test` and `bar.test`): + +/** Usage + +> mill selective.prepare __.test + +> echo '//' >> bar/src/bar/Bar.java # emulate the code change + +> mill selective.resolve __.test # dry-run selective execution to show what would get run +foo.test.test +bar.test.test + +> mill selective.run __.test +Test run foo.FooTests finished: 0 failed, 0 ignored, 1 total, ... +Test run bar.BarTests finished: 0 failed, 0 ignored, 1 total, ... + +*/ + +// Similarly, if we make a change `qux/`, using selective execution will only run tests +// in `qux.test`, and skip those in `foo.test` and `bar.test`. +// These examples all use `__.test` to selectively run tasks named `.test`, but you can +// use selective execution on any subset of tasks by specifying them in the selector. +// +// Selective execution is very useful for larger codebases, where you are usually changing +// only small parts of it, and thus only want to run the tests related to your changes. +// This keeps CI times fast and prevents unrelated breakages from affecting your CI runs. diff --git a/example/depth/large/9-selective-execution/foo/src/foo/Foo.java b/example/depth/large/9-selective-execution/foo/src/foo/Foo.java new file mode 100644 index 00000000000..71c0fb17769 --- /dev/null +++ b/example/depth/large/9-selective-execution/foo/src/foo/Foo.java @@ -0,0 +1,11 @@ +package foo; + +public class Foo { + + public static final String VALUE = "hello"; + + public static void mainFunction(String fooText, String barText) { + System.out.println("Foo.value: " + Foo.VALUE); + System.out.println("Bar.value: " + bar.Bar.generateHtml(barText)); + } +} diff --git a/example/depth/large/9-selective-execution/foo/test/src/bar/FooTests.java b/example/depth/large/9-selective-execution/foo/test/src/bar/FooTests.java new file mode 100644 index 00000000000..ad415adc415 --- /dev/null +++ b/example/depth/large/9-selective-execution/foo/test/src/bar/FooTests.java @@ -0,0 +1,12 @@ +package foo; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +public class FooTests { + @Test + public void simple() { + assertEquals(Foo.VALUE, "hello"); + } +} diff --git a/example/depth/large/9-selective-execution/qux/src/qux/Qux.java b/example/depth/large/9-selective-execution/qux/src/qux/Qux.java new file mode 100644 index 00000000000..96f515102eb --- /dev/null +++ b/example/depth/large/9-selective-execution/qux/src/qux/Qux.java @@ -0,0 +1,7 @@ +package qux; + +public class Qux { + public static String generateHtml(String text) { + return "

" + text + "

"; + } +} diff --git a/example/depth/large/9-selective-execution/qux/test/src/qux/QuxTests.java b/example/depth/large/9-selective-execution/qux/test/src/qux/QuxTests.java new file mode 100644 index 00000000000..8241e592065 --- /dev/null +++ b/example/depth/large/9-selective-execution/qux/test/src/qux/QuxTests.java @@ -0,0 +1,14 @@ +package qux; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +public class QuxTests { + + @Test + public void simple() { + String result = Qux.generateHtml("world"); + assertEquals("

world!

", result); + } +} diff --git a/integration/ide/bsp-modules/src/BspModulesTests.scala b/integration/ide/bsp-modules/src/BspModulesTests.scala index 0a59cb2daed..8927cef4dd0 100644 --- a/integration/ide/bsp-modules/src/BspModulesTests.scala +++ b/integration/ide/bsp-modules/src/BspModulesTests.scala @@ -27,7 +27,8 @@ object BspModulesTests extends UtestIntegrationTestSuite { "HelloBsp.test", "proj1", "proj2", - "proj3" + "proj3", + "selective" ).sorted assert(readModules == expectedModules) } diff --git a/integration/invalidation/multi-level-editing/src/MultiLevelBuildTests.scala b/integration/invalidation/multi-level-editing/src/MultiLevelBuildTests.scala index e0904fd427f..77e40d19af4 100644 --- a/integration/invalidation/multi-level-editing/src/MultiLevelBuildTests.scala +++ b/integration/invalidation/multi-level-editing/src/MultiLevelBuildTests.scala @@ -128,364 +128,373 @@ object MultiLevelBuildTests extends UtestIntegrationTestSuite { savedClassLoaderIds = currentClassLoaderIds } - test("validEdits") - integrationTest { tester => - import tester._ - runAssertSuccess(tester, "

hello

world

0.13.1

!") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - // First run all classloaders are new, except level 0 running user code - // which doesn't need generate a classloader which never changes - checkChangedClassloaders(tester, null, true, true, true) - - modifyFile(workspacePath / "foo/src/Example.scala", _.replace("!", "?")) - runAssertSuccess(tester, "

hello

world

0.13.1

?") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - // Second run with no build changes, all classloaders are unchanged - checkChangedClassloaders(tester, null, false, false, false) - - modifyFile(workspacePath / "build.mill", _.replace("hello", "HELLO")) - runAssertSuccess(tester, "

HELLO

world

0.13.1

?") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, false, false) - - modifyFile( - workspacePath / "mill-build/build.mill", - _.replace("def scalatagsVersion = ", "def scalatagsVersion = \"changed-\" + ") - ) - runAssertSuccess(tester, "

HELLO

world

changed-0.13.1

?") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, false) - - modifyFile( - workspacePath / "mill-build/mill-build/build.mill", - _.replace("0.13.1", "0.12.0") - ) - runAssertSuccess(tester, "

HELLO

world

changed-0.12.0

?") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, true) - - modifyFile( - workspacePath / "mill-build/mill-build/build.mill", - _.replace("0.12.0", "0.13.1") - ) - runAssertSuccess(tester, "

HELLO

world

changed-0.13.1

?") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, true) - - modifyFile( - workspacePath / "mill-build/build.mill", - _.replace("def scalatagsVersion = \"changed-\" + ", "def scalatagsVersion = ") - ) - runAssertSuccess(tester, "

HELLO

world

0.13.1

?") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, false) - - modifyFile(workspacePath / "build.mill", _.replace("HELLO", "hello")) - runAssertSuccess(tester, "

hello

world

0.13.1

?") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, false, false) - - modifyFile(workspacePath / "foo/src/Example.scala", _.replace("?", "!")) - runAssertSuccess(tester, "

hello

world

0.13.1

!") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, false, false, false) + val retryCount = if (sys.env.contains("CI")) 2 else 0 + test("validEdits") - retry(retryCount) { + integrationTest { tester => + import tester._ + runAssertSuccess(tester, "

hello

world

0.13.1

!") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + // First run all classloaders are new, except level 0 running user code + // which doesn't need generate a classloader which never changes + checkChangedClassloaders(tester, null, true, true, true) + + modifyFile(workspacePath / "foo/src/Example.scala", _.replace("!", "?")) + runAssertSuccess(tester, "

hello

world

0.13.1

?") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + // Second run with no build changes, all classloaders are unchanged + checkChangedClassloaders(tester, null, false, false, false) + + modifyFile(workspacePath / "build.mill", _.replace("hello", "HELLO")) + runAssertSuccess(tester, "

HELLO

world

0.13.1

?") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, false, false) + + modifyFile( + workspacePath / "mill-build/build.mill", + _.replace("def scalatagsVersion = ", "def scalatagsVersion = \"changed-\" + ") + ) + runAssertSuccess(tester, "

HELLO

world

changed-0.13.1

?") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, false) + + modifyFile( + workspacePath / "mill-build/mill-build/build.mill", + _.replace("0.13.1", "0.12.0") + ) + runAssertSuccess(tester, "

HELLO

world

changed-0.12.0

?") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, true) + + modifyFile( + workspacePath / "mill-build/mill-build/build.mill", + _.replace("0.12.0", "0.13.1") + ) + runAssertSuccess(tester, "

HELLO

world

changed-0.13.1

?") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, true) + + modifyFile( + workspacePath / "mill-build/build.mill", + _.replace("def scalatagsVersion = \"changed-\" + ", "def scalatagsVersion = ") + ) + runAssertSuccess(tester, "

HELLO

world

0.13.1

?") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, false) + + modifyFile(workspacePath / "build.mill", _.replace("HELLO", "hello")) + runAssertSuccess(tester, "

hello

world

0.13.1

?") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, false, false) + + modifyFile(workspacePath / "foo/src/Example.scala", _.replace("?", "!")) + runAssertSuccess(tester, "

hello

world

0.13.1

!") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, false, false, false) + } } - test("parseErrorEdits") - integrationTest { tester => - import tester._ - def causeParseError(p: os.Path) = - modifyFile(p, _.replace("extends", "extendx")) - - def fixParseError(p: os.Path) = - modifyFile(p, _.replace("extendx", "extends")) - - runAssertSuccess(tester, "

hello

world

0.13.1

!") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, true) - - causeParseError(workspacePath / "build.mill") - evalCheckErr(tester, "\n1 tasks failed", "\ngenerateScriptSources build.mill") - checkWatchedFiles(tester, Nil, buildPaths(tester), Nil, Nil) - // When one of the meta-builds still has parse errors, all classloaders - // remain null, because none of the meta-builds can evaluate. Only once - // all of them parse successfully do we get a new set of classloaders for - // every level of the meta-build - checkChangedClassloaders(tester, null, null, null, null) - - fixParseError(workspacePath / "build.mill") - causeParseError(workspacePath / "mill-build/build.mill") - evalCheckErr(tester, "\n1 tasks failed", "\ngenerateScriptSources mill-build/build.mill") - checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), Nil) - checkChangedClassloaders(tester, null, null, null, null) - - fixParseError(workspacePath / "mill-build/build.mill") - causeParseError(workspacePath / "mill-build/mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - "\ngenerateScriptSources mill-build/mill-build/build.mill" - ) - checkWatchedFiles(tester, Nil, Nil, Nil, buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, null, null) - - fixParseError(workspacePath / "mill-build/mill-build/build.mill") - causeParseError(workspacePath / "mill-build/build.mill") - evalCheckErr(tester, "\n1 tasks failed", "\ngenerateScriptSources mill-build/build.mill") - checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), Nil) - checkChangedClassloaders(tester, null, null, null, null) - - fixParseError(workspacePath / "mill-build/build.mill") - causeParseError(workspacePath / "build.mill") - evalCheckErr(tester, "\n1 tasks failed", "\ngenerateScriptSources build.mill") - checkWatchedFiles(tester, Nil, buildPaths(tester), Nil, Nil) - checkChangedClassloaders(tester, null, null, null, null) - - fixParseError(workspacePath / "build.mill") - runAssertSuccess(tester, "

hello

world

0.13.1

!") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, true) + test("parseErrorEdits") - retry(retryCount) { + integrationTest { tester => + import tester._ + def causeParseError(p: os.Path) = + modifyFile(p, _.replace("extends", "extendx")) + + def fixParseError(p: os.Path) = + modifyFile(p, _.replace("extendx", "extends")) + + runAssertSuccess(tester, "

hello

world

0.13.1

!") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, true) + + causeParseError(workspacePath / "build.mill") + evalCheckErr(tester, "\n1 tasks failed", "\ngenerateScriptSources build.mill") + checkWatchedFiles(tester, Nil, buildPaths(tester), Nil, Nil) + // When one of the meta-builds still has parse errors, all classloaders + // remain null, because none of the meta-builds can evaluate. Only once + // all of them parse successfully do we get a new set of classloaders for + // every level of the meta-build + checkChangedClassloaders(tester, null, null, null, null) + + fixParseError(workspacePath / "build.mill") + causeParseError(workspacePath / "mill-build/build.mill") + evalCheckErr(tester, "\n1 tasks failed", "\ngenerateScriptSources mill-build/build.mill") + checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), Nil) + checkChangedClassloaders(tester, null, null, null, null) + + fixParseError(workspacePath / "mill-build/build.mill") + causeParseError(workspacePath / "mill-build/mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + "\ngenerateScriptSources mill-build/mill-build/build.mill" + ) + checkWatchedFiles(tester, Nil, Nil, Nil, buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, null, null) + + fixParseError(workspacePath / "mill-build/mill-build/build.mill") + causeParseError(workspacePath / "mill-build/build.mill") + evalCheckErr(tester, "\n1 tasks failed", "\ngenerateScriptSources mill-build/build.mill") + checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), Nil) + checkChangedClassloaders(tester, null, null, null, null) + + fixParseError(workspacePath / "mill-build/build.mill") + causeParseError(workspacePath / "build.mill") + evalCheckErr(tester, "\n1 tasks failed", "\ngenerateScriptSources build.mill") + checkWatchedFiles(tester, Nil, buildPaths(tester), Nil, Nil) + checkChangedClassloaders(tester, null, null, null, null) + + fixParseError(workspacePath / "build.mill") + runAssertSuccess(tester, "

hello

world

0.13.1

!") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, true) + } } - test("compileErrorEdits") - integrationTest { tester => - import tester._ - def causeCompileError(p: os.Path) = - modifyFile(p, _ + "\nimport doesnt.exist") - - def fixCompileError(p: os.Path) = - modifyFile(p, _.replace("import doesnt.exist", "")) - - runAssertSuccess(tester, "

hello

world

0.13.1

!") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, true) - - causeCompileError(workspacePath / "build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - // Ensure the file path in the compile error is properly adjusted to point - // at the original source file and not the generated file - (workspacePath / "build.mill").toString, - "not found: value doesnt" - ) - checkWatchedFiles(tester, Nil, buildPaths(tester), buildPaths2(tester), buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, false, false) - - causeCompileError(workspacePath / "mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - (workspacePath / "mill-build/build.mill").toString, - "not found: object doesnt" - ) - checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, null, false) - - causeCompileError(workspacePath / "mill-build/mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - (workspacePath / "mill-build/mill-build/build.mill").toString, - "not found: object doesnt" - ) - checkWatchedFiles(tester, Nil, Nil, Nil, buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, null, null) - - fixCompileError(workspacePath / "mill-build/mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - (workspacePath / "mill-build/build.mill").toString, - "not found: object doesnt" - ) - checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, null, true) - - fixCompileError(workspacePath / "mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - (workspacePath / "build.mill").toString, - "not found: value doesnt" - ) - checkWatchedFiles(tester, Nil, buildPaths(tester), buildPaths2(tester), buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, true, false) - - fixCompileError(workspacePath / "build.mill") - runAssertSuccess(tester, "

hello

world

0.13.1

!") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, false, false) + test("compileErrorEdits") - retry(retryCount) { + integrationTest { tester => + import tester._ + def causeCompileError(p: os.Path) = + modifyFile(p, _ + "\nimport doesnt.exist") + + def fixCompileError(p: os.Path) = + modifyFile(p, _.replace("import doesnt.exist", "")) + + runAssertSuccess(tester, "

hello

world

0.13.1

!") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, true) + + causeCompileError(workspacePath / "build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + // Ensure the file path in the compile error is properly adjusted to point + // at the original source file and not the generated file + (workspacePath / "build.mill").toString, + "not found: value doesnt" + ) + checkWatchedFiles(tester, Nil, buildPaths(tester), buildPaths2(tester), buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, false, false) + + causeCompileError(workspacePath / "mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + (workspacePath / "mill-build/build.mill").toString, + "not found: object doesnt" + ) + checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, null, false) + + causeCompileError(workspacePath / "mill-build/mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + (workspacePath / "mill-build/mill-build/build.mill").toString, + "not found: object doesnt" + ) + checkWatchedFiles(tester, Nil, Nil, Nil, buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, null, null) + + fixCompileError(workspacePath / "mill-build/mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + (workspacePath / "mill-build/build.mill").toString, + "not found: object doesnt" + ) + checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, null, true) + + fixCompileError(workspacePath / "mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + (workspacePath / "build.mill").toString, + "not found: value doesnt" + ) + checkWatchedFiles(tester, Nil, buildPaths(tester), buildPaths2(tester), buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, true, false) + + fixCompileError(workspacePath / "build.mill") + runAssertSuccess(tester, "

hello

world

0.13.1

!") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, false, false) + } } - test("runtimeErrorEdits") - integrationTest { tester => - import tester._ - val runErrorSnippet = - """{ - |override def runClasspath = Task { - | throw new Exception("boom") - | super.runClasspath() - |}""".stripMargin - - def causeRuntimeError(p: os.Path) = - modifyFile(p, _.replaceFirst("\\{", runErrorSnippet)) - - def fixRuntimeError(p: os.Path) = - modifyFile(p, _.replaceFirst(Regex.quote(runErrorSnippet), "\\{")) - - runAssertSuccess(tester, "

hello

world

0.13.1

!") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, true) - - causeRuntimeError(workspacePath / "build.mill") - evalCheckErr(tester, "\n1 tasks failed", "foo.runClasspath java.lang.Exception: boom") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, false, false) - - causeRuntimeError(workspacePath / "mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - "build.mill", - "runClasspath java.lang.Exception: boom" - ) - checkWatchedFiles(tester, Nil, buildPaths(tester), buildPaths2(tester), buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, true, false) - - causeRuntimeError(workspacePath / "mill-build/mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - "build.mill", - "runClasspath java.lang.Exception: boom" - ) - checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, null, true) - - fixRuntimeError(workspacePath / "mill-build/mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - "build.mill", - "runClasspath java.lang.Exception: boom" - ) - checkWatchedFiles(tester, Nil, buildPaths(tester), buildPaths2(tester), buildPaths3(tester)) - checkChangedClassloaders(tester, null, null, true, true) - - fixRuntimeError(workspacePath / "mill-build/build.mill") - evalCheckErr( - tester, - "\n1 tasks failed", - "build.mill", - "foo.runClasspath java.lang.Exception: boom" - ) - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, true, false) - - fixRuntimeError(workspacePath / "build.mill") - runAssertSuccess(tester, "

hello

world

0.13.1

!") - checkWatchedFiles( - tester, - fooPaths(tester), - buildPaths(tester), - buildPaths2(tester), - buildPaths3(tester) - ) - checkChangedClassloaders(tester, null, true, false, false) + test("runtimeErrorEdits") - retry(retryCount) { + integrationTest { tester => + import tester._ + val runErrorSnippet = + """{ + |override def runClasspath = Task { + | throw new Exception("boom") + | super.runClasspath() + |}""".stripMargin + + def causeRuntimeError(p: os.Path) = + modifyFile(p, _.replaceFirst("\\{", runErrorSnippet)) + + def fixRuntimeError(p: os.Path) = + modifyFile(p, _.replaceFirst(Regex.quote(runErrorSnippet), "\\{")) + + runAssertSuccess(tester, "

hello

world

0.13.1

!") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, true) + + causeRuntimeError(workspacePath / "build.mill") + evalCheckErr(tester, "\n1 tasks failed", "foo.runClasspath java.lang.Exception: boom") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, false, false) + + causeRuntimeError(workspacePath / "mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + "build.mill", + "runClasspath java.lang.Exception: boom" + ) + checkWatchedFiles(tester, Nil, buildPaths(tester), buildPaths2(tester), buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, true, false) + + causeRuntimeError(workspacePath / "mill-build/mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + "build.mill", + "runClasspath java.lang.Exception: boom" + ) + checkWatchedFiles(tester, Nil, Nil, buildPaths2(tester), buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, null, true) + + fixRuntimeError(workspacePath / "mill-build/mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + "build.mill", + "runClasspath java.lang.Exception: boom" + ) + checkWatchedFiles(tester, Nil, buildPaths(tester), buildPaths2(tester), buildPaths3(tester)) + checkChangedClassloaders(tester, null, null, true, true) + + fixRuntimeError(workspacePath / "mill-build/build.mill") + evalCheckErr( + tester, + "\n1 tasks failed", + "build.mill", + "foo.runClasspath java.lang.Exception: boom" + ) + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, true, false) + + fixRuntimeError(workspacePath / "build.mill") + runAssertSuccess(tester, "

hello

world

0.13.1

!") + checkWatchedFiles( + tester, + fooPaths(tester), + buildPaths(tester), + buildPaths2(tester), + buildPaths3(tester) + ) + checkChangedClassloaders(tester, null, true, false, false) + } } } } diff --git a/integration/invalidation/selective-execution/resources/bar/bar.txt b/integration/invalidation/selective-execution/resources/bar/bar.txt new file mode 100644 index 00000000000..e5f279fe159 --- /dev/null +++ b/integration/invalidation/selective-execution/resources/bar/bar.txt @@ -0,0 +1 @@ +Qux Contents \ No newline at end of file diff --git a/integration/invalidation/selective-execution/resources/build.mill b/integration/invalidation/selective-execution/resources/build.mill new file mode 100644 index 00000000000..b304caea468 --- /dev/null +++ b/integration/invalidation/selective-execution/resources/build.mill @@ -0,0 +1,26 @@ +import mill._ + +object foo extends Module { + def fooTask = Task.Source(millSourcePath / "foo.txt") + + def fooHelper(p: os.Path) = { + "fooHelper " + os.read(p) + } + + def fooCommand() = Task.Command { + System.out.println("Computing fooCommand") + fooHelper(fooTask().path) + } +} +object bar extends Module { + def barTask = Task.Source(millSourcePath / "bar.txt") + + def barHelper(p: os.Path) = { + "barHelper " + os.read(p) + } + + def barCommand() = Task.Command { + System.out.println("Computing barCommand") + barHelper(barTask().path) + } +} \ No newline at end of file diff --git a/integration/invalidation/selective-execution/resources/foo/foo.txt b/integration/invalidation/selective-execution/resources/foo/foo.txt new file mode 100644 index 00000000000..631a2f92aea --- /dev/null +++ b/integration/invalidation/selective-execution/resources/foo/foo.txt @@ -0,0 +1 @@ +Foo Contents \ No newline at end of file diff --git a/integration/invalidation/selective-execution/src/SelectiveExecutionTests.scala b/integration/invalidation/selective-execution/src/SelectiveExecutionTests.scala new file mode 100644 index 00000000000..7b3cc808c51 --- /dev/null +++ b/integration/invalidation/selective-execution/src/SelectiveExecutionTests.scala @@ -0,0 +1,180 @@ +package mill.integration +import mill.testkit.UtestIntegrationTestSuite +import scala.concurrent.Future +import scala.concurrent.ExecutionContext.Implicits.global +import scala.concurrent.duration._ + +import utest._ +import utest.asserts.{RetryMax, RetryInterval} + +object SelectiveExecutionTests extends UtestIntegrationTestSuite { + implicit val retryMax: RetryMax = RetryMax(120.seconds) + implicit val retryInterval: RetryInterval = RetryInterval(1.seconds) + val tests: Tests = Tests { + test("changed-inputs") - integrationTest { tester => + import tester._ + + eval(("selective.prepare", "{foo.fooCommand,bar.barCommand}"), check = true) + modifyFile(workspacePath / "bar/bar.txt", _ + "!") + + val plan = eval(("selective.resolve", "{foo.fooCommand,bar.barCommand}"), check = true) + assert(plan.out == "bar.barCommand") + + val cached = + eval( + ("selective.run", "{foo.fooCommand,bar.barCommand}"), + check = true, + stderr = os.Inherit + ) + + assert(!cached.out.contains("Computing fooCommand")) + assert(cached.out.contains("Computing barCommand")) + } + test("changed-code") - integrationTest { tester => + import tester._ + + // Check method body code changes correctly trigger downstream evaluation + eval( + ("selective.prepare", "{foo.fooCommand,bar.barCommand}"), + check = true, + stderr = os.Inherit + ) + modifyFile(workspacePath / "build.mill", _.replace("\"barHelper \"", "\"barHelper! \"")) + val cached1 = + eval( + ("selective.run", "{foo.fooCommand,bar.barCommand}"), + check = true, + stderr = os.Inherit + ) + + assert(!cached1.out.contains("Computing fooCommand")) + assert(cached1.out.contains("Computing barCommand")) + + // Check module body code changes correctly trigger downstream evaluation + eval( + ("selective.prepare", "{foo.fooCommand,bar.barCommand}"), + check = true, + stderr = os.Inherit + ) + modifyFile( + workspacePath / "build.mill", + _.replace("object foo extends Module {", "object foo extends Module { println(123)") + ) + val cached2 = + eval( + ("selective.run", "{foo.fooCommand,bar.barCommand}"), + check = true, + stderr = os.Inherit + ) + + assert(cached2.out.contains("Computing fooCommand")) + assert(!cached2.out.contains("Computing barCommand")) + } + + test("watch") { + test("changed-inputs") - integrationTest { tester => + import tester._ + @volatile var output0 = List.empty[String] + def output = output0.mkString("\n") + Future { + eval( + ("--watch", "{foo.fooCommand,bar.barCommand}"), + check = true, + stdout = os.ProcessOutput.Readlines(line => output0 = output0 :+ line), + stderr = os.Inherit + ) + } + + eventually( + output.contains("Computing fooCommand") && output.contains("Computing barCommand") + ) + output0 = Nil + modifyFile(workspacePath / "bar/bar.txt", _ + "!") + eventually( + !output.contains("Computing fooCommand") && output.contains("Computing barCommand") + ) + eventually( + !output.contains("Computing fooCommand") && output.contains("Computing barCommand") + ) + } + test("show-changed-inputs") - integrationTest { tester => + import tester._ + @volatile var output0 = List.empty[String] + def output = output0.mkString("\n") + Future { + eval( + ("--watch", "show", "{foo.fooCommand,bar.barCommand}"), + check = true, + stderr = os.ProcessOutput.Readlines(line => output0 = output0 :+ line) + ) + } + + eventually( + output.contains("Computing fooCommand") && output.contains("Computing barCommand") + ) + output0 = Nil + modifyFile(workspacePath / "bar/bar.txt", _ + "!") + // For now, selective execution doesn't work with `show`, and always runs all provided + // tasks. This is necessary because we need all specified tasks to be run in order to + // get their value to render as JSON at the end of `show` + eventually( + output.contains("Computing fooCommand") && output.contains("Computing barCommand") + ) + eventually( + output.contains("Computing fooCommand") && output.contains("Computing barCommand") + ) + } + + test("changed-code") - integrationTest { tester => + import tester._ + + @volatile var output0 = List.empty[String] + def output = output0.mkString("\n") + Future { + eval( + ("--watch", "{foo.fooCommand,bar.barCommand}"), + check = true, + stdout = os.ProcessOutput.Readlines(line => output0 = output0 :+ line), + stderr = os.Inherit + ) + } + + eventually( + output.contains("Computing fooCommand") && output.contains("Computing barCommand") + ) + output0 = Nil + + // Check method body code changes correctly trigger downstream evaluation + modifyFile(workspacePath / "build.mill", _.replace("\"barHelper \"", "\"barHelper! \"")) + + eventually( + !output.contains("Computing fooCommand") && output.contains("Computing barCommand") + ) + output0 = Nil + + // Check module body code changes correctly trigger downstream evaluation + modifyFile( + workspacePath / "build.mill", + _.replace("object foo extends Module {", "object foo extends Module { println(123)") + ) + + eventually( + output.contains("Computing fooCommand") && !output.contains("Computing barCommand") + ) + } + } + test("failures") { + test("missing-prepare") - integrationTest { tester => + import tester._ + + val cached = eval( + ("selective.run", "{foo.fooCommand,bar.barCommand}"), + check = false, + stderr = os.Pipe + ) + + assert(cached.err.contains("`selective.run` can only be run after `selective.prepare`")) + } + } + } +} diff --git a/integration/invalidation/watch-source-input/src/WatchSourceInputTests.scala b/integration/invalidation/watch-source-input/src/WatchSourceInputTests.scala index 732d8b4efad..60076bc07e4 100644 --- a/integration/invalidation/watch-source-input/src/WatchSourceInputTests.scala +++ b/integration/invalidation/watch-source-input/src/WatchSourceInputTests.scala @@ -22,7 +22,7 @@ import scala.concurrent.ExecutionContext.Implicits.global */ object WatchSourceInputTests extends UtestIntegrationTestSuite { - val maxDuration = 60000 + val maxDuration = 120000 val tests: Tests = Tests { def awaitCompletionMarker(tester: IntegrationTester, name: String) = { val maxTime = System.currentTimeMillis() + maxDuration diff --git a/main/client/src/mill/main/client/OutFiles.java b/main/client/src/mill/main/client/OutFiles.java index e30f06b392f..51be8714b36 100644 --- a/main/client/src/mill/main/client/OutFiles.java +++ b/main/client/src/mill/main/client/OutFiles.java @@ -66,4 +66,11 @@ public class OutFiles { * Any active Mill command that is currently run, for debugging purposes */ public static final String millActiveCommand = "mill-active-command"; + + /** + * File used to store metadata related to selective execution, mostly + * input hashes and method code signatures necessary to determine what + * root tasks changed so Mill can decide which tasks to execute. + */ + public static final String millSelectiveExecution = "mill-selective-execution.json"; } diff --git a/main/eval/src/mill/eval/CodeSigUtils.scala b/main/eval/src/mill/eval/CodeSigUtils.scala new file mode 100644 index 00000000000..72b598dd808 --- /dev/null +++ b/main/eval/src/mill/eval/CodeSigUtils.scala @@ -0,0 +1,117 @@ +package mill.eval + +import mill.api.{BuildInfo, MillException} +import mill.define.{NamedTask, Task} +import mill.util.MultiBiMap + +import scala.reflect.NameTransformer.encode +import java.lang.reflect.Method + +private[mill] object CodeSigUtils { + def precomputeMethodNamesPerClass(sortedGroups: MultiBiMap[Terminal, Task[_]]) + : (Map[Class[_], IndexedSeq[Class[_]]], Map[Class[_], Map[String, Method]]) = { + def resolveTransitiveParents(c: Class[_]): Iterator[Class[_]] = { + Iterator(c) ++ + Option(c.getSuperclass).iterator.flatMap(resolveTransitiveParents) ++ + c.getInterfaces.iterator.flatMap(resolveTransitiveParents) + } + + val classToTransitiveClasses: Map[Class[?], IndexedSeq[Class[?]]] = sortedGroups + .values() + .flatten + .collect { case namedTask: NamedTask[?] => namedTask.ctx.enclosingCls } + .map(cls => cls -> resolveTransitiveParents(cls).toVector) + .toMap + + val allTransitiveClasses = classToTransitiveClasses + .iterator + .flatMap(_._2) + .toSet + + val allTransitiveClassMethods: Map[Class[?], Map[String, java.lang.reflect.Method]] = + allTransitiveClasses + .map { cls => + val cMangledName = cls.getName.replace('.', '$') + cls -> cls.getDeclaredMethods + .flatMap { m => + Seq( + m.getName -> m, + // Handle scenarios where private method names get mangled when they are + // not really JVM-private due to being accessed by Scala nested objects + // or classes https://github.com/scala/bug/issues/9306 + m.getName.stripPrefix(cMangledName + "$$") -> m, + m.getName.stripPrefix(cMangledName + "$") -> m + ) + }.toMap + } + .toMap + + (classToTransitiveClasses, allTransitiveClassMethods) + } + + def constructorHashSignatures(methodCodeHashSignatures: Map[String, Int]) + : Map[String, Seq[(String, Int)]] = + methodCodeHashSignatures + .toSeq + .collect { case (method @ s"$prefix#($args)void", hash) => (prefix, method, hash) } + .groupMap(_._1)(t => (t._2, t._3)) + + def codeSigForTask( + namedTask: => NamedTask[_], + classToTransitiveClasses: => Map[Class[?], IndexedSeq[Class[?]]], + allTransitiveClassMethods: => Map[Class[?], Map[String, java.lang.reflect.Method]], + methodCodeHashSignatures: => Map[String, Int], + constructorHashSignatures: => Map[String, Seq[(String, Int)]] + ): Iterable[Int] = { + + val encodedTaskName = encode(namedTask.ctx.segment.pathSegments.head) + + val methodOpt = for { + parentCls <- classToTransitiveClasses(namedTask.ctx.enclosingCls).iterator + m <- allTransitiveClassMethods(parentCls).get(encodedTaskName) + } yield m + + val methodClass = methodOpt + .nextOption() + .getOrElse(throw new MillException( + s"Could not detect the parent class of target ${namedTask}. " + + s"Please report this at ${BuildInfo.millReportNewIssueUrl} . " + )) + .getDeclaringClass.getName + + val name = namedTask.ctx.segment.pathSegments.last + val expectedName = methodClass + "#" + name + "()mill.define.Target" + val expectedName2 = methodClass + "#" + name + "()mill.define.Command" + + // We not only need to look up the code hash of the Target method being called, + // but also the code hash of the constructors required to instantiate the Module + // that the Target is being called on. This can be done by walking up the nested + // modules and looking at their constructors (they're `object`s and should each + // have only one) + val allEnclosingModules = Vector.unfold(namedTask.ctx) { + case null => None + case ctx => + ctx.enclosingModule match { + case null => None + case m: mill.define.Module => Some((m, m.millOuterCtx)) + case unknown => + throw new MillException(s"Unknown ctx of target ${namedTask}: $unknown") + } + } + + val constructorHashes = allEnclosingModules + .map(m => + constructorHashSignatures.get(m.getClass.getName) match { + case Some(Seq((singleMethod, hash))) => hash + case Some(multiple) => throw new MillException( + s"Multiple constructors found for module $m: ${multiple.mkString(",")}" + ) + case None => 0 + } + ) + + methodCodeHashSignatures.get(expectedName) ++ + methodCodeHashSignatures.get(expectedName2) ++ + constructorHashes + } +} diff --git a/main/eval/src/mill/eval/Evaluator.scala b/main/eval/src/mill/eval/Evaluator.scala index 912a25e6edc..6eae8393495 100644 --- a/main/eval/src/mill/eval/Evaluator.scala +++ b/main/eval/src/mill/eval/Evaluator.scala @@ -21,6 +21,7 @@ trait Evaluator extends AutoCloseable { def outPath: os.Path def externalOutPath: os.Path def pathsResolver: EvaluatorPathsResolver + def methodCodeHashSignatures: Map[String, Int] = Map.empty // TODO In 0.13.0, workerCache should have the type of mutableWorkerCache, // while the latter should be removed def workerCache: collection.Map[Segments, (Int, Val)] diff --git a/main/eval/src/mill/eval/EvaluatorCore.scala b/main/eval/src/mill/eval/EvaluatorCore.scala index 12fb433e097..4bdc2f48fbd 100644 --- a/main/eval/src/mill/eval/EvaluatorCore.scala +++ b/main/eval/src/mill/eval/EvaluatorCore.scala @@ -83,7 +83,7 @@ private[mill] trait EvaluatorCore extends GroupEvaluator { // and the class hierarchy, so during evaluation it is cheap to look up what class // each target belongs to determine of the enclosing class code signature changed. val (classToTransitiveClasses, allTransitiveClassMethods) = - precomputeMethodNamesPerClass(sortedGroups) + CodeSigUtils.precomputeMethodNamesPerClass(sortedGroups) def evaluateTerminals( terminals: Seq[Terminal], @@ -268,46 +268,6 @@ private[mill] trait EvaluatorCore extends GroupEvaluator { ) } - private def precomputeMethodNamesPerClass(sortedGroups: MultiBiMap[Terminal, Task[_]]) = { - def resolveTransitiveParents(c: Class[_]): Iterator[Class[_]] = { - Iterator(c) ++ - Option(c.getSuperclass).iterator.flatMap(resolveTransitiveParents) ++ - c.getInterfaces.iterator.flatMap(resolveTransitiveParents) - } - - val classToTransitiveClasses: Map[Class[?], IndexedSeq[Class[?]]] = sortedGroups - .values() - .flatten - .collect { case namedTask: NamedTask[?] => namedTask.ctx.enclosingCls } - .map(cls => cls -> resolveTransitiveParents(cls).toVector) - .toMap - - val allTransitiveClasses = classToTransitiveClasses - .iterator - .flatMap(_._2) - .toSet - - val allTransitiveClassMethods: Map[Class[?], Map[String, java.lang.reflect.Method]] = - allTransitiveClasses - .map { cls => - val cMangledName = cls.getName.replace('.', '$') - cls -> cls.getDeclaredMethods - .flatMap { m => - Seq( - m.getName -> m, - // Handle scenarios where private method names get mangled when they are - // not really JVM-private due to being accessed by Scala nested objects - // or classes https://github.com/scala/bug/issues/9306 - m.getName.stripPrefix(cMangledName + "$$") -> m, - m.getName.stripPrefix(cMangledName + "$") -> m - ) - }.toMap - } - .toMap - - (classToTransitiveClasses, allTransitiveClassMethods) - } - private def findInterGroupDeps(sortedGroups: MultiBiMap[Terminal, Task[_]]) : Map[Terminal, Seq[Terminal]] = { sortedGroups diff --git a/main/eval/src/mill/eval/EvaluatorImpl.scala b/main/eval/src/mill/eval/EvaluatorImpl.scala index 7045a05f98c..507497cc135 100644 --- a/main/eval/src/mill/eval/EvaluatorImpl.scala +++ b/main/eval/src/mill/eval/EvaluatorImpl.scala @@ -28,7 +28,7 @@ private[mill] case class EvaluatorImpl( failFast: Boolean, threadCount: Option[Int], scriptImportGraph: Map[os.Path, (Int, Seq[os.Path])], - methodCodeHashSignatures: Map[String, Int], + override val methodCodeHashSignatures: Map[String, Int], override val disableCallgraph: Boolean, override val allowPositionalCommandArgs: Boolean, val systemExit: Int => Nothing, diff --git a/main/eval/src/mill/eval/GroupEvaluator.scala b/main/eval/src/mill/eval/GroupEvaluator.scala index 4f62dcd4c1e..7ef94d96e7b 100644 --- a/main/eval/src/mill/eval/GroupEvaluator.scala +++ b/main/eval/src/mill/eval/GroupEvaluator.scala @@ -9,7 +9,6 @@ import mill.util._ import java.lang.reflect.Method import scala.collection.mutable -import scala.reflect.NameTransformer.encode import scala.util.control.NonFatal import scala.util.hashing.MurmurHash3 @@ -35,10 +34,8 @@ private[mill] trait GroupEvaluator { def systemExit: Int => Nothing def exclusiveSystemStreams: SystemStreams - lazy val constructorHashSignatures: Map[String, Seq[(String, Int)]] = methodCodeHashSignatures - .toSeq - .collect { case (method @ s"$prefix#($args)void", hash) => (prefix, method, hash) } - .groupMap(_._1)(t => (t._2, t._3)) + lazy val constructorHashSignatures: Map[String, Seq[(String, Int)]] = + CodeSigUtils.constructorHashSignatures(methodCodeHashSignatures) val effectiveThreadCount: Int = this.threadCount.getOrElse(Runtime.getRuntime().availableProcessors()) @@ -76,53 +73,14 @@ private[mill] trait GroupEvaluator { if (disableCallgraph) 0 else group .iterator - .collect { - case namedTask: NamedTask[_] => - val encodedTaskName = encode(namedTask.ctx.segment.pathSegments.head) - val methodOpt = for { - parentCls <- classToTransitiveClasses(namedTask.ctx.enclosingCls).iterator - m <- allTransitiveClassMethods(parentCls).get(encodedTaskName) - } yield m - - val methodClass = methodOpt - .nextOption() - .getOrElse(throw new MillException( - s"Could not detect the parent class of target ${namedTask}. " + - s"Please report this at ${BuildInfo.millReportNewIssueUrl} . " - )) - .getDeclaringClass.getName - - val name = namedTask.ctx.segment.pathSegments.last - val expectedName = methodClass + "#" + name + "()mill.define.Target" - - // We not only need to look up the code hash of the Target method being called, - // but also the code hash of the constructors required to instantiate the Module - // that the Target is being called on. This can be done by walking up the nested - // modules and looking at their constructors (they're `object`s and should each - // have only one) - val allEnclosingModules = Vector.unfold(namedTask.ctx) { - case null => None - case ctx => - ctx.enclosingModule match { - case null => None - case m: mill.define.Module => Some((m, m.millOuterCtx)) - case unknown => - throw new MillException(s"Unknown ctx of target ${namedTask}: $unknown") - } - } - - val constructorHashes = allEnclosingModules - .map(m => - constructorHashSignatures.get(m.getClass.getName) match { - case Some(Seq((singleMethod, hash))) => hash - case Some(multiple) => throw new MillException( - s"Multiple constructors found for module $m: ${multiple.mkString(",")}" - ) - case None => 0 - } - ) - - methodCodeHashSignatures.get(expectedName) ++ constructorHashes + .collect { case namedTask: NamedTask[_] => + CodeSigUtils.codeSigForTask( + namedTask, + classToTransitiveClasses, + allTransitiveClassMethods, + methodCodeHashSignatures, + constructorHashSignatures + ) } .flatten .sum diff --git a/main/src/mill/main/MainModule.scala b/main/src/mill/main/MainModule.scala index 34c1ec3e15a..0cf0f0b2507 100644 --- a/main/src/mill/main/MainModule.scala +++ b/main/src/mill/main/MainModule.scala @@ -1,8 +1,8 @@ package mill.main -import mill.api.{Ctx, _} -import mill.define.{BaseModule0, Command, NamedTask, Segments, Target, Task, _} -import mill.eval.{Evaluator, EvaluatorPaths, Terminal} +import mill.api._ +import mill.define._ +import mill.eval.{Evaluator, EvaluatorPaths} import mill.moduledefs.Scaladoc import mill.resolve.SelectMode.Separated import mill.resolve.{Resolve, SelectMode} @@ -120,7 +120,7 @@ trait MainModule extends BaseModule0 { */ def plan(evaluator: Evaluator, targets: String*): Command[Array[String]] = Task.Command(exclusive = true) { - plan0(evaluator, targets) match { + SelectiveExecution.plan0(evaluator, targets) match { case Left(err) => Result.Failure(err) case Right(success) => val renderedTasks = success.map(_.segments.render) @@ -129,19 +129,6 @@ trait MainModule extends BaseModule0 { } } - private def plan0(evaluator: Evaluator, targets: Seq[String]) = { - Resolve.Tasks.resolve( - evaluator.rootModule, - targets, - SelectMode.Multi - ) match { - case Left(err) => Left(err) - case Right(rs) => - val (sortedGroups, _) = evaluator.plan(rs) - Right(sortedGroups.keys().collect { case r: Terminal.Labelled[_] => r }.toArray) - } - } - /** * Prints out some dependency path from the `src` task to the `dest` task. * @@ -528,7 +515,7 @@ trait MainModule extends BaseModule0 { */ def visualizePlan(evaluator: Evaluator, targets: String*): Command[Seq[PathRef]] = Task.Command(exclusive = true) { - plan0(evaluator, targets) match { + SelectiveExecution.plan0(evaluator, targets) match { case Left(err) => Result.Failure(err) case Right(planResults) => visualize0( evaluator, @@ -596,7 +583,7 @@ trait MainModule extends BaseModule0 { private def visualize0( evaluator: Evaluator, targets: Seq[String], - ctx: Ctx, + ctx: mill.api.Ctx, vizWorker: VizWorker, planTasks: Option[List[NamedTask[_]]] = None ): Result[Seq[PathRef]] = { @@ -625,4 +612,11 @@ trait MainModule extends BaseModule0 { } } } + + /** + * Commands related to selective execution, where Mill runs tasks selectively + * depending on what task inputs or implementations changed + */ + lazy val selective: SelectiveExecutionModule = new SelectiveExecutionModule {} + } diff --git a/main/src/mill/main/RunScript.scala b/main/src/mill/main/RunScript.scala index 2478bcf4c8e..24020b7aef2 100644 --- a/main/src/mill/main/RunScript.scala +++ b/main/src/mill/main/RunScript.scala @@ -6,6 +6,7 @@ import mill.util.Watchable import mill.api.{PathRef, Result, Val} import mill.api.Strict.Agg import Evaluator._ +import mill.main.client.OutFiles import mill.resolve.{Resolve, SelectMode} object RunScript { @@ -19,6 +20,20 @@ object RunScript { ): Either[ String, (Seq[Watchable], Either[String, Seq[(Any, Option[(TaskName, ujson.Value)])]]) + ] = evaluateTasksNamed( + evaluator, + scriptArgs, + selectMode, + selectiveExecution = false + ) + def evaluateTasksNamed( + evaluator: Evaluator, + scriptArgs: Seq[String], + selectMode: SelectMode, + selectiveExecution: Boolean = false + ): Either[ + String, + (Seq[Watchable], Either[String, Seq[(Any, Option[(TaskName, ujson.Value)])]]) ] = { val resolved = mill.eval.Evaluator.currentEvaluator.withValue(evaluator) { Resolve.Tasks.resolve( @@ -28,9 +43,15 @@ object RunScript { evaluator.allowPositionalCommandArgs ) } - for (targets <- resolved) yield evaluateNamed(evaluator, Agg.from(targets)) + for (targets <- resolved) yield evaluateNamed(evaluator, Agg.from(targets), selectiveExecution) } + def evaluateNamed( + evaluator: Evaluator, + targets: Agg[NamedTask[Any]] + ): (Seq[Watchable], Either[String, Seq[(Any, Option[(TaskName, ujson.Value)])]]) = + evaluateNamed(evaluator, targets, selectiveExecution = false) + /** * @param evaluator * @param targets @@ -38,42 +59,72 @@ object RunScript { */ def evaluateNamed( evaluator: Evaluator, - targets: Agg[Task[Any]] + targets: Agg[NamedTask[Any]], + selectiveExecution: Boolean = false ): (Seq[Watchable], Either[String, Seq[(Any, Option[(TaskName, ujson.Value)])]]) = { - val evaluated: Results = evaluator.evaluate(targets, serialCommandExec = true) - val watched = evaluated.results - .iterator - .collect { - case (t: SourcesImpl, TaskResult(Result.Success(Val(ps: Seq[PathRef])), _)) => - ps.map(Watchable.Path(_)) - case (t: SourceImpl, TaskResult(Result.Success(Val(p: PathRef)), _)) => - Seq(Watchable.Path(p)) - case (t: InputImpl[_], TaskResult(result, recalc)) => - val pretty = t.ctx0.fileName + ":" + t.ctx0.lineNum - Seq(Watchable.Value(() => recalc().hashCode(), result.hashCode(), pretty)) - } - .flatten - .toSeq - - val errorStr = Evaluator.formatFailing(evaluated) + val selectedTargetsOrErr = + if (selectiveExecution && os.exists(evaluator.outPath / OutFiles.millSelectiveExecution)) { + SelectiveExecution + .diffMetadata(evaluator, targets.map(_.ctx.segments.render).toSeq) + .map { selected => + targets.filter { + case c: Command[_] if c.exclusive => true + case t => selected(t.ctx.segments.render) + } + } + } else Right(targets) - evaluated.failing.keyCount match { - case 0 => - val nameAndJson = for (t <- targets.toSeq) yield { - t match { - case t: mill.define.NamedTask[_] => - val jsonFile = EvaluatorPaths.resolveDestPaths(evaluator.outPath, t).meta - val metadata = upickle.default.read[Evaluator.Cached](ujson.read(jsonFile.toIO)) - Some((t.toString, metadata.value)) + selectedTargetsOrErr match { + case Left(err) => (Nil, Left(err)) + case Right(selectedTargets) => + val evaluated: Results = evaluator.evaluate(selectedTargets, serialCommandExec = true) + val watched = evaluated.results + .iterator + .collect { + case (t: SourcesImpl, TaskResult(Result.Success(Val(ps: Seq[PathRef])), _)) => + ps.map(Watchable.Path(_)) + case (t: SourceImpl, TaskResult(Result.Success(Val(p: PathRef)), _)) => + Seq(Watchable.Path(p)) + case (t: InputImpl[_], TaskResult(result, recalc)) => + val pretty = t.ctx0.fileName + ":" + t.ctx0.lineNum + Seq(Watchable.Value(() => recalc().hashCode(), result.hashCode(), pretty)) + } + .flatten + .toSeq - case _ => None + val allInputHashes = evaluated.results + .iterator + .collect { + case (t: InputImpl[_], TaskResult(Result.Success(Val(value)), _)) => + (t.ctx.segments.render, value.##) } + .toMap + + if (selectiveExecution) { + SelectiveExecution.saveMetadata( + evaluator, + SelectiveExecution.Metadata(allInputHashes, evaluator.methodCodeHashSignatures) + ) } - watched -> Right(evaluated.values.zip(nameAndJson)) - case n => watched -> Left(s"$n tasks failed\n$errorStr") + val errorStr = Evaluator.formatFailing(evaluated) + evaluated.failing.keyCount match { + case 0 => + val nameAndJson = for (t <- selectedTargets.toSeq) yield { + t match { + case t: mill.define.NamedTask[_] => + val jsonFile = EvaluatorPaths.resolveDestPaths(evaluator.outPath, t).meta + val metadata = upickle.default.read[Evaluator.Cached](ujson.read(jsonFile.toIO)) + Some((t.toString, metadata.value)) + + case _ => None + } + } + + watched -> Right(evaluated.values.zip(nameAndJson)) + case n => watched -> Left(s"$n tasks failed\n$errorStr") + } } } - } diff --git a/main/src/mill/main/SelectiveExecution.scala b/main/src/mill/main/SelectiveExecution.scala new file mode 100644 index 00000000000..d95d42da1c1 --- /dev/null +++ b/main/src/mill/main/SelectiveExecution.scala @@ -0,0 +1,150 @@ +package mill.main + +import mill.api.Strict +import mill.define.{InputImpl, NamedTask, Task} +import mill.eval.{CodeSigUtils, Evaluator, Plan, Terminal} +import mill.main.client.OutFiles +import mill.resolve.{Resolve, SelectMode} + +private[mill] object SelectiveExecution { + case class Metadata(inputHashes: Map[String, Int], methodCodeHashSignatures: Map[String, Int]) + implicit val rw: upickle.default.ReadWriter[Metadata] = upickle.default.macroRW + + object Metadata { + def apply(evaluator: Evaluator, tasks: Seq[String]): Either[String, Metadata] = { + for (transitive <- plan0(evaluator, tasks)) yield { + val inputTasksToLabels: Map[Task[_], String] = transitive + .collect { case Terminal.Labelled(task: InputImpl[_], segments) => + task -> segments.render + } + .toMap + + val results = evaluator.evaluate(Strict.Agg.from(inputTasksToLabels.keys)) + + new Metadata( + inputHashes = results + .results + .flatMap { case (task, taskResult) => + inputTasksToLabels.get(task).map { l => + l -> taskResult.result.getOrThrow.value.hashCode + } + } + .toMap, + methodCodeHashSignatures = evaluator.methodCodeHashSignatures + ) + } + } + } + + def plan0( + evaluator: Evaluator, + tasks: Seq[String] + ): Either[String, Array[Terminal.Labelled[_]]] = { + Resolve.Tasks.resolve( + evaluator.rootModule, + tasks, + SelectMode.Multi + ) match { + case Left(err) => Left(err) + case Right(rs) => + val (sortedGroups, _) = evaluator.plan(rs) + Right(sortedGroups.keys().collect { case r: Terminal.Labelled[_] => r }.toArray) + } + } + + def computeHashCodeSignatures( + res: Array[Terminal.Labelled[_]], + methodCodeHashSignatures: Map[String, Int] + ): Map[String, Int] = { + + val (sortedGroups, transitive) = Plan.plan(res.map(_.task).toSeq) + + val (classToTransitiveClasses, allTransitiveClassMethods) = + CodeSigUtils.precomputeMethodNamesPerClass(sortedGroups) + + lazy val constructorHashSignatures = CodeSigUtils + .constructorHashSignatures(methodCodeHashSignatures) + + sortedGroups.keys() + .collect { case Terminal.Labelled(namedTask: NamedTask[_], segments) => + segments.render -> CodeSigUtils + .codeSigForTask( + namedTask, + classToTransitiveClasses, + allTransitiveClassMethods, + methodCodeHashSignatures, + constructorHashSignatures + ) + .sum + } + .toMap + } + + def computeDownstream( + evaluator: Evaluator, + tasks: Seq[String], + oldHashes: Metadata, + newHashes: Metadata + ): Seq[Task[Any]] = { + val terminals = SelectiveExecution.plan0(evaluator, tasks).getOrElse(???) + val namesToTasks = terminals.map(t => (t.render -> t.task)).toMap + + def diffMap[K, V](lhs: Map[K, V], rhs: Map[K, V]) = { + (lhs.keys ++ rhs.keys) + .iterator + .distinct + .filter { k => lhs.get(k) != rhs.get(k) } + .toSet + } + + val changedInputNames = diffMap(oldHashes.inputHashes, newHashes.inputHashes) + val changedCodeNames = diffMap( + computeHashCodeSignatures(terminals, oldHashes.methodCodeHashSignatures), + computeHashCodeSignatures(terminals, newHashes.methodCodeHashSignatures) + ) + + val changedRootTasks = (changedInputNames ++ changedCodeNames).map(namesToTasks(_): Task[_]) + + val allNodes = breadthFirst(terminals.map(_.task: Task[_]))(_.inputs) + val downstreamEdgeMap = allNodes + .flatMap(t => t.inputs.map(_ -> t)) + .groupMap(_._1)(_._2) + + breadthFirst(changedRootTasks)(downstreamEdgeMap.getOrElse(_, Nil)) + } + + def breadthFirst[T](start: IterableOnce[T])(edges: T => IterableOnce[T]): Seq[T] = { + val seen = collection.mutable.Set.empty[T] + val seenList = collection.mutable.Buffer.empty[T] + val queued = collection.mutable.Queue.from(start) + + while (queued.nonEmpty) { + val current = queued.dequeue() + seen.add(current) + seenList.append(current) + + for (next <- edges(current).iterator) { + if (!seen.contains(next)) queued.enqueue(next) + } + } + seenList.toSeq + } + + def saveMetadata(evaluator: Evaluator, metadata: SelectiveExecution.Metadata): Unit = { + os.write.over( + evaluator.outPath / OutFiles.millSelectiveExecution, + upickle.default.write(metadata) + ) + } + + def diffMetadata(evaluator: Evaluator, tasks: Seq[String]): Either[String, Set[String]] = { + val oldMetadata = upickle.default.read[SelectiveExecution.Metadata]( + os.read(evaluator.outPath / OutFiles.millSelectiveExecution) + ) + for (newMetadata <- SelectiveExecution.Metadata(evaluator, tasks)) yield { + SelectiveExecution.computeDownstream(evaluator, tasks, oldMetadata, newMetadata) + .collect { case n: NamedTask[_] => n.ctx.segments.render } + .toSet + } + } +} diff --git a/main/src/mill/main/SelectiveExecutionModule.scala b/main/src/mill/main/SelectiveExecutionModule.scala new file mode 100644 index 00000000000..09d336e4bd7 --- /dev/null +++ b/main/src/mill/main/SelectiveExecutionModule.scala @@ -0,0 +1,71 @@ +package mill.main + +import mill.api.Result +import mill.define.{Command, Task} +import mill.eval.Evaluator +import mill.main.client.OutFiles +import mill.resolve.{Resolve, SelectMode} +import mill.resolve.SelectMode.Separated + +trait SelectiveExecutionModule extends mill.define.Module { + + /** + * Run to store a baseline snapshot of the Mill task inputs or implementations + * necessary to run [[tasks]], to be later compared against metadata computed + * after a code change to determine which tasks were affected and need to be re-run + */ + def prepare(evaluator: Evaluator, tasks: String*): Command[Unit] = + Task.Command(exclusive = true) { + val res: Either[String, Unit] = SelectiveExecution.Metadata(evaluator, tasks) + .map(SelectiveExecution.saveMetadata(evaluator, _)) + + res match { + case Left(err) => Result.Failure(err) + case Right(res) => Result.Success(()) + } + } + + /** + * Run after [[prepare]], prints out the tasks in [[tasks]] that are affected by + * any changes to the task inputs or task implementations since [[prepare]] + * was run. Effectively a dry-run version of [[run]] that lets you show the tasks + * that would be run without actually running them + */ + def resolve(evaluator: Evaluator, tasks: String*): Command[Array[String]] = + Task.Command(exclusive = true) { + val result = for { + resolved <- Resolve.Segments.resolve(evaluator.rootModule, tasks, SelectMode.Multi) + diffed <- SelectiveExecution.diffMetadata(evaluator, tasks) + } yield resolved.map(_.render).toSet.intersect(diffed).toArray.sorted + + result match { + case Left(err) => Result.Failure(err) + case Right(success) => + success.foreach(println) + Result.Success(success) + } + } + + /** + * Run after [[prepare]], selectively executes the tasks in [[tasks]] that are + * affected by any changes to the task inputs or task implementations since [[prepare]] + * was run + */ + def run(evaluator: Evaluator, tasks: String*): Command[Unit] = + Task.Command(exclusive = true) { + if (!os.exists(evaluator.outPath / OutFiles.millSelectiveExecution)) { + Result.Failure("`selective.run` can only be run after `selective.prepare`") + } else { + RunScript.evaluateTasksNamed( + evaluator, + tasks, + Separated, + selectiveExecution = true + ) match { + case Left(err) => Result.Failure(err) + case Right((watched, Left(err))) => Result.Failure(err) + case Right((watched, Right(res))) => Result.Success(res) + } + } + } +} diff --git a/runner/src/mill/runner/MillBuildBootstrap.scala b/runner/src/mill/runner/MillBuildBootstrap.scala index dc078a7b8e5..ab4be1dca6e 100644 --- a/runner/src/mill/runner/MillBuildBootstrap.scala +++ b/runner/src/mill/runner/MillBuildBootstrap.scala @@ -44,7 +44,8 @@ class MillBuildBootstrap( requestedMetaLevel: Option[Int], allowPositionalCommandArgs: Boolean, systemExit: Int => Nothing, - streams0: SystemStreams + streams0: SystemStreams, + selectiveExecution: Boolean ) { import MillBuildBootstrap._ @@ -215,7 +216,8 @@ class MillBuildBootstrap( evaluateWithWatches( rootModule, evaluator, - Seq("{runClasspath,compile,methodCodeHashSignatures}") + Seq("{runClasspath,compile,methodCodeHashSignatures}"), + selectiveExecution = false ) match { case (Left(error), evalWatches, moduleWatches) => val evalState = RunnerState.Frame( @@ -299,7 +301,7 @@ class MillBuildBootstrap( val (evaled, evalWatched, moduleWatches) = Evaluator.allBootstrapEvaluators.withValue( Evaluator.AllBootstrapEvaluators(Seq(evaluator) ++ nestedState.frames.flatMap(_.evaluator)) ) { - evaluateWithWatches(rootModule, evaluator, targetsAndParams) + evaluateWithWatches(rootModule, evaluator, targetsAndParams, selectiveExecution) } val evalState = RunnerState.Frame( @@ -392,15 +394,22 @@ object MillBuildBootstrap { def evaluateWithWatches( rootModule: BaseModule, evaluator: Evaluator, - targetsAndParams: Seq[String] + targetsAndParams: Seq[String], + selectiveExecution: Boolean ): (Either[String, Seq[Any]], Seq[Watchable], Seq[Watchable]) = { rootModule.evalWatchedValues.clear() val previousClassloader = Thread.currentThread().getContextClassLoader val evalTaskResult = try { Thread.currentThread().setContextClassLoader(rootModule.getClass.getClassLoader) - RunScript.evaluateTasksNamed(evaluator, targetsAndParams, SelectMode.Separated) + RunScript.evaluateTasksNamed( + evaluator, + targetsAndParams, + SelectMode.Separated, + selectiveExecution + ) } finally Thread.currentThread().setContextClassLoader(previousClassloader) + val moduleWatched = rootModule.watchedValues.toVector val addedEvalWatched = rootModule.evalWatchedValues.toVector diff --git a/runner/src/mill/runner/MillMain.scala b/runner/src/mill/runner/MillMain.scala index 580faa3283a..a501e68c7f6 100644 --- a/runner/src/mill/runner/MillMain.scala +++ b/runner/src/mill/runner/MillMain.scala @@ -33,7 +33,7 @@ object MillMain { err.println(e.getCause.getMessage()) (false, onError) case NonFatal(e) => - err.println("An unexpected error occurred " + e) + err.println("An unexpected error occurred " + e + "\n" + e.getStackTrace.mkString("\n")) throw e (false, onError) } @@ -222,6 +222,12 @@ object MillMain { repeatForBsp = false Using.resource(new TailManager(serverDir)) { tailManager => + if (config.watch.value) { + // When starting a --watch, clear the `mill-selective-execution.json` + // file, so that the first run always selects everything and only + // subsequent re-runs are selective depending on what changed. + os.remove(out / OutFiles.millSelectiveExecution) + } val (isSuccess, evalStateOpt) = Watching.watchLoop( ringBell = config.ringBell.value, watch = config.watch.value, @@ -268,7 +274,8 @@ object MillMain { requestedMetaLevel = config.metaLevel, config.allowPositional.value, systemExit = systemExit, - streams0 = streams0 + streams0 = streams0, + selectiveExecution = config.watch.value ).evaluate() } }