Remove betasty directory on successful compilation backgroundTasks #2410

jchyb · 2024-08-26T11:45:32Z

Fix to scalameta/metals#6628
Usually it was fine to always put betasty files at the end of the classpath, but in the reproduction for some reason despite the code being correct, those files were still being used, which resulted in an incoherent state in bloop (we obtained only betasty files on what was being considered non-best effort compilation). This lead to trying to apply incremental compilation there, producing singular betasty files, and losing information about symbols from other files, producing errors.

The fix is very simple, but so far I have been unable to properly add any tests for it, as the backgroundTasks do not seem to be run in the tests suites.

tgodzik · 2024-08-30T14:38:12Z

Looks like this indeed does help in case of the repository I sent your way, but the Scala compiler is failing with cryptic stuff like:

Could not verify that the method argument is transitively initialized (Hot). It was found to be the original object of type (class SymDenotation) where initialization checking started. Only transitively initialized arguments may be passed to methods (except constructors).
Non initialized field(s): variable myFlags, variable myPrivateWithin, variable myAnnotations, variable myParamss, variable myTargetName. Calling trace:
├── class SymDenotation private[SymDenotations] (	[ SymDenotations.scala:36 ]
│   ^
├── if (Config.checkNoSkolemsInInfo) assertNoSkolems(initInfo)	[ SymDenotations.scala:53 ]
│                                    ^^^^^^^^^^^^^^^^^^^^^^^^^
├── def assertNoSkolems(tp: Type): Unit =	[ SymDenotations.scala:1622 ]
│   ^
├── assert(!hasSkolems(tp), s"assigning type $tp containing skolems to $this")	[ SymDenotations.scala:1624 ]
│   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
├── 	[ Predef.scala:10 ]
│                                                                   ^^^^^^^
└── assert(!hasSkolems(tp), s"assigning type $tp containing skolems to $this")	[ SymDenotations.scala:1624 ]
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

any idea why?

jchyb · 2024-08-30T15:38:29Z

This is still pretty unfinished, however I am almost done with the working version. After just this change the incremental compiler interactions can be incorrect, so I also implemented a thing where we properly restart the compilation after the first unsuccessful one (this was also incorrect before, but it was not really noticable, due to us still having betasty we could use from the previous compilation - I'll try to explain this specific issue and the fix later, after I push the code here). Also one race condition I noticed caused a crash from one of the reports we got this week, so that also should be fixed.

With that said none of these fixes fix the behavior in the compiler unfortunately. The main issue there seems to be with the java source files, which I am pretty sure we do not handle yet in best effort compilation :(. When I tested the best effort compilation in the compiler I ended up running into a compiler crash and errors related to missing java symbols - now I can't be 100% sure whether the crash was also caused by the java files, but the compiler is not always able to recover gracefully from the missing symbols, so that might be it... I am sure there is some roundabout way to work with those even without updating the compiler, however it might take more time (I'll work on it next week).

tgodzik · 2024-08-30T15:44:16Z

When I tested the best effort compilation in the compiler I ended up running into a compiler crash and errors related to missing java symbols - now I can't be 100% sure

Wouldn't that be caused by the problems with compilation? So if the Scala source do not compile and produce classfiles, the Java sources which use them will not be able to compile either. Especially that the compilation mode in the compiler is "mixed". I would say the Java compilation issues are not relevant to the actual problem.

jchyb · 2024-08-30T15:44:39Z

Also, obviously it should always recover after fixing the issues, no idea why it does not do that in the compiler codebase, so I'll probably start investigating with this.

EDIT: I wrote this before I saw the comment above, but maybe it answers the doubt there? The java stuff should not matter for successful compilations, it only produces additional fake errors/other issues when the compilation is failing.

jchyb · 2024-08-30T16:27:46Z

(I pushed it but I still want to recheck it/ clean it up, so I am leaving this as a draft for now)

jchyb · 2024-08-30T16:39:15Z

In the meantime let me quickly explain why we do those recompilations here now:
Let's say we have a Utils module, consisting of:

UtilsA.scala
UtilsB.scala

Lets say the first Utils compilation would give us:
UtilsA.class/tasty... and UtilsB.class/tasty... <- we copy everything to client dir and to readonlyDir
We add an error to UtilsB.scala. Now we have from incremental compilation (without any data on what else to recompile) UtilsB.betasty.
Previously we would first copy everything (including betasty!) from readonlyDir, and copy UtilsB.betasty, giving an effect of complete best effort recompilation, but actually compiling only UtilsB.betasty. This was an issue on my part, that I did not notice. Now suddenly, after these fixes, we no longer have those outdated betasty in readonly, so we need to recompile the whole Utils, which we do in this PR. It's worth noting again that the previous behavior was also incorrect, but it was harder to notice.

jchyb · 2024-09-10T11:00:17Z

backend/src/main/scala/bloop/Compiler.scala

            case t: Throwable =>
              t.printStackTrace()
+              val sw = new StringWriter()
+              t.printStackTrace(new PrintWriter(sw))
+              logger.info(sw.toString())
              val backgroundTasks =
                toBackgroundTasks(backgroundTasksForFailedCompilation.toList)
-              Result.Failed(Nil, Some(t), elapsed, backgroundTasks, None)
+              val failedProblems = findFailedProblems(reporter, None)
+              Result.Failed(failedProblems, Some(t), elapsed, backgroundTasks, None)
          }


Changes here are not strictly related to best effort, but not returning failedProblems in the case of compiler crash caused those annoying leftover errors in the problem tab in the compiler codebase (I can't see why it wouldn't behave the same way even for crashes unrelated to best effort). I also figured that it might be useful to also log the stacktraces of those crashes, although maybe a different log level would be better (debug?).

jchyb · 2024-09-10T11:03:56Z

About the compiler crash itself (as seen while using best effort in the compiler codebase), the fix for that will have to be in the compiler (I already have a preliminary fix working there)

tgodzik

Thanks for working on this!

frontend/src/main/scala/bloop/engine/tasks/CompileTask.scala

backend/src/main/scala/bloop/Compiler.scala

tgodzik · 2024-09-17T13:38:53Z

backend/src/main/scala/bloop/Compiler.scala

-          Task
-            .gatherUnordered(List(firstTask, secondTask))
-            .map(_ => ())
+              .forEach(path => if (Files.exists(path)) Files.delete(path))


Suggested change

.forEach(path => if (Files.exists(path)) Files.delete(path))

.forEach(path => Files.delete(path)

Isn't this checked above? We could probably join the 2 filters and a foreach now

Looks like this is unchanged? Would you mind including the change?

Fixed now, sorry about that

backend/src/main/scala/bloop/Compiler.scala

frontend/src/main/scala/bloop/engine/tasks/CompileTask.scala

jchyb · 2024-10-16T14:41:48Z

Since the last review I figured out an elegant way to keep the previous successful artifacts along with CompileAnalysis and reuse those, while also being able to have the best effort information simultaneously. To do this, if we already have an analysis and successful artifacts, we always first try compiling everything changed/tasked with recompilation since last successful, and then if something fails along the way recompiling the whole project in best effort mode.

Compiling twice in case of every failure might not seem performant, but it's actually the opposite. The first compilation is going to be short, since (likely) very few files are going to be recompiled. Second will be a bit longer, but still manageable, as we end the compilation at an earlier phase. This is better then the alternative, when in case of failure we replace everything with betasty, and when we fix the errors, everything has to be recompiled, including long transform and backend phases.

To put this into perspective, in the reproduction repository I was given, when we add a typer error to a compiled file, the first erroring compilation on my machine takes around 1s, then the best effort recompilation takes around 5s, and if we fix the error we end up with a single 1s recompilation. However if we were to clean the successful output (like I tried before), then we end up having to withstand full 20s recompilations when we fix the errors, and switching back end forth between failing and successful programs becomes annoying.

I did make sure to test whether the standard incremental compilation behaviors still work - zinc schedules recompilations and updates the analyses only for successful compilations, so it all still works (e.g. I add an indirect error -> the file is successfully compiled -> another few files are scheduled for recompilation -> this recompilation still fails -> every "second" compilation will try to recompile those files and also newly changed ones). I made it a bit confusing here, but the main thing is that it I did not have to add any extra logic for that to work.

Since that compilation order aspect started to get a bit complex, I added a chapter in the contributors guide about it (which might be a bit weird, since there is not much else like it there, so having implementation details there might be a bit confusing, but it's better than nothing in my opinion).

backend/src/main/scala/bloop/Compiler.scala

frontend/src/main/scala/bloop/engine/tasks/compilation/CompileGraph.scala

tgodzik · 2024-10-22T13:36:45Z

frontend/src/main/scala/bloop/engine/tasks/CompileTask.scala

-            def compile(inputs: CompileInputs) =
-              Compiler.compile(inputs, isBestEffort, isBestEffortDep)
+            def compile(inputs: CompileInputs) = {
+              val firstResult = Compiler.compile(inputs, isBestEffort, isBestEffortDep, true)


Suggested change

val firstResult = Compiler.compile(inputs, isBestEffort, isBestEffortDep, true)

val firstResult = Compiler.compile(inputs, isBestEffort, isBestEffortDep, firstCompilation = true)

tgodzik · 2024-10-22T13:36:56Z

frontend/src/main/scala/bloop/engine/tasks/CompileTask.scala

+                    previousCompilerResult = result,
+                    previousResult = emptyResult
+                  )
+                  Compiler.compile(newInputs, isBestEffort, isBestEffortDep, false)


Suggested change

Compiler.compile(newInputs, isBestEffort, isBestEffortDep, false)

Compiler.compile(newInputs, isBestEffort, isBestEffortDep, firstCompilation = false)

tgodzik · 2024-10-22T13:44:58Z

backend/src/main/scala/bloop/Compiler.scala

-          Task
-            .gatherUnordered(List(firstTask, secondTask))
-            .map(_ => ())
+              .forEach(path => if (Files.exists(path)) Files.delete(path))


Looks like this is unchanged? Would you mind including the change?

…fter an error

…lyses Also better cleanup directories

frontend/src/main/scala/bloop/engine/tasks/compilation/CompileGraph.scala

tgodzik

LGTM!

…etasty" This reverts commit 57eb430, reversing changes made to fa29730.

jchyb force-pushed the fix-leftover-betasty branch from 5936fa2 to 9b1b8f6 Compare September 2, 2024 09:28

jchyb force-pushed the fix-leftover-betasty branch from 6a92110 to 615db68 Compare September 9, 2024 16:07

jchyb marked this pull request as ready for review September 10, 2024 10:54

jchyb commented Sep 10, 2024

View reviewed changes

tgodzik requested changes Sep 17, 2024

View reviewed changes

jchyb marked this pull request as draft October 11, 2024 09:59

jchyb force-pushed the fix-leftover-betasty branch 2 times, most recently from 5f822e1 to 1bd2b10 Compare October 16, 2024 13:49

jchyb marked this pull request as ready for review October 16, 2024 14:41

tgodzik requested changes Oct 22, 2024

View reviewed changes

jchyb added 6 commits October 23, 2024 14:29

Remove betasty directory on successful compilation backgroundTasks

0b291b3

Delete betasty in newClassesDir, completely restart BE compilations a…

31e62be

…fter an error

Allow problems to be cleaned up after compiler crash

251158a

Run OrganiseImports

2d576ea

Redesign best effort compilation to not have to remove successful ana…

449ae20

…lyses Also better cleanup directories

Address review comments

8ec5c8d

jchyb force-pushed the fix-leftover-betasty branch from 1bd2b10 to 8ec5c8d Compare October 23, 2024 12:30

tgodzik reviewed Oct 23, 2024

View reviewed changes

frontend/src/main/scala/bloop/engine/tasks/compilation/CompileGraph.scala Show resolved Hide resolved

tgodzik approved these changes Oct 24, 2024

View reviewed changes

tgodzik merged commit 57eb430 into scalacenter:main Oct 24, 2024
16 of 17 checks passed

jchyb added a commit to jchyb/bloop that referenced this pull request Oct 30, 2024

Revert "Merge pull request scalacenter#2410 from jchyb/fix-leftover-b…

82ff408

…etasty" This reverts commit 57eb430, reversing changes made to fa29730.

tgodzik mentioned this pull request Nov 20, 2024

Feature Request: Best Effort Compilation sbt/sbt#7900

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove betasty directory on successful compilation backgroundTasks #2410

Remove betasty directory on successful compilation backgroundTasks #2410

jchyb commented Aug 26, 2024 •

edited

Loading

tgodzik commented Aug 30, 2024

jchyb commented Aug 30, 2024

tgodzik commented Aug 30, 2024

jchyb commented Aug 30, 2024 •

edited

Loading

jchyb commented Aug 30, 2024

jchyb commented Aug 30, 2024

jchyb Sep 10, 2024

jchyb commented Sep 10, 2024

tgodzik left a comment

tgodzik Sep 17, 2024

tgodzik Oct 22, 2024

jchyb Oct 23, 2024

jchyb commented Oct 16, 2024

tgodzik Oct 22, 2024

tgodzik Oct 22, 2024

tgodzik Oct 22, 2024

tgodzik left a comment

	.forEach(path => if (Files.exists(path)) Files.delete(path))
	.forEach(path => Files.delete(path)

	val firstResult = Compiler.compile(inputs, isBestEffort, isBestEffortDep, true)
	val firstResult = Compiler.compile(inputs, isBestEffort, isBestEffortDep, firstCompilation = true)

	Compiler.compile(newInputs, isBestEffort, isBestEffortDep, false)
	Compiler.compile(newInputs, isBestEffort, isBestEffortDep, firstCompilation = false)

Remove betasty directory on successful compilation backgroundTasks #2410

Remove betasty directory on successful compilation backgroundTasks #2410

Conversation

jchyb commented Aug 26, 2024 • edited Loading

tgodzik commented Aug 30, 2024

jchyb commented Aug 30, 2024

tgodzik commented Aug 30, 2024

jchyb commented Aug 30, 2024 • edited Loading

jchyb commented Aug 30, 2024

jchyb commented Aug 30, 2024

jchyb Sep 10, 2024

Choose a reason for hiding this comment

jchyb commented Sep 10, 2024

tgodzik left a comment

Choose a reason for hiding this comment

tgodzik Sep 17, 2024

Choose a reason for hiding this comment

tgodzik Oct 22, 2024

Choose a reason for hiding this comment

jchyb Oct 23, 2024

Choose a reason for hiding this comment

jchyb commented Oct 16, 2024

tgodzik Oct 22, 2024

Choose a reason for hiding this comment

tgodzik Oct 22, 2024

Choose a reason for hiding this comment

tgodzik Oct 22, 2024

Choose a reason for hiding this comment

tgodzik left a comment

Choose a reason for hiding this comment

jchyb commented Aug 26, 2024 •

edited

Loading

jchyb commented Aug 30, 2024 •

edited

Loading