Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Starting a Gradle Daemon, x busy Daemons could not be reused, use --status for details, recursion until RAM floods #92

Open
lattice0 opened this issue Apr 23, 2022 · 18 comments

Comments

@lattice0
Copy link

When I added

tasks.whenTaskAdded { task ->
    if ((task.name == 'javaPreCompileDebug' || task.name == 'javaPreCompileRelease')) {
        task.dependsOn 'cargoBuild'
    }
}

to my flutter project's build.gradle and when I do flutter build apk I get things like

Starting a Gradle Daemon, 18 busy Daemons could not be reused, use --status for details
Starting a Gradle Daemon, 19 busy Daemons could not be reused, use --status for details
...

and then it keeps starting a new one until my RAM floods and the computer crashes. If I comment that code, it builds, but of course without gradle.

Is there a way to debug more to see what is hapening? Looking at the system monitor, it's starting lots of java gradle daemons

@lattice0
Copy link
Author

./gradlew assembleDebug --status
   PID STATUS   INFO
 17367 IDLE     7.4.2
 33232 IDLE     7.4.2
  8424 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
  8735 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
  8116 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
  9063 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
  9402 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
  9746 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 13826 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 14296 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 14781 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 13410 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 15287 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 10143 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 12966 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 15847 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 12584 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 10565 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 10944 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 12162 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 11744 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 11343 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
  4345 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 17623 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 17907 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 19255 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 18977 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
  7805 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 18705 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 18452 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 18187 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 20056 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 20337 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 20619 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 19773 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)
 19492 STOPPED  (other compatible daemons were started and after being idle for 0 minutes and not recently used)

@lattice0
Copy link
Author

This is part of the output of ./gradlew --info --stacktrace --debug assembleDebug

2022-04-24T17:14:45.981+0000 [QUIET] [system.out] > Configure project :
2022-04-24T17:14:45.981+0000 [QUIET] [system.out] Starting a Gradle Daemon, 2 busy and 23 stopped Daemons could not be reused, use --status for details
2022-04-24T17:14:51.958+0000 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager] 
2022-04-24T17:14:51.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2022-04-24T17:14:51.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2022-04-24T17:14:51.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2022-04-24T17:14:51.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2022-04-24T17:14:51.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2022-04-24T17:14:51.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2022-04-24T17:14:46.015+0000 [LIFECYCLE] [org.gradle.internal.logging.progress.ProgressLoggerFactory] 
2022-04-24T17:14:46.015+0000 [LIFECYCLE] [org.gradle.internal.logging.progress.ProgressLoggerFactory] > Configure project :
2022-04-24T17:14:54.279+0000 [QUIET] [system.out] 
2022-04-24T17:14:54.279+0000 [QUIET] [system.out] > Configure project :
2022-04-24T17:14:54.279+0000 [QUIET] [system.out] Starting a Gradle Daemon, 3 busy and 23 stopped Daemons could not be reused, use --status for details
2022-04-24T17:15:01.958+0000 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager] 
2022-04-24T17:15:01.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2022-04-24T17:15:01.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2022-04-24T17:15:01.958+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2022-04-24T17:15:01.959+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2022-04-24T17:15:01.959+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2022-04-24T17:15:01.959+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2022-04-24T17:14:54.315+0000 [LIFECYCLE] [org.gradle.internal.logging.progress.ProgressLoggerFactory] 
2022-04-24T17:14:54.315+0000 [LIFECYCLE] [org.gradle.internal.logging.progress.ProgressLoggerFactory] > Configure project :
2022-04-24T17:15:02.379+0000 [QUIET] [system.out] 
2022-04-24T17:15:02.379+0000 [QUIET] [system.out] > Configure project :
2022-04-24T17:15:02.379+0000 [QUIET] [system.out] Starting a Gradle Daemon, 4 busy and 23 stopped Daemons could not be reused, use --status for details
2022-04-24T17:15:10.279+0000 [QUIET] [system.out] 
2022-04-24T17:15:10.279+0000 [QUIET] [system.out] > Configure project :
2022-04-24T17:15:10.279+0000 [QUIET] [system.out] Starting a Gradle Daemon, 5 busy and 23 stopped Daemons could not be reused, use --status for details

@ncalexan
Copy link
Member

@lattice0: I am not aware of any particular interaction between rust-android-gradle and the Gradle daemon. You might try killing all of the daemons, perhaps with ./gradlew --stop, and then digging in. The fact that you have busy daemons might mean something -- perhaps some long cargo builds, or something is wedged?

@lattice0
Copy link
Author

lattice0 commented Apr 25, 2022 via email

@lattice0
Copy link
Author

@ncalexan I've found many issues like mine but all of them say something about deactivating windows hotspot (https://stackoverflow.com/questions/67044599/flutter-starting-gradle-daemon-loop-creating-endless-processes). However, I'm on Ubuntu and inside docker, and the issue only happens when I add


tasks.whenTaskAdded { task ->
    if ((task.name == 'javaPreCompileDebug' || task.name == 'javaPreCompileRelease')) {
        task.dependsOn 'cargoBuild'
    }
}

is there something in this plugin that tries to access something on the internet? I tried deactivating my VPN ad blocker and the VPN itself, but the problem persists. I also deactivated the internet on the VM itself and the problem persisted.

So, it seems like it happens only with flutter, and windows hotspot if windows, and in my case only when I add the cargoBuild task as a dependency

@ncalexan
Copy link
Member

Could you try removing the dependency block above, and instead invoking ./gradlew cargoBuild directly? That would give you some idea if it's a rust-android-gradle issue directly or if it's more likely to be a flutter interaction.

I will say that the plugin doesn't access anything on the internet itself (that I am aware of), but it will of course spark Gradle to fetch dependencies, and it will also spark Cargo to fetch crates, etc.

Can you link your project so that I can see what you're doing? Bonus points if you run this Docker flow in CI already and can link to logs!

@lattice0
Copy link
Author

lattice0 commented Apr 28, 2022 via email

@lattice0
Copy link
Author

@ncalexan here it is a minimum verifiable example that also recursively starts gradle daemons on the CI:

https://github.com/lattice0/flutter_bug/runs/6235250770?check_suite_focus=true

@ncalexan
Copy link
Member

@ncalexan here it is a minimum verifiable example that also recursively starts gradle daemons on the CI:

https://github.com/lattice0/flutter_bug/runs/6235250770?check_suite_focus=true

Maybe that repo isn't public? I certainly can't see it (or the link you provided).

@lattice0
Copy link
Author

@ncalexan ops, sorry, made public now, and also as a bonus, a branch with the rust-android-gradle stuff commented out from build.gradle building correctly https://github.com/lattice0/flutter_bug/runs/6235383129?check_suite_focus=true

@ncalexan
Copy link
Member

ncalexan commented May 5, 2022

@lattice0 sorry, I haven't been able to get to this. I wonder if #85 is relevant? Would you mind trying with the alternate dependency chain and see if it is related? Thanks!

@lattice0
Copy link
Author

lattice0 commented May 5, 2022

@ncalexan just tested and same problem. I tested all possible solutions, tried many different things this week :(

@lattice0
Copy link
Author

lattice0 commented May 6, 2022

@ncalexan does it possibly change an environment variable when trying to launch cargo? Some guy at gradle said that the criteria must be exactly as before, otherwise it launches a new daemon again

@ncalexan
Copy link
Member

ncalexan commented May 6, 2022

@ncalexan does it possibly change an environment variable when trying to launch cargo? Some guy at gradle said that the criteria must be exactly as before, otherwise it launches a new daemon again

Well, we set many environment variables for the Cargo invocations, but I think they're all managed by the exec spec, not globally. Can you get debug info from the Daemon re-use calculation? If it'll just tell you what changed, we can dig deeper.

@lattice0
Copy link
Author

lattice0 commented May 6, 2022

@ncalexan I called gradle with --stacktrace --info --debug, but when this daemon thing happens, there's no logs, just one after another

@ncalexan
Copy link
Member

ncalexan commented May 6, 2022

I had a chance to look into this. (It was really only possible for me to do so since you had arranged the Docker environment, etc -- thank you.)

Based on some hints in the Gradle daemon log files, which I found using gradlew --status and reading the per-PID log files in ~/.gradle/daemon/..., I conclude that the issue with the daemons is a flutter bug. If I insert only the following, I still see the failure you report:

tasks.whenTaskAdded { task ->
}

I haven't verified that the failure occurs when we remove rust-android-gradle entirely; when you report this to the flutter team, that would be helpful to determine. I.e., maybe this is an interaction with plugins and tasks.whenTaskAdded, although I doubt it.

Now, since everything is per-project, there's no need to do this in your root build.gradle file. Locally, the following patch addresses the issue:

diff --git a/flutter_bug/android/app/build.gradle b/flutter_bug/android/app/build.gradle
index a38331d..dd9b2a0 100644
--- a/flutter_bug/android/app/build.gradle
+++ b/flutter_bug/android/app/build.gradle
@@ -79,3 +79,9 @@ flutter {
 dependencies {
     implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version"
 }
+
+tasks.whenTaskAdded { task ->
+    if ((task.name == 'javaPreCompileDebug' || task.name == 'javaPreCompileRelease')) {
+        task.dependsOn 'cargoBuild'
+    }
+}
diff --git a/flutter_bug/android/build.gradle b/flutter_bug/android/build.gradle
index a69ddf5..cf2df46 100644
--- a/flutter_bug/android/build.gradle
+++ b/flutter_bug/android/build.gradle
@@ -30,12 +30,6 @@ subprojects {
     project.evaluationDependsOn(':app')
 }

-tasks.whenTaskAdded { task ->
-    if ((task.name == 'javaPreCompileDebug' || task.name == 'javaPreCompileRelease')) {
-        task.dependsOn 'cargoBuild'
-    }
-}
-
 task clean(type: Delete) {
     delete rootProject.buildDir
 }
diff --git a/libsomething/src/main.rs b/libsomething/src/main.rs
deleted file mode 100644
index e7a11a9..0000000
--- a/libsomething/src/main.rs
+++ /dev/null
@@ -1,3 +0,0 @@
-fn main() {
-    println!("Hello, world!");
-}

With this, flutter build apk produces:

# flutter build apk
fatal: unsafe repository ('/opt/flutter/development/flutter' is owned by someone else)
To add an exception for this directory, call:

	git config --global --add safe.directory /opt/flutter/development/flutter
Changing current working directory to: /home/dev/project/flutter_bug

💪 Building with sound null safety 💪

       Fresh libsomething v0.1.0 (/home/dev/project/libsomething)
warning: function is never used: `main`
 --> src/lib.rs:1:4
  |
1 | fn main() {
  |    ^^^^
  |
  = note: `#[warn(dead_code)]` on by default

warning: `libsomething` (lib) generated 1 warning
    Finished release [optimized] target(s) in 0.00s
Running Gradle task 'assembleRelease'...                            3.5s
✓  Built build/app/outputs/flutter-apk/app-release.apk (15.9MB).

and I see the Rust library libsomething.so as expected:

# unzip -l ../build/app/outputs/flutter-apk/app-release.apk | grep '\.so$'
  9364312  1981-01-01 01:01   lib/arm64-v8a/libflutter.so
 10347944  1981-01-01 01:01   lib/x86_64/libflutter.so
  6536388  1981-01-01 01:01   lib/armeabi-v7a/libflutter.so
  2851656  1981-01-01 01:01   lib/x86_64/libapp.so
  3228240  1981-01-01 01:01   lib/armeabi-v7a/libapp.so
  2851736  1981-01-01 01:01   lib/arm64-v8a/libapp.so
   288432  1981-01-01 01:01   lib/x86_64/libsomething.so

Good luck!

@lattice0
Copy link
Author

lattice0 commented May 7, 2022

thank you, it works. Gonna report this to flutter. Thank you very much!!!!

@ncalexan
Copy link
Member

ncalexan commented May 9, 2022

thank you, it works. Gonna report this to flutter. Thank you very much!!!!

You are welcome. What a strange bug! If it turns out this only repros with rust-android-gradle, please get back to me -- it's entirely possible there's something wonky with the plugin startup. Gradle has changed quite a bit since 4 (when I was building this) and 7/8 (current-ish), especially around lazy configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants