Quarkus crashing with "io.quarkus.arc.Arc.container() is null" while executing first MySQL query after startup

[mzasonski]

Describe the bug

We got application running application based on Quarkus 2.16.12.Final inside K8S cluster run on AWS.

Randomly when we're restarting the pods (application version update, database failover, cluster failover) we are experiencing crash back loop on the pods running Quarkus based application.

From our observations it's happening with the first queries done to the database after the service started and throwing exception like the problem happened after shutdown:

Error occurred while executing task for trigger %s [Error Occurred After Shutdown]: java.util.concurrent.CompletionException: java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.beanManager()" because the return value of "io.quarkus.arc.Arc.container()" is null

After startup applications randomly fails while performing query of the @Scheduled operation - sanitised relevant code snippets:

@ApplicationScoped
public class SessionService {
	
    private final SessionRepository sessionRepository;

    @Inject
    public SessionService(SessionRepository sessionRepository) {
        this.sessionRepository = sessionRepository;
    }

    @Transactional
    @Scheduled(every = "${interval:5m}")
    void deleteExpiredSessions() {
        sessionRepository.deleteExpired();
    }
}

@Transactional
@ApplicationScoped
public class SessionRepository implements PanacheRepositoryBase<SessionEntity, UUID> {

    public void deleteExpired() {
        delete("expirationTimestamp < :timestamp", Parameters.with("timestamp", Instant.now().getEpochSecond()));
    }
}

Something is closing the connection to the database and stopping Quarkus, resulting in the shutdown of the application, but we cannot find out what is the root cause of it, and it's happening only on the production environment so far. After few failed starts it's picking up and operating normally.

Database used: Aurora DB, engine version: 8.0-mysql_aurora.3.05.2

Startup logs: quarkus_failure_after_startup.txt

Expected behavior

Application continue running properly after startup.

Actual behavior

Null pointer exception on:
Error occurred while executing task for trigger %s [Error Occurred After Shutdown]: java.util.concurrent.CompletionException: java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.beanManager()" because the return value of "io.quarkus.arc.Arc.container()" is null saying down on the stack traces that db connection is closed and application shut down.

How to Reproduce?

Not able to provide reproducing steps, as I wasn't able to reproduce the issue locally.

Output of uname -a or ver

Linux <our_pod_details> 5.10.224-212.876.amzn2.x86_64 #1 SMP Thu Aug 22 16:55:24 UTC 2024 x86_64 GNU/Linux

Output of java -version

openjdk 17.0.2 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-86)
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)

Quarkus version or git rev

2.16.12.Final

Build tool (ie. output of mvnw --version or gradlew --version)

------------------------------------------------------------
Gradle 8.7
------------------------------------------------------------
Build time:   2024-03-22 15:52:46 UTC
Revision:     650af14d7653aa949fce5e886e685efc9cf97c10
Kotlin:       1.9.22
Groovy:       3.0.17
Ant:          Apache Ant(TM) version 1.10.13 compiled on January 4 2023
JVM:          17.0.11 (Eclipse Adoptium 17.0.11+9)
OS:           Linux 5.15.0-1070-aws amd64

Additional information

No response

[quarkus-bot]

/cc @Ladicek (arc), @geoand (kotlin), @manovotn (arc), @mkouba (arc)

[Ladicek]

Something is closing the connection to the database and stopping Quarkus, resulting in the shutdown of the application, but we cannot find out what is the root cause of it

Is the application running in Kubernetes? If so, I'd take a look at the liveness check(s). That's the first thing that comes to mind as to why the application would suddenly (in fact, practically immediately after startup) shut down.

[mzasonski]

It's as simple as below:

@Liveness
@ApplicationScoped
public class HttpHealthCheck implements HealthCheck {
    private final Logger logger;

    public HttpHealthCheck(Logger logger) {
        this.logger = logger;
    }
    @Override
    public HealthCheckResponse call() {
        logger.debug("HealthCheck called");
        return HealthCheckResponse.up("Service is live");
    }
}

and it actually manages to show the with UP value before it collapses, which actually causes problems on our side during the new version rollout.

[Ladicek]

There might be other liveness checks, added by Quarkus. I'm not exactly sure which ones, but I'd check that all of them report UP. But it looks like you've checked that already, so the issue is probably elsewhere...

[mzasonski]

That's right, both ours and built-in liveness and readiness checks completed successfully and reported UP, just before the crash.

[geoand]

Version 2.16.12.Final is very far out of date, so it would be good to know if the problem persists with the latest versions of Quarkus