java.lang.ClassCircularityError: jdk/internal/misc/VirtualThreads thrown when using virtual threads under load

[pzgadzaj-equinix]

Describe the bug

• java.lang.ClassCircularityError: jdk/internal/misc/VirtualThreads when running Spring based microservice on Kubernetes.
• This exception happens when in spring boot we enable virtual threads
• spring: threads: virtual: enabled: true
• Exception happens when service is under high load (many incomming HTTP requests)
• Only remediation is to restart the POD

To Reproduce

No steps to reproduce. Happens from time to time, and only when high load is there. So i cannot provide exact steps or reproducible example. Found two tickets over internet which might be related:

• Issue with virtual-threads open-telemetry/opentelemetry-java-instrumentation#10181
• https://bugs.java.com/bugdatabase/view_bug?bug_id=8322846 (Oracle JDK).

Expected behavior

No exceptions

Screenshots

N/A

Platform information

• Ubuntu 22.04.4 LTS
• Spring Boot 3.2.6
• Instrumentation with open telemetry java agent
• Instrumenation with app dynamics java agent
• JDK: openjdk 21.0.3 2024-04-16 LTSOpenJDK Runtime Environment Corretto-21.0.3.9.1 (build 21.0.3+9-LTS)OpenJDK 64-Bit Server VM Corretto-21.0.3.9.1 (build 21.0.3+9-LTS, mixed mode)

Additional context

Full stacktrace:
fulls_stacktrace.txt

[shipilev]

JDK-8322846 is already backported to 21.0.4 upstream, and would land in Corretto 21.0.4 release next week.

That said, java.lang.ClassCircularityError does not look like JDK-8322846. CCE implicates the class transformation code is some way. We have seen issues when agent tries to load the class that is currently under retransformation (JDK-8164165 shows the JVM throws CCE instead of more proper LinkageError in that case).

So I believe this is a bug in either OTel or AppDynamics java agent. Look around their bug-trackers for possibly related ClassCircularityError fixes.

[simonis]

I agree with @shipilev. This is most likely caused by incorrect usage of bytecode transformation in one of your instrumentation agents. Take a look at ByteBuddy issue #1666 for more details. That issue also references similar issues in the Data Dog dd-trace and the Glowroot library.

I've recently also opened JDK-8335619: Add an @apiNote to j.l.i.ClassFileTransformer to warn about recursive class loading and ClassCircularityErrors and submitted a PR for it.

[AlanBateman]

The supporting class jdk.internal.misc.VirtualThreads has been removed in the loom repo, as has jdk.internal.event.VirtualThreadPinnedEvent. This removes class loading from LockSupport.park that can happen when the first usage is in a class file transformer, which may be what is happening here. I hope to bring some of these changes into main line in advance of the object monitors work.