Sporadic java.lang.IllegalStateException: Field XXX defines an incompatible default value 20

[justinas-dabravolskas]

Hi,
java.lang.IllegalStateException sporadically occurred in two different classes on static final int declarations on. The error was observed once with a public static final int field and another time with a private static final int field on different machines, both running on Java 21.0.5 and ByteBuddy 1.15.10
I've disabled validation, but it's uncertain if this resolves the issue due to the very rare occurrence and inability to reproduce the error. Do you have insights what might be causing these sporadic errors?

java.lang.IllegalStateException: Field DEFAULT_SIZE defines an incompatible default value 1 at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ValidatingClassVisitor.visitField(TypeWriter.java:2535) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356) at net.bytebuddy.utility.visitor.MetadataAwareClassVisitor.onVisitField(MetadataAwareClassVisitor.java:293) at net.bytebuddy.utility.visitor.MetadataAwareClassVisitor.visitField(MetadataAwareClassVisitor.java:278) at net.bytebuddy.jar.asm.ClassReader.readField(ClassReader.java:1138) at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:740) at net.bytebuddy.utility.AsmClassReader$Default.accept(AsmClassReader.java:132) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:4039) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2246) at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$UsingTypeWriter.make(DynamicType.java:4092) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12726) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12661) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12370) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:13152) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:13082) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java:12556) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12604) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source) at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188) at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:610) at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)

[justinas-dabravolskas]

Looks like this talks about the same issue open-telemetry/opentelemetry-java-instrumentation#9744

[justinas-dabravolskas]

I'm curious how this one could happen
java.lang.IllegalStateException: Field DEFAULT_CORE_POOL_SIZE defines an incompatible default value 1 at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ValidatingClassVisitor.visitField(TypeWriter.java:2535) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356)
In this case DEFAULT_CORE_POOL_SIZE is defined as public static final int DEFAULT_CORE_POOL_SIZE = 1; Doesn't it mean that int minimum, maximum; had to be left at default values for it to happen? Though it could happen just in case none of switch statements was applied, including default that would indicate aJIT bug?

[raphw]

I improved the error message now. Can you improve this stabely? If so, try running withe the snapshot.

If you look into the complied code, all of these values are compile-time constants and added to the class file. I would also expect that this can only happen when a number is 0. I would assume a JIT bug, too.

[justinas-dabravolskas]

Exceptions are extremely rare and I can't reproduce them. I don't see your commit yet, but I'm happy to update to ByteBuddy1.15.11 with improved logging whenever it is available. I probably should submit an OpenJDK bug, but I don't have hopes without providing a reproducible.

[dogourd]

I tried reproducing this issue, and now I can reliably replicate the following error locally.

Exception in thread "main" java.lang.IllegalStateException: Field POOL_NORMAL defines an incompatible default value 0
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ValidatingClassVisitor.visitField(TypeWriter.java:2535)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining$WithFullProcessing$RedefinitionClassVisitor.onVisitField(TypeWriter.java:5164)
        at net.bytebuddy.utility.visitor.MetadataAwareClassVisitor.visitField(MetadataAwareClassVisitor.java:278)
        at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356)
        at net.bytebuddy.jar.asm.commons.ClassRemapper.visitField(ClassRemapper.java:169)
        at net.bytebuddy.jar.asm.ClassReader.readField(ClassReader.java:1138)
        at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:740)
        at net.bytebuddy.utility.AsmClassReader$Default.accept(AsmClassReader.java:132)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:4039)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2246)
        at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$UsingTypeWriter.make(DynamicType.java:4085)
        at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3769)
        at com.test.Test1732.main(Test1732.java:26)

Here is the corresponding reproduction example:

public class Test1732 {

    public static final int POOL_NORMAL = 0;
    public static final int POOL_SUSPENDED = 1;
    public static final int POOL_SHUTDOWN = 2;

    public volatile int poolState;

    public static void main(String[] args) throws Exception {
        DynamicType.Builder<Test1732> builder = new ByteBuddy().with(TypeValidation.ENABLED)
                .redefine(Test1732.class)
                .name(Test1732.class.getName() + "_Redefine");
        for (int i = 0; i < 100; i++) {
            builder.make();
        }
    }
}

You can reproduce it simply by explicitly adding the -Xcomp parameter, for example:
java -Xcomp -classpath ${classpath} Test1732

Here is some relevant environment information (in case it helps):

• OS: Windows 10
• ByteBuddy: 1.15.10
• Java:
  openjdk version "21" 2023-09-19
  OpenJDK Runtime Environment Zulu21.28+85-CA (build 21+35)
  OpenJDK 64-Bit Server VM Zulu21.28+85-CA (build 21+35, mixed mode, sharing)

[justinas-dabravolskas]

Interesting that running on Java 21 fails, but running on Java 17 doesn't.
Both tests on Mac OS:
openjdk version "21.0.5" 2024-10-15 LTS
OpenJDK Runtime Environment Temurin-21.0.5+11 (build 21.0.5+11-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (build 21.0.5+11-LTS, mixed mode, sharing)

openjdk version "17.0.8" 2023-07-18
OpenJDK Runtime Environment Temurin-17.0.8+7 (build 17.0.8+7)
OpenJDK 64-Bit Server VM Temurin-17.0.8+7 (build 17.0.8+7, mixed mode)

[raphw]

I pushed to a branch by accident but now added the change to master. Could you run it? I do not have Windows available, unfortunately. I assume that this is due to a JIT compiler bug, indeed. Likely one that was introduces with Java 21.

[justinas-dabravolskas]

Confirms a theory. How so ByteBuddy is effected by it so consistently?
/Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java -Xcomp -cp .:/Users/jdabravolskas/Downloads/byte-buddy-1.15.10.jar Test1732 Exception in thread "main" java.lang.IllegalStateException: Field POOL_NORMAL defines an incompatible default value 0 (-2147483648-2147483647) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ValidatingClassVisitor.visitField(TypeWriter.java:2535) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining$WithFullProcessing$RedefinitionClassVisitor.onVisitField(TypeWriter.java:5164) at net.bytebuddy.utility.visitor.MetadataAwareClassVisitor.visitField(MetadataAwareClassVisitor.java:278) at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356) at net.bytebuddy.jar.asm.commons.ClassRemapper.visitField(ClassRemapper.java:169) at net.bytebuddy.jar.asm.ClassReader.readField(ClassReader.java:1138) at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:740) at net.bytebuddy.utility.AsmClassReader$Default.accept(AsmClassReader.java:132) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:4039) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2246) at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$UsingTypeWriter.make(DynamicType.java:4085) at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3769) at Test1732.main(Test1732.java:17)

[raphw]

The JIT compiler can be more deterministic than one thinks. Does the problem occur with Java 24-EA? it might be worth filing a bug.

[justinas-dabravolskas]

Issue can be reproduced on Java 18-23. Submitted an OpenJDk bug with a reference to minimal reproducer and link to this discussion.

[dogourd]

Looks like this talks about the same issue open-telemetry/opentelemetry-java-instrumentation#9744

However, unexpectedly, the previously mentioned reference explicitly states they are using Java 11(JDK: Temurin 11.0.15). If the current case cannot be reproduced on versions earlier than 17, the worst-case scenario is that there are multiple JIT issues, and we are only reproducing one of them.

Additionally, it's worth noting that the range of Java versions affected by this problem seems quite broad. @raphw I am more curious whether we can make some equivalent variations in ByteBuddy's code to steer clear of this JIT bug.

[raphw]

Two thoughts:

1. Yes, we can try. I replaced the switch statement with an if/else. Could you try with the latest master? I cannot reproduce this bug on Linux, so I assume it is Windows related.
2. Ideally, type validation should be deactivated in production environments.

[dogourd]

That's strange—I was able to successfully reproduce it in a Linux environment as well. I tried using the if-else version, but it doesn’t seem to work.

Additionally, when I added the following parameters in the Windows environment
-XX:+UnlockDiagnosticVMOptions -XX:CompileCommand="exclude net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ValidatingClassVisitor::visitField"
it executed successfully. However, it still doesn't seem to work in my Linux environment.

Here is some execution information in Linux:

[root@www project]# ls
byte-buddy-1.15.11-SNAPSHOT.jar  Test1732.java

[root@www project]# ../zulu21/bin/java -version
openjdk version "21.0.5" 2024-10-15 LTS
OpenJDK Runtime Environment Zulu21.38+21-CA (build 21.0.5+11-LTS)
OpenJDK 64-Bit Server VM Zulu21.38+21-CA (build 21.0.5+11-LTS, mixed mode, sharing)

[root@www project]# ../zulu21/bin/javac -cp .:./byte-buddy-1.15.11-SNAPSHOT.jar Test1732.java 

[root@www project]# ls
byte-buddy-1.15.11-SNAPSHOT.jar  Test1732.class  Test1732.java

[root@www project]# ../zulu21/bin/java -Xcomp -cp .:byte-buddy-1.15.11-SNAPSHOT.jar Test1732
Exception in thread "main" java.lang.IllegalStateException: Field POOL_NORMAL defines an incompatible default value 0 (-2147483648-2147483647)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ValidatingClassVisitor.visitField(TypeWriter.java:2531)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining$WithFullProcessing$RedefinitionClassVisitor.onVisitField(TypeWriter.java:5160)
        at net.bytebuddy.utility.visitor.MetadataAwareClassVisitor.visitField(MetadataAwareClassVisitor.java:278)
        at net.bytebuddy.jar.asm.ClassVisitor.visitField(ClassVisitor.java:356)
        at net.bytebuddy.jar.asm.commons.ClassRemapper.visitField(ClassRemapper.java:169)
        at net.bytebuddy.jar.asm.ClassReader.readField(ClassReader.java:1138)
        at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:740)
        at net.bytebuddy.utility.AsmClassReader$Default.accept(AsmClassReader.java:132)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:4035)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2246)
        at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$UsingTypeWriter.make(DynamicType.java:4085)
        at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3769)
        at Test1732.main(Test1732.java:18)

[root@www project]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

It seems that @justinas-dabravolskas has already reported this issue to the JIT team. Could you provide the link to the corresponding issue?

[raphw]

I ran an too old VM, I guess, and got to reproduce it. I gave it another try with a different implementation and I think it is related to the different data types of the MAX_VALUE and MIN_VALUE fields.

With the new implementation, the error does no longer occure.

[justinas-dabravolskas]

@dogourd I'll add OpenJDK ticket after they accept it. @raphw I can confirm that error now longer occurs with your latest commit on Java 21,23 MacOS. I'm curious what is an actual idea of changing two int assignments to a single boolean?

[raphw]

I wanted to avoid mixing different datatypes (boolean, byte, short, char and int), I think that is where the JIT mixes things up somewhere and does not get the comparison right. More of a hinch, I have not looked at the assembly.