Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jdk22 october for poc for virtual threads DO NOT COMMIT, DO NOT REVIEW, IGNORE IT #1319

Draft
wants to merge 34 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
042d8af
trash comit just for testing
ekaterinadimitrova2 May 22, 2024
959d48d
Substitute ant script and scriptdef with custom ant tasks
ekaterinadimitrova2 Jan 6, 2022
833ee22
Set correctly keepBriefBrief property
ekaterinadimitrova2 Apr 10, 2023
ae1da17
Assuming this is just test branch for newer JDK 17+, hacked G1 in bui…
ekaterinadimitrova2 May 30, 2024
defd65c
allow security manager which is deprecated
ekaterinadimitrova2 May 30, 2024
3543d64
Align options with what we currently use in cndb for 11,17,20 to test
ekaterinadimitrova2 Jun 1, 2024
5585c6f
Removes the use of CASSANDRA_USE_JDK11 and introduces the build.xml p…
michaelsembwever Jan 21, 2023
1f39566
Revisited add-exports and add-opens. Need to change the order in the …
ekaterinadimitrova2 Jun 4, 2024
c56115d
Apply same versions for dependencies as cndb/pom.xml
ekaterinadimitrova2 Jun 6, 2024
648c4a3
Fixes
ekaterinadimitrova2 Jun 6, 2024
2a9a439
Fix SSLFactoryTest
ekaterinadimitrova2 Jun 6, 2024
974991e
Fix WaitQueueTest,OutOfSpaceTest, PreparedStatementsTest, QueryWithIn…
ekaterinadimitrova2 Jun 7, 2024
9c6f32c
Fix tests post JDK-8210522 (rewrite reflection of modifiers field)
ekaterinadimitrova2 Jun 10, 2024
f0779fe
Upgrade Mockito to fix UnifiedCompactionStrategyTest
ekaterinadimitrova2 Jun 10, 2024
a5ff6df
Fix some internals access
ekaterinadimitrova2 Jun 10, 2024
945f100
Fix tools tests
ekaterinadimitrova2 Jun 10, 2024
ebd1f32
Bump netty - technically we need a full port of CASSANDRA-17992 and C…
ekaterinadimitrova2 Jun 10, 2024
5fa9826
Update mockito/bytebuddy; Fix our pom to really use the bytebuddy.ver…
ekaterinadimitrova2 Jun 11, 2024
c85b80d
Remove mocking in InternalNodeProbe spying on StorageServiceMBean
smiklosovic Jan 16, 2023
6632905
CASSANDRA-18180 Fix bulkLoaderSuccessfullyStreamsOverSsl fails with C…
djatnieks Apr 20, 2023
c75ac5b
Fix a few tests that had one-time issues with jamm
ekaterinadimitrova2 Jul 18, 2024
8536266
Temporarily make some constants not final to unblock testing DO NOT C…
ekaterinadimitrova2 Jul 19, 2024
d074ddf
Make it compile on JDK22
ekaterinadimitrova2 Jul 22, 2024
9938112
Fix tools/bin/cassandra.in.sh; fix some JDK internals openings for au…
ekaterinadimitrova2 Jul 22, 2024
a786f24
Port CASSANDRA-18190, this fixes the tests for now, but maybe we stil…
ekaterinadimitrova2 Sep 14, 2024
54d2d92
MessagePayloadTest fixed
ekaterinadimitrova2 Sep 16, 2024
6e1eb3b
Upgrade Jamm version to 0.4.0
blerer Jul 13, 2023
b160b80
cherry-pick conflict solved, the jamm upgrade may need some thorough …
ekaterinadimitrova2 Sep 17, 2024
441297d
Fix at bulk org.apache.cassandra.index.sai.disk tests
ekaterinadimitrova2 Sep 17, 2024
ce27ec1
Fix EmptyValuesTest
ekaterinadimitrova2 Sep 17, 2024
7c2bdc0
Revert "cherry-pick conflict solved, the jamm upgrade may need some t…
ekaterinadimitrova2 Oct 1, 2024
c0a3d69
Revert "Upgrade Jamm version to 0.4.0"
ekaterinadimitrova2 Oct 1, 2024
9c8cdc9
Ran into an odd situation where, on jdk21, antlr crashed first time t…
ekaterinadimitrova2 Oct 2, 2024
8cd964f
Bump ASM and add a few add-opens
ekaterinadimitrova2 Oct 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
Future version (tbd)
* Require only MODIFY permission on base when updating table with MV (STAR-564)
Merged from 5.0:
* Fix ClassCastException from jdk GaloisCounterMode when using JDK17 provider (CASSANDRA-18180)
* Remove mocking in InternalNodeProbe spying on StorageServiceMBean (CASSANDRA-18152)
* Prevent InaccessibleObjectException when the Leak Detector is traversing objects (CASSANDRA-18708)
* Remove Scripted UDFs internals; hooks to be added later in CASSANDRA-17281 (CASSANDRA-18252)
* Add guardrail for vector dimensions (CASSANDRA-18730)
Expand Down
4 changes: 3 additions & 1 deletion bin/cassandra.in.sh
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,9 @@ esac

# Read user-defined JVM options from jvm-server.options file
JVM_OPTS_FILE=$CASSANDRA_CONF/jvm${jvmoptions_variant:--clients}.options
if [ $JAVA_VERSION -ge 17 ] ; then
if [ $JAVA_VERSION -ge 22 ] ; then
JVM_DEP_OPTS_FILE=$CASSANDRA_CONF/jvm22${jvmoptions_variant:--clients}.options
elif [ $JAVA_VERSION -ge 17 ] ; then
JVM_DEP_OPTS_FILE=$CASSANDRA_CONF/jvm17${jvmoptions_variant:--clients}.options
elif [ $JAVA_VERSION -ge 11 ] ; then
JVM_DEP_OPTS_FILE=$CASSANDRA_CONF/jvm11${jvmoptions_variant:--clients}.options
Expand Down
467 changes: 256 additions & 211 deletions build.xml

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions conf/jvm-server.options
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,9 @@
#-Xms4G
#-Xmx4G

# Need experimental bytebuddy for JDK21
-Dnet.bytebuddy.experimental

# Young generation size is automatically calculated by cassandra-env
# based on this formula: min(100 * num_cores, 1/4 * heap size)
#
Expand Down
2 changes: 1 addition & 1 deletion conf/jvm11-server.options
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@
# Java 11 (and newer) GC logging options:
# See description of https://bugs.openjdk.java.net/browse/JDK-8046148 for details about the syntax
# The following is the equivalent to -XX:+PrintGCDetails -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-Xlog:gc=info,heap*=debug,age*=debug,safepoint=info,promotion*=debug:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
# -Xlog:gc=info,heap*=debug,age*=debug,safepoint=info,promotion*=debug:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760

# Notes for Java 8 migration:
#
Expand Down
2 changes: 1 addition & 1 deletion conf/jvm17-server.options
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@
# Java 11 (and newer) GC logging options:
# See description of https://bugs.openjdk.java.net/browse/JDK-8046148 for details about the syntax
# The following is the equivalent to -XX:+PrintGCDetails -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-Xlog:gc=info,heap*=debug,age*=debug,safepoint=info,promotion*=debug:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
# -Xlog:gc=info,heap*=debug,age*=debug,safepoint=info,promotion*=debug:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760

# Notes for Java 8 migration:
#
Expand Down
31 changes: 31 additions & 0 deletions conf/jvm22-clients.options
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
###########################################################################
# jvm22-clients.options #
# #
# See jvm-clients.options. This file is specific for Java 22 and newer. #
###########################################################################

###################
# JPMS SETTINGS #
###################

-Djdk.attach.allowAttachSelf=true
--add-exports java.base/jdk.internal.misc=ALL-UNNAMED
--add-exports java.base/jdk.internal.ref=ALL-UNNAMED
--add-exports java.base/sun.nio.ch=ALL-UNNAMED
--add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED
--add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED
--add-exports java.rmi/sun.rmi.server=ALL-UNNAMED
--add-exports java.sql/java.sql=ALL-UNNAMED

--add-opens java.base/java.lang.module=ALL-UNNAMED
--add-opens java.base/jdk.internal.loader=ALL-UNNAMED
--add-opens java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens java.base/jdk.internal.reflect=ALL-UNNAMED
--add-opens java.base/jdk.internal.math=ALL-UNNAMED
--add-opens java.base/jdk.internal.module=ALL-UNNAMED
--add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens java.base/java.io=ALL-UNNAMED
--add-opens java.base/sun.nio.ch=ALL-UNNAMED

# The newline in the end of file is intentional
131 changes: 131 additions & 0 deletions conf/jvm22-server.options
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
###########################################################################
# jvm22-server.options #
# #
# See jvm-server.options. This file is specific for Java 22 and newer. #
###########################################################################

#################
# GC SETTINGS #
#################



### CMS Settings
#-XX:+UseConcMarkSweepGC
#-XX:+CMSParallelRemarkEnabled
#-XX:SurvivorRatio=8
#-XX:MaxTenuringThreshold=1
#-XX:CMSInitiatingOccupancyFraction=75
#-XX:+UseCMSInitiatingOccupancyOnly
#-XX:CMSWaitDuration=10000
#-XX:+CMSParallelInitialMarkEnabled
#-XX:+CMSEdenChunksRecordAlways
## some JVMs will fill up their heap when accessed via JMX, see CASSANDRA-6541
#-XX:+CMSClassUnloadingEnabled



### G1 Settings
## Use the Hotspot garbage-first collector.
-XX:+UseG1GC
-XX:+ParallelRefProcEnabled

#
## Have the JVM do less remembered set work during STW, instead
## preferring concurrent GC. Reduces p99.9 latency.
-XX:G1RSetUpdatingPauseTimePercent=5
#
## Main G1GC tunable: lowering the pause target will lower throughput and vise versa.
## 200ms is the JVM default and lowest viable setting
## 1000ms increases throughput. Keep it smaller than the timeouts in cassandra.yaml.
-XX:MaxGCPauseMillis=500

## Optional G1 Settings
# Save CPU time on large (>= 16GB) heaps by delaying region scanning
# until the heap is 70% full. The default in Hotspot 8u40 is 40%.
#-XX:InitiatingHeapOccupancyPercent=70

# For systems with > 8 cores, the default ParallelGCThreads is 5/8 the number of logical cores.
# Otherwise equal to the number of cores when 8 or less.
# Machines with > 10 cores should try setting these to <= full cores.
#-XX:ParallelGCThreads=16
# By default, ConcGCThreads is 1/4 of ParallelGCThreads.
# Setting both to the same value can reduce STW durations.
#-XX:ConcGCThreads=16


### JPMS

-Djdk.attach.allowAttachSelf=true
--add-exports java.base/jdk.internal.misc=ALL-UNNAMED
--add-exports java.base/jdk.internal.ref=ALL-UNNAMED
--add-exports java.base/jdk.internal.perf=ALL-UNNAMED
--add-exports java.base/sun.nio.ch=ALL-UNNAMED
--add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED
--add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED
--add-exports java.rmi/sun.rmi.server=ALL-UNNAMED
# --add-exports java.sql/java.sql=ALL-UNNAMED --> not seen in
# Below exist in the other script
--add-exports jdk.naming.dns/com.sun.jndi.dns=java.naming
--add-exports jdk.unsupported/sun.misc=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED
--add-exports java.base/jdk.internal.util.jar=ALL-UNNAMED

--add-opens java.base/jdk.internal.loader=ALL-UNNAMED
--add-opens java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens java.base/jdk.internal.reflect=ALL-UNNAMED
--add-opens java.base/jdk.internal.math=ALL-UNNAMED
--add-opens java.base/jdk.internal.module=ALL-UNNAMED
--add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
# Added in
--add-opens java.base/java.lang.module=ALL-UNNAMED
--add-opens jdk.management.jfr/jdk.management.jfr=ALL-UNNAMED
--add-opens jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED
--add-opens jdk.compiler/com.sun.tools.javac=ALL-UNNAMED
--add-opens java.base/java.nio.file.spi=ALL-UNNAMED
--add-opens java.base/java.net=ALL-UNNAMED
--add-opens java.base/jdk.internal.math=ALL-UNNAMED
--add-opens java.base/java.lang.reflect=ALL-UNNAMED
--add-opens java.base/jdk.internal.vm=ALL-UNNAMED
--add-opens java.base/java.util.concurrent.locks=ALL-UNNAMED

# required for org.apache.cassandra.Util.getSupportedMTimeGranularity
--add-opens java.base/java.nio.file.attribute=ALL-UNNAMED

--add-opens jdk.compiler/com.sun.tools.javac=ALL-UNNAMED


### GC logging options -- uncomment to enable

# Java 11 (and newer) GC logging options:
# See description of https://bugs.openjdk.java.net/browse/JDK-8046148 for details about the syntax
# The following is the equivalent to -XX:+PrintGCDetails -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
# -Xlog:gc=info,heap*=debug,age*=debug,safepoint=info,promotion*=debug:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760

# Notes for Java 8 migration:
#
# -XX:+PrintGCDetails maps to -Xlog:gc*:... - i.e. add a '*' after "gc"
# -XX:+PrintGCDateStamps maps to decorator 'time'
#
# -XX:+PrintHeapAtGC maps to 'heap' with level 'trace'
# -XX:+PrintTenuringDistribution maps to 'age' with level 'debug'
# -XX:+PrintGCApplicationStoppedTime maps to 'safepoint' with level 'info'
# -XX:+PrintPromotionFailure maps to 'promotion' with level 'trace'
# -XX:PrintFLSStatistics=1 maps to 'freelist' with level 'trace'

### Netty Options

# On Java >= 9 Netty requires the io.netty.tryReflectionSetAccessible system property to be set to true to enable
# creation of direct buffers using Unsafe. Without it, this falls back to ByteBuffer.allocateDirect which has
# inferior performance and risks exceeding MaxDirectMemory
-Dio.netty.tryReflectionSetAccessible=true

### Enable vector incubator feature (simd support)

--add-modules jdk.incubator.vector

### Compatibility Options
--add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED
-Djava.security.manager=allow

# The newline in the end of file is intentional
118 changes: 17 additions & 101 deletions doc/modules/cassandra/pages/new/java11.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,15 @@ Table 1 : Support Matrix for Java
|===
| |Java 8 (Run) |Java 11 (Run)
|Java 8 (Build) |Supported |Supported
|Java 11(Build) |Not Supported |Supported
|Java 11(Build) |Not Supported |Experimental
|===

Essentially Apache 4.0 source code built with Java 11 cannot be run with
Java 8. Next, we shall discuss using each of Java 8 and 11 to build and
Apache 4.0 source code built with Java 11 cannot be run with
Java 8.

All binary releases are built with Java 8.

Next, we shall discuss using each of Java 8 and 11 to build and
run Apache Cassandra 4.0.

== Using Java 8 to Build
Expand All @@ -42,18 +46,10 @@ RedHat Linux the command is as follows:
$ sudo yum install java-1.8.0-openjdk-devel
....

Set `JAVA_HOME` and `JRE_HOME` environment variables in the shell bash
script. First, open the bash script:

....
$ sudo vi ~/.bashrc
....

Set the environment variables including the `PATH`.
Set the environment variables `JAVA_HOME` and `PATH`.

....
$ export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
$ export JRE_HOME=/usr/lib/jvm/java-1.8.0-openjdk/jre
$ export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
....

Expand All @@ -68,7 +64,7 @@ If Cassandra is already running stop Cassandra with the following
command.

....
[ec2-user@ip-172-30-3-146 bin]$ ./nodetool stopdaemon
$ ./nodetool stopdaemon
....

Build the source code from the `cassandra` directory, which has the
Expand Down Expand Up @@ -143,53 +139,19 @@ and install Java 11 on RedHat Linux run the following command.
$ yum install java-11-openjdk-devel
....

Set the environment variables in the bash script for Java 11. The first
command is to open the bash script.
Set the environment variables `JAVA_HOME` and `PATH`.

....
$ sudo vi ~/.bashrc
$ export JAVA_HOME=/usr/lib/jvm/java-11-openjdk
$ export JRE_HOME=/usr/lib/jvm/java-11-openjdk/jre
$ export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
....

To build source code with Java 11 one of the following two options must
be used.

____
[arabic]
. {blank}
+
Include Apache Ant command-line option `-Duse.jdk=11` as follows:::
....
$ ant -Duse.jdk=11
....
. {blank}
+
Set environment variable `CASSANDRA_USE_JDK11` to `true`:::
....
$ export CASSANDRA_USE_JDK11=true
....
____

As an example, set the environment variable `CASSANDRA_USE_JDK11` to
`true`.

....
[ec2-user@ip-172-30-3-146 cassandra]$ export CASSANDRA_USE_JDK11=true
[ec2-user@ip-172-30-3-146 cassandra]$ ant
Buildfile: /home/ec2-user/cassandra/build.xml
....

Or, set the command-line option.

....
[ec2-user@ip-172-30-3-146 cassandra]$ ant -Duse.jdk11=true
....

The build output should include the following.
The build output should now include the following.

....
[echo] Non default JDK version used: 11
...
...
_build_java:
[echo] Compiling for Java 11
...
Expand All @@ -215,36 +177,20 @@ jar:

BUILD SUCCESSFUL
Total time: 1 minute 3 seconds
[ec2-user@ip-172-30-3-146 cassandra]$
....

== Common Issues

One of the two options mentioned must be used to compile with JDK 11 or
the build fails and the following error message is output.

....
[ec2-user@ip-172-30-3-146 cassandra]$ ant
Buildfile: /home/ec2-user/cassandra/build.xml
validate-build-conf:

BUILD FAILED
/home/ec2-user/cassandra/build.xml:293: -Duse.jdk11=true or $CASSANDRA_USE_JDK11=true must
be set when building from java 11
Total time: 1 second
[ec2-user@ip-172-30-3-146 cassandra]$
....

The Java 11 built Apache Cassandra 4.0 source code may be run with Java
11 only. If a Java 11 built code is run with Java 8 the following error
message gets output.

....
[root@localhost ~]# ssh -i cassandra.pem [email protected]
# ssh -i cassandra.pem [email protected]
Last login: Wed Jul 31 20:47:26 2019 from 75.155.255.51
[ec2-user@ip-172-30-3-146 ~]$ echo $JAVA_HOME
$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk
[ec2-user@ip-172-30-3-146 ~]$ cassandra
$ cassandra
...
...
Error: A JNI error has occurred, please check your installation and try again
Expand All @@ -258,33 +204,3 @@ class file versions up to 52.0
...
....

The `CASSANDRA_USE_JDK11` variable or the command-line option
`-Duse.jdk11` cannot be used to build with Java 8. To demonstrate set
`JAVA_HOME` to version 8.

....
[root@localhost ~]# ssh -i cassandra.pem [email protected]
Last login: Wed Jul 31 21:41:50 2019 from 75.155.255.51
[ec2-user@ip-172-30-3-146 ~]$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk
....

Set the `CASSANDRA_USE_JDK11=true` or command-line option
`-Duse.jdk11=true`. Subsequently, run Apache Ant to start the build. The
build fails with error message listed.

....
[ec2-user@ip-172-30-3-146 ~]$ cd
cassandra
[ec2-user@ip-172-30-3-146 cassandra]$ export CASSANDRA_USE_JDK11=true
[ec2-user@ip-172-30-3-146 cassandra]$ ant
Buildfile: /home/ec2-user/cassandra/build.xml

validate-build-conf:

BUILD FAILED
/home/ec2-user/cassandra/build.xml:285: -Duse.jdk11=true or $CASSANDRA_USE_JDK11=true cannot
be set when building from java 8

Total time: 0 seconds
....
Loading