Triage for Amazon Linux 2023 ( Docker static) #5300
Assignees
Comments
|
Aarch64 has one more test failure - https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_aarch64_linux/82/testReport/junit/jdk_jfr_jcmd_TestJcmdDump/java/TestJcmdDump/ , which happened to other platforms before. Other than that those two platforms have exactly same failures. Will triage x64. |
|
@sxa According to jenkins builds mentioned above- running against jdk21
So Temurin has same validation on Amazon (docker static) as on other docker static. Let me know if any further validation needed. |
|
The Ubuntu 24.04 machine is also new (and will also require more triage of the results on there - there are some failures in those runs which I hadn't seen before), possibly also true for the Debian 12 one which I think is also new, so they may not be the best ones to compare with. Are those 16 + 32 failures expected and seen on the GA release triage last month? I didn't think I'd seen that many failures in the suite when I was running it recently on builds on other machines. EDIT: From
does this mean that Amazon Linux 2023 is showing failures that do not occur on RHEL7? |
test-ibmcloud-rhel7-x64-1 is not a docker static one. |
|
Does that mean we're seeing significantly different results under docker than normal machines with the same tests? I wasn't aware we still had that and that is quite concerning. I'd be more comfortable if we understood why we're getting failures here before declaring success. |
|
https://ci.adoptium.net/label/ci.role.test&&hw.arch.x86&&sw.os.linux&&!%28centos6%7C%7Crhel6%29/ Which docker static is a good one to compare? We do have some tests failures are related with docker ones, which should not be the issue with the os, arch platform. So I think it's fair to limit the comparison to docker static ones. |
|
jdk_lang
|
|
jdk_lang
|
test-docker-ubuntu2004-x64-2 was the one I had a clean run on fir my 22+35 tests so should be a good stable choice. |
|
jdk_lang
|
|
@smlambert Are these failures already known about? I'm just nervous that I didn't see any of them in my sanity/extended runs on jdk-22+35 (So, of course, they could be specific to 21). Also I see that the RHEL7 test was done on the latest build ( What's your feeling on this @smlambert (since you're working through them based on the comments!)? I'm inclined to suggest either running again with |
Which job/machine was that from @smlambert? The Amazon Linux 2023 x64 machine seems to have it in /usr/bin so it should have been available without any issues. |
|
Tests unstable on docker containers #2138 |
Given only one machine is configured, test-docker-amazon2023-x64-1, its that one. Output from the test: |
To be clear, we've been running the tests on an amazon linux x64 and an aarch64 machine.
OK yeah that's one we've seen before. I guess we don't have the upstream fix referred to in adoptium/infrastructure#2886 (comment) but the OS choosing to use a different implementation for the sleep command should definitely be considered non-blocking for the purposes of this. |
@sophia-guo It would be good to go through that list and see how many are still valid. While that lists quite a few failing tests it was last updated over two years ago and I don't think we're seeing many failures that are specific to docker containers now. |
Noting that similar to my earler comment these both seem to have been run with the |
|
I've kicked off an extended.openjdk run with Also kicked off some re-runs including jdk_lang and jdk_security from the AL2023 sanity run sanity run (with the sleep / coreutils issue) on aarch64 but I don't expect this to show anything too useful (I initially thought that Fedora39 would be the same, but I was mistaken and it uses a normal coreutils package for
On AL2023 aarch64 machine: On CentOS8: |
|
Keeping the aarch64 machine busy with re-runs with
NOTE: The intermittent finalizer test on JDK21 extended.openjdk is a |
Noting that this is not an OutOfMemoryError causing the test to fail (Therefore is unlikely to be related to insufficient resources on the machine) but is a failure due to the absence of such an error. EDIT: I've run Grinders 9969 through 9974 on several machines, some docker, some non-docker, some with many iterations, and also grinders 4323-4325 on the Temurin Compliance machines with the same release build and v1.0.1-release material. 100% failure rate on this test so it's not an intermittent failure. NOTE: For some reason the ones after 9974 don't seem to have a "Test Results" link but from the console log they all failed too EDIT2: The ones after 9974, and the ones on the TC server were run with the |
Unclear as I don't have the job any more, but it did not fail in the Grinder from the v1.0.1-release branch run which was also using "releases" so I would expect that setup to have automatically pulled the correct things across. Seems most likely another case of incorrect test material, although it may be worth a multi-iteration x64 Grinder to see if it's intermittent so:
|
autotest.shTesting on two other relatively new distributions: Fedora 39 (Passing ok) and Ubuntu 24.04 - (Passing ok - sophiaguo) sophia- Testing autotest.sh only on test-docker-ubuntu2204-armv8l-2 https://ci.adoptium.net/job/Grinder/10006/ - [100 iteration] - (Failed)
TestCurvesTestECDSA``` [2024-05-14T05:05:44.033Z] PATH=/bin:/usr/bin:/usr/sbin \ [2024-05-14T05:05:44.033Z] /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/javac \ [2024-05-14T05:05:44.033Z] -J-ea \ [2024-05-14T05:05:44.033Z] -J-esa \ [2024-05-14T05:05:44.033Z] -J-Xmx512m \ [2024-05-14T05:05:44.033Z] -J-XX:+UseCompressedOops \ [2024-05-14T05:05:44.033Z] -J-Dtest.vm.opts='-ea -esa -Xmx512m -XX:+UseCompressedOops' \ [2024-05-14T05:05:44.033Z] -J-Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-XX:+UseCompressedOops' \ [2024-05-14T05:05:44.033Z] -J-Dtest.compiler.opts= \ [2024-05-14T05:05:44.033Z] -J-Dtest.java.opts= \ [2024-05-14T05:05:44.034Z] -J-Dtest.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \ [2024-05-14T05:05:44.034Z] -J-Dcompile.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \ [2024-05-14T05:05:44.034Z] -J-Dtest.timeout.factor=8.0 \ [2024-05-14T05:05:44.034Z] -J-Dtest.nativepath=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \ [2024-05-14T05:05:44.034Z] -J-Dtest.root=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk \ [2024-05-14T05:05:44.034Z] -J-Dtest.name=sun/security/pkcs11/ec/TestECDSA.java \ [2024-05-14T05:05:44.034Z] -J-Dtest.file=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec/TestECDSA.java \ [2024-05-14T05:05:44.034Z] -J-Dtest.src=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec \ [2024-05-14T05:05:44.034Z] -J-Dtest.src.path=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11 \ [2024-05-14T05:05:44.034Z] -J-Dtest.classes=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d \ [2024-05-14T05:05:44.034Z] -J-Dtest.class.path=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11 \ [2024-05-14T05:05:44.034Z] -J-Dtest.class.path.prefix=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11 \ [2024-05-14T05:05:44.034Z] -J-Dtest.modules=jdk.crypto.cryptoki \ [2024-05-14T05:05:44.034Z] --add-modules jdk.crypto.cryptoki \ [2024-05-14T05:05:44.034Z] -d /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d \ [2024-05-14T05:05:44.034Z] -sourcepath /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11 \ [2024-05-14T05:05:44.034Z] -classpath /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11 /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec/TestECDSA.java [2024-05-14T05:05:44.034Z] direct: [2024-05-14T05:05:44.034Z] /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/PKCS11Test.java:780: warning: [removal] Policy in java.security has been deprecated and marked for removal [2024-05-14T05:05:44.034Z] Policy.setPolicy(null); // Clear the policy created by JIB if any [2024-05-14T05:05:44.034Z] ^ [2024-05-14T05:05:44.034Z] /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/PKCS11Test.java:799: warning: [removal] SecurityManager in java.lang has been deprecated and marked for removal [2024-05-14T05:05:44.034Z] System.setSecurityManager(new SecurityManager()); [2024-05-14T05:05:44.034Z] ^ [2024-05-14T05:05:44.034Z] /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/PKCS11Test.java:799: warning: [removal] setSecurityManager(SecurityManager) in System has been deprecated and marked for removal [2024-05-14T05:05:44.034Z] System.setSecurityManager(new SecurityManager()); [2024-05-14T05:05:44.034Z] ^ [2024-05-14T05:05:44.034Z] /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/PKCS11Test.java:811: warning: [removal] setSecurityManager(SecurityManager) in System has been deprecated and marked for removal [2024-05-14T05:05:44.034Z] System.setSecurityManager(null); [2024-05-14T05:05:44.034Z] ^ [2024-05-14T05:05:44.034Z] Note: /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib/jdk/test/lib/artifacts/ArtifactResolver.java uses or overrides a deprecated API. [2024-05-14T05:05:44.034Z] Note: Recompile with -Xlint:deprecation for details. [2024-05-14T05:05:44.034Z] Note: /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib/jdk/test/lib/artifacts/JibArtifactManager.java uses unchecked or unsafe operations. [2024-05-14T05:05:44.034Z] Note: Recompile with -Xlint:unchecked for details. [2024-05-14T05:05:44.034Z] 4 warnings [2024-05-14T05:05:44.034Z] [2024-05-14T05:05:44.034Z] ACTION: main -- Failed. Execution failed: `main' threw exception: java.lang.Exception: Sign/verify 3 failed [2024-05-14T05:05:44.034Z] REASON: User specified action: run main/othervm TestECDSA ``` |
6 tasks
|
Also triggered x64 jdk17, 8 - waiting for machine , need to confirm if it's necessary? @jiekang
|
|
Summary for amazon x64
jdk8, jdk17, jdk11 waiting for result #5300 (comment) Summary for aarch64: #5300 (comment) |
|
ref #5300 (comment) We're seeing sun/security/krb5/auto/rcache_usemd5.sh fail on other platforms, jdk8 arm32 linux adoptium/infrastructure#3043 (comment) |
|
sun/security/krb5/auto/rcache_usemd5.sh has a related test fix that was backported to JDK 8 last month. It will be in the July CPU. You may be able to ignore it for now. openjdk/jdk8u-dev@6b53212 |
|
Latest status
jdk 17:
jdk11:
jdk 8:
sun/security/pkcs11/ec/TestCurves is common issue for jdk17, 11 and 8 with aarch64 and linux64, will open an issue to investigate sun/security/tools/keytool/autotest.sh is common issue for jdk8 linux64 and aarch64. All failed tests are jdk_security tests. @jiekang |
|
Repeating the same failures using Corretto jdk jdk17 linux x64 sun/security/pkcs11/ec/TestCurves.java https://ci.adoptium.net/job/Grinder/10053/console - 10 out 10 failed jdk11 linux x64 jdk_security3 https://ci.adoptium.net/job/Grinder/10055/console jdk8 linux x64 jdk_security3 https://ci.adoptium.net/job/Grinder/10056/console All using Sophia's v1.0.1-release branch |
|
Confirmed Corretto has same failures with jdk_security3. |
|
Thank you @sophia-guo and @Haroon-Khel ! |
|
In https://ci.adoptium.net/job/Grinder/10053/testReport/, jdk17 corretto linux x64, we have 1 pass out of 10 iterations sun/security/pkcs11/ec/TestECDSA.java.TestECDSA But still has a standard error of java.lang.Exception: Sign/verify 3 failed Not sure why the test is passed sun/security/pkcs11/ec/TestCurves.java.TestCurves |
|
Without prior expertise, this NSS fuzziness could probably be a multi-week investigation of it's own :| |
|
Those security testcases mention the recommendation for using the latest NSS version, given https://wiki.mozilla.org/NSS:Release_Versions that would be 3.100 and that jives with seeing those tests pass on OS's with v3.98. If Amazon Linux 2023 can only be updated to 3.90, we may consider those failing tests as not necessarily valid for this platform. As long as this is documented and understood (for Zeke when in discussion with others), I think we can/should mark this as acceptable. Ideally the upstream test code would include a check that said, if NSS version is less than X, then skip or mark as invalid against lesser versions. |
|
Triage done, close it. |
|
Some extra notes:
It's possible that RHEL has some backports of fixes from later libnss3 versions, but it does mean there is at least some doubt as to whether 3.90 in AL2023 is the cause of the failures. |
|
Ref libnss3:
While this would suggest we're ok for this particular test, we know that the master branch is not suitable for testing the GA level as per the original attempts to run from the aqa-tests
running infotest JDK17 on multiple machines as grinder 10235 through 10239 - all passed extended.openjdk other than java/lang/ProcessBuilder/PipelineLeaksFD.java.PipelineLeaksFD so the sleep problem does not occur there. It is a known problem with CentOS and UBI too. And x64:
[*] - The multiple runs which show a 15% pass rate on the AL2023/aarch64 machine used the command in the twisty below, and one passing and one failing run output is shown below: Command line usedPassing outputFailing outputBy comparison the
Based on earlier comments about failures in JDK11/x64 I'm also trying the following runs:
|
Summary for aarch64Most of the failures seen on the branch in the PKCS11 tests do not occur on the latest nightly and latest test code, indicating that changes in the test code may have resolved some of the issues after the last release which we have been testing on. There are still failures in Summary for x64testJcmdDump had failed but that test seems to show intermittent failures and is not believed to be specific to Amazon. The InfoTest/sleep failure is as mentioned in the aarch64 summary above. JDK8 e/o is also showing a number of pkcs11 and nio (character set related) failures which are not showing on aarch64. Re-running 50 iterations of all the failures on three static docker containers on the same host: AL2023, Ubuntu2204-7, Debian12 (pkcs failures only on AL2023, nio on all three) ... running 5000 iterations over the weekend on AL2023 at https://ci.adoptium.net/job/Grinder/10271/) ... running 5000 iterations over the weekend on AL2023 at https://ci.adoptium.net/job/Grinder/10271/) Why the differences?The pkcs11 failures on JDK8/x64 do not occur on aarch64 because there is code that explicitly skips it on Linux/aarch64 - it reports Related issues: |
This was referenced Jun 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
adoptium/infrastructure#3041 (comment)
adoptium/infrastructure#3041 (comment)
The text was updated successfully, but these errors were encountered: