Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal JITServer test failures #14538

Open
cjjdespres opened this issue Feb 17, 2022 · 165 comments
Open

Internal JITServer test failures #14538

cjjdespres opened this issue Feb 17, 2022 · 165 comments
Labels
comp:jitserver Artifacts related to JIT-as-a-Service project

Comments

@cjjdespres
Copy link
Contributor

cjjdespres commented Feb 17, 2022

This thread collects various failures in internal JITServer tests that have been observed recently, for platforms other than Z. I'll be going through them to see if they can be reproduced without JITServer enabled.

Unable to reproduce

  • jdk_util_1
    In: job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/271/
    Comment here. The sub-test LoggingDeadlock2 failed with "Agent 7 timed out with a timeout of 960 seconds".

  • SharedClasses.SCM01.MultiCL_1
    In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/
    Comment here, here. Segfaults.
    Unable to reproduce in Grinder.

  • MiniMix_5m_0
    In: job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/, job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/
    Comment here, here. Segfaults.
    Unable to reproduce in Grinder.

  • DBBLoadTest_5m_0
    In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/
    Comment here, here. Segfaults.
    Unable to reproduce in Grinder.

  • HCRLateAttachWorkload_previewEnabled_0
    In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/
    Comment here, here. Segfaults.
    Unable to reproduce in Grinder.

  • ObjectTreeLoadTest_5m_1
    In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/
    Comment here, here. Segfaults.
    Unable to reproduce in Grinder.

  • testOSMXBeanRemote_1, TestManagementAgent_0
    In: job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/290/
    Comment here. Tests fail to connect to remote server that was created.

  • LambdaLoadTest_CS_5m_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/
    Comment here. Segfault.

  • MathLoadTest_bigdecimal_5m_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/
    Comment here. "org.apache.logging.log4j.message.DefaultFlowMessageFactory not found"

  • ParallelStreamsLoadTest_J9_1, MauveMultiThrdLoad_5m_0, CpMp_MP_0, AutoMod1_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/
    Comment here. Path/URI exceptions, string index out of bounds errors.

  • cmdLineTester_jvmtitests_hcr_OSRG_nongold_4
    In: job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/278/
    Comment here. "Exception in thread "Attach API initializer" java.lang.ExceptionInInitializerError".

  • testSCCMLTests1_openj9_1
    In: job/Test_openjdk8_j9_sanity.functional_ppc64le_linux_jit_Personal/840/
    Comment here. Various success and failure conditions not found.

  • cmdLineTester_SCHelperCompatTests_unix_0
    In: job/Test_openjdk8_j9_sanity.functional_ppc64le_linux_jit_Personal/840/
    Comment here. Various success and failure conditions not found.

  • LambdaLoadTest_special_J9_5m_9
    In: job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/195/
    Comment here. Segfault.

  • DaaLoadTest_daa2_CS_5m_0, MathLoadTest_autosimd_CS_5m_0, MathLoadTest_bigdecimal_CS_5m_0, CpMp_CpMp_0, SLTest_0, PatMod_PlatMod_0, PatMod_AppMod_0, PatMod_Unex_0, PatMod_Adv_0, PatModImg_AppMod_0, UpgModPath_Exp_0, UpgModPath_ExpImg_0, UpgModPath_JarImg_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276
    Comment here. Segfaults

  • DaaLoadTest_daa1_5m_1, LambdaLoadTest_J9_5m_1, AutoMod_Impl2_0, NioLoadTest_5m_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276
    Comment here. String index out of bounds errors.

  • MauveMultiThrdLoad_5m_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276
    Comment here. "Unable to parse --add-opens /: java.base/java.lang"

  • DaaLoadTest_daa3_CS_5m_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276
    Comment here, here. Segfault.

  • ParallelStreamsLoadTest_CS_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276
    Comment here, here. Segfault.

  • CpMp3_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276
    Comment here, here. Segfault.

  • AutoMod2_0
    In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276
    Comment here, here. Segfault.

  • cmdLineTester_classesdbgddrext_1
    In: job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/261/
    Console log here. The error encountered is JVMSHRC023E Cache does not exist.

  • testSCCacheManagement_0, cmdLineTester_classesdbgddrext_1, cmdLineTester_fastClassHashTable_0, cmdLineTester_javaAssertions_0, cmdLineTester_decompilationTests_nongold_3, cmdLineTester_lockWordAlignment_Object_Standard_0, cmdLineTester_lockWordAlignment_Object_iii_0, cmdLineTester_lockWordAlignment_Object_d_0, cmdLineTester_modularityddrtests17_1, cmdLineTester_shareClassesBadStackMap_0, testSCCMLTests3_1, testSCCMLTests4_1, testSCCMLTests6_1, testSCCMLSnapshot_1, testSCCMLAotMethodOperation_1, cmdLineTester_shrcdbgddrext_1, cmdLineTest_stackSizeInfoTest_0, cmdLineTester_verbosetest_5, cmdLineTester_vmRuntimeState_0, cmdLineTester_xlogTests_0, StackWalkerTestJava10_0, JCL_TEST_Java-Lang-Invoke_0, TestAttachAPIEnabling_0, TestSunAttachClasses_0, gcNotificationTest_Optthruput_1, gcNotificationTest_OptAvgpause_1, gcNotificationTest_Balanced_0, gcNotificationTest_Balanced_1, gcNotificationTest_Metronome_1, ContendedFieldsTests_90_0, threadMXBeanTestSuite2_2, threadMXBeanTimedParkTest_2, threadMXBeanTimedParkTest_3, memoryCategories_0, HashConsistency_0, jniOnLoadExceptions_1, badUTF8inJNI_1, badUTF8inJNI_2, testvmcheck_3, testvmcheck_6, TestSoftMxCallBackOOM_1, TestSoftMxCallBackExtend_1, MethodVisibilityTests_0, UnreflectTests_0
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/
    Comment here. Segfaults.

  • SecurityTests_0, CDSAdaptorTest_0, cmdLineTester_bootStrapStaticVerify_0, cmdLineTester_EnableAssertionStatusTest_0, cmdLineTest_J9test_extended_0, cmdLineTest_J9test_consoleUpTo17_0, cmdLineTester_dumpromclasstests_0, cmdLineTester_getCallerClassTests_SE80_0, cmdLineTester_ignoreunrecognizedvmoptions_0, cmdLineTester_jep178_staticLinking_SE80_0, cmdLineTester_fieldwatchtests_0
    In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/
    Comment here. Segfaults.

  • DaaLoadTest_all_5m_1, SharedClasses.SCM01.SingleCL_1, SharedClasses.SCM01.MultiThread_1, SharedClasses.SCM23.MultiThreadMultiCL_1
    In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/
    Comment here. Segfaults.

  • SecurityTests_0, cmdLineTester_dumpromclasstests_0, cmdLineTester_locales_0, cmdLineTester_lockWordAlignment_Object_i_0, testDDRExtJunit_MonitorsAndDeadlock2_1, testDDRExtJunit_MonitorsAndDeadlock3_1, testDDRExtJunit_MonitorsAndDeadlock6_1, TestAttachAPI_0, TestAttachErrorHandling_0, gcNotificationTest_GenCon_0, threadMXBeanTimersTest_3, HashPerformance_0, ContendedFieldsTests_90_1, ContendedFieldsTests_90_3
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/
    Comment here. Null pointer exceptions.

  • StackWalkerTest_1
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/
    Comment here. "Exception in thread "Attach API initializer" java.lang.InternalError: not invoker: invokeExact"

  • MathLoadTest_all_5m_0
    In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/
    Comment here. "java.lang.NoClassDefFoundError: org.apache.logging.log4j.message.DefaultFlowMessageFactory"

  • cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2
    In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/
    Comment here. "Error: Could not find or load main class MainTester"

  • DaaLoadTest_daa1_special_5m_17, DaaLoadTest_daa3_special_5m_16, LambdaLoadTest_special_J9_5m_16, MathLoadTest_all_special_5m_16, MathLoadTest_all_special_5m_21, MathLoadTest_autosimd_special_5m_16, ClassLoadingTest_special_5m_15, ClassLoadingTest_special_5m_22,
    In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/
    Comment here. Segfaults.

  • DaaLoadTest_daa1_special_5m_18, DaaLoadTest_daa1_special_5m_19, DaaLoadTest_daa1_special_5m_25, DaaLoadTest_daa1_special_5m_26, DaaLoadTest_daa1_special_5m_27, DaaLoadTest_daa2_special_5m_14, DaaLoadTest_daa2_special_5m_19, DaaLoadTest_daa3_special_5m_14, DaaLoadTest_daa3_special_5m_19, DaaLoadTest_daa3_special_5m_25, DaaLoadTest_all_special_5m_23, DaaLoadTest_all_special_5m_26, LambdaLoadTest_special_J9_5m_25, ParallelStreamsLoadTest_special_J9_17, ParallelStreamsLoadTest_special_J9_18, MathLoadTest_all_special_5m_24, MathLoadTest_autosimd_special_5m_23, MathLoadTest_autosimd_special_5m_24, MathLoadTest_bigdecimal_special_5m_23, MathLoadTest_bigdecimal_special_5m_24, ClassLoadingTest_special_5m_18
    In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/
    Comment here. "Null property: java.home".

  • DaaLoadTest_daa2_special_5m_22, DaaLoadTest_all_special_5m_22, MathLoadTest_autosimd_special_5m_22, LambdaLoadTest_special_J9_5m_22, LambdaLoadTest_special_J9_5m_29, MathLoadTest_all_special_5m_28, ClassLoadingTest_special_5m_29
    In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/
    Comment here. Various path/URI errors, string index out of bounds errors.

  • DaaLoadTest_daa1_special_5m_21
    In: job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/192/
    Comment here. Segfault.

  • SC_Softmx_UpDown_1, SC_Softmx_Increase_1
    In: job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/
    Comment here. Unexpected crash, no other detail in the log.

  • MiniMix_aot_5m_0, SC_Softmx_JitAot_Linux_1, ConcurrentLoadTest_5m_0, LockingLoadTest_0, UtilLoadTest_5m_0, SharedClasses.SCM23.SingleCL_1
    In: job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/
    Comment here. Segfaults.

  • DaaLoadTest_all_CS_5m_0, MathLoadTest_all_CS_5m_0, SharedClasses.SCM01.SingleCL_1
    In: job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/
    Comment here. Null pointer exceptions.

  • MiniMix_aot_5m_0
    In: job/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal/274/
    Comment here. An exception - the relevant log line is too corrupted to tell what kind.

  • ParallelStreamsLoadTest_special_J9_13
    In: job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/
    Comment here, here. "Null property: java.home".

  • DaaLoadTest_daa3_special_5m_13, DaaLoadTest_daa3_special_5m_17, DaaLoadTest_all_special_5m_16, DaaLoadTest_all_special_5m_21, ParallelStreamsLoadTest_special_J9_12
    In: job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/193/
    Comment here. Segfaults.

  • DaaLoadTest_all_special_5m_29, DaaLoadTest_daa1_special_5m_21
    In: job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/193/
    Comment here. "org.apache.logging.log4j.message.DefaultFlowMessageFactory not found"

  • SyntheticGCWorkload_DoubleMap_J9_0
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/, job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/
    Comment here, here. Segfaults.
    Unable to reproduce in Grinder.

  • cmdLineTester_defaultLazySymbolResolution_1
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/, job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/
    Comment here, here. Segfaults.
    Unable to reproduce in Grinder.

  • cmdLineTester_hangTest_0
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/, job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/
    Comment here, here. Segfaults.
    Unable to reproduce in Grinder.

  • jdk_lang_1
    In: job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/280/
    Comment here. Some sort of thrown exception during the test, java.lang.Throwable: x[20] =0.

  • testSCCacheManagement_0
    In: `job/Test_openjdk17_j9_extended.functional_ppc64le_linux_jit_Personal/199/.
    Comment here. "Invalid number of cache found".

  • shrtest_linux_SE80_1
    In: job/Test_openjdk8_j9_sanity.functional_x86-64_linux_jit_Personal/918/
    Comment here. "Failure to unlink directory". May be a system error.

  • jdk_lang_0
    In: job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/310/
    Comment here. Test time out.

  • AutoMod2_0, ClassLoadingTest_special_5m_27, ClassLoadingTest_special_5m_29, CpMpModJar_0, CpMp_MP_0, DaaLoadTest_all_special_5m_27, DaaLoadTest_all_special_5m_29, DaaLoadTest_daa1_special_5m_20, DaaLoadTest_daa1_special_5m_25, DaaLoadTest_daa2_5m_1, DaaLoadTest_daa2_special_5m_29, DaaLoadTest_daa3_CS_5m_0, DaaLoadTest_daa3_special_5m_26, DaaLoadTest_daa3_special_5m_28, InternalAPIs_0, Jlink_AddMLimitM_0, Jlink_ReqMod_0, LambdaLoadTest_J9_5m_1, MathLoadTest_all_special_5m_24, MathLoadTest_autosimd_special_5m_21, MathLoadTest_autosimd_special_5m_28, MathLoadTest_bigdecimal_5m_0, MathLoadTest_bigdecimal_CS_5m_0, MathLoadTest_bigdecimal_special_5m_24, MathLoadTest_bigdecimal_special_5m_26, MathLoadTest_bigdecimal_special_5m_28, MauveSingleInvocLoad_J9_5m_1, PatModImg_AppMod_0, PatMod_AppMod_0, SLTest_0, UpgModPath_Exp_0, UpgModPath_JarImg_0, jdk11_tier1_buffer_0, jdk11_tier1_cipher_0, jdk11_tier1_iso8859_0, jdk_foreign_0, jdk_lang_0, jdk_lang_VarHandleTest_j9_0, jdk_lang_native_0, jdk_math_0, jdk_util_0
    In: job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/250/, job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/247/, job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/249/
    Comments here, here, here. Assertion failure, ((sp <= endOfStack)), also manifesting as a segfault related to stack walking.

  • cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_0
    In: job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/317/
    Comment here. A couple of segfaults.

  • jdk_util_0
    In: job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/334/, job/Test_openjdk8_j9_sanity.openjdk_ppc64le_linux_jit_Personal/336/
    Comment here. "java.lang.RuntimeException: Incorrect hashCode computation."

  • LambdaLoadTest_special_J9_5m_8
    In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/366/
    Comment here. A segfault.

  • testSCCMLSoftmx_1
    In: /job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/423
    Comment here. Test timeout.

  • LambdaLoadTest_J9_5m_0
    In: /job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/321
    Comment here. Assertion failure "block->next() == NULL".

  • jdk11_tier1_cipher_0, jdk11_tier1_cipher_1
    In: /job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/335
    Comment here. Possibly some kind of FIPS-related error, and also test output mismatches.

  • Jep389Tests_testClinkerFfi_UpCall_0
    In: /job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/422
    Comment here. An assertion failure.

  • jdk_util_0
    In: /job/Test_openjdk8_j9_sanity.openjdk_ppc64le_linux_jit_Personal/573
    Comment here. A deadlock occurred.

  • LambdaLoadTest_special_J9_5m_12. Segfault. Comment here.

Resolved

  • testDefaultDisclaimMemory_3
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/185/
    Console log here. Crash - assertion failure.
    Appears to be a duplicate of another issue.

  • ParallelStreamsLoadTest_special_J9_11
    In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/254/
    Console log here. Crash - assertion failure; segmentation error.
    Reproduced in view/Test_grinder/job/Grinder/21194/ without JITAAS with a 10% failure rate.

  • testSoftMxLocal_LP4k_3,testSoftMxDisclaimMemory_LP4k_3,testDefaultDisclaimMemory_3
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/188/
    Comment here. Crash - assertion failure; segmentation error.
    Appears to be a duplicate of another issue.

  • jdk_lang_0,jdk_lang_1
    In: job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/186/, job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/265/, job/Test_openjdk17_j9_sanity.openjdk_ppc64le_linux_jit_Personal/180/
    Console log here.
    The failure seems similar to that reported in another issue. Fixed by this commit.

  • cmdLineTester_pltest_j9sig_ext_0, testSCCMLTests2_0, testSCCMLTests4_0, testSCCMLTests4_1, cmdLineTester_shrcdbgddrext_1, testDDRExt_SharedClasses_0, testDDRExtJunit_StackMap_0, testDDRExtJunit_MonitorsAndDeadlock1_1, testDDRExtJunit_MonitorsAndDeadlock2_0, testDDRExtJunit_MonitorsAndDeadlock2_1, testDDRExtJunit_MonitorsAndDeadlock3_0, testDDRExtJunit_MonitorsAndDeadlock5_0, testDDRExtJunit_MonitorsAndDeadlock5_1, TestJcmd_0, testOpenJ9DiagnosticsMXBean_0
    In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/268/
    Some logs here. All the failures seem to relate to the failure of a core dump to be created. Possibly all are related to Test-extended.functional-JDK11-linux_x86-64 testSCCMLTests2_0 failures #5310.

  • cmdLineTester_getPid_0
    In: job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/191/, job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/185/
    Console log here. Test failure [OUT] FAIL: RuntimeMXBean.getPID() returned 0 instead of 15990794.
    Duplicate of cmdLineTester_getPid_0 RuntimeMXBean.getPID() returned 0 instead of #13085.

  • SharedClassesAPI_1
    In: job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/265/
    Console log here. Test failure Incorrect number of caches found. Expected at least 4, found 2.
    Duplicate of SharedClassesAPI_1_FAILED - **FAILED** at step 71 (Verify Classes Caches using JVMTI) #13239.

  • Various TestJlm* and Test IBMJlm* tests.
    In: job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/276/, job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/275/, job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/191/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/203/
    Console log here, and here. All are either a crash while attempting to connect to a VM, or a failure of a VM to start. Haven't attempted to reproduce, but Marius here says that they've been happening for a while, probably aren't JITServer related, and are low priority.

  • cmdLineTester_fieldwatchtests_0
    In: job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/271/
    Console log and comments here. Test failure Invalid number of field reports detected: 0. Expected: 5.
    I've reproduced the failure with JITAAS, but not without it. Tracked by cmdLineTester_fieldwatchtests_0: unexpected number of fields #14633.

  • jdk_lang_1
    In: job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/267/
    Comment here. Segfault.
    Tracked by jdk_lang_0_FAILED jdk/internal/math/FloatingDecimal/TestFloatingDecimal.java - Segmentation error vmState=0x00000000 #14182, not JITServer-specific.

  • cmdLineTester_jvmtitests_debug_3, cmdLineTester_jvmtitests_debug_5, cmdLineTester_jvmtitests_debug_7, cmdLineTester_jvmtitests_debug_8, cmdLineTester_jvmtitests_debug_openj9_none_SCC_11, cmdLineTester_jython_6
    In: job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/899/, job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/902/
    Comment here, here. Segfaults.
    Reproduction: with JITAAS, job/Grinder/21630/, 418/918 failures; without JITAAS, job/Grinder/21638/, 0/1170 failures.
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • cmdLineTester_jvmtitests_debug_10, cmdLineTester_jvmtitests_debug_11, cmdLineTester_jvmtitests_debug_openj9_none_SCC_3, cmdLineTester_jvmtitests_debug_openj9_none_SCC_5, cmdLineTester_jvmtitests_debug_openj9_none_SCC_7, cmdLineTester_jvmtitests_debug_openj9_none_SCC_8, cmdLineTester_jvmtitests_debug_openj9_none_SCC_10
    In: job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/898/, job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/899/, job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/902/
    Comment here, here. Segfaults.
    Each has been reproduced with JITAAS and none without (see comment for details).
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • TestFileLocking_SE80_0
    In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/281/
    Comment here. Segfault.
    Reproduced with 4/50 failure rate in job/Grinder/21553/ with JITAAS, unable to reproduce without it.
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • testDefaultDisclaimMemory_2
    In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/281/, job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/285/
    Comment here. Segfault.
    Reproduction: with JITAAS, job/Grinder/21631/, 26/195 failures); without JITAAS, job/Grinder/21633/, 0/195 failures.
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • testSoftMxDisclaimMemory_2
    In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/289/
    Comment here. Segfault.
    Reproduction: with JITAAS, job/Grinder/21673/, 6/195 failures; without JITAAS, job/Grinder/21674/, 0/195 failures.
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • TestArrayCopy_3
    In: job/Test_openjdk8_j9_sanity.functional_x86-64_linux_jit_Personal/895/
    Comment here. Segfault.
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • jdk_util_0
    In: job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/213/.
    Comment here. Segfault.
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • MathLoadTest_autosimd_CS_5m_0
    In job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/292/
    Comment here. Test output did not match expected value.
    Reproduction: 1/80 failures in job/Grinder/21725/, 0/200 failures in job/Grinder/21737/. Tracked by Internal test failures in MathLoadTest_autosimd #14755.

  • MathLoadTest_autosimd_special_5m_29
    In: job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/213/
    Comment here. Test output did not match expected value.
    Tracked by Internal test failures in MathLoadTest_autosimd #14755.

  • jdk_foreign_0, jdk_foreign_1
    In: job/Test_openjdk17_j9_sanity.openjdk_ppc64le_linux_jit_Personal/217/.
    Comment here. "Method TestUnsupportedPlatform.testNoInitialization()[pri:0, instance:TestUnsupportedPlatform@8fe597f0] should have thrown an exception of type class java.lang.ExceptionInInitializerError".
    Tracked by OpenJDK java/foreign/TestUnsupportedPlatform failing on jdk17, jdk18 aix, plinux, zlinux #14828.

  • MiniMix_3h_0
    In: job/Test_openjdk8_j9_special.system_x86-64_linux_jit_Personal/304/
    Comment here. "Inconsistency detected by ld.so".
    Probably a glibc issue.

  • NioLoadTest_5m_0
    In: job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/232/
    Comment here. Segmentation error.
    Appears to be A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • jdk_lang_0, jdk_util_0, jdk_util_1, MathLoadTest_all_CS_5m_0
    In: job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/233/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/235/, job/Test_openjdk8_j9_sanity.openjdk_x86-64_linux_jit_Personal/306/, job/Test_openjdk11_j9_extended.system_x86-64_linux_jit_Personal/313/
    Comment here. Various math test failures (test output mismatches, stack overflow errors).
    Tracked by Math failures and stack overflows #14864.

  • DaaLoadTest_daa2_CS_5m_0
    In: job/Test_openjdk8_j9_sanity.system_ppc64le_linux_jit_Personal/309/
    Comment here. Segfault.
    Tracked by JDK8 DaaLoadTest_daa2_ConcurrentScavenge_0_FAILED : ASSERTION FAILED ** at OwnableSynchronizerObjectBuffer.cpp:100 ((false)) #10489.

  • testSCCMLAotMethodOperation_1
    In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/315/
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • testvmcheck_3
    In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/316/
    Tracked by A cluster of segfaults in internal functional_x86-64 tests with JITServer #14704.

  • TestJps_0
    In: job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/317/
    "Missing argument: foo".
    Looks like it's tracked by TestJps_SE80_0 testComposedArguments Missing argument: foo #11509.

  • threadMXBeanTimedParkTest_0, threadMXBeanTimedParkTest_1
    In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/246/
    Comment here. Assertion failure - wait time is higher than expected.
    Tracked by threadMXBeanTimedParkTest Blocked time is lower than expected #14470.

  • jdk_lang_0
    In: job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/237/
    Comment here. "java/nio/charset/CoderMalfunctionError: java.lang.IllegalArgumentException: newPosition > limit".
    I think this may be an instance of java.nio.charset.CoderMalfunctionError: IllegalArgumentException: newPosition > limit: (2 > 1) & NPE Cannot invoke "java.nio.Buffer.arrayOffset()" because "dst" is null #14654.

  • cmdLineTest_sigabrtHandlingTest_0
    In: job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/256/
    Comment here. A java core failed to be created.
    Tracked by cmdLineTest_sigabrtHandlingTest_0 Default abort testing hang requesting java dump javacore #11979.

  • TestSoftMxCallBackOOM_1, JCL_Test_OutOfMemoryError_1
    In: job/Test_openjdk17_j9_extended.functional_ppc64le_linux_jit_Personal/280, job/Test_openjdk11_j9_sanity.functional_ppc64le_linux_jit_Personal/939/
    Comment here. Assertion failure. Fixed after the reversion of Fix halted thread being allowed to continue execution #15201.

  • testRASAPI_0
    In: job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/367/
    Comment here. Dump couldn't be written to file.
    Tracked by testRASAPI_0_FAILED due to The (dump) file exists. #11800.

  • cmdLineTester_GCCheck_1, cmdLineTester_GCCheck_2, cmdLineTester_classesdbgddrext_0, cmdLineTester_modularityddrtests11_0, cmdLineTester_shrcdbgddrext_1, testRASAPI_0
    In: job/Test_openjdk8_j9_sanity.functional_x86-64_linux_jit_Personal/983/, job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/389/
    Comment here. Seems to be some kind of test setup failure.

  • DaaLoadTest_daa1_special_5m_20
    In: /job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/300/
    Comment here. Possible hang detected.
    Tracked by DaaLoadTest_daa1 tests - **FAILED** Process DLT has hung #13477.

  • jdk_lang_1
    In: job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/376/
    Comment here. Segfault.
    Tracked by jdk19 OpenJDK crash vmState=0x00000000 #15321.

  • ConcurrentLoadTest_5m_1
    In: job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/375/
    Comment here. Tracked by ConcurrentLoadTest or MiniMix testStress IllegalStateException #8493.

  • ClassLoadingTest_CS_5m_0, DaaLoadTest_all_CS_5m_0, DaaLoadTest_daa1_CS_5m_0, DaaLoadTest_daa2_CS_5m_0, DaaLoadTest_daa3_CS_5m_0, HCRLateAttachWorkload_0, HCRLateAttachWorkload_1, HCRLateAttachWorkload_previewEnabled_0, HCRLateAttachWorkload_previewEnabled_1, LambdaLoadTest_CS_5m_0, MathLoadTest_all_CS_5m_0, MathLoadTest_autosimd_CS_5m_0, MathLoadTest_bigdecimal_CS_5m_0, ParallelStreamsLoadTest_CS_0
    In: /job/Test_openjdk11_j9_extended.system_x86-64_linux_jit_Personal/386, /job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/307/, /job/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal/384/, /job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/379/, /job/Test_openjdk8_j9_sanity.system_ppc64le_linux_jit_Personal/380/, /job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/293/, /job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/293/
    Comment here. Assertion failed: _methodInfo->getMethodInfo() == reinterpret_cast<TR_OpaqueMethodBlock *>(_j9method).
    Tracked by Assertion failed: _methodInfo->getMethodInfo() == reinterpret_cast<TR_OpaqueMethodBlock *>(_j9method) #15426.

  • cmdLineTest_J9test_common_0
    In: job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/325
    Comment here. Segfault. Tracked by VMBench.FibBench with -Xlockword:mode=minimizeFootprint Segfaults Intermittently #13577.

  • LambdaLoadTest_special_J9_5m_21
    In: job/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal/405/
    Comment here. Segfault. Tracked by Crash in JIT with vmState=0x0002000f #13617.

  • ClassLoadingTest_special_5m_18, ClassLoadingTest_special_5m_19, ClassLoadingTest_special_5m_19 , ClassLoadingTest_special_5m_24, DaaLoadTest_all_special_5m_18, DaaLoadTest_all_special_5m_19, DaaLoadTest_all_special_5m_24, DaaLoadTest_daa1_special_5m_18, DaaLoadTest_daa1_special_5m_19, DaaLoadTest_daa1_special_5m_24, DaaLoadTest_daa2_special_5m_18, DaaLoadTest_daa2_special_5m_19, DaaLoadTest_daa2_special_5m_23, DaaLoadTest_daa2_special_5m_24, DaaLoadTest_daa3_special_5m_24, LambdaLoadTest_special_J9_5m_18, LambdaLoadTest_special_J9_5m_24, MathLoadTest_all_special_5m_18, MathLoadTest_all_special_5m_19, MathLoadTest_all_special_5m_24, MathLoadTest_autosimd_special_5m_18, MathLoadTest_autosimd_special_5m_19, MathLoadTest_bigdecimal_special_5m_18, MathLoadTest_bigdecimal_special_5m_19, MathLoadTest_bigdecimal_special_5m_24, ParallelStreamsLoadTest_special_J9_18
    In: /job/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal/411, /job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/318
    Comment here. Segfault.
    Tracked by Segmentation error vmState=0x0005ffff in internal special.system tests with JITServer #15601.

  • DaaLoadTest_all_special_5m_27, DaaLoadTest_daa1_special_5m_26, LambdaLoadTest_special_J9_5m_25, MathLoadTest_all_special_5m_25, MathLoadTest_all_special_5m_27, MathLoadTest_autosimd_special_5m_23, MathLoadTest_bigdecimal_special_5m_25, MathLoadTest_bigdecimal_special_5m_26, MauveMultiThrdLoad_special_5m_13, MauveMultiThrdLoad_special_5m_24, MauveMultiThrdLoad_special_5m_25
    In: /job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/341, /job/Test_openjdk8_j9_special.system_ppc64le_linux_jit_Personal/421, /job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/356, /job/Test_openjdk8_j9_special.system_ppc64le_linux_jit_Personal/420, /job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/340, /job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/421
    Comment here. Memory corruption/malloc errors.
    Tracked by JITServer special.system test failures: malloc/memory corruption errors #15751.

  • SharedClasses.SCM23.MultiThreadMultiCL_0, SharedClasses.SCM23.MultiThread_0, jdk_util_0
    In: /job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/376, /job/Test_openjdk11_j9_extended.system_x86-64_linux_jit_Personal/448, /job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/374
    Comment here. Segfault.
    Tracked by crash vmState=0x00020021 #15852.

  • cmdLineTest_J9test_common_0
    In: /job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/386
    Comment here. Segfault.
    Tracked by VMBench.FibBench with -Xlockword:mode=minimizeFootprint Segfaults Intermittently #13577.

  • jdk_util_1
    In: /job/Test_openjdk8_j9_sanity.openjdk_x86-64_linux_jit_Personal/475, /job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/478
    Comment here. Assertion failed, "guardOkForExpr: should not intern OSR guard n3908n".
    Tracked by jdk_util - Assertion failed: guardOkForExpr: should not intern OSR guard #16115.

  • DaaLoadTest_daa1_CS_5m_0, DaaLoadTest_daa3_CS_5m_0, LambdaLoadTest_CS_5m_0, MathLoadTest_bigdecimal_CS_5m_0, DaaLoadTest_daa2_CS_5m_0, ParallelStreamsLoadTest_CS_0, MathLoadTest_autosimd_CS_5m_0, ClassLoadingTest_CS_5m_0 , DaaLoadTest_all_CS_5m_0, MathLoadTest_all_CS_5m_0
    In: /job/Test_openjdk11_j9_extended.system_x86-64_linux_jit_Personal/488, /job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/415, /job/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal/484, /job/Test_openjdk11_j9_sanity.system_x86-64_linux_jit_Personal/491, /job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/416, /job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/401, /job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/484, /job/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal/486, /job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/401, /job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/487
    Comment here. Segfault.
    Tracked by Segfault in various system tests #16149.

  • TestJcmd_0, threadMXBeanTestSuite1_2, threadMXBeanTestSuite1_3, threadMXBeanTestSuite1_5, threadMXBeanTestSuite2_0, threadMXBeanTestSuite2_2, threadMXBeanTestSuite2_3, testJCMMXBeanRemote_0, threadMXBeanTestSuite1_0, threadMXBeanTestSuite1_1, threadMXBeanTestSuite1_4, threadMXBeanTestSuite1_6, threadMXBeanTestSuite1_7, threadMXBeanTestSuite2_1, testJCMMXBeanRemote_1, TestJlmLocal_0, TestJlmLocal_1
    In: /job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/426, /job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/506, /job/Test_openjdk11_j9_sanity.system_x86-64_linux_jit_Personal/494, /job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/419
    Comment here. Test failures related to threading.
    Has been resolved.

  • cmdLineTester_criu_random_0
    In: /job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/480/, among others
    Comment here. The initial checkpoint does not seem to be created in the first place.
    Tracked by cmdLineTester_criu_random test failure with JITServer #16502.

  • jdk_lang_j9, on ppc64le. Null pointer exception in SealedClassesReflectionTest. Comment here. Tracked by OpenJDK java/lang/reflect/sealed_classes/SealedClassesReflectionTest NullPointerException #17254.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 17, 2022

From build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/185/, failure in testDefaultDisclaimMemory_3, with console output

testDefaultDisclaimMemory_3 Start Time: Sat Feb 12 02:28:27 2022 Epoch Time (ms): 1644661707905
variation: Mode501
JVM_OPTIONS: -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs
[IncludeExcludeTestAnnotationTransformer] [INFO] EXCLUDE_FILE environment variable: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/latest_exclude_17.txt,/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/feature_ojdkmh_exclude.txt
[IncludeExcludeTestAnnotationTransformer] [INFO] Processing exclude file: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/latest_exclude_17.txt
[IncludeExcludeTestAnnotationTransformer] [INFO] Processing exclude file: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/feature_ojdkmh_exclude.txt
...
... TestNG 6.14.2 by Cédric Beust ([email protected])
...

Invalid class in objectPtr=00007F82A63115A0
10:28:29.463 0x7f823c003a00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK17_x86-64_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:1258: ((false))
JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/12 02:28:29 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/core.20220212.022829.32526.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/core.20220212.022829.32526.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/javacore.20220212.022829.32526.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/javacore.20220212.022829.32526.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/Snap.20220212.022829.32526.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/Snap.20220212.022829.32526.0003.trc
JVMDUMP013I Processed dump event "traceassert", detail "".

testDefaultDisclaimMemory_3_FAILED

Seems to be related to #14396, and shows up without JITServer enabled.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 17, 2022

From build job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/261/, failure in cmdLineTester_classesdbgddrext_1, with excerpted console output

 ===============================================
 Running test cmdLineTester_classesdbgddrext_1 ...
 ===============================================
 cmdLineTester_classesdbgddrext_1 Start Time: Sun Feb 13 01:17:51 2022 Epoch Time (ms): 1644743871414
 variation: Mode610
 JVM_OPTIONS: -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon
 Start SysV clean up for user:all
 Clean up is finished:
 	Shared Memory Segments Removed:0
 	Shared Semaphores Removed:1
 Parsing exclude list...
   add excludes for [all]
   add excludes for [linux_ppc-64]
 DONE in 0 ms.

 No tests are excluded.

 *** Starting test suite: J9 shared cache dbgext Tests ***
 Testing: Create new shared cache with partition
 Test start time: 2022/02/13 01:17:52 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xaot:forceaot,count=0,disableAsyncCompilation -Xshareclasses:name=shareddbgext,reset,modified=mod1
 Time spent starting: 90 milliseconds
 Time spent executing: 10087 milliseconds
 Test result: PASSED

 Executing command: rm -f j9core.dmp

 Testing: Create core file
 Test start time: 2022/02/13 01:18:02 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xmx4m -Xaot:forceaot,count=0 -Xshareclasses:name=shareddbgext -cp /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/utils/utils.jar -Xdump:system:file=j9core.dmp,events=vmstop org.openj9.test.ivj.Hanoi 2
 Time spent starting: 8 milliseconds
 Time spent executing: 13650 milliseconds
 Test result: PASSED

 Testing: Remove shared cache
 Test start time: 2022/02/13 01:18:16 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=shareddbgext,destroy
 Time spent starting: 19 milliseconds
 Time spent executing: 106 milliseconds
 Test result: FAILED
  [ERR] JVMSHRC023E Cache does not exist
  [ERR] JVMSHRC023E Cache does not exist
 >> Success condition was not found: [Output match: destroyed]

I haven't been able to reproduce this with the grinders that I've run.

@cjjdespres
Copy link
Contributor Author

From build job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/254/, failure in ParallelStreamsLoadTest_special_J9_11 with console output

===============================================
Running test ParallelStreamsLoadTest_special_J9_11 ...
===============================================
ParallelStreamsLoadTest_special_J9_11 Start Time: Sun Feb 13 03:52:10 2022 Epoch Time (ms): 1644753130302
variation: Mode501
JVM_OPTIONS: -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs
STF 03:52:10.589 - =========================   S T F   =========================
systemtest-prereqs has been processed, and set to: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqsRetrieving amount of free space on drive containing /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11
There is 204283 Mb free
STF 03:52:10.607 - ==================   G E N E R A T I O N   ==================
STF 03:52:10.609 - Checking JVM: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image
STF 03:52:10.610 - Starting process to generate scripts: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java  -Dlog4j.skipJansi=true -Djava.system.class.loader=net.adoptopenjdk.stf.runner.StfClassLoader -classpath /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/system/lambdaLoadTest/..//systemtest_prereqs/log4j/log4j-api.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/system/lambdaLoadTest/..//systemtest_prereqs/log4j/log4j-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/scripts/../bin net.adoptopenjdk.stf.runner.StfRunner -properties "/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/stf_parameters.properties, , /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/config/stf.properties" -testDir "/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest"
GEN 03:52:13.980 - Found test. Project: 'openjdk.test.load' class: 'ParallelStreamsLoadTest.class' Dir: '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.load/bin'
GEN 03:52:14.053 - Found test. Project: 'openjdk.test.load' class: 'net.adoptopenjdk.stf.ParallelStreamsLoadTest'
GEN Classpath directories used by project 'openjdk.test.load':
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.load/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.load/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-api.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-core.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/junit.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/hamcrest-core.jar
GEN 03:52:14.606 - Using Mode NoOptions. Values = ''
GEN 03:52:14.750 -
GEN 03:52:14.750 - Test command summary:
GEN 03:52:14.751 -   Step  Stage   Command           Description
GEN 03:52:14.752 -  -----+--------+-----------------+------------
GEN 03:52:14.752 -     1  execute  Run java          Run parallel streams load test
STF 03:52:14.803 -
STF 03:52:14.803 - Script generation completed
STF 03:52:14.803 -
STF 03:52:14.805 -
STF 03:52:14.805 - =======================   S E T U P   =======================
STF 03:52:14.805 - Running setup: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/setUp.pl
STF 03:52:14.932 - SETUP stage completed
STF 03:52:14.937 -
STF 03:52:14.937 - ====================   E X E C U T E -   ====================
STF 03:52:14.937 - Running execute: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/execute.pl
STF 03:52:15.055 -
STF 03:52:15.055 - Java version
STF 03:52:15.055 - Running: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java -version
openjdk version "11.0.15-internal" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15-internal+0-adhoc.jenkins.BuildJDK11ppc64lelinuxjitPersonal)
Eclipse OpenJ9 VM (build master-c462b8b9d0c, JRE 11 Linux ppc64le-64-Bit Compressed References 20220213_936 (JIT enabled, AOT enabled)
OpenJ9   - c462b8b9d0c
OMR      - ae234b55240
JCL      - 24e7209ed42 based on jdk-11.0.15+1)
STF 03:52:15.311 -
STF 03:52:15.312 - +------ Step 1 - Run parallel streams load test
STF 03:52:15.312 - | Run foreground process
STF 03:52:15.312 - |   Program:     /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java
STF 03:52:15.312 - |   Mnemonic:    LT
STF 03:52:15.312 - |   Echo:        ECHO_ON
STF 03:52:15.312 - |   Expectation: CLEAN_RUN within 1h
STF 03:52:15.312 - |
STF 03:52:15.315 - Running command: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java -Xmx1024M -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs -classpath /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.load/bin:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/bin:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-api.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/junit.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/hamcrest-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.lambdasAndStreams/bin net.adoptopenjdk.loadTest.LoadTest -resultsDir /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results -resultsPrefix 1.LT. -inactivityLimit 60m -reportFailureLimit 1 -abortAtFailureLimit 10 -maxTotalLogFileSpace 200M -maxSingleLogSize 1/25 -suite.parallelStreams.threadCount 2 -suite.parallelStreams.inventoryFile /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory/openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml -suite.parallelStreams.inventoryExcludeFile none -suite.parallelStreams.totalNumberTests 2 -suite.parallelStreams.selection sequential -suite.parallelStreams.seed -1 -suite.parallelStreams.repeatCount 1 -suite.parallelStreams.thinkingTime 0ms..0ms
STF 03:52:15.315 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.stderr
STF 03:52:15.315 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.stdout
STF 03:52:15.337 - Monitoring processes: LT
LT  03:52:18.683 - Load test parameters
LT  03:52:18.692 -   Time limited         = false
LT  03:52:18.693 -   abortIfOutOfMemory   = true
LT  03:52:18.694 -   reportFailureLimit   = 1
LT  03:52:18.695 -   abortAtFailureLimit  = 10
LT  03:52:18.698 -   maxTotalLogFileSpace = 209715200
LT  03:52:18.699 -   maxSingleLogSize     = 8388608
LT  03:52:18.700 - Parameters for suite 0
LT  03:52:18.702 -   Suite name     = parallelStreams
LT  03:52:18.703 -   Number threads = 2
LT  03:52:18.704 -   Supplied seed  = -1
LT  03:52:18.705 -   Inventory file = /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory/openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml
LT  03:52:18.706 -   Exclude file   = none
LT  03:52:18.707 -   Number tests   = 2
LT  03:52:18.708 -   Repeat count   = 1
LT  03:52:18.709 -   Thinking time  = 0ms..0ms
LT  03:52:18.711 -   Selection mode = sequential
LT  03:52:18.726 - Parsing inventory file. Root=/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory File=openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml
LT  03:52:18.740 - Final test list:
LT  03:52:18.745 -   0 JUnit[net.adoptopenjdk.test.streams.TestParallelStreamOperations]  Weighting=1
LT  03:52:18.899 - Starting thread. Suite=0 thread=0
LT  03:52:18.902 - Starting thread. Suite=0 thread=1
LT  03:52:38.936 - Completed 100.0%. Number of tests started=2
LT  03:52:58.971 - Completed 100.0%. Number of tests started=2 (+0)
LT  stderr 11:53:18.527 0x7ff830002400    j9mm.107    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64le_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3577: ((false && (isObjectInEvacuateMemory(object))))
LT  stderr 11:53:18.527 0x7ff83c002400    j9mm.107    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64le_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3577: ((false && (isObjectInEvacuateMemory(object))))
LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/13 03:53:18 - please wait.
LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/13 03:53:18 - please wait.
LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp' in response to an event
LT  stderr JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.3133622.
LT  stderr
LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt' in response to an event
STF 03:53:22.925 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
STF 03:53:22.926 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
LT  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp' in response to an event
LT  stderr JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
LT  stderr JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.3133632.
LT  stderr
LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt' in response to an event
LT  stderr JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc' in response to an event
STF 03:53:31.952 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt
STF 03:53:31.953 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
STF 03:53:31.953 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt
LT  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
LT  stderr Snap file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr JVMDUMP013I Processed dump event "traceassert", detail "".
STF 03:53:38.476 - **FAILED** Process LT  ended with exit code (255) and not the expected exit code/s (0)
STF 03:53:38.476 - Monitoring Report Summary:
STF 03:53:38.476 -   o Process LT  has crashed unexpectedly
STF 03:53:38.477 - Killing processes: LT
STF 03:53:38.477 -   o Process LT  pid 3133542 is not running
**FAILED** at step 1 (Run parallel streams load test). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/execute.pl line 97.
STF 03:53:38.904 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 03:53:38.904 -
STF 03:53:38.904 - ====================   T E A R D O W N   ====================
STF 03:53:38.904 - Running teardown: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/tearDown.pl
STF 03:53:39.027 - TEARDOWN stage completed
STF 03:53:39.030 -
STF 03:53:39.030 - =====================   R E S U L T S   =====================
STF 03:53:39.030 - Stage results:
STF 03:53:39.030 -   setUp:     pass
STF 03:53:39.030 -   execute:  *fail*
STF 03:53:39.030 -   teardown:  pass
STF 03:53:39.031 -
STF 03:53:39.031 - Overall result: **FAILED**

ParallelStreamsLoadTest_special_J9_11_FAILED

I haven't been able to reproduce this with the grinders that I've run.

@cjjdespres
Copy link
Contributor Author

From build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/188/, failures in testSoftMxLocal_LP4k_3,testSoftMxDisclaimMemory_LP4k_3,testDefaultDisclaimMemory_3. I think this one is a duplicate of #14396 as well, based on that thread.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 17, 2022

From build job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/186/, failures in jdk_lang_0 and jdk_lang_1. I believe the relevant console output is

03:51:13  java.lang.Throwable
03:51:13  	at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13  	at Basic.fail(Basic.java:2817)
03:51:13  	at Basic.fail(Basic.java:2818)
03:51:13  	at Basic.equal(Basic.java:2824)
03:51:13  	at Basic.realMain(Basic.java:1385)
03:51:13  	at Basic.main(Basic.java:2827)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13  	at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  >'=C:=\,OPENJ9_JAVA_COMMAND_LINE=/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/openjdkbinary/j2sdk-image/bin/java -XX:+DisplayVMOutputToStderr -classpath /home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/TKG/output_16450868001065/jdk_lang_1/work/classes/1/java/lang/ProcessBuilder/Basic_id1.d:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/ProcessBuilder:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/TKG/output_16450868001065/jdk_lang_1/work/classes/1/test/lib:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/jvmtest/openjdk/jtreg/lib/jtreg.jar Basic$JavaChild System.getenv(),'< not equal to '=C:=\,'
03:51:13  java.lang.Throwable
03:51:13  	at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13  	at Basic.fail(Basic.java:2817)
03:51:13  	at Basic.fail(Basic.java:2818)
03:51:13  	at Basic.equal(Basic.java:2824)
03:51:13  	at Basic.realMain(Basic.java:1902)
03:51:13  	at Basic.main(Basic.java:2827)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13  	at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  Incorrect handling of envstrings containing NULs
03:51:13  java.lang.Throwable
03:51:13  	at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13  	at Basic.fail(Basic.java:2817)
03:51:13  	at Basic.fail(Basic.java:2818)
03:51:13  	at Basic.check(Basic.java:2821)
03:51:13  	at Basic.realMain(Basic.java:1956)
03:51:13  	at Basic.main(Basic.java:2827)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13  	at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  java.lang.AssertionError: Some tests failed
03:51:13  	at Basic.main(Basic.java:2829)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13  	at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  
03:51:13  JavaTest Message: Test threw exception: java.lang.AssertionError: Some tests failed
03:51:13  JavaTest Message: shutting down test
03:51:13  
03:51:13  STATUS:Failed.`main' threw exception: java.lang.AssertionError: Some tests failed

@cjjdespres
Copy link
Contributor Author

Builds job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/265/ and job/Test_openjdk17_j9_sanity.openjdk_ppc64le_linux_jit_Personal/180/ fail in exactly the same way that #14538 (comment) does.

@mpirvu mpirvu added the comp:jitserver Artifacts related to JIT-as-a-Service project label Feb 17, 2022
@mpirvu
Copy link
Contributor

mpirvu commented Feb 17, 2022

For easier tracking I suggest we have a list in the description of this issue with some brief comments about what we've learned. E.g. fails without JITServer too, already tracked by issue N, crash or not, etc. I am suggesting this because this issue is going to become very large, very soon :-)

@cjjdespres
Copy link
Contributor Author

Definitely a good idea.

@pshipton
Copy link
Member

The recent java/lang/ProcessBuilder/Basic.java failures are caused by changes merged yesterday and I expect they will be fixed by changes to the OpenJDK repos within the next hour. Fixes are under review now.
#14520
#14528

@pshipton
Copy link
Member

The fixes are merged now.

@cjjdespres
Copy link
Contributor Author

The grinder https://hyc-runtimes-jenkins.swg-devops.com/view/Test_grinder/job/Grinder/21194/ reproduces the ParallelStreamsLoadTest_special_J9_11 failure without JITAAS.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 18, 2022

The build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/268/ has failures in:

  • cmdLineTester_pltest_j9sig_ext_0
  • testSCCMLTests2_0
  • testSCCMLTests4_0
  • testSCCMLTests4_1
  • cmdLineTester_shrcdbgddrext_1
  • testDDRExt_SharedClasses_0
  • testDDRExtJunit_StackMap_0
  • testDDRExtJunit_MonitorsAndDeadlock1_1
  • testDDRExtJunit_MonitorsAndDeadlock2_0
  • testDDRExtJunit_MonitorsAndDeadlock2_1
  • testDDRExtJunit_MonitorsAndDeadlock3_0
  • testDDRExtJunit_MonitorsAndDeadlock5_0
  • testDDRExtJunit_MonitorsAndDeadlock5_1
  • TestJcmd_0
  • testOpenJ9DiagnosticsMXBean_0

I believe that testSCCMLTests2_0, testSCCMLTests4_0, testSCCMLTests4_1 are all being tracked by #5310. They fail because certain core files aren't created that should be. The rest all seem to fail in ways vaguely similar to those three tests. An excerpt from the (very large) output of cmdLineTester_pltest_j9sig_ext_0:

 [ERR] ----------------------------------------
 [ERR] dump tests
 [ERR] ----------------------------------------
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_verify_functiontable_slots
 [ERR] Ending test j9dump_verify_functiontable_slots
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_with_name
 [ERR]   calling j9dump_create with filename: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/j9dump_test_create_dump_with_name
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  213: j9dump_test_create_dump_with_name j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252389 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/core"
 [ERR] 
 [ERR] 		LastErrorNumber: -108
 [ERR] 		LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_with_name
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_from_signal_handler
 [ERR]   calling j9dump_create with filename: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/j9dump_test_create_dump_from_signal_handler
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  311: j9dump_test_create_dump_from_signal_handler j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252395 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/core"
 [ERR] 		LastErrorNumber: -108
 [ERR] 		LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_from_signal_handler
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_with_NO_name
 [ERR]   calling j9dump_create with empty filename
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  153: j9dump_test_create_dump_with_NO_name j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252401 was not found. Expected to find core file with name "core"
 [ERR] 		LastErrorNumber: -108
 [ERR] 		LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_with_NO_name
 [ERR] 
 [ERR] Dump test done, failures detected.

And an excerpt from testDDRExtJunit_MonitorsAndDeadlock5_1:

     [exec] JVMDUMP039I Processing dump event "throw", detail "j9vm/test/corehelper/HelperExceptionForCoreGeneration" at 2022/02/18 02:07:24 - please wait.
     [exec] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP' in response to an event
     [exec] JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.1275170.
     [exec] 
     [exec] HelperExceptionForCoreGeneration is thrown and caught successfuly.
     [exec] Forcefully terminating process.
     [exec] addTestJitHint adding hint to openj9/internal/tools/attach/target/AttachHandler$1.<clinit>
     [exec] addTestJitHint adding hint to java/util/Vector$Itr.<init>
     [exec] addTestJitHint adding hint to java/lang/Thread$State.values
     [exec] JVMDUMP012E Error in System dump: The core file created by child process with pid = 1275170 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/core"
     [exec] JVMDUMP013I Processed dump event "throw", detail "j9vm/test/corehelper/HelperExceptionForCoreGeneration".

TCK.destroy.cache:
     [echo] Destroying cache
     [echo] Running j9vm.test.corehelper.StackMapCoreGenerator to destroy the cache
     [echo] Using JVM : /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java
     [echo] classname = "j9vm.test.corehelper.StackMapCoreGenerator"
     [echo] Java VM Args:
     [echo]   jvmarg = -Xshareclasses:name=ddrextjunitSCC,destroy
     [java] JVMSHRC806I Compressed references persistent shared cache "ddrextjunitSCC" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
     [java] Java Result: 1

TCK.run.tests.ddrext:
     [echo] Running the DDR Extension Test
     [java] openjdk version "1.8.0_332-internal"
     [java] OpenJDK Runtime Environment (build 1.8.0_332-internal-_2022_02_18_08_00-b00)
     [java] Eclipse OpenJ9 VM (build master-c0f5771, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20220218_965 (JIT enabled, AOT enabled)
     [java] OpenJ9   - c0f5771
     [java] OMR      - 64690f6
     [java] JCL      - bd2e6cf based on jdk8u332-b02)
     [java] 
     [java] [AutoRun] [INFO] Warning: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP is not directory or file!
     [java] [AutoRun] [ERROR] No dump file under /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP

BUILD FAILED
/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/tck_ddrext.xml:152: The following error occurred while executing this line:
/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/tck_ddrext.xml:124: Java returned: 255

The only one that's a little different is cmdLineTester_shrcdbgddrext_1. That one starts with the failure:

Testing: Check whether dump file exist
Test start time: 2022/02/18 01:56:32 Pacific Standard Time
Running command: ls
Time spent starting: 3 milliseconds
Time spent executing: 18 milliseconds
Test result: FAILED
Output from test:
 [OUT] Calculation.class
 [OUT] Calculation.java
 [OUT] cmdlinetestresources.jar
 [OUT] com
 [OUT] Dump.class
 [OUT] Dump.java
 [OUT] j9vm
 [OUT] JavaRuntimeVersion.class
 [OUT] JavaRuntimeVersion.java
 [OUT] META-INF
 [OUT] SystemProperties.class
 [OUT] SystemProperties.java
 [OUT] VMBench
 [OUT] WithDebugInfo.class
 [OUT] WithDebugInfo.java
 [OUT] WithoutDebugInfo.class
 [OUT] WithoutDebugInfo.java
>> Success condition was not found: [Output match: j9core1.dmp]

and the subsequent tests fail because j9core1.dmp doesn't exist. But even here it's a problem with the creation of a core dump that causes the tests to fail.

In #5310 it is suspected that the failures are caused by machine issues. Perhaps that is the case here as well. The child build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/259/ had all the failures, running on computer ub20x64rt1-1.

@cjjdespres
Copy link
Contributor Author

Related to these failures, I ran a subset of the failing tests a few times on the same machine in this run and got all failures, and ran the same tests on a few different automatically-allocated machines in this run and got all successes. So it does seem like a problem with ub20x64rt1-1.

@cjjdespres
Copy link
Contributor Author

I tried to reproduce the cmdLineTester_classesdbgddrext_1 failure from job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/261/ again, including running on the same machine, and I still can't reproduce it. I think it may just be highly intermittent.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 21, 2022

From job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/191/, failure in cmdLineTester_getPid_0, with output

cmdLineTester_getPid_0 Start Time: Sat Feb 19 02:32:37 2022 Epoch Time (ms): 1645266757691
variation: NoOptions
JVM_OPTIONS: -XX:+UseJITServer 
*** Starting test suite: getPid test ***
Testing: Compare RuntimeMXBean.getPid() with getting PID from system
Test start time: 2022/02/19 02:32:48 Pacific Standard Time
Running command: perl /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/runtimemxbeanTests/getPidTest.pl "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer  -cp /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/runtimemxbeanTests/runtimemxbeanTests.jar GetPid
Time spent starting: 88 milliseconds
Time spent executing: 5059 milliseconds
Test result: FAILED
Output from test:
 [OUT] FAIL: RuntimeMXBean.getPID() returned 0 instead of 31948
>> Success condition was not found: [Output match: PASS]
>> Failure condition was found: [Output match: FAIL]

Appears to be a duplicate of #13085.

Build job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/185/ experienced exactly the same failure.

@cjjdespres
Copy link
Contributor Author

From job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/269/, failure in testSoftMxDisclaimMemory_3. In addition, the build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/193/ has a failure in testSoftMxDisclaimMemory_LP4k_3. Both have been experienced before and are being tracked elsewhere. I suppose I'll not report failures like these in future.

@cjjdespres
Copy link
Contributor Author

From job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/265/, failure in SharedClassesAPI_1, with output

JVMT Sun Feb 20 02:26:54 2022
JVMT Incorrect number of caches found. Expected at least 4, found 2
JVMT Sun Feb 20 02:26:54 2022
JVMT Iterating over caches: ERROR
STF 02:26:55.620 - Monitoring Report Summary:
STF 02:26:55.620 -   o Process JVMT ended with the expected exit code (0)
**FAILED** at step 71 (Verify Classes Caches using JVMTI). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal_testList_2/aqa-tests/TKG/../TKG/output_16453495033600/SharedClassesAPI_1/20220220-022207-SharedClassesAPI/execute.pl line 2534.

Version information:

04:19:38  openjdk version "1.8.0_332-internal"
04:19:38  OpenJDK Runtime Environment (build 1.8.0_332-internal-jenkins_2022_02_19_23_57-b00)
04:19:38  Eclipse OpenJ9 VM (build master-ce1c984f065, JRE 1.8.0 Linux ppc64le-64-Bit Compressed References 20220220_921 (JIT enabled, AOT enabled)
04:19:38  OpenJ9   - ce1c984f065
04:19:38  OMR      - 64690f622a7
04:19:38  JCL      - bd2e6cf7984 based on jdk8u332-b02)

Appears to be tracked by #13239.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 24, 2022

From build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/274/, failures in

  • cmdLineTester_dumpromclasstests_0
  • cmdLineTester_pltest_j9sig_ext_0
  • testSCCMLTests4_0
  • cmdLineTester_shrcdbgddrext_0
  • testDDRExt_General_0
  • testDDRExt_Callsites_0
  • testDDRExt_JITExt_0
  • testDDRExtJunit_CollisionResilientHashtable_0
  • testDDRExtJunit_MonitorsAndDeadlock1_0
  • testDDRExtJunit_MonitorsAndDeadlock2_0
  • testDDRExtJunit_MonitorsAndDeadlock3_0
  • testDDRExtJunit_MonitorsAndDeadlock3_1
  • testDDRExtJunit_MonitorsAndDeadlock4_0
  • testDDRExtJunit_MonitorsAndDeadlock5_0
  • testDDRExtJunit_MonitorsAndDeadlock5_1
  • testDDRExtJunit_MonitorsAndDeadlock6_0
  • testDDRExtJunit_MonitorsAndDeadlock6_1
  • testRASAPI_0

These have a moderate overlap with the failures in this build. It looks like they all fail because a core dump file is not created, so they all fall under #5310 if I'm right about the connection between that issue and the previous failures. Note that these failures happened on ub20x64rt1-1 again.

@cjjdespres
Copy link
Contributor Author

From job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/271/, failure in cmdLineTester_fieldwatchtests_0, with output:

Testing: fw001-5
Test start time: 2022/02/24 02:15:55 Pacific Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_0/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer  -Xdump  -agentlib:jvmtitest=test:fw001 -XX:+JITInlineWatches -cp "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 6 milliseconds
Time spent executing: 45897 milliseconds
Test result: FAILED
Output from test:

 [OUT] *ERROR*: Invalid number of field reports detected: 0. Expected: 5
 [OUT] 
 [OUT] 
 [OUT] ===============Disabling watches for instanceDoubleField
 [OUT] 
 [OUT] 
 [OUT] *ERROR*: Unable to remove field watches for instanceDoubleField
 [OUT] 
 [OUT] ===============adding field read watches for instanceIntField
 [OUT] 
 [OUT] 
 [OUT] *ERROR*: Unable to add field watches for instanceIntField

 [OUT] =============== Ending - STATIC AND INSTANCE FIELD WATCH TESTS
 [OUT] 
 [OUT] 
 [OUT] *** Test took 45609 milliseconds
 [OUT] FAILED
 [OUT] 
 [OUT] ield Modification Report #5: method: 0x79dd98160de8 location: 22, fieldID: 0x79dd98160e88, signature: F
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceSingleField. The fieldID is: 0x79dd98160e88 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceSingleField, signature: F, isStatic: No, modificationType: remove
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceLongField. The fieldID is: 0x79dd98160eb0 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceLongField, signature: J, isStatic: No, modificationType: add
 [OUT] Field Modification Report #1: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #2: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #3: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #4: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #5: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceLongField. The fieldID is: 0x79dd98160eb0 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceLongField, signature: J, isStatic: No, modificationType: remove
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
>> Success condition was not found: [Return code: 0]

There is more in the log that may be relevant, including more error lines, but I hope that is representative.

Version information:

04:43:16  openjdk version "1.8.0_332-internal"
04:43:16  OpenJDK Runtime Environment (build 1.8.0_332-internal-jenkins_2022_02_24_00_02-b00)
04:43:16  Eclipse OpenJ9 VM (build master-016ce8a1e28, JRE 1.8.0 Linux ppc64le-64-Bit Compressed References 20220224_925 (JIT enabled, AOT enabled)
04:43:16  OpenJ9   - 016ce8a1e28
04:43:16  OMR      - bb570045546
04:43:16  JCL      - bd2e6cf7984 based on jdk8u332-b02)

Reproduced with a 1/50 failure rate with JITAAS in view/Test_grinder/job/Grinder/21338/, and not reproduced with a 100x job in view/Test_grinder/job/Grinder/21375/ without JITAAS.

@cjjdespres
Copy link
Contributor Author

Very large number of test failures over the past few days. It will take a little while to sort through it all.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 28, 2022

I've noticed that there's a large overlap between the failing tests and the ones in this comment, but that may just be coincidence?


Edit: Even though there is such a large overlap (including tests that I haven't had a chance to look at yet), the ones in that comment all failed due to the disabling of SecurityManager, and so it's not directly related.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 28, 2022

Two jobs had tests that all failed in the same way - the processes could not connect to a VM, as Monitored VM not ready, then eventually gave up and failed. These failures occurred in:

job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/276/

  • TestJlmRemoteClassAuth_1
  • TestJlmRemoteThreadAuth_1
  • TestIBMJlmRemoteMemoryAuth_SE80_Linux_0
  • TestIBMJlmRemoteMemoryNoAuth_SE80_Linux_0
  • TestIBMJlmRemoteMemoryNoAuth_SE80_Linux_1

job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/275/

  • TestJlmRemoteClassAuth_0
  • TestJlmRemoteClassNoAuth_1
  • TestJlmRemoteMemoryNoAuth_0
  • TestJlmRemoteNotifierProxyAuth_1
  • TestJlmRemoteThreadAuth_0
  • TestJlmRemoteThreadNoAuth_0
  • TestIBMJlmRemoteClassAuth_SE80_Linux_1
  • TestIBMJlmRemoteClassNoAuth_SE80_Linux_1

Representative log output:

STF 04:44:26.517 - +------ Step 5 - Wait for the processes to complete
STF 04:44:26.517 - | Wait for processes to meet expectations
STF 04:44:26.517 - |   Processes: [LT1, CL1]
STF 04:44:26.517 - |
STF 04:44:26.517 - Monitoring processes: CL1 LT1
CL1 j> 2022/02/27 04:44:26.692 ServerURL=service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi
CL1 j> 2022/02/27 04:44:26.728 Attempting to connect
CL1 j> 2022/02/27 04:44:26.910 Monitored VM not ready at Feb 27, 2022 4:44:26 AM (attempt 1, elapsed 163ms).
CL1 j> 2022/02/27 04:44:26.910 Waiting 5 secs and trying again...
CL1 j> 2022/02/27 04:44:31.912 Attempting to connect
CL1 j> 2022/02/27 04:44:33.498 Monitored VM not ready at Feb 27, 2022 4:44:33 AM (attempt 2, elapsed 6769ms).
CL1 j> 2022/02/27 04:44:33.498 Waiting 5 secs and trying again...
CL1 j> 2022/02/27 04:44:38.499 Attempting to connect
CL1 j> 2022/02/27 04:44:38.587 Monitored VM not ready at Feb 27, 2022 4:44:38 AM (attempt 3, elapsed 11858ms).

CL1 j> 2022/02/27 04:46:55.214 Failed to connect to Monitored VM after 30 attempts in 148 seconds - giving up.  Connection Exception received is below:
CL1 stderr java.rmi.ConnectException: Connection refused to host: 9.24.248.182; nested exception is: 
CL1 stderr 	java.net.ConnectException: Connection refused (Connection refused)
CL1 stderr 	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
CL1 stderr 	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
CL1 stderr 	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
CL1 stderr 	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
CL1 stderr 	at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
CL1 stderr 	at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
CL1 stderr 	at com.sun.proxy.$Proxy0.newClient(Unknown Source)
CL1 stderr 	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
CL1 stderr 	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
CL1 stderr 	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ServerConnector.doConnect(ServerConnector.java:273)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ServerConnector.getSecureServerConnection(ServerConnector.java:170)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ServerConnector.<init>(ServerConnector.java:82)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ClassProfiler.<init>(ClassProfiler.java:72)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ClassProfiler.main(ClassProfiler.java:89)
CL1 stderr Caused by: java.net.ConnectException: Connection refused (Connection refused)
CL1 stderr 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
CL1 stderr 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
CL1 stderr 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
CL1 stderr 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
CL1 stderr 	at java.net.Socket.connect(Socket.java:607)
CL1 stderr 	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:288)
CL1 stderr 	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:147)
CL1 stderr 	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
CL1 stderr 	at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:121)
CL1 stderr 	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
CL1 stderr 	... 14 more
CL1 stderr Exception in thread "main" java.lang.NullPointerException
CL1 stderr 	at com.ibm.java.lang.management.internal.ManagementUtils.isANotificationEmitter(ManagementUtils.java:536)
CL1 stderr 	at java.lang.management.ManagementFactory.newPlatformMXBeanProxy(ManagementFactory.java:289)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ClassProfiler.getStatsViaProxy(ClassProfiler.java:117)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ClassProfiler.main(ClassProfiler.java:102)
STF 04:46:55.496 - **FAILED** Process CL1 ended with exit code (1) and not the expected exit code/s (0)
STF 04:46:55.496 - Monitoring Report Summary:
STF 04:46:55.496 -   o Process CL1 ended with exit code (1) and not the expected exit code/s (0)
STF 04:46:55.496 -   o Process LT1 is still running as expected
STF 04:46:55.496 - Killing processes: CL1 LT1
STF 04:46:55.496 -   o Process LT1 pid 2478 stop()
STF 04:47:05.549 -   o Process LT1 pid 2478 terminate()
STF 04:47:06.550 -   o Process LT1 pid 2478 killed
STF 04:47:06.550 -   o Process CL1 pid 2479 is not running
**FAILED** at step 5 (Wait for the processes to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TKG/output_16459519958390/TestJlmRemoteClassAuth_1/20220227-044425-TestJlmRemoteClassAuth/execute.pl line 178.
STF 04:47:06.792 - **FAILED** execute script failed. Expected return value=0 Actual=1

Haven't attempted to reproduce.

@cjjdespres
Copy link
Contributor Author

From job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/191/, failures in

  • TestJlmRemoteMemoryNoAuth_0
  • TestJlmRemoteNotifierProxyAuth_0
  • TestJlmRemoteThreadNoAuth_0
  • TestJlmRemoteThreadNoAuth_1
  • TestIBMJlmRemoteClassAuth_0
  • TestIBMJlmRemoteClassAuth_1
  • TestJlmRemoteClassAuth_1

These all fail in similar ways - the tests attempt to connect to a particular ServerURL, and either just before or just after that a child process fails unexpectedly and the test is aborted.

Representative console log:

STF 02:00:23.344 - +------ Step 3 - Wait for processes to complete
STF 02:00:23.344 - | Wait for processes to meet expectations
STF 02:00:23.344 - |   Processes: [LT1, CL1]
STF 02:00:23.344 - |
STF 02:00:23.345 - Monitoring processes: CL1 LT1
CL1 j> 2022/02/26 02:00:25.833 ServerURL=service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi
CL1 j> 2022/02/26 02:00:26.710 Attempting to connect
STF 02:00:28.587 - **FAILED** Process LT1 has ended unexpectedly
STF 02:00:28.587 - Monitoring Report Summary:
STF 02:00:28.587 -   o Process CL1 is still running
STF 02:00:28.587 -   o Process LT1 has ended unexpectedly
STF 02:00:28.588 - Killing processes: CL1 LT1
STF 02:00:28.588 -   o Process LT1 pid 26108 is not running
STF 02:00:28.588 -   o Process CL1 pid 26109 stop()
STF 02:00:38.682 -   o Process CL1 pid 26109 terminate()
STF 02:00:39.700 -   o Process CL1 pid 26109 killed
CL1 j> 2022/02/26 02:00:29.574 Connection established!
CL1 j> 2022/02/26 02:00:30.503 Starting to write data
**FAILED** at step 3 (Wait for processes to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal_testList_2/aqa-tests/TKG/../TKG/output_1645867773491/TestJlmRemoteMemoryNoAuth_0/20220226-020012-TestJlmRemoteMemoryNoAuth/execute.pl line 163.
STF 02:00:39.773 - **FAILED** execute script failed. Expected return value=0 Actual=1

Haven't attempted to reproduce.

@pshipton
Copy link
Member

I've noticed that there's a large overlap between the failing tests and the ones in #14412 (comment), but that may just be coincidence?

If running Java 18+ you need the following set -Djava.security.manager=allow. We do that for the regular tests by setting the JAVA_TOOL_OPTIONS environment variable to prepend the option adoptium/aqa-tests#3352

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Feb 28, 2022

The next jobs all fail due to a segfault. They also have some tests that seem to fail for other reasons, so I'll post those separately. The two running the functional tests all have identical stack backtraces

----------- Stack Backtrace -----------
_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x133da (0x00007F0F643EF1BA [libj9vm29.so+0xa61ba])
bytecodeLoopCompressed+0x95 (0x00007F0F643DBDD5 [libj9vm29.so+0x92dd5])
 (0x00007F0F6447F5B2 [libj9vm29.so+0x1365b2])

aside from what I assume are the memory addresses. The one running the system tests has more varied stack backtraces. The jobs and tests are:

job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/

  • SyntheticGCWorkload_DoubleMap_J9_0
  • testSCCacheManagement_0
  • cmdLineTester_classesdbgddrext_1
  • cmdLineTester_defaultLazySymbolResolution_1
  • cmdLineTester_fastClassHashTable_0
  • cmdLineTester_hangTest_0
  • cmdLineTester_javaAssertions_0
  • cmdLineTester_decompilationTests_nongold_3
  • cmdLineTester_lockWordAlignment_Object_Standard_0
  • cmdLineTester_lockWordAlignment_Object_iii_0
  • cmdLineTester_lockWordAlignment_Object_d_0
  • cmdLineTester_modularityddrtests17_1
  • cmdLineTester_shareClassesBadStackMap_0
  • testSCCMLTests3_1
  • testSCCMLTests4_1
  • testSCCMLTests6_1
  • testSCCMLSnapshot_1
  • testSCCMLAotMethodOperation_1
  • cmdLineTester_shrcdbgddrext_1
  • cmdLineTest_stackSizeInfoTest_0
  • cmdLineTester_verbosetest_5
  • cmdLineTester_vmRuntimeState_0
  • cmdLineTester_xlogTests_0
  • StackWalkerTestJava10_0
  • JCL_TEST_Java-Lang-Invoke_0
  • TestAttachAPIEnabling_0
  • TestSunAttachClasses_0
  • gcNotificationTest_Optthruput_1
  • gcNotificationTest_OptAvgpause_1
  • gcNotificationTest_Balanced_0
  • gcNotificationTest_Balanced_1
  • gcNotificationTest_Metronome_1
  • ContendedFieldsTests_90_0
  • threadMXBeanTestSuite2_2
  • threadMXBeanTimedParkTest_2
  • threadMXBeanTimedParkTest_3
  • memoryCategories_0
  • HashConsistency_0
  • jniOnLoadExceptions_1
  • badUTF8inJNI_1
  • badUTF8inJNI_2
  • testvmcheck_3
  • testvmcheck_6
  • TestSoftMxCallBackOOM_1
  • TestSoftMxCallBackExtend_1
  • MethodVisibilityTests_0
  • UnreflectTests_0

job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/

  • SyntheticGCWorkload_DoubleMap_J9_0
  • SecurityTests_0
  • CDSAdaptorTest_0
  • cmdLineTester_bootStrapStaticVerify_0
  • cmdLineTester_EnableAssertionStatusTest_0
  • cmdLineTest_J9test_extended_0
  • cmdLineTest_J9test_consoleUpTo17_0
  • cmdLineTester_defaultLazySymbolResolution_1
  • cmdLineTester_dumpromclasstests_0
  • cmdLineTester_getCallerClassTests_SE80_0
  • cmdLineTester_hangTest_0
  • cmdLineTester_ignoreunrecognizedvmoptions_0
  • cmdLineTester_jep178_staticLinking_SE80_0
  • cmdLineTester_fieldwatchtests_0

job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/

  • DaaLoadTest_all_5m_1
  • MiniMix_5m_0
  • DBBLoadTest_5m_0
  • HCRLateAttachWorkload_previewEnabled_0
  • ObjectTreeLoadTest_5m_1
  • SharedClasses.SCM01.SingleCL_1
  • SharedClasses.SCM01.MultiCL_1
  • SharedClasses.SCM01.MultiThread_1
  • SharedClasses.SCM23.MultiThreadMultiCL_1

Haven't been able to reproduce.

@cjjdespres
Copy link
Contributor Author

A lot of recent test failures in jdk_security2_0, jdk_security2_1. The tests fail with the message:

TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.IllegalArgumentException: Key must be at least 160 bits

This issue appears to be tracked by #17551.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Jun 16, 2023

Failure in DaaLoadTest_all_special_5m_0 in /job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/613/. Segfault, with "invalid JIT return address". Console log:

DLT stderr Unhandled exception
DLT stderr Type=Segmentation error vmState=0x00000000
DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
DLT stderr Handler1=00007FFF99D6F9E0 Handler2=00007FFF99C98AA0
DLT stderr R0=0000000000000000 R1=00007FFEC84DB700 R2=00007FFF205BF4D8 R3=0000000000000040
DLT stderr R4=00007FFF22B84B48 R5=00007FFF1DD332F0 R6=00007FFF1DD332F0 R7=0000000000000001
DLT stderr R8=0000000000000000 R9=00007FFE04055880 R10=0000000000000000 R11=00007FFDC4012C00
DLT stderr R12=00007FFF99D69A90 R13=00007FFEC84E68E0 R14=00007FFE200082E0 R15=00007FFF94B8C000
DLT stderr R16=00007FFF22290A60 R17=00007FFF1E6282F0 R18=00007FFF22B83B10 R19=00007FFF22B7CE60
DLT stderr R20=00007FFF22B7D5A8 R21=00007FFF22B84A98 R22=00007FFF22B84AC8 R23=00007FFF22B84B48
DLT stderr R24=00007FFF22B84B48 R25=00007FFF1DC86410 R26=00007FFF22B845D0 R27=00007FFF22B84AC8
DLT stderr R28=0000000000000000 R29=000000000000DEAD R30=000000000000DEAD R31=000000000000DEAD
DLT stderr NIP=00007FFF02B9DB2C MSR=800000000280F033 ORIG_GPR3=00007FFF029B1C08 CTR=00007FFF99D69A90
DLT stderr LINK=00007FFF029B1C0C XER=0000000000000000 CCR=0000000024084222 SOFTE=0000000000000001
DLT stderr TRAP=0000000000000300 DAR=0000000000000040 dsisr=0000000042000000 RESULT=0000000000000000
DLT stderr FPR0 0000000000000004 (f: 4.000000, d: 1.976263e-323)
DLT stderr FPR1 4028000000000000 (f: 0.000000, d: 1.200000e+01)
DLT stderr FPR2 41d0000000000000 (f: 0.000000, d: 1.073742e+09)
DLT stderr FPR3 4030000000000000 (f: 0.000000, d: 1.600000e+01)
DLT stderr FPR4 3fe8000000000000 (f: 0.000000, d: 7.500000e-01)
DLT stderr FPR5 3fe8000000000000 (f: 0.000000, d: 7.500000e-01)
DLT stderr FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
DLT stderr FPR7 3faaa5aa5df25984 (f: 1576163712.000000, d: 5.204518e-02)
DLT stderr FPR8 006900720061002e (f: 6357038.000000, d: 1.112614e-306)
DLT stderr FPR9 0073006300690074 (f: 6881396.000000, d: 1.691191e-306)
DLT stderr FPR10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR11 0000000000000001 (f: 1.000000, d: 4.940656e-324)
DLT stderr FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR13 4034000000000000 (f: 0.000000, d: 2.000000e+01)
DLT stderr FPR14 7f7fffff7f7fffff (f: 2139095040.000000, d: 1.404447e+306)
DLT stderr FPR15 47efffffe0000000 (f: 3758096384.000000, d: 3.402823e+38)
DLT stderr FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR17 7ff0000000000000 (f: 0.000000, d: inf)
DLT stderr FPR18 00bd19d700bd19d7 (f: 12392919.000000, d: 4.144111e-305)
DLT stderr FPR19 0000001500000015 (f: 21.000000, d: 4.456191e-313)
DLT stderr FPR20 4035000000000000 (f: 0.000000, d: 2.100000e+01)
DLT stderr FPR21 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
DLT stderr FPR22 40342b443866f3b6 (f: 946271168.000000, d: 2.016901e+01)
DLT stderr FPR23 4047386e22edf4a6 (f: 586020032.000000, d: 4.644086e+01)
DLT stderr FPR24 40026bb1bbb55516 (f: 3149223168.000000, d: 2.302585e+00)
DLT stderr FPR25 4050c00000000000 (f: 0.000000, d: 6.700000e+01)
DLT stderr FPR26 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
DLT stderr FPR27 c1612a8800000000 (f: 0.000000, d: -9.000000e+06)
DLT stderr FPR28 c022000000000000 (f: 0.000000, d: -9.000000e+00)
DLT stderr FPR29 41612a8800000000 (f: 0.000000, d: 9.000000e+06)
DLT stderr FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr 
DLT stderr Compiled_method=net/openj9/test/binaryData/TestByteArray2Long.testInvalidOffset()V
DLT stderr Target=2_90_20230615_663 (Linux 4.18.0-477.13.1.el8_8.ppc64le)
DLT stderr CPU=ppc64le (32 logical CPUs) (0x1da680000 RAM)
DLT stderr ----------- Stack Backtrace -----------
DLT stderr  (0x00007FFF02B9DB2C [<unknown>+0x0])
DLT stderr runJavaThread+0x240 (0x00007FFF99D46CB0 [libj9vm29.so+0x16cb0])
DLT stderr javaProtectedThreadProc+0x138 (0x00007FFF99DE3828 [libj9vm29.so+0xb3828])
DLT stderr omrsig_protect+0x3f4 (0x00007FFF99C99F74 [libj9prt29.so+0x39f74])
DLT stderr javaThreadProc+0x60 (0x00007FFF99DDF120 [libj9vm29.so+0xaf120])
DLT stderr thread_wrapper+0x190 (0x00007FFF99C2CBC0 [libj9thr29.so+0xcbc0])
DLT stderr start_thread+0xf8 (0x00007FFF9A719678 [libpthread-2.28.so+0x9678])
DLT stderr clone+0x74 (0x00007FFF9A608738 [libc-2.28.so+0x138738])
DLT stderr ---------------------------------------
DLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2023/06/16 06:12:34 - please wait.
DLT stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_3/aqa-tests/TKG/output_16869044229161/DaaLoadTest_all_special_5m_0/20230616-060916-DaaLoadTest/results/core.20230616.061234.46053.0001.dmp' in response to an event
DLT stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_3/aqa-tests/TKG/output_16869044229161/DaaLoadTest_all_special_5m_0/20230616-060916-DaaLoadTest/results/core.20230616.061234.46053.0001.dmp
DLT stderr 
DLT stderr 
DLT stderr *** Invalid JIT return address 00007FFF22B7CE60 in 00007FFF0087BA08

Couldn't reproduce in grinder.

@cjjdespres
Copy link
Contributor Author

Quite a few failures in SecurityTests_0 (e.g., https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/736/), but it might be some sort of infra issue (expired certificate).

FAILED: testTls13
javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:360)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:298)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1511)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:572)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168)
	at net.adoptopenjdk.test.Tls13FunctionalTest.testTls13(Tls13FunctionalTest.java:94)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.testng.TestNG.privateMain(TestNG.java:1354)
	at org.testng.TestNG.main(TestNG.java:1323)
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
	at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:263)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341)
	... 39 more
Caused by: java.security.cert.CertPathValidatorException: validity check failed
	at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83)
	at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)
	at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364)
	... 45 more
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Mon Jun 26 09:55:42 EDT 2023
	at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277)
	at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:675)
	at java.base/sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190)
	at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144)
	at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
	... 50 more

@cjjdespres
Copy link
Contributor Author

There have been recent failures in jdk_security4_0, with some unhelpful test exception thrown:

 STDERR:
05:06:29  Nsanity started
05:06:29  Na started
05:06:29  Nb started
05:06:29  java.lang.Exception
05:06:29  	at ReplayCacheTestProc.main0(ReplayCacheTestProc.java:279)
05:06:29  	at ReplayCacheTestProc.main(ReplayCacheTestProc.java:326)
05:06:29  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
05:06:29  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
05:06:29  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
05:06:29  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
05:06:29  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
05:06:29  	at java.base/java.lang.Thread.run(Thread.java:839)
05:06:29  java.lang.Exception
05:06:29  	at ReplayCacheTestProc.main0(ReplayCacheTestProc.java:279)
05:06:29  	at ReplayCacheTestProc.main(ReplayCacheTestProc.java:326)
05:06:29  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
05:06:29  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
05:06:29  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
05:06:29  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
05:06:29  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
05:06:29  	at java.base/java.lang.Thread.run(Thread.java:839)
05:06:29  
05:06:29  JavaTest Message: Test threw exception: java.lang.Exception
05:06:29  JavaTest Message: shutting down test

This failure occurred in, for instance, https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/716/. I believe this is tracked by #13756.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Aug 15, 2023

Some recent failures in testDDREExtJunit_FindExtThread_0, e.g. in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_extended.functional_ppc64le_linux_jit_Personal/677/ and https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/708/.

Console log:

     [java] junit.framework.AssertionFailedError: Failed to construct parameter list for command : !stackslots thread,sp,a0,pc,literals,els
     [java] StackSlots Output being used : 
     [java] <1c6100> *** BEGIN STACK WALK, flags = 00400001 walkThread = 1,859,840 ***
     [java] <1c6100> 	ITERATE_O_SLOTS
     [java] <1c6100> 	RECORD_BYTECODE_PC_OFFSET
     [java] <1c6100> Initial values: walkSP = 0x00000000001C5EE8, PC = 0x0000000000000003, literals = 0x0000000000000000, A0 = 0x00000000001C5F20, j2iFrame = 0x0000000000000000, decomp = 0x0000000000000000
     [java] <1c6100> INL native method frame: bp = 0x00000000001C5F08, sp = 0x00000000001C5EE8, pc = 0x0000000000000003, cp = 0x0000000000000000, arg0EA = 0x00000000001C5F20, flags = 0x0000000000000000
     [java] <1c6100> 	Method: java/lang/Thread.sleepImpl(JI)V !j9method 0x0000000000074C40
     [java] <1c6100> 	Using signature mapper
     [java] <1c6100> 	Arguments starting at 0x00000000001C5F20 for UDATA(0x0000000000000003) slots
     [java] <1c6100> 		I-Slot: a0[0x00000000001C5F20] = 0x0000000000000000
     [java] <1c6100> 		I-Slot: a1[0x00000000001C5F18] = 0x00000000000003E8
     [java] <1c6100> 		I-Slot: a2[0x00000000001C5F10] = 0x0000000000000000
     [java] <1c6100> Bytecode frame: bp = 0x00000000001C5F38, sp = 0x00000000001C5F28, pc = 0x00007F94E5268332, cp = 0x0000000000073A90, arg0EA = 0x00000000001C5F50, flags = 0x0000000000000000
     [java] <1c6100> 	Method: java/lang/Thread.sleep(JI)V !j9method 0x0000000000074C20
     [java] <1c6100> 	Bytecode index = 2
     [java] <1c6100> 	Using local mapper
     [java] <1c6100> 	Locals starting at 0x00000000001C5F50 for 0x0000000000000003 slots
     [java] <1c6100> 		I-Slot: a0[0x00000000001C5F50] = 0x00007F94E52083A9
     [java] <1c6100> 		I-Slot: a1[0x00000000001C5F48] = 0x00000000000003E8
     [java] <1c6100> 		I-Slot: a2[0x00000000001C5F40] = 0x0000000000000000
     [java] <1c6100> Bytecode frame: bp = 0x00000000001C5F68, sp = 0x00000000001C5F58, pc = 0x00007F94E526830A, cp = 0x0000000000073A90, arg0EA = 0x00000000001C5F78, flags = 0x0000000000000000
     [java] <1c6100> 	Method: java/lang/Thread.sleep(J)V !j9method 0x0000000000074C00
     [java] <1c6100> 	Bytecode index = 2
     [java] <1c6100> 	Using local mapper
     [java] <1c6100> 	Locals starting at 0x00000000001C5F78 for 0x0000000000000002 slots
     [java] <1c6100> 		I-Slot: a0[0x00000000001C5F78] = 0x0000000000000006
     [java] <1c6100> 		I-Slot: a1[0x00000000001C5F70] = 0x00000000000003E8
     [java] <1c6100> Bytecode frame: bp = 0x00000000001C5F90, sp = 0x00000000001C5F80, pc = 0x00007F94E54B7885, cp = 0x00000000001FAEB0, arg0EA = 0x00000000001C5FA8, flags = 0x0000000000000000
     [java] <1c6100> 	Method: j9vm/test/corehelper/ThreadWorker.run()V !j9method 0x00000000001FAF38
     [java] <1c6100> 	Bytecode index = 9
     [java] <1c6100> 	Using local mapper
     [java] <1c6100> 	Locals starting at 0x00000000001C5FA8 for 0x0000000000000003 slots
     [java] <1c6100> 		I-Slot: a0[0x00000000001C5FA8] = 0x00000000FFFB5DB8
     [java] <1c6100> 		I-Slot: t1[0x00000000001C5FA0] = 0x0000000000000006
     [java] <1c6100> 		I-Slot: t2[0x00000000001C5F98] = 0x00000000FFF982A8
     [java] <1c6100> JNI call-in frame: bp = 0x00000000001C5FD0, sp = 0x00000000001C5FB0, pc = 0x00007F953E4231B0, cp = 0x0000000000000000, arg0EA = 0x00000000001C5FD0, flags = 0x0000000000000000
     [java] <1c6100> 	New ELS = 0x0000000000000000
     [java] <1c6100> JNI native method frame: bp = 0x00000000001C5FF8, sp = 0x00000000001C5FD8, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x00000000001C5FF8, flags = 0x0000000000000000
     [java] <1c6100> <end of stack>
     [java] <1c6100> *** END STACK WALK (rc = NONE) ***
     [java] 
     [java] 	at junit.framework.Assert.fail(Assert.java:50)
     [java] 	at j9vm.test.ddrext.junit.TestThread.testStackSlots(TestThread.java:92)
     [java] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
     [java] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     [java] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
     [java] 	at junit.framework.TestCase.runTest(TestCase.java:168)
     [java] 	at junit.framework.TestCase.runBare(TestCase.java:134)
     [java] 	at junit.framework.TestResult$1.protect(TestResult.java:110)
     [java] 	at junit.framework.TestResult.runProtected(TestResult.java:128)
     [java] 	at junit.framework.TestResult.run(TestResult.java:113)
     [java] 	at junit.framework.TestCase.run(TestCase.java:124)
     [java] 	at junit.framework.TestSuite.runTest(TestSuite.java:243)
     [java] 	at j9vm.test.ddrext.junit.DDRTestSuite.run(DDRTestSuite.java:74)
     [java] 	at j9vm.test.ddrext.junit.DDRTestSuite.run(DDRTestSuite.java:69)
     [java] 	at j9vm.test.ddrext.junit.DDRTestSuite.run(DDRTestSuite.java:55)
     [java] 	at j9vm.test.ddrext.AutoRun.runTest(AutoRun.java:275)
     [java] 	at j9vm.test.ddrext.AutoRun.main(AutoRun.java:110)

Tracked by #17962.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Aug 28, 2023

There were a lot of failures over the weekend (starting on the 26th it seems) in SC_Softmx_JitAot_Linux_1, and some failures in testSCCMLTests3_1, testSCCMLSoftmx_1, testSCCMLAotMethodOperation_1, cmdLineTester_shrcdbgddrext_1 that may be related. Example runs for that one test:

and for that one group:

Given the timing this certainly could be related to my changes, but I have no idea how. I see that SC_Softmx_JitAot_Linux_1 looks for a particular error string defined in j9shr.nls - could the addition of a new error somehow have disturbed that? None of my new code should have activated in the test.

Console log for one of the SC_Softmx_JitAot_Linux_1 failures:

SC_Softmx_JitAot_Linux_1 Start Time: Sun Aug 27 02:22:38 2023 Epoch Time (ms): 1693128158943
variation: Mode610
JVM_OPTIONS: -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon 

STF 02:25:36.811 - +------ Step 8 - Make sure the space for AOT data in shared cache is full
STF 02:25:36.811 - | Count file matches
STF 02:25:36.811 - |   TargetFile:         /home/jenkins/workspace/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16931240696944/SC_Softmx_JitAot_Linux_1/20230827-022239-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/stderr_combined.txt
STF 02:25:36.811 - |   ExpectedNumMatches: 1
STF 02:25:36.811 - |   SearchStrings:      ["JVMSHRC773I The space for AOT data in shared cache "SCC_SysTest_Workload_Cache" is full."]
STF 02:25:36.811 - |
**FAILED** at step 8 (Make sure the space for AOT data in shared cache is full). Expected return value=1 Actual=0 at /home/jenkins/workspace/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TKG/output_16931240696944/SC_Softmx_JitAot_Linux_1/20230827-022239-SharedClassesWorkloadTest_Softmx_Increase_JitAot/execute.pl line 291.

Console log for a testSCCMLSoftmx_1 failure:

Testing: Test 3-a: Add ROMClasses, AOT methods and JITHints into the cache
Test start time: 2023/08/26 03:05:45 Pacific Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=ShareClassesCMLTests,verboseAOT,verboseJITDATA,addtestjithints -Xaot:forceAot,count=1 -Xjit:disableAsyncCompilation -version
Time spent starting: 2 milliseconds
***[TEST INFO 2023/08/26 03:15:45] ProcessKiller detected a timeout after 600000 milliseconds!***
INFO: Cannot find '/usr/bin/gdb' using 'gdb' from the path.
***[TEST INFO 2023/08/26 03:15:45] executing gdb -batch -x /tmp/debugger17473389253726545395.txt /home/jenkins/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java 29607***
java.io.IOException: Cannot run program "gdb": error=2, No such file or directory
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
	at java.base/java.lang.Runtime.exec(Runtime.java:592)
	at java.base/java.lang.Runtime.exec(Runtime.java:451)
	at Test$ProcessKiller.captureCoreForProcess(Test.java:697)
	at Test$ProcessKiller.captureCoreForProcess(Test.java:646)
	at Test$ProcessKiller.run(Test.java:596)
Caused by: java.io.IOException: error=2, No such file or directory
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
	... 6 more
***[TEST INFO 2023/08/26 03:15:45] executing kill -ABRT 29607***
***[TEST INFO 2023/08/26 03:15:45] kill -ABRT signal sent***
***[TEST INFO 2023/08/26 03:15:46] ABRT completed***
Time spent executing: 604334 milliseconds
Test result: FAILED



Testing: Test 3-b: Verify in the cache stats that ROMClasses, AOT methods and JIT data have been added into the cache
Test start time: 2023/08/26 03:15:49 Pacific Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=ShareClassesCMLTests,printDetails
Time spent starting: 2 milliseconds
Time spent executing: 173 milliseconds
Test result: FAILED
 [ERR] 
 [ERR] Current statistics for cache "ShareClassesCMLTests": 
 [ERR] 
 [ERR] Cache created with:
 [ERR] 	-Xnolinenumbers                      = false
 [ERR] 	BCI Enabled                          = true
 [ERR] 	Restrict Classpaths                  = false
 [ERR] 	Feature                              = cr
 [ERR] 	Processor Features                   = vsx dfp vec
 [ERR] 
 [ERR] Cache contains only classes with line numbers
 [ERR] 
 [ERR] base address                         = 0x00007FFF88FB0000
 [ERR] end address                          = 0x00007FFF89F90000
 [ERR] allocation pointer                   = 0x00007FFF8919C428
 [ERR] 
 [ERR] metadata start address               = 0x00007FFF89E3D040
 [ERR] runtime flags                        = 0xA00C2020F43680B3
 [ERR] cache generation                     = 43
 [ERR] 
 [ERR] cache layer                          = 0
 [ERR] cache size                           = 16776608
 [ERR] softmx bytes                         = 16777216
 [ERR] free bytes                           = 13241368
 [ERR] Reserved space for AOT bytes         = -1
 [ERR] Maximum space for AOT bytes          = -1
 [ERR] Reserved space for JIT data bytes    = -1
 [ERR] Maximum space for JIT data bytes     = -1
 [ERR] ReadWrite bytes                      = 130464
 [ERR] Metadata bytes                       = 40808
 [ERR] Metadata % used                      = 1%
 [ERR] Class debug area size                = 1310720
 [ERR] Class LineNumberTable bytes          = 176168
 [ERR] Class LocalVariableTable bytes       = 298248
 [ERR] Class debug area % used              = 36%
 [ERR] 
 [ERR] ROMClass bytes                       = 2016296
 [ERR] AOT code bytes                       = 0
 [ERR] AOT data bytes                       = 288
 [ERR] AOT class hierarchy bytes            = 20128
 [ERR] AOT thunk bytes                      = 2684
 [ERR] JIT hint bytes                       = 2204
 [ERR] JIT profile bytes                    = 11648
 [ERR] Java Object bytes                    = 0
 [ERR] Zip cache bytes                      = 0
 [ERR] Startup hint bytes                   = 0
 [ERR] JCL data bytes                       = 0
 [ERR] Byte data bytes                      = 0
 [ERR] 
 [ERR] # ROMClasses                         = 572
 [ERR] # AOT Data Entries                   = 1
 [ERR] # AOT Class Hierarchy                = 331
 [ERR] # AOT Thunks                         = 36
 [ERR] # JIT Hints                          = 83
 [ERR] # JIT Profiles                       = 95
 [ERR] # AOT Methods                        = 0
 [ERR] # Classpaths                         = 1
 [ERR] # URLs                               = 0
 [ERR] # Tokens                             = 0
 [ERR] # Java Objects                       = 0
 [ERR] # Zip caches                         = 0
 [ERR] # Startup hints                      = 0
 [ERR] # JCL Entries                        = 0
 [ERR] # Stale classes                      = 0
 [ERR] % Stale classes                      = 0%
 [ERR] 
 [ERR] 
 [ERR] Cache is 21% soft full
 [ERR] 
 [ERR] Cache is accessible to current user = true
>> Required condition was not found: [Output match: AOT Methods\s+= [^0][0-9]*]
>> Required condition was found: [Output match: JIT Hints\s+= [^0][0-9]*]
>> Required condition was found: [Output match: ROMClasses\s+= [^0][0-9]*]
>> Success condition was found: [Output match: Cache is [^0][0-9]*% soft full]
>> Failure condition was found: [Output match: AOT Methods\s+= 0]
>> Failure condition was not found: [Output match: JIT Hints\s+= 0]
>> Failure condition was not found: [Output match: ROMClasses\s+= 0]
>> Failure condition was not found: [Output match: Cache is 0% soft full]
>> Failure condition was not found: [Output match: error]
>> Failure condition was not found: [Output match: Unhandled Exception]
>> Failure condition was not found: [Output match: Exception:]
>> Failure condition was not found: [Output match: corrupt]
>> Failure condition was not found: [Output match: Processing dump event]

Testing: Test 4-a: Test setting maxAOT/maxJIT to a value that is smaller than AOT/JIT bytes already in the cache
Test start time: 2023/08/26 03:15:50 Pacific Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=ShareClassesCMLTests,adjustmaxaot=1,adjustmaxjitdata=1
Time spent starting: 2 milliseconds
Time spent executing: 38 milliseconds
Test result: FAILED
 [ERR] JVMSHRC782W The size of maximum JIT data space should not be smaller than the JIT bytes (9884) that is already stored in the cache.
 [ERR] JVMSHRC786I The maximum allowed AOT bytes is set to 8.
 [ERR] JVMSHRC788I The maximum allowed JIT data bytes is set to 9884.
>> Required condition was not found: [Output match: The size of maximum AOT space should not be smaller than the AOT bytes \([^0][0-9]*\) that is already stored in the cache]
>> Success condition was found: [Output match: The size of maximum JIT data space should not be smaller than the JIT bytes \([^0][0-9]*\) that is already stored in the cache]
>> Failure condition was not found: [Output match: error]
>> Failure condition was not found: [Output match: Unhandled Exception]
>> Failure condition was not found: [Output match: Exception:]
>> Failure condition was not found: [Output match: corrupt]
>> Failure condition was not found: [Output match: Processing dump event]

EDIT: It seems likely that not storing received AOT methods in the SCC by default has caused these failures - the JVMs that are supposed to add methods to the local SCC no longer do so when requesting remote AOT from the JITServer. This is probably why the lookups and cache filling are failing.

@cjjdespres
Copy link
Contributor Author

It seems as though there have been more timeouts recently in certain tests, e.g., jdk_util_0, jdk_math_0, jdk_security2_0, jdk_foreign_0 variously in

and also cmdLineTester_jython_6 repeatedly in, e.g., https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk8_j9_sanity.functional_x86-64_linux_jit_Personal/1388/.

Console log for one of the jdk_util_0 failures:

04:02:31  ===============================================
04:02:31  Running test jdk_util_0 ...
04:02:31  ===============================================
04:02:31  jdk_util_0 Start Time: Wed Aug 30 09:02:31 2023 Epoch Time (ms): 1693382551048
04:02:31  variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache Mode150
04:02:31  JVM_OPTIONS: -XX:+UseJITServer -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:+UseCompressedOops 

04:39:58  JRE 17 Linux amd64-64-Bit Compressed References 20230829_753 (JIT enabled, AOT enabled)
04:39:58  OpenJ9   - 39071983a8a
04:39:58  OMR      - 9659cbcdcef
04:39:58  JCL      - e6f0d592f7e based on jdk-17.0.9+4

04:39:58  ACTION: testng -- Error. Program `/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal_testList_2/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 967728ms).
04:39:58  REASON: Assumed action based on file name: run testng org.openjdk.tests.java.util.stream.CountLargeTest 
04:39:58  TIME:   967.733 seconds
04:39:58  messages:
04:39:58  command: testng org.openjdk.tests.java.util.stream.CountLargeTest
04:39:58  reason: Assumed action based on file name: run testng org.openjdk.tests.java.util.stream.CountLargeTest 
04:39:58  Mode: othervm [test requires non-default system module, test or library overrides a system module]
04:39:58  Additional options from @modules: --add-modules jdk.incubator.foreign
04:39:58  Timeout information:
04:39:58  Running jcmd on process 908715
04:39:58  Dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal_testList_2/aqa-tests/TKG/output_16933818326424/jdk_util_0/work/java/util/stream/test/org/openjdk/tests/java/util/stream/CountLargeTest/core.20230830.093947.908715.0001.dmp
04:39:58  Dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal_testList_2/aqa-tests/TKG/output_16933818326424/jdk_util_0/work/java/util/stream/test/org/openjdk/tests/java/util/stream/CountLargeTest/javacore.20230830.093953.908715.0002.txt
04:39:58  Running jstack on process 908715
04:39:58  2023-08-30T09:39:53.645097230
04:39:58  Virtual machine: 908715 JVM information:
04:39:58  JRE 17 Linux amd64-64-Bit Compressed References 20230829_753 (JIT enabled, AOT enabled)
04:39:58  OpenJ9   - 39071983a8a
04:39:58  OMR      - 9659cbcdcef
04:39:58  JCL      - e6f0d592f7e based on jdk-17.0.9+4
04:39:58  
04:39:58  "main" prio=5 Id=1 WAITING
04:39:58  	at [email protected]/java.lang.Object.waitImpl(Native Method)
04:39:58  	at [email protected]/java.lang.Object.wait(Object.java:251)
04:39:58  	at [email protected]/java.lang.Object.wait(Object.java:219)
04:39:58  	at [email protected]/java.lang.Thread.join(Thread.java:784)
04:39:58  	at [email protected]/java.lang.Thread.join(Thread.java:739)
04:39:58  	at app//com.sun.javatest.regtest.agent.MainWrapper.main(MainWrapper.java:74)
04:39:58  
04:39:58  "JIT Compilation Thread-000" prio=10 Id=3 RUNNABLE
04:39:58  
04:39:58  "JIT Compilation Thread-001 Suspended" prio=10 Id=4 RUNNABLE
04:39:58  
04:39:58  "JIT Compilation Thread-002 Suspended" prio=10 Id=5 RUNNABLE
04:39:58  
04:39:58  "JIT Compilation Thread-003 Suspended" prio=10 Id=6 RUNNABLE
04:39:58  
04:39:58  "JIT Compilation Thread-004 Suspended" prio=10 Id=7 RUNNABLE
04:39:58  
04:39:58  "JIT Compilation Thread-005 Suspended" prio=10 Id=8 RUNNABLE
04:39:58  
04:39:58  "JIT Compilation Thread-006 Suspended" prio=10 Id=9 RUNNABLE
04:39:58  
04:39:58  "JIT Diagnostic Compilation Thread-007 Suspended" prio=10 Id=10 RUNNABLE
04:39:58  
04:39:58  "JIT-SamplerThread" prio=10 Id=11 TIMED_WAITING
04:39:58  
04:39:58  "IProfiler" prio=5 Id=12 RUNNABLE
04:39:58  
04:39:58  "Common-Cleaner" prio=8 Id=2 TIMED_WAITING
04:39:58  	at [email protected]/java.lang.Object.waitImpl(Native Method)
04:39:58  	at [email protected]/java.lang.Object.wait(Object.java:251)
04:39:58  	at [email protected]/java.lang.Object.wait(Object.java:219)
04:39:58  	at [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:137)
04:39:58  	at [email protected]/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
04:39:58  	at [email protected]/java.lang.Thread.run(Thread.java:857)
04:39:58  	at [email protected]/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:162)
04:39:58  
04:39:58  "Finalizer thread" prio=5 Id=13 RUNNABLE
04:39:58  
04:39:58  "Concurrent Mark Helper" prio=1 Id=14 RUNNABLE
04:39:58  
04:39:58  "GC Worker" prio=5 Id=15 RUNNABLE
04:39:58  
04:39:58  "GC Worker" prio=5 Id=16 RUNNABLE
04:39:58  
04:39:58  "GC Worker" prio=5 Id=17 RUNNABLE
04:39:58  
04:39:58  "Attach API wait loop" prio=10 Id=20 RUNNABLE
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.IPC.waitSemaphore(Native Method)
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.CommonDirectory.waitSemaphore(CommonDirectory.java:264)
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.WaitLoop.waitForNotification(WaitLoop.java:66)
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.WaitLoop.run(WaitLoop.java:157)
04:39:58  
04:39:58  "MainThread" prio=5 Id=22 RUNNABLE
04:39:58  	at [email protected]/java.util.stream.LongPipeline$4$1.accept(LongPipeline.java:251)
04:39:58  	at [email protected]/java.util.stream.Streams$RangeLongSpliterator.forEachRemaining(Streams.java:233)
04:39:58  	at [email protected]/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:775)
04:39:58  	at [email protected]/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:522)
04:39:58  	at [email protected]/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:512)
04:39:58  	at [email protected]/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
04:39:58  	at [email protected]/java.util.stream.ReduceOps$9.evaluateSequential(ReduceOps.java:434)
04:39:58  	at [email protected]/java.util.stream.ReduceOps$9.evaluateSequential(ReduceOps.java:424)
04:39:58  	at [email protected]/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:239)
04:39:58  	at [email protected]/java.util.stream.IntPipeline.count(IntPipeline.java:516)
04:39:58  	at app//org.openjdk.tests.java.util.stream.CountLargeTest.testIntLarge(CountLargeTest.java:68)
04:39:58  	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
04:39:58  	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
04:39:58  	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
04:39:58  	at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
04:39:58  	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
04:39:58  	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
04:39:58  	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
04:39:58  	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
04:39:58  	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
04:39:58  	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
04:39:58  	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
04:39:58  	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
04:39:58  	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
04:39:58  	at org.testng.TestRunner.privateRun(TestRunner.java:764)
04:39:58  	at org.testng.TestRunner.run(TestRunner.java:585)
04:39:58  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
04:39:58  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
04:39:58  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
04:39:58  	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
04:39:58  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
04:39:58  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
04:39:58  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
04:39:58  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
04:39:58  	at org.testng.TestNG.runSuites(TestNG.java:1069)
04:39:58  	at org.testng.TestNG.run(TestNG.java:1037)
04:39:58  	at app//com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:94)
04:39:58  	at app//com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:54)
04:39:58  	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
04:39:58  	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
04:39:58  	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
04:39:58  	at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
04:39:58  	at app//com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
04:39:58  	at [email protected]/java.lang.Thread.run(Thread.java:857)
04:39:58  
04:39:58  "file lock watchdog" prio=10 Id=24 TIMED_WAITING
04:39:58  	at [email protected]/java.lang.Object.waitImpl(Native Method)
04:39:58  	at [email protected]/java.lang.Object.wait(Object.java:251)
04:39:58  	at [email protected]/java.lang.Object.wait(Object.java:219)
04:39:58  	at [email protected]/java.util.TimerThread.mainLoop(Timer.java:600)
04:39:58  	at [email protected]/java.util.TimerThread.run(Timer.java:543)
04:39:58  
04:39:58  "Attachment portNumber: 41347" prio=10 Id=26 RUNNABLE
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.DiagnosticUtils.dumpAllThreadsImpl(Native Method)
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.DiagnosticUtils.getThreadInfo(DiagnosticUtils.java:249)
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.DiagnosticUtils.executeDiagnosticCommand(DiagnosticUtils.java:185)
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.Attachment.doCommand(Attachment.java:248)
04:39:58  	at [email protected]/openj9.internal.tools.attach.target.Attachment.run(Attachment.java:159)
04:39:58  
04:39:58  
04:39:58  --- Timeout information end.

These sorts of timeouts have been seen before in the JITServer nightly tests, but they always seem to be unreproducible manually. I'll see if I can reproduce any of them now.

@cjjdespres
Copy link
Contributor Author

I can reproduce the timeouts on grinder. The run https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/34591/ has failures in cmdLineTester_jython_6, though they were all on a single machine; it succeeded on the other four. The run https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/34590 for jdk_util_0 is still going, but appears to have failures in it as well.

I was unable to reproduce this locally, unfortunately. There's also the possibility that this might be related to my recent changes, so I'm going to see if it still shows up now that my latest PR has been merged.

@cjjdespres
Copy link
Contributor Author

Tests with the latest nightly do not seem to be timing out (in the nightly tests themselves, or in grinder), so the timeouts were either related to my changes (I'm still unsure how they could be) or they were caused by some transient factor.

@cjjdespres
Copy link
Contributor Author

Since it's the new year and I had a bit of a backlog to get through, I thought I'd add a quick summary of the unstable nightly tests that have failed recently and already have been reported elsewhere, just to get a sense of what's still failing:

There are a few possibly new ones that I will be posting shortly.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Jan 9, 2024

Test failure in DaaLoadTest_daa1_special_5m_10 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal/900/ due to a java.lang.NullPointerException. Console log:

DaaLoadTest_daa1_special_5m_10 Start Time: Tue Jan  2 13:25:16 2024 Epoch Time (ms): 1704201916425
variation: Mode301
JVM_OPTIONS: -XX:+UseJITServer -Xgcpolicy:metronome -Xnocompressedrefs 

STF 13:25:18.620 - ====================   E X E C U T E -   ====================
STF 13:25:18.620 - Running execute: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TKG/output_17041854359949/DaaLoadTest_daa1_special_5m_10/20240102-132516-DaaLoadTest/execute.pl
STF 13:25:18.671 - 
STF 13:25:18.671 - Java version
STF 13:25:18.671 - Running: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java -version
openjdk version "11.0.22-internal" 2024-01-16
OpenJDK Runtime Environment (build 11.0.22-internal+0-adhoc..BuildJDK11x86-64linuxjitPersonal)
Eclipse OpenJ9 VM (build master-f052734, JRE 11 Linux amd64-64-Bit Compressed References 20240102_1703 (JIT enabled, AOT enabled)
OpenJ9   - f052734
OMR      - 787725a
JCL      - 6fd19af based on jdk-11.0.22+6)

DLT class net.openj9.test.PDMoveShifts.TestShiftsAndConvert Total: 64 Fail: 1 Ignore: 0
DLT 	FAILURE: testShiftRightSimpleRound(net.openj9.test.PDMoveShifts.TestShiftsAndConvert) - null
DLT Test failed:
DLT java.lang.NullPointerException
DLT 	at java.base/java.util.regex.Matcher.groupCount(Matcher.java:700)
DLT 	at java.base/java.util.regex.Matcher.start(Matcher.java:482)
DLT 	at java.base/java.util.Formatter$FormatSpecifier.<init>(Formatter.java:2855)
DLT 	at java.base/java.util.Formatter.parse(Formatter.java:2713)
DLT 	at java.base/java.util.Formatter.format(Formatter.java:2655)
DLT 	at java.base/java.io.PrintStream.format(PrintStream.java:1053)
DLT 	at java.base/java.io.PrintStream.printf(PrintStream.java:949)
DLT 	at net.openj9.test.PDMoveShifts.ShiftTestRunner.runTests(ShiftTestRunner.java:50)
DLT 	at net.openj9.test.PDMoveShifts.ShiftTestRunner.invokeTest(ShiftTestRunner.java:32)
DLT 	at jdk.internal.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
DLT 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
DLT 	at java.base/java.lang.reflect.Method.invoke(Method.java:572)
DLT 	at net.adoptopenjdk.loadTest.adaptors.ArbitraryJavaAdaptor.executeTest(ArbitraryJavaAdaptor.java:102)
DLT 	at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:182)
DLT 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
DLT 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
DLT 	at java.base/java.lang.Thread.run(Thread.java:839)

Unable to reproduce.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Jan 9, 2024

Test failure in HeapHogLoadTest_5m_0 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal/903/. Console log:

HeapHogLoadTest_5m_0 Start Time: Fri Jan  5 10:02:29 2024 Epoch Time (ms): 1704448949825
variation: Mode110
JVM_OPTIONS: -XX:+UseJITServer -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

openjdk version "11.0.22-internal" 2024-01-16
OpenJDK Runtime Environment (build 11.0.22-internal+0-adhoc.jenkins.BuildJDK11ppc64lelinuxjitPersonal)
Eclipse OpenJ9 VM (build master-42d63ef93b0, JRE 11 Linux ppc64le-64-Bit Compressed References 20240105_1612 (JIT enabled, AOT enabled)
OpenJ9   - 42d63ef93b0
OMR      - 3a4787401ff
JCL      - 6fd19af878b based on jdk-11.0.22+6)

HHLT >>> Captured test output >>>
HHLT testStarted : testTailMap(net.adoptopenjdk.test.util.treemap.TreeMapAPITest)
HHLT testFinished: testTailMap(net.adoptopenjdk.test.util.treemap.TreeMapAPITest)
HHLT testStarted : testMapConstructor(net.adoptopenjdk.test.math.MathAPITest)
HHLT testFailure: testMapConstructor(net.adoptopenjdk.test.math.MathAPITest): Method "testMapConstructor" not found
HHLT junit.framework.AssertionFailedError: Method "testMapConstructor" not found
HHLT 	at junit.framework.Assert.fail(Assert.java:57)
HHLT 	at junit.framework.TestCase.fail(TestCase.java:227)
HHLT 	at junit.framework.TestCase.runTest(TestCase.java:169)
HHLT 	at junit.framework.TestCase.runBare(TestCase.java:141)
HHLT 	at junit.framework.TestResult$1.protect(TestResult.java:122)
HHLT 	at junit.framework.TestResult.runProtected(TestResult.java:142)
HHLT 	at junit.framework.TestResult.run(TestResult.java:125)
HHLT 	at junit.framework.TestCase.run(TestCase.java:129)
HHLT 	at junit.framework.TestSuite.runTest(TestSuite.java:252)
HHLT 	at junit.framework.TestSuite.run(TestSuite.java:247)
HHLT 	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
HHLT 	at org.junit.runners.Suite.runChild(Suite.java:128)
HHLT 	at org.junit.runners.Suite.runChild(Suite.java:27)
HHLT 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
HHLT 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
HHLT 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
HHLT 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
HHLT 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
HHLT 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
HHLT 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
HHLT 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
HHLT 	at net.adoptopenjdk.loadTest.adaptors.JUnitAdaptor.executeTest(JUnitAdaptor.java:130)
HHLT 	at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:182)
HHLT 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
HHLT 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
HHLT 	at java.base/java.lang.Thread.run(Thread.java:839)

There were various failures of the same kind in that test, e.g., a junit.framework.AssertionFailedError: Method "testComparator" not found was also thrown. Unable to reproduce.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Jan 9, 2024

Test failure in DaaLoadTest_daa1_special_5m_8 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/899/. Both a test failure (subtractPackedDecimal: arrays first differed at element [5]; expected:<1> but was:<0>) and a later Invalid JIT return address. Console log:

DaaLoadTest_daa1_special_5m_8 Start Time: Sat Jan  6 03:40:16 2024 Epoch Time (ms): 1704530416736
variation: Mode145
JVM_OPTIONS: -XX:+UseJITServer -Xgcpolicy:gencon -Xshareclasses -Xjit -Xnocompressedrefs 

openjdk version "11.0.22-internal" 2024-01-16
OpenJDK Runtime Environment (build 11.0.22-internal+0-adhoc.jenkins.BuildJDK11ppc64lelinuxjitPersonal)
Eclipse OpenJ9 VM (build master-a9b050b4bad, JRE 11 Linux ppc64le-64-Bit Compressed References 20240106_1613 (JIT enabled, AOT enabled)
OpenJ9   - a9b050b4bad
OMR      - 3a4787401ff
JCL      - 87557b83a60 based on jdk-11.0.22+6)

DLT >>> Captured test output >>>
DLT testStarted : testIntrinsicPerformance(net.openj9.test.arithmetics.TestPerformance)
DLT testFailure: testIntrinsicPerformance(net.openj9.test.arithmetics.TestPerformance): subtractPackedDecimal: arrays first differed at element [5]; expected:<1> but was:<0>
DLT subtractPackedDecimal: arrays first differed at element [5]; expected:<1> but was:<0>
DLT 	at org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:55)
DLT 	at org.junit.Assert.internalArrayEquals(Assert.java:532)
DLT 	at org.junit.Assert.assertArrayEquals(Assert.java:341)
DLT 	at net.openj9.test.arithmetics.TestPerformance.verify(TestPerformance.java:465)
DLT 	at net.openj9.test.arithmetics.TestPerformance.testSub(TestPerformance.java:140)
DLT 	at net.openj9.test.arithmetics.TestPerformance.testIntrinsicPerformance(TestPerformance.java:77)
DLT 	at jdk.internal.reflect.GeneratedMethodAccessor106.invoke(Unknown Source)
DLT 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
DLT 	at java.base/java.lang.reflect.Method.invoke(Method.java:572)
DLT 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
DLT 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
DLT 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
DLT 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
DLT 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
DLT 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
DLT 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
DLT 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
DLT 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
DLT 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
DLT 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
DLT 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
DLT 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
DLT 	at org.junit.runners.Suite.runChild(Suite.java:128)
DLT 	at org.junit.runners.Suite.runChild(Suite.java:27)
DLT 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
DLT 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
DLT 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
DLT 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
DLT 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
DLT 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
DLT 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
DLT 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
DLT 	at net.adoptopenjdk.loadTest.adaptors.JUnitAdaptor.executeTest(JUnitAdaptor.java:130)
DLT 	at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:182)
DLT 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
DLT 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
DLT 	at java.base/java.lang.Thread.run(Thread.java:839)
DLT Caused by: java.lang.AssertionError: expected:<1> but was:<0>
DLT 	at org.junit.Assert.fail(Assert.java:88)
DLT 	at org.junit.Assert.failNotEquals(Assert.java:834)
DLT 	at org.junit.Assert.assertEquals(Assert.java:118)
DLT 	at org.junit.Assert.assertEquals(Assert.java:144)
DLT 	at org.junit.internal.ExactComparisonCriteria.assertElementsEqual(ExactComparisonCriteria.java:8)
DLT 	at org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:53)
DLT 	... 36 more
DLT testFinished: testIntrinsicPerformance(net.openj9.test.arithmetics.TestPerformance)
DLT 
DLT JUnit Test Results for: net.openj9.test.arithmetics.TestPerformance
DLT   Ran    : 1
DLT   Passed : 0
DLT   Failed : 1
DLT   Ignored: 0
DLT   Result : FAILED
DLT <<<
DLT 
DLT 03:43:43.967 - suite.getInventory().getInventoryFileRef(): openj9.test.load/config/inventories/daa/daa1.xml
DLT 03:43:43.968 - suite.isCreateDump() : false
DLT 03:43:43.968 - Test failed. Details recorded in execution log.
DLT 03:43:43.970 - suite.getInventory().getInventoryFileRef(): openj9.test.load/config/inventories/daa/daa1.xml
DLT 03:43:43.970 - suite.isCreateDump() : false
DLT 03:43:43.971 - Test failed. Details recorded in execution log.
DLT 03:43:43.992 - suite.getInventory().getInventoryFileRef(): openj9.test.load/config/inventories/daa/daa1.xml
DLT 03:43:43.993 - suite.isCreateDump() : false
DLT 03:43:43.993 - Test failed. Details recorded in execution log.
DLT stderr 
DLT stderr 
DLT stderr *** Invalid JIT return address 00007FFEF37E3B58 in 00007FFEF14BB9E8
DLT stderr 
DLT stderr 08:43:44.034 0x7ffe4c002400    j9vm.249    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64le_linux_jit_Personal/openj9/runtime/vm/swalk.c:1633: ((0 ))

Unable to reproduce.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Jan 9, 2024

Test failure in jdk_util_1 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_sanity.openjdk_ppc64le_linux_jit_Personal/813/, in java/util/concurrent/ArrayBlockingQueue/WhiteBox. Console log:

04:20:17  ===============================================
04:20:17  Running test jdk_util_1 ...
04:20:17  ===============================================
04:20:17  jdk_util_1 Start Time: Tue Jan  9 01:20:13 2024 Epoch Time (ms): 1704792013825
04:20:17  variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache Mode650
04:20:17  JVM_OPTIONS: -XX:+UseJITServer -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-UseCompressedOops -Xverbosegclog 

04:25:28  test WhiteBox.incrementalSweepingOfUnreachableIterators(): failure
04:25:28  java.lang.AssertionError: failed to do a "full" gc
04:25:28  	at WhiteBox.forceFullGc(WhiteBox.java:226)
04:25:28  	at WhiteBox.incrementalSweepingOfUnreachableIterators(WhiteBox.java:555)
04:25:28  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
04:25:28  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
04:25:28  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
04:25:28  	at java.base/java.lang.reflect.Method.invoke(Method.java:574)
04:25:28  	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
04:25:28  	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
04:25:28  	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
04:25:28  	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
04:25:28  	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
04:25:28  	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
04:25:28  	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
04:25:28  	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
04:25:28  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
04:25:28  	at org.testng.TestRunner.privateRun(TestRunner.java:764)
04:25:28  	at org.testng.TestRunner.run(TestRunner.java:585)
04:25:28  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
04:25:28  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
04:25:28  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
04:25:28  	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
04:25:28  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
04:25:28  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
04:25:28  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
04:25:28  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
04:25:28  	at org.testng.TestNG.runSuites(TestNG.java:1069)
04:25:28  	at org.testng.TestNG.run(TestNG.java:1037)
04:25:28  	at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:102)
04:25:28  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
04:25:28  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
04:25:28  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
04:25:28  	at java.base/java.lang.reflect.Method.invoke(Method.java:574)
04:25:28  	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:333)
04:25:28  	at java.base/java.lang.Thread.run(Thread.java:857)

Unable to reproduce.

@cjjdespres
Copy link
Contributor Author

None of the recent test failures could be reproduced in Grinder.

@cjjdespres
Copy link
Contributor Author

The recent high level of test failures are due to the ROM class packing issue. I'm still looking into that.

@cjjdespres
Copy link
Contributor Author

The tests appear to have settled down after the ROM class packing fix and the revert of the PR that was causing failing builds.
Tests that have failed so far all seem to be the usual failures or are otherwise known:

Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport -XX:+UseJITServer -Xjit:count=0  -cp /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.CRIUSimpleTest 1 1
 [OUT] Total checkpoint(s) 1:
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Wed Feb 21 00:59:13 PST 2024, System.currentTimeMillis(): 1708505953382, System.nanoTime(): 1708505841481212193
 [OUT] Post-checkpoint 1
 [OUT] finished script
 [ERR] cat: criuOutput: No such file or directory
>> Success condition was not found: [Output match: Killed]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Failure condition was found: [Output match: Post-checkpoint]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Removed testOutput file]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Failure condition was not found: [Output match: ERR]
>> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]
>> Failure condition was not found: [Output match: User requested Java dump using]

which has been seen before, I think.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Mar 4, 2024

Failures since I've been away:

There were test failures in CRIU-related tests with the following error:

 [OUT] (00.029294) Error (criu/proc_parse.c:474): Can't open map_files: Permission denied
 [OUT] (00.029314) Error (criu/proc_parse.c:694): Can't open 9568's mapfile link 7fc7a8cca000: Permission denied
 [OUT] (00.029336) Error (criu/cr-dump.c:1558): Collect mappings (pid: 9568) failed with -1
 [OUT] (00.029654) net: Unlock network
 [OUT] (00.029677) Unfreezing tasks into 1
 [OUT] (00.029695) 	Unseizing 9568 into 1
 [OUT] (00.030079) Error (criu/cr-dump.c:2093): Dumping FAILED.

in

that feels like some kind of infrastructure problem. I haven't seen an issue opened for it. Appears to have been reported in #19073 as well.

@cjjdespres
Copy link
Contributor Author

Failure in testSCCacheManagement_0, in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/908/. Error was No file found at '/home/jenkins/.cache/javasharedresources/C290M17F1A64P_Foo_G43L00', which appears to be tracked here: #8511.

@cjjdespres
Copy link
Contributor Author

A lot of failures on ppc64le and s390x, e.g. in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk8_j9_sanity.functional_s390x_linux_jit_Personal/1241/. All of them appear to have the same message JVMJ9VM007E Command-line option unrecognised: -Xgcpolicy:metronome. Appears to be tracked by #19115.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Mar 22, 2024

Failure in DaaLoadTest_daa1_special_5m_8 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal/968/. A segfault. Console log:

        STF 07:47:31.937 - ====================   E X E C U T E -   ====================
        STF 07:47:31.937 - Running execute: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/../TKG/output_1711006906892/DaaLoadTest_daa1_special_5m_8/20240321-074728-DaaLoadTest/execute.pl
        STF 07:47:31.995 - 
        STF 07:47:31.995 - Java version
        STF 07:47:31.995 - Running: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java -version
        openjdk version "11.0.23-internal" 2024-04-16
        OpenJDK Runtime Environment (build 11.0.23-internal+0-adhoc.jenkins.BuildJDK11s390xlinuxjitPersonal)
        Eclipse OpenJ9 VM (build master-f2c434f2126, JRE 11 Linux s390x-64-Bit Compressed References 20240320_1373 (JIT enabled, AOT enabled)
        OpenJ9   - f2c434f2126
        OMR      - 73f2f2035d3
        JCL      - b59f8e52b58 based on jdk-11.0.23+6)
        STF 07:47:32.115 - 
        STF 07:47:32.115 - +------ Step 1 - Run daa load test
        STF 07:47:32.115 - | Run foreground process
        STF 07:47:32.115 - |   Program:     /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java
        STF 07:47:32.115 - |   Mnemonic:    DLT
        STF 07:47:32.115 - |   Echo:        ECHO_ON
        STF 07:47:32.115 - |   Expectation: CLEAN_RUN within 35m
        STF 07:47:32.115 - |
        STF 07:47:32.115 - Running command: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java -XX:+UseJITServer -Xgcpolicy:gencon -Xshareclasses -Xjit -Xnocompressedrefs -Xdump:system:events=user -classpath /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jvmtest/system/STF/stf.load/bin:/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jvmtest/system/STF/stf.core/bin:/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jvmtest/system/systemtest_prereqs/log4j/log4j-api.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jvmtest/system/systemtest_prereqs/log4j/log4j-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jvmtest/system/systemtest_prereqs/junit/junit.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jvmtest/system/systemtest_prereqs/junit/hamcrest-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jvmtest/system/openj9-systemtest/openj9.test.daa/bin net.adoptopenjdk.loadTest.LoadTest -resultsDir /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/output_1711006906892/DaaLoadTest_daa1_special_5m_8/20240321-074728-DaaLoadTest/results -resultsPrefix 1.DLT. -timeLimit 5m -dumpRequested false -reportFailureLimit 1 -abortAtFailureLimit 10 -maxTotalLogFileSpace 200M -maxSingleLogSize 1/25 -suite.daa.threadCount 2 -suite.daa.inventoryFile /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/output_1711006906892/DaaLoadTest_daa1_special_5m_8/20240321-074728-DaaLoadTest/results/1.DLT.inventory/openj9.test.load/config/inventories/daa/daa1.xml -suite.daa.inventoryExcludeFile none -suite.daa.selection random -suite.daa.seed -1 -suite.daa.repeatCount 1 -suite.daa.thinkingTime 0ms..0ms
        STF 07:47:32.115 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/output_1711006906892/DaaLoadTest_daa1_special_5m_8/20240321-074728-DaaLoadTest/results/1.DLT.stderr
        STF 07:47:32.115 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/output_1711006906892/DaaLoadTest_daa1_special_5m_8/20240321-074728-DaaLoadTest/results/1.DLT.stdout
        STF 07:47:32.125 - Monitoring processes: DLT
        DLT 07:47:35.620 - Load test parameters
        DLT 07:47:35.655 -   Time limited         = true
        DLT 07:47:35.656 -   Time limit         = 5m
        DLT 07:47:35.656 -   abortIfOutOfMemory   = true
        DLT 07:47:35.657 -   reportFailureLimit   = 1
        DLT 07:47:35.658 -   abortAtFailureLimit  = 10
        DLT 07:47:35.664 -   maxTotalLogFileSpace = 209715200
        DLT 07:47:35.664 -   maxSingleLogSize     = 8388608
        DLT 07:47:35.665 - Parameters for suite 0
        DLT 07:47:35.665 -   Suite name     = daa
        DLT 07:47:35.665 -   Number threads = 2
        DLT 07:47:35.666 -   Supplied seed  = -1
        DLT 07:47:35.666 -   Inventory file = /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/output_1711006906892/DaaLoadTest_daa1_special_5m_8/20240321-074728-DaaLoadTest/results/1.DLT.inventory/openj9.test.load/config/inventories/daa/daa1.xml
        DLT 07:47:35.667 -   Exclude file   = none
        DLT 07:47:35.667 -   Number tests   = -1
        DLT 07:47:35.668 -   Repeat count   = 1
        DLT 07:47:35.668 -   Thinking time  = 0ms..0ms
        DLT 07:47:35.668 -   Selection mode = random
        DLT 07:47:35.670 -   Actual seed    = 1711046756415
        DLT 07:47:35.678 - Parsing inventory file. Root=/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/output_1711006906892/DaaLoadTest_daa1_special_5m_8/20240321-074728-DaaLoadTest/results/1.DLT.inventory File=openj9.test.load/config/inventories/daa/daa1.xml
        DLT 07:47:35.701 - Final test list:
        DLT 07:47:35.704 -   0 ArbitraryJava[net.openj9.test.simple.ConvertDecimal invokeTest]  Weighting=1 
        DLT 07:47:35.704 -   1 ArbitraryJava[net.openj9.test.simple.MarshalUnmarshalBinary invokeTest]  Weighting=1 
        DLT 07:47:35.705 -   2 JUnit[net.openj9.test.arithmetics.TestArithmeticOperations]  Weighting=1 
        DLT 07:47:35.705 -   3 JUnit[net.openj9.test.arithmetics.TestArithmeticInline]  Weighting=1 
        DLT 07:47:35.705 -   4 JUnit[net.openj9.test.arithmetics.TestArithmetics]  Weighting=1 
        DLT 07:47:35.706 -   5 JUnit[net.openj9.test.arithmetics.TestComparisonEquals]  Weighting=1 
        DLT 07:47:35.706 -   6 JUnit[net.openj9.test.arithmetics.TestPDComparisons]  Weighting=1 
        DLT 07:47:35.706 -   7 JUnit[net.openj9.test.arithmetics.TestPerformance]  Weighting=1 
        DLT 07:47:35.706 -   8 JUnit[net.openj9.test.arithmetics.TestSubExceptions]  Weighting=1 
        DLT 07:47:35.707 -   9 JUnit[net.openj9.test.arithmetics.TestValidityChecking]  Weighting=1 
        DLT 07:47:35.707 -   10 JUnit[net.openj9.test.binaryData.LongIntegerComparison]  Weighting=1 
        DLT 07:47:35.707 -   11 JUnit[net.openj9.test.binaryData.TestOptimizer]  Weighting=1 
        DLT 07:47:35.707 -   12 JUnit[net.openj9.test.decimals.TestBD2PD2BD]  Weighting=1 
        DLT 07:47:35.707 -   13 JUnit[net.openj9.test.decimals.TestDecimalData]  Weighting=1 
        DLT 07:47:35.707 -   14 JUnit[net.openj9.test.decimals.TestDecimalData2]  Weighting=1 
        DLT 07:47:35.707 -   15 JUnit[net.openj9.test.PD2Primitive.TestPD2Primitives2PD]  Weighting=1 
        DLT 07:47:35.708 -   16 JUnit[net.openj9.test.PDMoveShifts.TestShiftsAndConvert]  Weighting=1 
        DLT 07:47:35.708 -   17 JUnit[net.openj9.test.PDMoveShifts.TestPDMove]  Weighting=1 
        DLT 07:47:35.708 -   18 ArbitraryJava[net.openj9.test.PDMoveShifts.ShiftTestRunner invokeTest]  Weighting=1 
        DLT 07:47:35.824 - Starting thread. Suite=0 thread=0
        DLT 07:47:35.826 - Starting thread. Suite=0 thread=1
        DLT 07:47:55.872 - Completed 6.7%. Number of tests started=227
        DLT stderr #0: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa6552e) [0x3ff9ac6552e]
        DLT stderr #1: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa72e98) [0x3ff9ac72e98]
        DLT stderr #2: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x17084a) [0x3ff9a37084a]
        DLT stderr #3: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x3373c) [0x3ffa01b373c]
        DLT stderr #4: function __kernel_rt_sigreturn+0 [0]
        DLT stderr #5: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b1946) [0x3ff9a4b1946]
        DLT stderr #6: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b1df2) [0x3ff9a4b1df2]
        DLT stderr #7: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x89e074) [0x3ff9aa9e074]
        DLT stderr #8: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x89ea22) [0x3ff9aa9ea22]
        DLT stderr #9: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8b93e0) [0x3ff9aab93e0]
        DLT stderr #10: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c7ffe) [0x3ff9aac7ffe]
        DLT stderr #11: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c8ad6) [0x3ff9aac8ad6]
        DLT stderr #12: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6a3f44) [0x3ff9a8a3f44]
        DLT stderr #13: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x685640) [0x3ff9a885640]
        DLT stderr #14: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1899be) [0x3ff9a3899be]
        DLT stderr #15: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18aaa0) [0x3ff9a38aaa0]
        DLT stderr #16: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34626) [0x3ffa01b4626]
        DLT stderr #17: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1880aa) [0x3ff9a3880aa]
        DLT stderr #18: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x188752) [0x3ff9a388752]
        DLT stderr #19: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18715e) [0x3ff9a38715e]
        DLT stderr #20: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18764e) [0x3ff9a38764e]
        DLT stderr #21: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187732) [0x3ff9a387732]
        DLT stderr #22: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34626) [0x3ffa01b4626]
        DLT stderr #23: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187bce) [0x3ff9a387bce]
        DLT stderr #24: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x5a14) [0x3ffa0105a14]
        DLT stderr #25: /lib/s390x-linux-gnu/libc.so.6(+0x96296) [0x3ffa0816296]
        DLT stderr #26: /lib/s390x-linux-gnu/libc.so.6(+0x10ff8e) [0x3ffa088ff8e]
        DLT stderr Unhandled exception
        DLT stderr Type=Segmentation error vmState=0x000532ff
        DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
        DLT stderr Handler1=000003FF9BD4D9E0 Handler2=000003FFA01B34B8 InaccessibleAddress=0000000000000000
        DLT stderr gpr0=0000000000000010 gpr1=0000000000000000 gpr2=000003FF14571330 gpr3=0000000000000E50
        DLT stderr gpr4=0000000000000C89 gpr5=0000000000000000 gpr6=00000000000001CA gpr7=0000000051954306
        DLT stderr gpr8=000003FEFCBC9660 gpr9=000003FF14571330 gpr10=000003FEA8052800 gpr11=00000000000001CA
        DLT stderr gpr12=000003FF9B094000 gpr13=000003FF9AEBCBC0 gpr14=000003FF9A4B1DF2 gpr15=000003FEFFFF6C00
        DLT stderr psw=000003FF9A4B1946 mask=0705200180000000 fpc=00880000 bea=000003FF9A4B1DEC
        DLT stderr fpr0 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
        DLT stderr fpr1 0000000000000007 (f: 7.000000, d: 3.458460e-323)
        DLT stderr fpr2 000003fef4000140 (f: 4093640960.000000, d: 2.170702e-311)
        DLT stderr fpr3 000003fef4000918 (f: 4093643008.000000, d: 2.170702e-311)
        DLT stderr fpr4 000003fef40008ee (f: 4093643008.000000, d: 2.170702e-311)
        DLT stderr fpr5 45781b3dd66ea263 (f: 3597574656.000000, d: 4.662858e+26)
        DLT stderr fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr7 401524482f152d05 (f: 789916928.000000, d: 5.285432e+00)
        DLT stderr fpr8 000003feffeff000 (f: 4293914624.000000, d: 2.170801e-311)
        DLT stderr fpr9 0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
        DLT stderr fpr10 000003ff9c0ebc78 (f: 2618211328.000000, d: 2.172095e-311)
        DLT stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr12 000003fefffff900 (f: 4294965504.000000, d: 2.170802e-311)
        DLT stderr fpr13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
        DLT stderr fpr14 000003feffeff000 (f: 4293914624.000000, d: 2.170801e-311)
        DLT stderr fpr15 0000000000e55380 (f: 15029120.000000, d: 7.425372e-317)
        DLT stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
        DLT stderr Module_base_address=000003FF9A200000
        DLT stderr 
        DLT stderr Method_being_compiled=org/junit/runners/BlockJUnit4ClassRunner.runChild(Lorg/junit/runners/model/FrameworkMethod;Lorg/junit/runner/notification/RunNotifier;)V
        DLT stderr Target=2_90_20240320_1373 (Linux 5.15.0-100-generic)
        DLT stderr CPU=s390x (4 logical CPUs) (0x1e8e5b000 RAM)
        DLT stderr ----------- Stack Backtrace -----------
        DLT stderr #0: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa6552e) [0x3ff9ac6552e]
        DLT stderr #1: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa72e98) [0x3ff9ac72e98]
        DLT stderr #2: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x17084a) [0x3ff9a37084a]
        DLT stderr #3: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x3373c) [0x3ffa01b373c]
        DLT stderr #4: function __kernel_rt_sigreturn+0 [0]
        DLT stderr #5: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b1946) [0x3ff9a4b1946]
        DLT stderr #6: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b1df2) [0x3ff9a4b1df2]
        DLT stderr #7: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x89e074) [0x3ff9aa9e074]
        DLT stderr #8: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x89ea22) [0x3ff9aa9ea22]
        DLT stderr #9: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8b93e0) [0x3ff9aab93e0]
        DLT stderr #10: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c7ffe) [0x3ff9aac7ffe]
        DLT stderr #11: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c8ad6) [0x3ff9aac8ad6]
        DLT stderr #12: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6a3f44) [0x3ff9a8a3f44]
        DLT stderr #13: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x685640) [0x3ff9a885640]
        DLT stderr #14: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1899be) [0x3ff9a3899be]
        DLT stderr #15: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18aaa0) [0x3ff9a38aaa0]
        DLT stderr #16: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34626) [0x3ffa01b4626]
        DLT stderr #17: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1880aa) [0x3ff9a3880aa]
        DLT stderr #18: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x188752) [0x3ff9a388752]
        DLT stderr #19: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18715e) [0x3ff9a38715e]
        DLT stderr #20: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18764e) [0x3ff9a38764e]
        DLT stderr #21: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187732) [0x3ff9a387732]
        DLT stderr #22: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34626) [0x3ffa01b4626]
        DLT stderr #23: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187bce) [0x3ff9a387bce]
        DLT stderr #24: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x5a14) [0x3ffa0105a14]
        DLT stderr #25: /lib/s390x-linux-gnu/libc.so.6(+0x96296) [0x3ffa0816296]
        DLT stderr #26: /lib/s390x-linux-gnu/libc.so.6(+0x10ff8e) [0x3ffa088ff8e]
        DLT stderr Unhandled exception
        DLT stderr Type=Segmentation error vmState=0x000532ff
        DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
        DLT stderr Handler1=000003FF9BD4D9E0 Handler2=000003FFA01B34B8 InaccessibleAddress=0000000000000000
        DLT stderr gpr0=0000000000000010 gpr1=0000000000000000 gpr2=000003FF14571330 gpr3=0000000000000E50
        DLT stderr gpr4=0000000000000C89 gpr5=0000000000000000 gpr6=00000000000001CA gpr7=0000000051954306
        DLT stderr gpr8=000003FEFD1CCBF0 gpr9=000003FF14571330 gpr10=000003FEA8052800 gpr11=00000000000001CA
        DLT stderr gpr12=000003FF9B094000 gpr13=000003FF9AEBCBC0 gpr14=000003FF9A4B1DF2 gpr15=000003FEFFEF5C00
        DLT stderr psw=000003FF9A4B1946 mask=0705200180000000 fpc=00880000 bea=000003FF9A4B1DEC
        DLT stderr fpr0 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
        DLT stderr fpr1 0000000000000007 (f: 7.000000, d: 3.458460e-323)
        DLT stderr fpr2 000003feec000620 (f: 3959424512.000000, d: 2.170636e-311)
        DLT stderr fpr3 000003feec000928 (f: 3959425280.000000, d: 2.170636e-311)
        DLT stderr fpr4 000003feec0008f2 (f: 3959425280.000000, d: 2.170636e-311)
        DLT stderr fpr5 45781b3dd66ea263 (f: 3597574656.000000, d: 4.662858e+26)
        DLT stderr fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr7 401524482f152d05 (f: 789916928.000000, d: 5.285432e+00)
        DLT stderr fpr8 000003feffdfe000 (f: 4292861952.000000, d: 2.170801e-311)
        DLT stderr fpr9 0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
        DLT stderr fpr10 000003ff9c0ec170 (f: 2618212608.000000, d: 2.172095e-311)
        DLT stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr12 000003feffefe900 (f: 4293912832.000000, d: 2.170801e-311)
        DLT stderr fpr13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
        DLT stderr fpr14 000003feffdfe000 (f: 4292861952.000000, d: 2.170801e-311)
        DLT stderr fpr15 0000000000e55380 (f: 15029120.000000, d: 7.425372e-317)
        DLT stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
        DLT stderr Module_base_address=000003FF9A200000
        DLT stderr 
        DLT stderr Method_being_compiled=org/junit/runners/BlockJUnit4ClassRunner.methodBlock(Lorg/junit/runners/model/FrameworkMethod;)Lorg/junit/runners/model/Statement;
        DLT stderr Target=2_90_20240320_1373 (Linux 5.15.0-100-generic)
        DLT stderr CPU=s390x (4 logical CPUs) (0x1e8e5b000 RAM)
        DLT stderr ----------- Stack Backtrace -----------
        DLT stderr #0: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa6552e) [0x3ff9ac6552e]
        DLT stderr #1: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa72e98) [0x3ff9ac72e98]
        DLT stderr #2: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x17084a) [0x3ff9a37084a]
        DLT stderr #3: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x3373c) [0x3ffa01b373c]
        DLT stderr #4: function __kernel_rt_sigreturn+0 [0]
        DLT stderr #5: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b1946) [0x3ff9a4b1946]
        DLT stderr #6: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b1df2) [0x3ff9a4b1df2]
        DLT stderr #7: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x89e074) [0x3ff9aa9e074]
        DLT stderr #8: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x89ea22) [0x3ff9aa9ea22]
        DLT stderr #9: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8b93e0) [0x3ff9aab93e0]
        DLT stderr #10: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c7ffe) [0x3ff9aac7ffe]
        DLT stderr #11: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c8ad6) [0x3ff9aac8ad6]
        DLT stderr #12: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6a3f44) [0x3ff9a8a3f44]
        DLT stderr #13: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x685640) [0x3ff9a885640]
        DLT stderr #14: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1899be) [0x3ff9a3899be]
        DLT stderr #15: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18aaa0) [0x3ff9a38aaa0]
        DLT stderr #16: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34626) [0x3ffa01b4626]
        DLT stderr #17: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1880aa) [0x3ff9a3880aa]
        DLT stderr #18: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x188752) [0x3ff9a388752]
        DLT stderr #19: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18715e) [0x3ff9a38715e]
        DLT stderr #20: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18764e) [0x3ff9a38764e]
        DLT stderr #21: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187732) [0x3ff9a387732]
        DLT stderr #22: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34626) [0x3ffa01b4626]
        DLT stderr #23: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187bce) [0x3ff9a387bce]
        DLT stderr #24: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x5a14) [0x3ffa0105a14]
        DLT stderr #25: /lib/s390x-linux-gnu/libc.so.6(+0x96296) [0x3ffa0816296]
        DLT stderr #26: /lib/s390x-linux-gnu/libc.so.6(+0x10ff8e) [0x3ffa088ff8e]
        DLT stderr Unhandled exception
        DLT stderr Type=Segmentation error vmState=0x000532ff
        DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
        DLT stderr Handler1=000003FF9BD4D9E0 Handler2=000003FFA01B34B8 InaccessibleAddress=0000000000000000
        DLT stderr gpr0=0000000000000010 gpr1=0000000000000000 gpr2=000003FF14571330 gpr3=0000000000000E50
        DLT stderr gpr4=0000000000000C89 gpr5=0000000000000000 gpr6=00000000000001CA gpr7=0000000051954306
        DLT stderr gpr8=000003FEFDDCD400 gpr9=000003FF14571330 gpr10=000003FEA8052800 gpr11=00000000000001CA
        DLT stderr gpr12=000003FF9B094000 gpr13=000003FF9AEBCBC0 gpr14=000003FF9A4B1DF2 gpr15=000003FF041F4C00
        DLT stderr psw=000003FF9A4B1946 mask=0705200180000000 fpc=00880000 bea=000003FF9A4B1DEC
        DLT stderr #0: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa6552e) [0x3ff9ac6552e]
        DLT stderr #1: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa72e98) [0x3ff9ac72e98]
        DLT stderr #2: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x17084a) [0x3ff9a37084a]
        DLT stderr #3: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x3373c) [0x3ffa01b373c]
        DLT stderr fpr0 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
        DLT stderr fpr1 0000000000000007 (f: 7.000000, d: 3.458460e-323)
        DLT stderr fpr2 45362000f8000720 (f: 4160751360.000000, d: 2.674750e+25)
        DLT stderr fpr3 000003fef80010d8 (f: 4160753920.000000, d: 2.170735e-311)
        DLT stderr fpr4 000003fef8000ade (f: 4160752384.000000, d: 2.170735e-311)
        DLT stderr fpr5 45781b3dd66ea263 (f: 3597574656.000000, d: 4.662858e+26)
        DLT stderr fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr7 401524482f152d05 (f: 789916928.000000, d: 5.285432e+00)
        DLT stderr fpr8 000003ff040fd000 (f: 68145152.000000, d: 2.170835e-311)
        DLT stderr fpr9 0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
        DLT stderr fpr10 000003ff9c0eb780 (f: 2618210304.000000, d: 2.172095e-311)
        DLT stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr12 000003ff041fd900 (f: 69196032.000000, d: 2.170836e-311)
        DLT stderr fpr13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
        DLT stderr fpr14 000003ff040fd000 (f: 68145152.000000, d: 2.170835e-311)
        DLT stderr fpr15 0000000000e55380 (f: 15029120.000000, d: 7.425372e-317)
        DLT stderr #4: function __kernel_rt_sigreturn+0 [0]
        DLT stderr #5: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b1946) [0x3ff9a4b1946]
        DLT stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
        DLT stderr Module_base_address=000003FF9A200000
        DLT stderr #6: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b1df2) [0x3ff9a4b1df2]
        DLT stderr #7: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x89e074) [0x3ff9aa9e074]
        DLT stderr 
        DLT stderr Method_being_compiled=org/junit/runners/BlockJUnit4ClassRunner.withRules(Lorg/junit/runners/model/FrameworkMethod;Ljava/lang/Object;Lorg/junit/runners/model/Statement;)Lorg/junit/runners/model/Statement;
        DLT stderr #8: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x89ea22) [0x3ff9aa9ea22]
        DLT stderr Target=2_90_20240320_1373 (Linux 5.15.0-100-generic)
        DLT stderr #9: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8b93e0) [0x3ff9aab93e0]
        DLT stderr CPU=s390x (4 logical CPUs) (0x1e8e5b000 RAM)
        DLT stderr #10: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c7ffe) [0x3ff9aac7ffe]
        DLT stderr ----------- Stack Backtrace -----------
        DLT stderr #11: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c8ad6) [0x3ff9aac8ad6]
        DLT stderr #12: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6a3f44) [0x3ff9a8a3f44]
        DLT stderr #13: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x685640) [0x3ff9a885640]
        DLT stderr #14: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1899be) [0x3ff9a3899be]
        DLT stderr #15: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18aaa0) [0x3ff9a38aaa0]
        DLT stderr #16: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34626) [0x3ffa01b4626]
        DLT stderr #17: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1880aa) [0x3ff9a3880aa]
        DLT stderr #18: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x188752) [0x3ff9a388752]
        DLT stderr #19: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18715e) [0x3ff9a38715e]
        DLT stderr #20: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18764e) [0x3ff9a38764e]
        DLT stderr #21: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187732) [0x3ff9a387732]
        DLT stderr #22: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34626) [0x3ffa01b4626]
        DLT stderr #23: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187bce) [0x3ff9a387bce]
        DLT stderr #24: /home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x5a14) [0x3ffa0105a14]
        DLT stderr #25: /lib/s390x-linux-gnu/libc.so.6(+0x96296) [0x3ffa0816296]
        DLT stderr #26: /lib/s390x-linux-gnu/libc.so.6(+0x10ff8e) [0x3ffa088ff8e]
        DLT stderr Unhandled exception
        DLT stderr Type=Segmentation error vmState=0x000532ff
        DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
        DLT stderr Handler1=000003FF9BD4D9E0 Handler2=000003FFA01B34B8 InaccessibleAddress=0000000000000000
        DLT stderr gpr0=0000000000000010 gpr1=0000000000000000 gpr2=000003FF14571330 gpr3=0000000000000E50
        DLT stderr gpr4=0000000000000C89 gpr5=0000000000000000 gpr6=00000000000001CA gpr7=0000000051954306
        DLT stderr gpr8=000003FEFE450400 gpr9=000003FF14571330 gpr10=000003FEA8052800 gpr11=00000000000001CA
        DLT stderr gpr12=000003FF9B094000 gpr13=000003FF9AEBCBC0 gpr14=000003FF9A4B1DF2 gpr15=000003FF044F5C00
        DLT stderr psw=000003FF9A4B1946 mask=0705200180000000 fpc=00880000 bea=000003FF9A4B1DEC
        DLT stderr fpr0 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
        DLT stderr fpr1 0000000000000007 (f: 7.000000, d: 3.458460e-323)
        DLT stderr fpr2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr3 000003ff000010d8 (f: 4312.000000, d: 2.170802e-311)
        DLT stderr fpr4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr5 45781b3dd66ea263 (f: 3597574656.000000, d: 4.662858e+26)
        DLT stderr fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr7 401524482f152d05 (f: 789916928.000000, d: 5.285432e+00)
        DLT stderr fpr8 000003ff043fe000 (f: 71294976.000000, d: 2.170837e-311)
        DLT stderr fpr9 0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
        DLT stderr fpr10 000003ff9c01ef70 (f: 2617372416.000000, d: 2.172095e-311)
        DLT stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr fpr12 000003ff044fe900 (f: 72345856.000000, d: 2.170837e-311)
        DLT stderr fpr13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
        DLT stderr fpr14 000003ff043fe000 (f: 71294976.000000, d: 2.170837e-311)
        DLT stderr fpr15 0000000000e55380 (f: 15029120.000000, d: 7.425372e-317)
        DLT stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
        DLT stderr Module_base_address=000003FF9A200000
        DLT stderr 
        DLT stderr Method_being_compiled=org/junit/runners/BlockJUnit4ClassRunner.getMethodRules(Ljava/lang/Object;)Ljava/util/List;
        DLT stderr Target=2_90_20240320_1373 (Linux 5.15.0-100-generic)
        DLT stderr CPU=s390x (4 logical CPUs) (0x1e8e5b000 RAM)
        DLT stderr ----------- Stack Backtrace -----------
        DLT stderr _ZN26JITClientPersistentCHTable9markDirtyEP19TR_OpaqueClassBlock+0x46 (0x000003FF9A4B1946 [libj9jit29.so+0x2b1946])
        DLT stderr _ZN31TR_JITClientPersistentClassInfo21incNumPrexAssumptionsEv+0x32 (0x000003FF9A4B1DF2 [libj9jit29.so+0x2b1df2])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence19processIndirectCallEPN2TR4NodeEPNS0_7TreeTopEt+0x167c (0x000003FF9AA9E074 [libj9jit29.so+0x89e074])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence11processNodeEPN2TR4NodeEPNS0_7TreeTopEt+0x5a (0x000003FF9AA9EA22 [libj9jit29.so+0x89ea22])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence7performEv+0xf68 (0x000003FF9AAB93E0 [libj9jit29.so+0x8b93e0])
        DLT stderr _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii+0x184e (0x000003FF9AAC7FFE [libj9jit29.so+0x8c7ffe])
        DLT stderr _ZN3OMR9Optimizer8optimizeEv+0x1f6 (0x000003FF9AAC8AD6 [libj9jit29.so+0x8c8ad6])
        DLT stderr _ZN3OMR20ResolvedMethodSymbol5genILEP11TR_FrontEndPN2TR11CompilationEPNS3_20SymbolReferenceTableERNS3_12IlGenRequestE+0x444 (0x000003FF9A8A3F44 [libj9jit29.so+0x6a3f44])
        DLT stderr _ZN3OMR11Compilation7compileEv+0x1e8 (0x000003FF9A885640 [libj9jit29.so+0x685640])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x4c6 (0x000003FF9A3899BE [libj9jit29.so+0x1899be])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x378 (0x000003FF9A38AAA0 [libj9jit29.so+0x18aaa0])
        DLT stderr omrsig_protect+0x366 (0x000003FFA01B4626 [libj9prt29.so+0x34626])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x332 (0x000003FF9A3880AA [libj9jit29.so+0x1880aa])
        DLT stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1d2 (0x000003FF9A388752 [libj9jit29.so+0x188752])
        DLT stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3c6 (0x000003FF9A38715E [libj9jit29.so+0x18715e])
        DLT stderr _ZN2TR24CompilationInfoPerThread3runEv+0x66 (0x000003FF9A38764E [libj9jit29.so+0x18764e])
        DLT stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x9a (0x000003FF9A387732 [libj9jit29.so+0x187732])
        DLT stderr omrsig_protect+0x366 (0x000003FFA01B4626 [libj9prt29.so+0x34626])
        DLT stderr _Z21compilationThreadProcPv+0x1fe (0x000003FF9A387BCE [libj9jit29.so+0x187bce])
        DLT stderr thread_wrapper+0x114 (0x000003FFA0105A14 [libj9thr29.so+0x5a14])
        DLT stderr  (0x000003FFA0816296 [libc.so.6+0x96296])
        DLT stderr  (0x000003FFA088FF8E [libc.so.6+0x10ff8e])
        DLT stderr ---------------------------------------
        DLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2024/03/21 07:48:02 - please wait.
        DLT stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/output_1711006906892/DaaLoadTest_daa1_special_5m_8/20240321-074728-DaaLoadTest/results/core.20240321.074802.815853.0001.dmp' in response to an event
        DLT stderr _ZN26JITClientPersistentCHTable9markDirtyEP19TR_OpaqueClassBlock+0x46 (0x000003FF9A4B1946 [libj9jit29.so+0x2b1946])
        DLT stderr _ZN31TR_JITClientPersistentClassInfo21incNumPrexAssumptionsEv+0x32 (0x000003FF9A4B1DF2 [libj9jit29.so+0x2b1df2])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence19processIndirectCallEPN2TR4NodeEPNS0_7TreeTopEt+0x167c (0x000003FF9AA9E074 [libj9jit29.so+0x89e074])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence11processNodeEPN2TR4NodeEPNS0_7TreeTopEt+0x5a (0x000003FF9AA9EA22 [libj9jit29.so+0x89ea22])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence7performEv+0xf68 (0x000003FF9AAB93E0 [libj9jit29.so+0x8b93e0])
        DLT stderr _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii+0x184e (0x000003FF9AAC7FFE [libj9jit29.so+0x8c7ffe])
        DLT stderr _ZN3OMR9Optimizer8optimizeEv+0x1f6 (0x000003FF9AAC8AD6 [libj9jit29.so+0x8c8ad6])
        DLT stderr _ZN3OMR20ResolvedMethodSymbol5genILEP11TR_FrontEndPN2TR11CompilationEPNS3_20SymbolReferenceTableERNS3_12IlGenRequestE+0x444 (0x000003FF9A8A3F44 [libj9jit29.so+0x6a3f44])
        DLT stderr _ZN3OMR11Compilation7compileEv+0x1e8 (0x000003FF9A885640 [libj9jit29.so+0x685640])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x4c6 (0x000003FF9A3899BE [libj9jit29.so+0x1899be])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x378 (0x000003FF9A38AAA0 [libj9jit29.so+0x18aaa0])
        DLT stderr omrsig_protect+0x366 (0x000003FFA01B4626 [libj9prt29.so+0x34626])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x332 (0x000003FF9A3880AA [libj9jit29.so+0x1880aa])
        DLT stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1d2 (0x000003FF9A388752 [libj9jit29.so+0x188752])
        DLT stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3c6 (0x000003FF9A38715E [libj9jit29.so+0x18715e])
        DLT stderr _ZN2TR24CompilationInfoPerThread3runEv+0x66 (0x000003FF9A38764E [libj9jit29.so+0x18764e])
        DLT stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x9a (0x000003FF9A387732 [libj9jit29.so+0x187732])
        DLT stderr omrsig_protect+0x366 (0x000003FFA01B4626 [libj9prt29.so+0x34626])
        DLT stderr _Z21compilationThreadProcPv+0x1fe (0x000003FF9A387BCE [libj9jit29.so+0x187bce])
        DLT stderr thread_wrapper+0x114 (0x000003FFA0105A14 [libj9thr29.so+0x5a14])
        DLT stderr  (0x000003FFA0816296 [libc.so.6+0x96296])
        DLT stderr  (0x000003FFA088FF8E [libc.so.6+0x10ff8e])
        DLT stderr ---------------------------------------
        DLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2024/03/21 07:48:03 - please wait.
        DLT stderr _ZN26JITClientPersistentCHTable9markDirtyEP19TR_OpaqueClassBlock+0x46 (0x000003FF9A4B1946 [libj9jit29.so+0x2b1946])
        DLT stderr _ZN31TR_JITClientPersistentClassInfo21incNumPrexAssumptionsEv+0x32 (0x000003FF9A4B1DF2 [libj9jit29.so+0x2b1df2])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence19processIndirectCallEPN2TR4NodeEPNS0_7TreeTopEt+0x167c (0x000003FF9AA9E074 [libj9jit29.so+0x89e074])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence11processNodeEPN2TR4NodeEPNS0_7TreeTopEt+0x5a (0x000003FF9AA9EA22 [libj9jit29.so+0x89ea22])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence7performEv+0xf68 (0x000003FF9AAB93E0 [libj9jit29.so+0x8b93e0])
        DLT stderr _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii+0x184e (0x000003FF9AAC7FFE [libj9jit29.so+0x8c7ffe])
        DLT stderr _ZN3OMR9Optimizer8optimizeEv+0x1f6 (0x000003FF9AAC8AD6 [libj9jit29.so+0x8c8ad6])
        DLT stderr _ZN3OMR20ResolvedMethodSymbol5genILEP11TR_FrontEndPN2TR11CompilationEPNS3_20SymbolReferenceTableERNS3_12IlGenRequestE+0x444 (0x000003FF9A8A3F44 [libj9jit29.so+0x6a3f44])
        DLT stderr _ZN3OMR11Compilation7compileEv+0x1e8 (0x000003FF9A885640 [libj9jit29.so+0x685640])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x4c6 (0x000003FF9A3899BE [libj9jit29.so+0x1899be])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x378 (0x000003FF9A38AAA0 [libj9jit29.so+0x18aaa0])
        DLT stderr omrsig_protect+0x366 (0x000003FFA01B4626 [libj9prt29.so+0x34626])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x332 (0x000003FF9A3880AA [libj9jit29.so+0x1880aa])
        DLT stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1d2 (0x000003FF9A388752 [libj9jit29.so+0x188752])
        DLT stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3c6 (0x000003FF9A38715E [libj9jit29.so+0x18715e])
        DLT stderr _ZN2TR24CompilationInfoPerThread3runEv+0x66 (0x000003FF9A38764E [libj9jit29.so+0x18764e])
        DLT stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x9a (0x000003FF9A387732 [libj9jit29.so+0x187732])
        DLT stderr omrsig_protect+0x366 (0x000003FFA01B4626 [libj9prt29.so+0x34626])
        DLT stderr _Z21compilationThreadProcPv+0x1fe (0x000003FF9A387BCE [libj9jit29.so+0x187bce])
        DLT stderr thread_wrapper+0x114 (0x000003FFA0105A14 [libj9thr29.so+0x5a14])
        DLT stderr  (0x000003FFA0816296 [libc.so.6+0x96296])
        DLT stderr  (0x000003FFA088FF8E [libc.so.6+0x10ff8e])
        DLT stderr ---------------------------------------
        DLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2024/03/21 07:48:03 - please wait.
        DLT stderr _ZN26JITClientPersistentCHTable9markDirtyEP19TR_OpaqueClassBlock+0x46 (0x000003FF9A4B1946 [libj9jit29.so+0x2b1946])
        DLT stderr _ZN31TR_JITClientPersistentClassInfo21incNumPrexAssumptionsEv+0x32 (0x000003FF9A4B1DF2 [libj9jit29.so+0x2b1df2])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence19processIndirectCallEPN2TR4NodeEPNS0_7TreeTopEt+0x167c (0x000003FF9AA9E074 [libj9jit29.so+0x89e074])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence11processNodeEPN2TR4NodeEPNS0_7TreeTopEt+0x5a (0x000003FF9AA9EA22 [libj9jit29.so+0x89ea22])
        DLT stderr _ZN32TR_InvariantArgumentPreexistence7performEv+0xf68 (0x000003FF9AAB93E0 [libj9jit29.so+0x8b93e0])
        DLT stderr _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii+0x184e (0x000003FF9AAC7FFE [libj9jit29.so+0x8c7ffe])
        DLT stderr _ZN3OMR9Optimizer8optimizeEv+0x1f6 (0x000003FF9AAC8AD6 [libj9jit29.so+0x8c8ad6])
        DLT stderr _ZN3OMR20ResolvedMethodSymbol5genILEP11TR_FrontEndPN2TR11CompilationEPNS3_20SymbolReferenceTableERNS3_12IlGenRequestE+0x444 (0x000003FF9A8A3F44 [libj9jit29.so+0x6a3f44])
        DLT stderr _ZN3OMR11Compilation7compileEv+0x1e8 (0x000003FF9A885640 [libj9jit29.so+0x685640])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x4c6 (0x000003FF9A3899BE [libj9jit29.so+0x1899be])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x378 (0x000003FF9A38AAA0 [libj9jit29.so+0x18aaa0])
        DLT stderr omrsig_protect+0x366 (0x000003FFA01B4626 [libj9prt29.so+0x34626])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x332 (0x000003FF9A3880AA [libj9jit29.so+0x1880aa])
        DLT stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1d2 (0x000003FF9A388752 [libj9jit29.so+0x188752])
        DLT stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3c6 (0x000003FF9A38715E [libj9jit29.so+0x18715e])
        DLT stderr _ZN2TR24CompilationInfoPerThread3runEv+0x66 (0x000003FF9A38764E [libj9jit29.so+0x18764e])
        DLT stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x9a (0x000003FF9A387732 [libj9jit29.so+0x187732])
        DLT stderr omrsig_protect+0x366 (0x000003FFA01B4626 [libj9prt29.so+0x34626])
        DLT stderr _Z21compilationThreadProcPv+0x1fe (0x000003FF9A387BCE [libj9jit29.so+0x187bce])
        DLT stderr thread_wrapper+0x114 (0x000003FFA0105A14 [libj9thr29.so+0x5a14])
        DLT stderr  (0x000003FFA0816296 [libc.so.6+0x96296])
        DLT stderr  (0x000003FFA088FF8E [libc.so.6+0x10ff8e])
        DLT stderr ---------------------------------------

50x grinder showed 0 failures.

@cjjdespres
Copy link
Contributor Author

Failure in MathLoadTest_bigdecimal_special_5m_7 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_special.system_s390x_linux_jit_Personal/833/. Assertion failure. Console log:

        MathLoadTest_bigdecimal_special_5m_7 Start Time: Fri Mar 22 05:44:14 2024 Epoch Time (ms): 1711111454712
        variation: Mode122
        JVM_OPTIONS: -XX:+UseJITServer -Xgcpolicy:optavgpause -Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xnocompressedrefs 

        STF 05:44:21.896 - ====================   E X E C U T E -   ====================
        STF 05:44:21.896 - Running execute: perl /home/jenkins/workspace/Test_openjdk17_j9_special.system_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_17110960169821/MathLoadTest_bigdecimal_special_5m_7/20240322-054414-MathLoadTest/execute.pl
        STF 05:44:21.958 - 
        STF 05:44:21.958 - Java version
        STF 05:44:21.958 - Running: /home/jenkins/workspace/Test_openjdk17_j9_special.system_s390x_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/bin/java -version
        openjdk version "17.0.11-internal" 2024-04-16
        OpenJDK Runtime Environment (build 17.0.11-internal+0-adhoc.jenkins.BuildJDK17s390xlinuxjitPersonal)
        Eclipse OpenJ9 VM (build master-5447bca467f, JRE 17 Linux s390x-64-Bit Compressed References 20240322_933 (JIT enabled, AOT enabled)
        OpenJ9   - 5447bca467f
        OMR      - ecae9e5a637
        JCL      - ea48af27e19 based on jdk-17.0.11+7)

        MLT 05:44:50.910 - Starting thread. Suite=0 thread=0
        MLT 05:44:50.947 - Starting thread. Suite=0 thread=1
        MLT 05:44:50.954 - Starting thread. Suite=0 thread=2
        MLT stderr 12:44:52.189 0x3ff88807b00j9codertvm(j9ji.105    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK17_s390x_linux_jit_Personal/openj9/runtime/codert_vm/cnathelp.cpp:125: (!((0 != (((iTableOffset)) & ((1))))))
        MLT stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2024/03/22 05:44:52 - please wait.
        MLT stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17110960169821/MathLoadTest_bigdecimal_special_5m_7/20240322-054414-MathLoadTest/results/core.20240322.054452.6983.0001.dmp' in response to an event
        MLT stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17110960169821/MathLoadTest_bigdecimal_special_5m_7/20240322-054414-MathLoadTest/results/core.20240322.054452.6983.0001.dmp
        MLT stderr 000003FF88807B00: Object neither in heap nor stack-allocated in thread load-0
        MLT stderr 000003FF88807B00:	O-Slot=000003FE8C00A608
        MLT stderr 000003FF88807B00:	O-Slot value=000003FF00000074
        MLT stderr 000003FF88807B00:	PC=0000000000000005
        MLT stderr 000003FF88807B00:	framesWalked=0
        MLT stderr 000003FF88807B00:	arg0EA=000003FE8C00A598
        MLT stderr 000003FF88807B00:	walkSP=000003FE8C00A608
        MLT stderr 000003FF88807B00:	literals=0000000000000000
        MLT stderr 000003FF88807B00:	jitInfo=0000000000000000
        MLT stderr 000003FF88807B00:	stack=000003FE8C005A00-000003FE8C00B230

The same assertion failed in #12696.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Apr 1, 2024

Failure in jdk11_tier1_buffer_1 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal/974/. Segmentation fault. Console log:

        jdk11_tier1_buffer_1 Start Time: Sat Mar 30 01:41:44 2024 Epoch Time (ms): 1711788104547
        variation: Mode650
        JVM_OPTIONS: -XX:+UseJITServer -XX:-UseCompressedOops -Xverbosegclog 
        
        TEST SETUP:
        JVMSHRC005I No shared class caches available
        JVMSHRC005I No shared class caches available
        cache cleanup done
        
        TESTING:
        Directory "/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/../TKG/output_17117845953492/jdk11_tier1_buffer_1/work" not found: creating
        Directory "/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/../TKG/output_17117845953492/jdk11_tier1_buffer_1/report" not found: creating
        XML output with verification to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/aqa-tests/TKG/output_17117845953492/jdk11_tier1_buffer_1/work
        Test results: passed: 10
        #0: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa63abe) [0x3ff99b63abe]
        #1: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa71428) [0x3ff99b71428]
        #2: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x17075a) [0x3ff9927075a]
        #3: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x3379c) [0x3ffac0b379c]
        #4: function __kernel_rt_sigreturn+0 [0]
        #5: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x914b24) [0x3ff99a14b24]
        #6: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x911946) [0x3ff99a11946]
        #7: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x91866a) [0x3ff99a1866a]
        #8: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x781486) [0x3ff99881486]
        #9: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c88de) [0x3ff999c88de]
        #10: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c76a2) [0x3ff999c76a2]
        #11: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c76a2) [0x3ff999c76a2]
        #12: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c93b6) [0x3ff999c93b6]
        #13: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x686902) [0x3ff99786902]
        #14: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x188d1e) [0x3ff99288d1e]
        #15: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x189e00) [0x3ff99289e00]
        #16: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34686) [0x3ffac0b4686]
        #17: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18740a) [0x3ff9928740a]
        #18: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x187ab2) [0x3ff99287ab2]
        #19: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1864be) [0x3ff992864be]
        #20: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1869ae) [0x3ff992869ae]
        #21: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x186a92) [0x3ff99286a92]
        #22: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34686) [0x3ffac0b4686]
        #23: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x186f2e) [0x3ff99286f2e]
        #24: /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x5a14) [0x3ffa7c85a14]
        #25: /lib64/libc.so.6(+0x9e2e6) [0x3ffac39e2e6]
        #26: /lib64/libc.so.6(+0x2fdbe) [0x3ffac32fdbe]
        Unhandled exception
        Type=Segmentation error vmState=0x000522ff
        J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
        Handler1=000003FFA794D9E0 Handler2=000003FFAC0B3518 InaccessibleAddress=0000000000000000
        gpr0=000003FF00000013 gpr1=0000000000000000 gpr2=000003FF1F97F000 gpr3=000003FF1FB321E0
        gpr4=0000000000000020 gpr5=0000000000000020 gpr6=000003FF1F42D8A0 gpr7=04004840000000A0
        gpr8=000003FF1EE09090 gpr9=000003FF000004A3 gpr10=000003FF7E569380 gpr11=0000000000000ED8
        gpr12=000003FF99F93000 gpr13=000003FF99DC26F0 gpr14=000003FF99A14996 gpr15=000003FF7F5F3F18
        psw=000003FF99A14B24 mask=0705100180000000 fpc=00880000 bea=000003FF99A14B92
        fpr0 408f400000000000 (f: 0.000000, d: 1.000000e+03)
        fpr1 000003ff7f5f36b0 (f: 2136946304.000000, d: 2.171857e-311)
        fpr2 4059000000000000 (f: 0.000000, d: 1.000000e+02)
        fpr3 000003ff78000908 (f: 2013268224.000000, d: 2.171796e-311)
        fpr4 40c28c0000000000 (f: 0.000000, d: 9.496000e+03)
        fpr5 1ca285f2ae1ee5e6 (f: 2921260544.000000, d: 9.586262e-171)
        fpr6 0000000000000064 (f: 100.000000, d: 4.940656e-322)
        fpr7 401524482f152d05 (f: 789916928.000000, d: 5.285432e+00)
        fpr8 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
        fpr9 0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
        fpr10 000003ffa801e6d0 (f: 2818696960.000000, d: 2.172194e-311)
        fpr11 000003ffe0177e9b (f: 3759636224.000000, d: 2.172659e-311)
        fpr12 000003ff7f5fe900 (f: 2136992000.000000, d: 2.171858e-311)
        fpr13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
        fpr14 000003ff7f4fe000 (f: 2135941120.000000, d: 2.171857e-311)
        fpr15 000000000161ed00 (f: 23194880.000000, d: 1.145979e-316)
        Module=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
        Module_base_address=000003FF99100000
        
        Method_being_compiled=com/sun/javatest/finder/HTMLCommentStream.readComment()Ljava/lang/String;
        Target=2_90_20240330_1382 (Linux 5.14.0-362.24.1.el9_3.s390x)
        CPU=s390x (4 logical CPUs) (0x1e5cae000 RAM)
        ----------- Stack Backtrace -----------
        _ZN3OMR17RegisterCandidate24processLiveOnEntryBlocksEPPN2TR5BlockEPiPNS1_11CompilationER8TR_ArrayIiESA_SA_P12TR_BitVectorRS8_IS3_Eb+0x36c (0x000003FF99A14B24 [libj9jit29.so+0x914b24])
        _ZN3OMR17RegisterCandidate9setWeightEPPN2TR5BlockEPiPNS1_11CompilationER8TR_ArrayIiESA_SA_P12TR_BitVectorRS8_IS3_ERSB_SF_+0x5fe (0x000003FF99A11946 [libj9jit29.so+0x911946])
        _ZN3OMR18RegisterCandidates6assignEPPN2TR5BlockEiRiS5_+0xa72 (0x000003FF99A1866A [libj9jit29.so+0x91866a])
        _ZN26TR_GlobalRegisterAllocator7performEv+0xd56 (0x000003FF99881486 [libj9jit29.so+0x781486])
        _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii+0x184e (0x000003FF999C88DE [libj9jit29.so+0x8c88de])
        _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii+0x612 (0x000003FF999C76A2 [libj9jit29.so+0x8c76a2])
        _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii+0x612 (0x000003FF999C76A2 [libj9jit29.so+0x8c76a2])
        _ZN3OMR9Optimizer8optimizeEv+0x1f6 (0x000003FF999C93B6 [libj9jit29.so+0x8c93b6])
        _ZN3OMR11Compilation7compileEv+0xbca (0x000003FF99786902 [libj9jit29.so+0x686902])
        _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x4c6 (0x000003FF99288D1E [libj9jit29.so+0x188d1e])
        _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x378 (0x000003FF99289E00 [libj9jit29.so+0x189e00])
        omrsig_protect+0x366 (0x000003FFAC0B4686 [libj9prt29.so+0x34686])
        _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x332 (0x000003FF9928740A [libj9jit29.so+0x18740a])
        _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1d2 (0x000003FF99287AB2 [libj9jit29.so+0x187ab2])
        _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3c6 (0x000003FF992864BE [libj9jit29.so+0x1864be])
        _ZN2TR24CompilationInfoPerThread3runEv+0x66 (0x000003FF992869AE [libj9jit29.so+0x1869ae])
        _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x9a (0x000003FF99286A92 [libj9jit29.so+0x186a92])
        omrsig_protect+0x366 (0x000003FFAC0B4686 [libj9prt29.so+0x34686])
        _Z21compilationThreadProcPv+0x1fe (0x000003FF99286F2E [libj9jit29.so+0x186f2e])
        thread_wrapper+0x114 (0x000003FFA7C85A14 [libj9thr29.so+0x5a14])
        start_thread+0x146 (0x000003FFAC39E2E6 [libc.so.6+0x9e2e6])
         (0x000003FFAC32FDBE [libc.so.6+0x2fdbe])

I will attempt to reproduce. It is a known issue.

@mpirvu
Copy link
Contributor

mpirvu commented Apr 1, 2024

This last one is a known issue that Annabelle works on.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented Apr 9, 2024

In https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/904/, there was a failure in testJitserverArguments_0, in two components, the "Test SSL condition" component and the "Test SSL Failure Case with mismatched certificate" component. I'm not exactly sure what happened, but I think the server may have crashed partway into the initial connection with the client. Console log for one of the components:

        Testing: Test SSL success condition
        Test start time: 2024/04/03 01:17:01 Pacific Standard Time
        Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jitserver/jitserverScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jitserver /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/bin "-Xjit" "-XX:+UseJITServer -Xjit:count=0,verbose={JITServer},verbose={JITServerConns},verbose={compilePerformance} -XX:-JITServerLocalSyncCompiles -XX:JITServerSSLRootCerts=cert.pem" false false
        Time spent starting: 5 milliseconds
        Time spent executing: 11010 milliseconds
        Test result: FAILED
        Output from test:
         [OUT] start running script
         [OUT] Generate SSL certificates
         [OUT] Creating SSL certificates
         [OUT] Certificates generated
         [OUT] Starting /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=55294 -XX:JITServerHealthProbePort=47986  -Xjit -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem
         [OUT] 3425415 ?        00:00:00 jitserver
         [OUT] JITSERVER EXISTS
         [OUT] JITSERVER NO LONGER EXISTS
         [OUT] Terminating /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=55294 -XX:JITServerHealthProbePort=47986  -Xjit -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem
         [OUT] finished script
         [ERR] ....+.......+..+.+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.....+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.....+.+............+........+.+.....+.+.....+.......+..+.........+................+.........+..+...+.+...+...+..+.............+...+.....+.........+.+.....+.......+.........+...+...+.................+............+...+.+......+......+..+...+.........+....+.....+............+...+....+.................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
         [ERR] .+.+......+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+...+.+.....+...+............+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+......+.+......+.................+...+...+.+...+...+..+...+....+......+............+...+.....+.........+.........+....+...........+......+....+......+........+.......+...........+...+.+.....+....+............+..+.+......+...+.......................+.......+..+.+........+.+..............+.......+............+......+.........+........+....+.....+...+...+....+.........+.....+.+...+............+.....+.+...+..+...+......+...+...+...+.......+.....+...+................+...............+.........+.........+.....+.+...........+....+..+...+.........+...+.........+....+.........+..+...............+.............+.....+......+.+.........+.....+......+.+..+...+...+....+..+..........+.....+.+............+.....+..........+........+....+........+...+..........+........+......+....+......+...+...........+.+...+..+.+........+....+............+...+........+....+...............+...+.....+.+..............+......+.+...+...+..+............+...............+...+.......+.......................+...+....+.....+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
         [ERR] -----
         [ERR] 
         [ERR] JITServer is ready to accept incoming requests
         [ERR] #JITServer: JITServer version: 1.60.0
         [ERR] #JITServer: JITServer Client Mode. Server address: localhost port: 55294. Connection Timeout 30000ms
         [ERR] #JITServer: Identifier for current client JVM: 15848033599062044792
         [ERR] 
         [ERR] #JITServer: Built against (OpenSSL 3.0.13 30 Jan 2024); Loaded with (OpenSSL 3.0.7 1 Nov 2022)
         [ERR] 
         [ERR] #JITServer: Successfully initialized SSL context (OpenSSL 3.0.7 1 Nov 2022) 
         [ERR] 
         [ERR] #INFO:  _______________________________________
         [ERR] #INFO:  Version Information:
         [ERR] #INFO:       JIT Level  - j9jit_20240403_0308_jenkins
         [ERR] #INFO:       JVM Level  - 20240403_956
         [ERR] #INFO:       GC Level   - 975e8764d60
         [ERR] #INFO:  
         [ERR] #INFO:  Processor Information:
         [ERR] #INFO:       Platform Info:PPCp8
         [ERR] #INFO:       Supports HardwareSQRT:1
         [ERR] #INFO:       Supports HardwareRound:1
         [ERR] #INFO:       Supports HardwareCopySign:1
         [ERR] #INFO:       Supports FPU:1
         [ERR] #INFO:       Supports DFP:1
         [ERR] #INFO:       Supports VMX:1
         [ERR] #INFO:       Supports VSX:1
         [ERR] #INFO:       Supports AES:1
         [ERR] #INFO:       Supports  TM:1
         [ERR] #INFO:       Vendor:Unknown
         [ERR] #INFO:       numProc=4
         [ERR] #INFO:  
         [ERR] #INFO:  _______________________________________
         [ERR] #INFO:  AOT 
         [ERR] #INFO:  options specified:
         [ERR] #INFO:       samplingFrequency=2
         [ERR] #INFO:  
         [ERR] #INFO:  options in effect:
         [ERR] #INFO:       count=1000
         [ERR] #INFO:       disableSuffixLogs
         [ERR] #INFO:       samplingFrequency=2
         [ERR] #INFO:       verbose={options|compilePerformance|JITServer|JITServerConns}
         [ERR] #INFO:       compressedRefs shiftAmount=0
         [ERR] #INFO:  aggressivenessLevel=5
         [ERR] #INFO:  _______________________________________
         [ERR] #INFO:  JIT 
         [ERR] #INFO:  options specified:
         [ERR] #INFO:       count=0,verbose={JITServer},verbose={JITServerConns},verbose={compilePerformance}
         [ERR] ,disableSuffixLogs#INFO:  
         [ERR] #INFO:  options in effect:
         [ERR] #INFO:       count=0
         [ERR] #INFO:       disableSuffixLogs
         [ERR] #INFO:       samplingFrequency=2
         [ERR] #INFO:       verbose={options|compilePerformance|JITServer|JITServerConns}
         [ERR] #INFO:       compressedRefs shiftAmount=0
         [ERR] #INFO:  aggressivenessLevel=5
         [ERR] #INFO:  StartTime: Apr 03 01:17:07 2024
         [ERR] #INFO:  Free Physical Memory: 5247 MB 
         [ERR] #INFO:  Shared Class Cache Information:
         [ERR] #INFO:  	SCCname:sharedcc_jenkins   SCCpath:/home/jenkins/.cache/javasharedresources/C290M17F1A64P_sharedcc_jenkins_G43L00
         [ERR] #INFO:  	SCC_stats_bytes: size=314572192 free=60477164 ROMClass=1838896 AOTCode=3820624 AOTData=288 JITHint=3316 JITProfile=0
         [ERR] #INFO:  	SCC_stats_#:     ROMClasses=486 AOTMethods=1329 AOTDataEntries=1 AOTHints=127 AOTJitProfiles=0
         [ERR] #INFO:  	AOT header compressedRefs shiftAmount=0
         [ERR] #INFO:  CPU entitlement = 400.00
         [ERR] #INFO:  Method java/lang/String.<clinit>()V will continue as interpreted
         [ERR] #INFO:  Method com/ibm/jit/JITHelpers.<clinit>()V will continue as interpreted
         [ERR]  (warm) Compiling com/ibm/jit/JITHelpers.classIsInterfaceFlag()I  OrdinaryMethod j9m=00000000000BCC30 remote t=0 compThreadID=0 memLimit=262144 KB freePhysicalMemory=5247 MB
         [ERR] JVMDUMP039I Processing dump event "abort", detail "" at 2024/04/03 01:17:07 - please wait.
         [ERR] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17121311425559/testJitserverArguments_0/core.20240403.011707.3425415.0001.dmp' in response to an event
         [ERR] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17121311425559/testJitserverArguments_0/core.20240403.011707.3425415.0001.dmp
         [ERR] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17121311425559/testJitserverArguments_0/javacore.20240403.011707.3425415.0002.txt' in response to an event
         [ERR] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17121311425559/testJitserverArguments_0/javacore.20240403.011707.3425415.0002.txt
         [ERR] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17121311425559/testJitserverArguments_0/Snap.20240403.011707.3425415.0003.trc' in response to an event
         [ERR] JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17121311425559/testJitserverArguments_0/Snap.20240403.011707.3425415.0003.trc
         [ERR] JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17121311425559/testJitserverArguments_0/jitdump.20240403.011707.3425415.0004.dmp' in response to an event
         [ERR] JVMDUMP051I JIT dump occurred in '(unnamed thread)' thread 0x00000000001C6100
         [ERR] JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_17121311425559/testJitserverArguments_0/jitdump.20240403.011707.3425415.0004.dmp
         [ERR] JVMDUMP013I Processed dump event "abort", detail "".
         [ERR] 70F1E7ACFF7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:321:
         [ERR] #FAILURE:  JITServer::StreamFailure: Failed to SSL_connect for com/ibm/jit/JITHelpers.classIsInterfaceFlag()I @ warm
         [ERR] #JITServer: t=  1091 Could not connect to a server. Next attempt in 2000 ms.
         [ERR] #JITServer: t=  1091 Resetting activation policy to AGGRESSIVE because client has lost connection to server
         [ERR] #JITServer: compThreadID=0 JITServer StreamFailure: Generic stream failure
         [ERR] ! (warm) com/ibm/jit/JITHelpers.classIsInterfaceFlag()I Q_SZ=0 Q_SZI=0 QW=2 j9m=00000000000BCC30 time=1092483us compilationStreamFailure memLimit=262144 KB freePhysicalMemory=5272 MB mem=[region=64 system=2048]KB compThreadID=0

The rest of the log seems to be the remainder of the client's verbose log, showing only local compilations. The other component failed similarly. I haven't attempted to reproduce yet.

Edit: This appears to have occurred in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/906/ as well.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented May 14, 2024

The tests cmdLineTester_jvmtitests_hcr_OSRG_nongold_0 and cmdLineTester_jvmtitests_hcr_OSRG_nongold_2 have recently begun failing. The May 8 run https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/1040/ is the earliest failure I can see. It seems to occur only on x86 with openjdk11. Both tests fail in the same way, in subtest rtc003. Console log for that May 8 failure:

03:32:06  Eclipse OpenJ9 VM (build master-bfaeffa, JRE 11 Linux amd64-64-Bit Compressed References 20240508_1838 (JIT enabled, AOT enabled)
03:32:06  OpenJ9   - bfaeffa
03:32:06  OMR      - 1bcf969
03:32:06  JCL      - b8f6121 based on jdk-11.0.24+1)

        Testing: rtc003
        Test start time: 2024/05/08 01:12:04 Pacific Standard Time
        Running command: "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation  -Xdump    --add-opens=java.base/java.lang.reflect=ALL-UNNAMED -agentlib:jvmtitest=test:rtc003 -cp "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar:/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/../../testDependency/lib/asm-all.jar" com.ibm.jvmti.tests.util.TestRunner
        Time spent starting: 6 milliseconds
        Time spent executing: 1422 milliseconds
        Test result: FAILED
        Output from test:
         [OUT] *** Testing [1/1]:	testRedefineImplementedInterface
         [ERR] Exception in thread "main" java.lang.NoClassDefFoundError: com.ibm.oti.util.Msg (initialization failure)
         [ERR] 	at java.base/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:160)
         [ERR] 	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:865)
         [ERR] 	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:859)
         [ERR] 	at java.base/java.lang.Thread.uncaughtException(Thread.java:1363)
         [ERR] Caused by: java.lang.ClassCastException: K0340
         [ERR] 	at java.base/java.lang.invoke.HandleCache.getStaticCache(HandleCache.java:124)
         [ERR] 	at java.base/java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:644)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:313)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:216)
         [ERR] 	at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.open(SystemModuleFinders.java:448)
         [ERR] 	at java.base/jdk.internal.loader.BuiltinClassLoader.findResourceAsStream(BuiltinClassLoader.java:434)
         [ERR] 	at java.base/jdk.internal.loader.BootLoader.findResourceAsStream(BootLoader.java:146)
         [ERR] 	at java.base/java.lang.Module.getResourceAsStream(Module.java:1590)
         [ERR] 	at java.base/com.ibm.oti.vm.MsgHelp.loadMessages(MsgHelp.java:185)
         [ERR] 	at java.base/com.ibm.oti.util.PriviAction.run(PriviAction.java:172)
         [ERR] 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:690)
         [ERR] 	at java.base/com.ibm.oti.util.Msg.<clinit>(Msg.java:63)
         [ERR] 	at java.base/java.lang.ClassCastException.<init>(ClassCastException.java:71)
         [ERR] 	at java.base/java.lang.invoke.HandleCache.getStaticCache(HandleCache.java:124)
         [ERR] 	at java.base/java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:644)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:313)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:216)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003$TestFunction.apply(rtc003.java:49)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003$TestFunction.apply(rtc003.java:47)
         [ERR] 	at java.base/java.util.Map.computeIfAbsent(Map.java:1003)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003.testRedefineImplementedInterface(rtc003.java:38)
         [ERR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         [ERR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         [ERR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         [ERR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:572)
         [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
         [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
         [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
         [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
        >> Success condition was not found: [Return code: 0]

I'll see if I can reproduce without JITServer. An initial small (x25) grinder without JITServer did not result in any failures.

@cjjdespres
Copy link
Contributor Author

cjjdespres commented May 22, 2024

The test org.openjdk.tests.java.util.stream.CountLargeTest in jdk_util_0 in sanity.openjdk seems to be timing out somewhat regularly again. This was last seen in #14538 (comment). The failures ended up going away on their own the last time. I'll see if I can reproduce the failures again.


Update: The test seems to have stopped failing again. The last failure was May 27, which was the last test before #19545 was merged, so the performance regression fixed by that PR could have been the cause.

@cjjdespres
Copy link
Contributor Author

The nightly tests have been quite unstable recently, I believe due to #19947 (it's the same segfault with collapseIntoParent at the top of the trace). The suspected change was reverted today, so hopefully I'll get a better sense of what failures (if any) are new and not caused by that.

@cjjdespres
Copy link
Contributor Author

A lot of recent nightly test failures have been in SharedClassesAPI, which uses jvmti. There was an unintended change to how the -agentpath: option was processed (see #20194) that's caused this test to fail on all platforms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:jitserver Artifacts related to JIT-as-a-Service project
Projects
None yet
Development

No branches or pull requests

7 participants