Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SVM AOT support for resolved invokeHandle/invokeDynamic dispatch #20373

Merged
merged 8 commits into from
Nov 12, 2024

Conversation

dsouzai
Copy link
Contributor

@dsouzai dsouzai commented Oct 17, 2024

  • Add validation for TR_ValidateDynamicMethodFromCallsiteIndex
  • Add validation for TR_ValidateHandleMethodFromCPIndex
  • Add relocations for TR_CallsiteTableEntryAddress
  • Add relocations for TR_MethodTypeTableEntryAddress
  • Conditionally support storing LambdaForms into the SCC if -Xshareclasses:shareLambdaForm is enabled.

Depends on eclipse-omr/omr#7494

AOT MH is only enabled under -Xaot:enableMHRelocatableCompile. To enable resolved invokeHandle/invokeDynamic dispatch you also need to add shareLambdaForm to the -Xshareclasses option.

@dsouzai dsouzai added comp:jit comp:jit:aot project:MH Used to track Method Handles related work depends:omr Pull request is dependent on a corresponding change in OMR labels Oct 17, 2024
@dsouzai
Copy link
Contributor Author

dsouzai commented Oct 17, 2024

@hangshao0 does f9c7eba look like the proper way to conditionally support storing LambdaForms into the SCC? I wanted something like this so that we can add AOT MH tests without having to enable by default both LambdaForms in the SCC & Resolved invokeHandle/invokeDynamic dispatch in AOT code.

@hangshao0
Copy link
Contributor

does f9c7eba look like the proper way to conditionally support storing LambdaForms into the SCC?

Looks mostly fine. But you may want to set J9_FINDCLASS_FLAG_LAMBDAFORM | J9_FINDCLASS_FLAG_DO_NOT_SHARE in the case shareLambdaForm is off. Set flag J9_FINDCLASS_FLAG_LAMBDAFORM in the case shareLambdaForm is on.

@dsouzai dsouzai marked this pull request as ready for review October 17, 2024 19:20
@dsouzai
Copy link
Contributor Author

dsouzai commented Oct 17, 2024

@jdmpapin could you please review?

@hangshao0 could you please review the VM changes?

@hangshao0
Copy link
Contributor

For now, is the new option shareLambdaForm a public option for everyone or a private option just to be used by OpenJ9 developers/builds ?

I remember the reason of not storing lambdaForms class is that it has a performance issue. Storing lambdaForms requires the VM to do extra class comparisons before we can return the correct lamdaForm class. There is a PR #19763 that could address the performance issue. But it is not merged so the performance issue is still there.

@dsouzai
Copy link
Contributor Author

dsouzai commented Oct 17, 2024

For now, is the new option shareLambdaForm a public option for everyone or a private option just to be used by OpenJ9 developers/builds ?

I'm treating this as an internal option for devs/testing. It can be deleted in the future when LambdaForms are stored into the SCC.

Is there a better way to create/use an option for dev/testing purposes?

@hangshao0
Copy link
Contributor

I'm treating this as an internal option for devs/testing. It can be deleted in the future when LambdaForms are stored into the SCC.
Is there a better way to create/use an option for dev/testing purposes?

The reason I was asking is because public option needs to be documented in the user guide, but since it it private option, we don't need documentation change. The way you are doing it here is fine.

runtime/bcutil/ROMClassBuilder.cpp Outdated Show resolved Hide resolved
runtime/shared_common/shrinit.h Outdated Show resolved Hide resolved
Copy link
Contributor

@hangshao0 hangshao0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@dsouzai
Copy link
Contributor Author

dsouzai commented Oct 30, 2024

@jdmpapin review reminder.

Copy link
Contributor

@jdmpapin jdmpapin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for the most part. I have some very minor comments, some extremely minor, so feel free to decide where the line is

IMO the biggest naming problem is "callee." I didn't leave a regular comment for this because it's pervasive throughout these changes, but I think it's really misleading. Everything here is related to some call site, and w.r.t. to that call site, the method denoted by "callee" in this PR is always actually the caller

runtime/compiler/runtime/SymbolValidationManager.cpp Outdated Show resolved Hide resolved
runtime/compiler/env/j9method.cpp Outdated Show resolved Hide resolved
runtime/compiler/env/j9method.cpp Outdated Show resolved Hide resolved
runtime/compiler/env/j9method.cpp Outdated Show resolved Hide resolved
runtime/compiler/runtime/RelocationRecord.hpp Outdated Show resolved Hide resolved
runtime/compiler/runtime/RelocationRuntime.hpp Outdated Show resolved Hide resolved
runtime/compiler/runtime/RelocationRuntime.hpp Outdated Show resolved Hide resolved
runtime/compiler/runtime/SymbolValidationManager.hpp Outdated Show resolved Hide resolved
@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 1, 2024

the method denoted by "callee" in this PR is always actually the caller

Huh, yeah I'm not sure why I wrote "callee", in my head it was definitely the caller method (since the callee is the thing we're validating). I know that the HandleMethod relo stuff was mostly a copy of the DynamicMethod relo stuff, so I guess the mistake I made just propagated everywhere.

I'll make the fix since it is actually, more than just misleading, incorrect heh.

@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 1, 2024

@jdmpapin made all the necessary changes. I also realized I was missing JITServer support, so I added that as well.

I see that there's a conflict with the MINOR_NUMBER, but I'll fix that after you're done looking at the changes, since the force push will contain more than just my updates.

runtime/compiler/env/j9method.cpp Outdated Show resolved Hide resolved
runtime/compiler/runtime/RelocationRecord.hpp Outdated Show resolved Hide resolved
@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 5, 2024

@jdmpapin updated the PR to fix the JITServer issue; it essentially involved refactoring the code that dereferenced invokeCacheArray into its own TR_ResolvedMethod API (which involved adding it to OMR as well) and then adding a new message type for JITServer.

Good for review again.

runtime/compiler/env/j9method.h Outdated Show resolved Hide resolved
runtime/compiler/env/j9method.cpp Outdated Show resolved Hide resolved
runtime/compiler/env/j9method.cpp Outdated Show resolved Hide resolved
runtime/compiler/env/j9method.cpp Outdated Show resolved Hide resolved
@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 6, 2024

@jdmpapin made the requested changes.

@jdmpapin
Copy link
Contributor

jdmpapin commented Nov 6, 2024

Just need to fix the conflict now

getTargetMethodFromMemberName is used for to get both a resolved dynamic
method as well as a resolved handle method. It is a refactoring of code
to enable resolved dispatch of invokeDynamic/invokeHandle in AOT in both
local as well as remote compilation.

Signed-off-by: Irwin D'Souza <[email protected]>
Add new TR_RelocationErrorCode kinds to indicate validation failures for
TR_ValidateDynamicMethodFromCallsiteIndex and
TR_ValidateHandleMethodFromCPIndex.

Signed-off-by: Irwin D'Souza <[email protected]>
Signed-off-by: Irwin D'Souza <[email protected]>
@jdmpapin
Copy link
Contributor

jdmpapin commented Nov 6, 2024

Jenkins test sanity.functional+aot all jdk17 depends eclipse-omr/omr#7494

@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 7, 2024

zlinux failures due to

aix failure due to

The linux failures are all of the form

handleServerMessage(JITServer::ClientStream*, TR_J9VM*, JITServer::MessageType&) 0x22e0 (0x0000F1EFEC96CD60 [libj9jit29.so 0x16cd60])
remoteCompile(J9VMThread*, TR::Compilation*, TR_ResolvedMethod*, J9Method*, TR::IlGeneratorMethodDetails&, TR::CompilationInfoPerThreadBase*).localalias 0xe18 (0x0000F1EFEC97DCE8 [libj9jit29.so 0x17dce8])
TR::CompilationInfoPerThreadBase::compile(J9VMThread*, TR::Compilation*, TR_ResolvedMethod*, TR_J9VMBase&, TR_OptimizationPlan*, TR::SegmentAllocator const&) 0x5c0 (0x0000F1EFEC93FA14 [libj9jit29.so 0x13fa14])
TR::CompilationInfoPerThreadBase::wrappedCompile(J9PortLibrary*, void*) 0x2c8 (0x0000F1EFEC940688 [libj9jit29.so 0x140688])
omrsig_protect 0x21c (0x0000F1EFEDCE889C [libj9prt29.so 0x2889c])
TR::CompilationInfoPerThreadBase::compile(J9VMThread*, TR_MethodToBeCompiled*, J9::J9SegmentProvider&) 0x2cc (0x0000F1EFEC93E390 [libj9jit29.so 0x13e390])
TR::CompilationInfoPerThread::processEntry(TR_MethodToBeCompiled&, J9::J9SegmentProvider&) 0x118 (0x0000F1EFEC93E848 [libj9jit29.so 0x13e848])
TR::CompilationInfoPerThread::processEntries() 0x2f8 (0x0000F1EFEC93D5D8 [libj9jit29.so 0x13d5d8])
TR::CompilationInfoPerThread::run() 0x50 (0x0000F1EFEC93DA84 [libj9jit29.so 0x13da84])
protectedCompilationThreadProc(J9PortLibrary*, TR::CompilationInfoPerThread*) 0x7c (0x0000F1EFEC93DB2C [libj9jit29.so 0x13db2c])
omrsig_protect 0x21c (0x0000F1EFEDCE889C [libj9prt29.so 0x2889c])
compilationThreadProc(void*) 0x148 (0x0000F1EFEC93DEF8 [libj9jit29.so 0x13def8])
thread_wrapper 0xcc (0x0000F1EFEDC873BC [libj9thr29.so 0x73bc])
 (0x0000F1EFEE26597C [libc.so.6 0x8597c])
 (0x0000F1EFEE2CBA4C [libc.so.6 0xeba4c])

which I can't find an existing issue for, so probably related to a change I made.

@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 7, 2024

which I can't find an existing issue for, so probably related to a change I made.

My guess is it's the change

         vmInfo._shareLambdaForm
            = J9_ARE_ALL_BITS_SET(javaVM->sharedClassConfig->runtimeFlags2, J9SHR_RUNTIMEFLAG2_SHARE_LAMBDAFORM);

which is odd because this is an AOT test so you'd think javaVM->sharedClassConfig would not be NULL. But I'll look into it and confirm.

@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 7, 2024

which is odd because this is an AOT test so you'd think javaVM->sharedClassConfig would not be NULL.

Ah...

[2024-11-07T01:17:00.341Z] testJITServer_1 Start Time: Wed Nov  6 21:16:59 2024 Epoch Time (ms): 1730942219954
[2024-11-07T01:17:00.341Z] variation: Mode610 -Xshareclasses:none -Xjit:optLevel=hot
[2024-11-07T01:17:00.341Z] JVM_OPTIONS: -Xshareclasses:name=test_aot -Xscmx400M -Xscmaxaot256m  -Xcompressedrefs -Xjit -Xgcpolicy:gencon -Xshareclasses:none -Xjit:optLevel=hot 

@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 7, 2024

Added NULL check (see force push).

Jenkins test sanity.functional+aot xlinux,plinux,zlinux,alinux jdk17 depends eclipse-omr/omr#7494

@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 8, 2024

alinux tests failed because of

Calling Pipeline was cancelled
[2024-11-07T22:11:17.886Z] Sending interrupt signal to process
[2024-11-07T22:11:18.758Z] make[6]: *** [autoGen.mk:288: cmdLineTester_jvmtitests_hcr_openi9_none_SCC_1] Error 143
[2024-11-07T22:11:18.758Z] make[6]: Leaving directory '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests/functional/cmdLineTests/jvmtitests'
[2024-11-07T22:11:18.758Z] make[5]: *** [/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests/TKG/../TKG/settings.mk:361: testList-jvmtitests] Error 2
[2024-11-07T22:11:18.758Z] make[5]: Leaving directory '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests/functional/cmdLineTests'
[2024-11-07T22:11:18.758Z] make[4]: *** [/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests/TKG/../TKG/settings.mk:361: testList-cmdLineTests] Error 2
[2024-11-07T22:11:18.758Z] make[4]: Leaving directory '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests/functional'
[2024-11-07T22:11:18.759Z] make[3]: *** [/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests/TKG/../TKG/settings.mk:361: testList-functional] Error 2
[2024-11-07T22:11:18.759Z] make[3]: Leaving directory '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests'
[2024-11-07T22:11:18.759Z] make[2]: *** [settings.mk:361: testList-..] Error 2
[2024-11-07T22:11:18.759Z] make[2]: Leaving directory '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests/TKG'
[2024-11-07T22:11:18.759Z] make[1]: *** [makefile:74: _testList] Error 2
[2024-11-07T22:11:18.759Z] make[1]: Leaving directory '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/aqa-tests/TKG'
[2024-11-07T22:11:18.759Z] make: *** [parallelList.mk:8: testList_0] Error 2
[2024-11-07T22:11:18.764Z] script returned exit code 2

Can't really determine why it was aborted.

Pipeline_Build_Test shows

1:26:16  Scheduling project: [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal/)
11:26:16  The parameter 'KEEP_REPORTDIR' did not have the type expected by [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal/). Converting to Boolean Parameter.
11:26:23  Starting building: [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal #530](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal/530/)
17:11:46  Build [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal #530](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal/530/) completed: ABORTED

Pipeline_Test shows

11:30:02  Scheduling project: [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/)
[Pipeline] build (Building Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_1)
11:30:02  Scheduling project: [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_1](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_1/)
11:30:08  Starting building: [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0 #520](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/520/)
11:30:08  Starting building: [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_1 #517](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_1/517/)
15:47:25  Build [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_1 #517](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_1/517/) completed: SUCCESS
[Pipeline] }
Calling Pipeline was cancelled
[Click here to forcibly terminate running steps](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal/530/console#)
17:11:44  Build [Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0 #520](https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_Personal_testList_0/520/) completed: ABORTED

@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 8, 2024

Jenkins test sanity.functional+aot alinux jdk17 depends eclipse-omr/omr#7494

@dsouzai
Copy link
Contributor Author

dsouzai commented Nov 8, 2024

@jdmpapin all tests passed. Btw, the OMR PR that this PR depends on does not need a coordinated merge with this PR.

@jdmpapin
Copy link
Contributor

OK, I've merged the OMR PR. Waiting for it to promote before merging this one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:jit:aot comp:jit comp:vm depends:omr Pull request is dependent on a corresponding change in OMR project:MH Used to track Method Handles related work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants