Shared Classes Cache / AOT Testing

[dsouzai]

This issue (epic) will track the plan for improving Shared Classes Cache (SCC) / AOT Testing.

Enable more Shared Classes Test modes

One of the best tests we have is the shared classes test found here https://github.com/eclipse/openj9-systemtest/tree/master/openj9.test.sharedClasses . It internally has 5 tests and 25 different modes; some of these modes look like they're either irrelevant or out of date, but effectively there are at least a couple hundred different tests that could be run. Unfortunately, the systemtest playlist only runs 8 of these. @smlambert and @llxia are working towards enabling extra modes in this test bucket.

Update Shared Classes Test to run twice

https://github.com/eclipse/openj9-systemtest/blob/master/openj9.test.sharedClasses/src/test.sharedClasses/net/openj9/stf/SharedClasses.java#L245-L253 will need to be modified to be invoked twice. The problem with the sharedClasses test right now is that even though there are 5 JVMs invoked, which means that some JVMs will load AOT code compiled by a different JVM, it doesn't fully test the AOT load infrastructure becuase after the test is completed, the SCC gets destroyed before the next test runs.

EDIT: This has been addressed by adoptium/openj9-systemtest#54

Augment non SCC/AOT Tests for SCC/AOT verification

Existing Tests can be augmented to verify the SCC / AOT infrastructure and generated code. The general approach would be modifying existing tests in the following way:

1. Destroy SCC
2. Run Test (this will be the AOT compile run)
3. Run Test some number of times (this will be the AOT load run)
4. Destroy SCC

AOT/SCC PR Testing

#4875

Fix AOT/SCC PR Tests Failures

#6057

---

fyi @smlambert @llxia @mstoodle @andrewcraik

[dsouzai]

I'll tackle the "Update Shared Classes Test to run twice" task; it looks like a relatively simple problem that I (as someone with no experience in the test infrastructure) can address :)

[andrewcraik]

@dsouzai So in relation to #3752 do we expect to have some basic shared classes testing available in the next few days that we can use to test the pull requests this item will entail. Pull request a in the sequence is open pending testing and I was hopiong we could use some of the work described here to do that, but that requires this to be available sooner rather than later.

[dsouzai]

So in relation to #3752 do we expect to have some basic shared classes testing available in the next few days that we can use to test the pull requests this item will entail.

Yeah I believe so (assuming the "Updated Shared Classes Test to run twice" change I'll have soon gets merged quickly); The systemTest bucket already has SCC tests (https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/systemtest/playlist.xml#L725-L923) which can be used for testing so my change should improve the coverage provided by these existing tests.

[dsouzai]

I'm not sure if these tests can be triggered from a PR though; that's something an infra person would know.

[andrewcraik]

@AdamBrousseau or @smlambert do you know the answer to the above about triggering these improved SCC tests from a PR - since there are going to be > 26 of them made over the next week or two having the tests available would be a big help (see the merge plan in #3752)

[llxia]

No, we donot have PR build for system tests at this point. However, you can run a grinder build by changing the following file to point into your own branch.
https://github.com/AdoptOpenJDK/openjdk-tests/blob/7f959e7be0ea44844918dc1e65949aef8939764a/systemtest/build.xml#L99

[AdamBrousseau]

PR support for system tests will come in #2836

[llxia]

PR support for system tests (#2836) works for changes in openj9 repo.
I do not think it will work for changes in the system test repos (i.e., https://github.com/eclipse/openj9-systemtest or https://github.com/AdoptOpenJDK/openjdk-systemtest).

[jdmpapin]

However, you can run a grinder build by changing the following file to point into your own branch [...]

Changing that line looks like it would affect the version of openj9-systemtest used for testing, but I don't have a branch of openj9-systemtest. Rather, I'm modifying openj9 itself. So I suppose the way to run the tests is as a grinder, but with no need to edit the linked file—is that right? If so, where can I find instructions for running a grinder?

[llxia]

Changing that line looks like it would affect the version of openj9-systemtest used for testing, but I don't have a branch of openj9-systemtest. Rather, I'm modifying openj9 itself. So I suppose the way to run the tests is as a grinder, but with no need to edit the linked file—is that right? If so, where can I find instructions for running a grinder?

Yes, you can grinder for any type of tests. Or PR build for functional for now.
Link to How-to-Run-a-Grinder-Build-on-Jenkins
#3697 (comment) has more detail in terms of testing in case you are interested

In general, what test to run really depends on what is your change.

• If your changes are in the openj9 repo, you can launch PR builds to run functional test or Grinder to run any type of tests (i.e., system test, openjdk regression, etc)
• If your changes are in system test repo (https://github.com/eclipse/openj9-systemtest or https://github.com/AdoptOpenJDK/openjdk-systemtest), then you need do as mentioned in Shared Classes Cache / AOT Testing #3774 (comment)

[smlambert]

Related work:

1. Flush SCC prior to each test target run - Add test setUp/tearDown to TKG #3950
2. Set correct options, run tests X number of iterations (default X=2) - Support AOT testing #4096

With the above 2 pieces of work, a committer can now run a Grinder against particular/individual test targets with the TEST_FLAG=AOT. The committer at this point should assess the TEST_TARGET as to whether it is appropriate to run (some tests in the sanity or extended target are not applicable).

Last step is to tag tests so that the ones that are not relevant or appropriate to run multiple times are skipped if trying to run under AOT test flag. After this is done, then running the higher level targets like the sanity.functional target will make sense.

[dsouzai]

Closing this as the general infrastructure has been in place for a while now.