Prototype a 'rerun' script

[smlambert]

User story:
As a developer, I want to recreate a test failure that was reported 2 months ago. I know which JDK binary was being tested, and I have downloaded and installed that binary locally on my machine. I want to set up the same set of test material that was used 2 months ago. I have the SHA.txt file from that test run from 2 months ago. How do I setup the correct test material so that I can rerun the test that failed locally?

---

Related to the reproducible tests epic, it would be good to write a script that can read and parse a 'rerun in Grinder link' and/or the contents of a SHA.txt file, set the appropriate env vars based on those 2 types of inputs, and then 'rerun' the test target (defined in the 'rerun in Grinder link').

A good starting point for this work, is to first try to run a test target locally (instructions for how to do that can be found here). Once can run locally, try changing some variables and see what happens when you next compile and run.

Example:

• install your test JDK and export TEST_JDK_HOME and try compiling and running different test material and running different TARGETs
• export BUILD_LIST=openjdk, make compile and then make _jdk_math
• then change and export BUILD_LIST=system, make compile and make _MathLoadTest_all_5m

rerun.sh --help

rerun.sh --link someRerunURL [--shas someSHAtxtFile]

example:
rerun.sh --link https://ci.adoptopenjdk.net/job/Grinder/parambuild/?SDK_RESOURCE=nightly&TARGET=jdk_imageio_0&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=-v+%24%7BTEST_JDK_HOME%7D%3A%2Fopt%2Fjava%2Fopenjdk&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&TEST_PARALLELIZATION_PARAMS=&PLATFORM=x86-64_linux --shas /tmp/SHA.txt (edited)

where a SHA.txt file is an optional input to the rerun.sh script but if present, would set some environment variables to specify which specific test repos we want to use

[smlambert]

Some additional reference:
SHA.txt files are produced by test runs in Jenkins (an example: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/73/artifact/openjdk-tests/TKG/SHA.txt) where contents look like:

================================================
timestamp: 20210414-155933
repo dir: /home/jenkins/workspace/Grinder/openjdk-tests
git repo: 
  Fetch URL: https://github.com/AdoptOpenJDK/openjdk-tests.git
sha:
7ba66b4847f1df8c76f316a07a1b1c0a01b16454
================================================
timestamp: 20210414-155933
repo dir: /home/jenkins/workspace/Grinder/openjdk-tests/TKG
git repo: 
  Fetch URL: https://github.com/AdoptOpenJDK/TKG.git
sha:
07171520f8b0849d922125d35380d45193f9c5bf
================================================
timestamp: 20210414-155941
repo dir: /home/jenkins/workspace/Grinder/openjdk-tests/openj9
git repo: 
  Fetch URL: https://github.com/eclipse/openj9.git
sha:
51fc64e9189c7972a6649fcbeb407c18120d3e6b
================================================
timestamp: 20210414-160111
repo dir: openjdk-jdk
git repo: 
  Fetch URL: https://github.com/ibmruntimes/openj9-openjdk-jdk11.git
sha:
ba1caf576ee57574ee49e4e73e77107ddcedb5be

Rerun links are generated by the parameters passed in, in fact more than 1 rerun link is created. Rerun link is a convenience to prepopulate the parameters used in a job, and allow the user to change some parameters to suit the needs of the 'next run'. They are created by 2 mechanisms:

• One by the Jenkinsfilebase script, which captures everything passed in to run a job in Jenkins, at the time of triggering the test job (so this rerun link does not know which tests failed, but captures ALL parameters that are explicitly set in the job). This link is found at the top of the test job page in Jenkins (as shown in screen shot of this Jenkins job):

Example link (when clicked on, sets up a prepopulated Grinder job that can then be further edited and then launched):

https://ci.adoptopenjdk.net/job/Grinder/parambuild/?SDK_RESOURCE=nightly&TARGET=jdk_imageio_0&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=-v+%24%7BTEST_JDK_HOME%7D%3A%2Fopt%2Fjava%2Fopenjdk&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&TEST_PARALLELIZATION_PARAMS=&PLATFORM=x86-64_linux&KEEP_REPORTDIR=true&PERSONAL_BUILD=false&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2FAdoptOpenJDK%2Fopenjdk-tests.git&LABEL=test-ibmcloud-ubuntu1604-x64-1&TEST_OPTIONS_PARAMS=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&NON_AQA_TEST_REPOS_HELP_TEXT=&ADOPTOPENJDK_BRANCH=master&LIGHT_WEIGHT_CHECKOUT=false&NON_AQA_TEST_REPOS=&ARTIFACTORY_SERVER=&TEST_REPO_PARAMS=&TEST_SELECTION_PARAMS=&TEST_PARALLELIZATION_PARAMS_HELP_TEXT=&JDK_SELECTION_PARAMS=&KEEP_WORKSPACE=false&USER_CREDENTIALS_ID=&JDK_VERSION=11&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=&PLATFORM_AND_MACHINE_HELP_TEXT=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&UPSTREAM_JOB_NAME=&PLATFORM_AND_MACHINE=&CUSTOM_TARGET=&VENDOR_TEST_SHAS=&JDK_BRANCH=&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&POST_RUN_PARAMS_HELP_TEXT=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_SELECTION_PARAMS_HELP_TEXT=&JDK_IMPL=openj9&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&TKG_SHA=&TEST_SELECTION_PARAMS_HELP_TEXT=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&OPENJDK_SHA=&NUM_MACHINES=&BUILD_LIST=openjdk&UPSTREAM_JOB_NUMBER=&TEST_REPO_PARAMS_HELP_TEXT=&POST_RUN_PARAMS=&TIME_LIMIT=10&TEST_OPTIONS_PARAMS_HELP_TEXT=&JVM_OPTIONS=&PARALLEL=None

• one or more links are created by TKG based on which targets failed. These links are found at the end of the console output, and are captured/created for when you are running locally. It does not take and transform all locally set environment variables and convert them to Jenkins test job parameters, that is currently up to the user to do. You see them when you run tests locally and tests fail, sample looks same as the example link above, but not all parameters are captured.

[brngylni]

Hey, I can work on it.

[llxia]

Future enhancement suggestion:
Instead of getting info from URL and SHA file, does it make sense to auto-generate a rerun script at the end of the failed test build? The rerun script should contain all necessary info (repo, branch, SHA, TARGET, BUILD_LIST, etc).
Advantage:

• This rerun script can be attached to the issue without requiring additional information from the users
• No need to parse rerun in Grinder link

[smlambert]

Yes, that is even better @llxia ... that was why we called this a prototype... wanted to see what it looked like to take SHAs and rerun link info and try to run again. Having it autogenerate and attach as an artifact (and be present in local or automated runs) is sweet.

[smlambert]

Closing this as completed since #2531 was merged. We will revisit enhancements under a separate feature enhancement (likely in TKG repo).