Test history refactoring and improvements #625
Conversation
More logging. Parallel history stream.
… tests. Add timeout and build count limit.
Reduce trend chart loading times. Display same count of builds in trend chart as in table. Workaround for jQuery issue.
Support decimal duration. Increase builds in view to 100.
Round maximum up if over 0.5.
Reuse the retrieved history for chart generator to avoid multiple requests.
Fix click on chart.
…it goofy). Improve chart appearance. Show both charts.
Add history size links. Improve history page load time.
Add Total line. More distinct line colors. Improve overall appearance.
Increase age computation window to 25 to skip over builds without results.
Fix test compilation error. Update POM.
Fix history chart layout with only few results.
…-plugin into test-history-refactor-1265
…er/junit-plugin into test-history-refactor-1265
|
I discovered that properties are not loaded properly, I am fixing it (this was added by another PR in the last year or so and I forgot to implement the parsing). I will also see if I can implement a test for writing+parsing. |
Implement related test.
|
Any final concerns or shall we ship? |
|
Going to ship now to benefit from some of @mdealer's availability if required. |
|
Thanks @mdealer! Released in https://github.com/jenkinsci/junit-plugin/releases/tag/1279.v72cf99b_25c43 |
jenkinsci/junit-plugin#625 includes history improvemnts and performance improvements. Especially helpful for large JUnit results.
| <dependency> | ||
| <groupId>ca.umontreal.iro.simul</groupId> | ||
| <artifactId>ssj</artifactId> | ||
| <version>3.3.1</version> |
There was a problem hiding this comment.
This was outdated at the time of integration. I think I have managed to upgrade it in #628.
| <dependency> | ||
| <groupId>com.pivovarit</groupId> | ||
| <artifactId>parallel-collectors</artifactId> | ||
| <version>2.6.1</version> |
There was a problem hiding this comment.
This is up-to-date, but Renovate is trying to upgrade it to version 3.x in e.g. #629, which won't work because 3.x requires Java 21 or newer. Can we update the Renovate configuration to exclude 3.x until we are ready to require Java 21 or newer? That way, we will continue to get updates to version 2.x.
|
Breaks |
| return r; | ||
| } | ||
|
|
||
| static ConcurrentHashMap<String, SoftReference<TestResult>> resultCache = new ConcurrentHashMap<>(); |
There was a problem hiding this comment.
This is incorrect FTR: Jenkins plugins should never keep state in static fields. Rather use instance fields of some @Extension. Noticed because of a test failure in jenkinsci/parallel-test-executor-plugin#296 where we try to simulate a Jenkins restart without using RealJenkinsRule and thus a new JVM.
There was a problem hiding this comment.
I am not that familiar with Jenkins extension mechanism, I just assumed that it's OK because
- it is in the gray, between the state of the plugin and that of the file system
- plugins cannot be unloaded
Maybe there is a good example on how to do it?
There was a problem hiding this comment.
In practice this mainly causes trouble for functional tests. If there is not already an @Extension which would be a natural home for the state, you can simply add one
@Extension public static final class TheCache {
static TheCache get() {
return ExtensionList.lookupSingleton(TheCache.class);
}
private final Cache<Whatever, Else> data = Caffeine.newCache().orSomething();
// etc.
}
This is an update of #446
Links to HPI iterations if you want to try it:
https://github.com/mdealer/junit-plugin/releases/
TL;DR:
Major performance and usability improvements to the test history file storage, history charts and test failure age computation.
Improve large JUnit test history performance and usability. RAM and CPU resources are better utilized to parse and cache the test results for faster retrieval.
JUnit SQL Storage plugin does not solve the problem fully as there are too many requests generated by the existing implementation.
To fully benefit from this, assign more CPU cores, e.g. 8+, and increase the Java heap size, e.g. -Xmx8g.
Changes:
Binary is now bigger due to a 3rd party math library for processing historical results.
The more CPU cores, the faster the result XML file parsing, the quicker the initial and subsequent history page load. The more heap size, the more test results will remain cached (using SoftReferences), the less CPU and Disk IO usage when browsing the results back and forth.
New parameter in 'junit' pipeline step: keepTestNames
Allow disabling the test name mangling when uploading results from multiple stages or parallel branches in a single build via 'keepTestNames' parameter. There was a change in the plugin a few years ago that basically broke the result URLs (e.g. if we are linking to the results from some external system), this is a way to undo this change.
Related tickets that are potentially fixed
There various already reported issues that I also encountered on the way. This list must be double checked, but based on the code I looked at, there is a very high chance that most of the 'closed' tickets below are wrongly closed and this PR should improve the situation for them.
Status
TODOs:
Testing done
Live 'tested' for over 1 year, also being resynced multiple times inbetween, so we are talking multiple Jenkins versions and thousands of builds with thousands of test results each.
The UI changes are functional, but I am not that familiar with Jenkins to make it fully clean. I think that should be done as part of another PR.
The existing tests should cover nearly all of the changes.
There is no 'before' screenshots because before it didn't work, the history page loaded forever.
See it in action
If you want to test it with large number of tests, then create a Pipeline Job, call it 'junit_test' and add this Groovy script into it:
Then run the job for... thousands of times or as much as you like. Running takes some time, but navigating the results should be still fast. The NUM parameter is only required to have some builds without test results inbetween, you can use it if you want.
To speed things up, set up a second trigger job that executes this: