Packit API enhancement - make /testing-farm/results parametrizable, expose ref

[danmyway]

Description

Gathering data for test results from the Testing Farm is very time demanding and tedious with growing number of pipelines. One needs to go over each check, click the respective hyperlink and scroll through the results. We have our own way to easily parse the results from the result page/artifact URL, however getting to the pipeline_id and or URL is the pain point.

One way to get to the result page/pipeline ID "faster" is to parse the Packit's API /testing-farm/results and/or /projects/{forge}/{namespace}/{repo_name}/prs namespace.

1. The issue with the /testing-farm/results is, that it contains all the projects' testing-farm requests handled by packit with no way to parametrize the query. Being able to parametrize the query to narrow the results to a specific project would speed up the data gathering immensely.

2. The other option, however not exclusive to the testing-farm namespace enhancement is exposing ref and possibly any type of timestamp for the /projects/{forge}/{namespace}/{repo_name}/prs response body. Specifically inside the tests field.

Benefit

If proven working, we could provide the artifact URL gathering and reporting functionality to packit itself. This could help to resolve #2205
Additionally, there could be /packit report feature provided in the future to report the test results by Packit in a comment.

❯ tesar report -f tesar_latest_jobs_20230920143852 
Reporting for the requested tasks:
AlmaLinux OS 8.6.20220901 x86_64 tier0/non-destructive COMPLETE
Rocky-8-EC2-Base-8.8-20230518.0.x86_64 tier0/non-destructive COMPLETE
+--------------------------------------+----------------------------------------+-----------------------------+--------+
| UUID                                 | Target                                 | Test Plan                   | Result |
+--------------------------------------+----------------------------------------+-----------------------------+--------+
| 2ae608e4-3456-47ca-93b5-1dbf2c420b97 | AlmaLinux OS 8.6.20220901 x86_64       |                             |        |
|                                      |                                        | plans/tier0/non-destructive | FAILED |
| 87ae5111-27ea-4718-9c5a-ab1e03e01718 | Rocky-8-EC2-Base-8.8-20230518.0.x86_64 |                             |        |
|                                      |                                        | plans/tier0/non-destructive | FAILED |
+--------------------------------------+----------------------------------------+-----------------------------+--------+

❯ tesar report -f tesar_latest_jobs_20230920143852 -l2 --short
Reporting for the requested tasks:
AlmaLinux OS 8.6.20220901 x86_64 tier0/non-destructive COMPLETE
Rocky-8-EC2-Base-8.8-20230518.0.x86_64 tier0/non-destructive COMPLETE
+--------------------------------------+----------------------------------------+-----------------+---------------------------------------------+--------+
| UUID                                 | Target                                 | Test Plan       | Test Case                                   | Result |
+--------------------------------------+----------------------------------------+-----------------+---------------------------------------------+--------+
| 2ae608e4-3456-47ca-93b5-1dbf2c420b97 | AlmaLinux OS 8.6.20220901 x86_64       |                 |                                             |        |
|                                      |                                        | non-destructive |                                             | FAILED |
|                                      |                                        |                 | simultaneous_runs                           | ERROR  |
|                                      |                                        |                 | convert_successful_report                   | ERROR  |
|                                      |                                        |                 | failures_and_skips_in_report                | ERROR  |
|                                      |                                        |                 | successful_report                           | ERROR  |
|                                      |                                        |                 | clean_cache                                 | ERROR  |
|                                      |                                        |                 | c2r_version_latest_or_newer                 | ERROR  |
|                                      |                                        |                 | c2r_version_older_no_envar                  | ERROR  |
|                                      |                                        |                 | c2r_version_older_with_envar                | ERROR  |
|                                      |                                        |                 | data_collection_acknowledgement             | ERROR  |
|                                      |                                        |                 | disable_data_collection                     | ERROR  |
|                                      |                                        |                 | incomplete_rollback_in_analyze              | ERROR  |
|                                      |                                        |                 | log_file_exists                             | ERROR  |
|                                      |                                        |                 | log_rhsm_error                              | ERROR  |
|                                      |                                        |                 | manpage                                     | ERROR  |
|                                      |                                        |                 | root_privileges                             | ERROR  |
|                                      |                                        |                 | smoke                                       | ERROR  |
|                                      |                                        |                 | variant_message                             | ERROR  |
|                                      |                                        |                 | config_cli_priority                         | ERROR  |
|                                      |                                        |                 | config_custom_path_custom_filename          | ERROR  |
|                                      |                                        |                 | config_custom_path_standard_filename        | ERROR  |
|                                      |                                        |                 | config_password_file_priority               | ERROR  |
|                                      |                                        |                 | config_standard_paths_priority              | ERROR  |
|                                      |                                        |                 | config_standard_paths_priority_diff_methods | ERROR  |
|                                      |                                        |                 | custom_invalid_repo_provided                | ERROR  |
|                                      |                                        |                 | custom_valid_repo_provided                  | ERROR  |
|                                      |                                        |                 | passing_activation_key_to_submgr            | ERROR  |
|                                      |                                        |                 | passing_password_to_submgr                  | ERROR  |
|                                      |                                        |                 | available_connection                        | ERROR  |
|                                      |                                        |                 | unavailable_connection                      | ERROR  |
|                                      |                                        |                 | custom_kernel                               | ERROR  |
|                                      |                                        |                 | failed_repoquery                            | ERROR  |
|                                      |                                        |                 | non_latest_kernel_error                     | ERROR  |
|                                      |                                        |                 | yum_exclude_kernel                          | ERROR  |
|                                      |                                        |                 | custom_module_loaded                        | ERROR  |
|                                      |                                        |                 | custom_module_not_loaded                    | ERROR  |
|                                      |                                        |                 | unsupported_kmod_with_envar                 | ERROR  |
|                                      |                                        |                 | force_loaded_kmod                           | ERROR  |
|                                      |                                        |                 | tainted_kernel                              | ERROR  |
|                                      |                                        |                 | logfile_starts_with_command                 | ERROR  |
|                                      |                                        |                 | modified_releasever_in_configs              | ERROR  |
|                                      |                                        |                 | modified_releasever_to_unknown_release      | ERROR  |
|                                      |                                        |                 | packages_untracked_graceful_rollback        | ERROR  |
|                                      |                                        |                 | rhsm_cleanup                                | ERROR  |
|                                      |                                        |                 | registration_start                          | ERROR  |
|                                      |                                        |                 | registration_success                        | ERROR  |
|                                      |                                        |                 | test_missing_credentials_rollback           | ERROR  |
|                                      |                                        |                 | package_download_error                      | ERROR  |
|                                      |                                        |                 | transaction_validation_error                | ERROR  |
|                                      |                                        |                 | missing_system_release                      | ERROR  |
|                                      |                                        |                 | backup_os_release_no_envar                  | ERROR  |
|                                      |                                        |                 | backup_os_release_with_envar                | ERROR  |
|                                      |                                        |                 | unsuccessful_satellite_registration         | ERROR  |
+--------------------------------------+----------------------------------------+-----------------+---------------------------------------------+--------+

Importance

high
It's not blocking us to use Packit per se, but it's faster to query for results if we use our wrapper to schedule test jobs and parse the resulting request pipeline IDs.

What is the impacted category (job)?

Testing Farm tests

Workaround

• There is an existing workaround that can be used until this feature is implemented.

Participation

• I am willing to submit a pull request for this issue. (Packit team is happy to help!)

[lbarcziova]

hi @danmyway ! Thanks for the issue. I see this could improve your situation a lot. The only thing that might cause trouble is our current DB architecture, with more complicated queries we have bumped into performance issues previously.

You can see the current DB models architecture here.

As for the 1.option, how do you propose the parametrizing? Would ref be enough here as well?

As for the second one, if we would do this in general (not only for /prs), this could enable some improvements for dashboard as well (somehow related are packit/dashboard#196 and packit/dashboard#257). However, we will need to have a closer look how demanding these queries would be.

[lbarcziova]

hi @danmyway ! What is the status of this issue for you and its current importance?

[danmyway]

Hi @lbarcziova, I'm sorry for the unresponsiveness, I have somehow missed this.
Given there is no result retention in the GitHub GUI and getting to the results through the Packit Dashboard sifting through dozens of PRs is very time-consuming, I would say it is pretty important to me.
We have a tooling ready, which could parse the testing farm results based on the request UUID, the data leading to the UUIDs exist, but there is no easy way to connect these endpoints to get the desired results nowadays.