Start Weekly Run & Reporting of ARS Performance Tests

[sstemann]

No description provided.

[sierra-moxon]

From @ShervinAbd92:

On ARS side, our pytest will run series of queries and check out their results after 5 min,
Only taking the tools that have “Done” status with results greater than 0. Then going through each result, it will detect “allow_curies” and “deny_curies” results and map them with their normalized score into a report_card. So you will get a json report.
But here are some ongoing issues:

1. Lets say you are interested in Natalizumab. looking it up in the name resolver, I found MESH:D000069442 to be a top level CURIE for it and looking at the results i see BTE and robokop return this CURIE but ARAX and Improving show CHEMBL.COMPOUND:CHEMBL120160 which is no where in the list of name resolver or node normalizer equivalent identifiers --> contacted Gaurav Vaidya —> Natalizumab is split into two cliques TranslatorSRI/Babel#157

2. Another issue I am seeing is having the same drug one upper case and one lower case appearing on the same tool, one has the matching CURIE but the other one has the second CURIE item from the name resolver list (which is not wrong) —>Gaurav working on this ticket to resolve the issue Add RxNorm TranslatorSRI/Babel#150

3. Next issue is with Molepro returning the same drug multiple time with no score, in the case of “What drugs may treat Multiple Sclerosis?” We are interested in “Fingolimod” and currently we get 6 results from Molepro with score being None, for now i am just assigning zero when the code detect them —> contacted Vlado Dancik —> Returning superclasses in treats responses broadinstitute/molecular-data-provider#308 and Don't do subclass_of stuff on a subclass_of query ranking-agent/reasoner-transpiler#77

4. This report card is done for multiple sclerosis and castleman’s disease, we can discuss how to move forward with them?

[sierra-moxon]

From @edeutsch -

This is great progress! If you can make the results available via API or some other HTTP mechanism, we could try to display them via the ARAX UI if that is desirable

[edeutsch]

Just an offer, not a requirement. But I do think that getting a visually accessible report card in front of everyone is good way to get problems noticed and fixed.

[sierra-moxon]

@ShervinAbd92 - does this run weekly? can we close this? Do you know who is responding to the weekly runs? :) I am not sure what to do with this ticket.

[ShervinAbd92]

Hi @sierra-moxon, I am not sure if we should close it yet. with @edeutsch suggestion i started working on developing a django server that has endpoints for running the semantic smoke testing and load testing and showing the resultant report. I am hoping that i can finish it by this week and get ITRB to deploy it for us.

[ShervinAbd92]

Hi @sierra-moxon, i have started running the weekly load testing since Friday. tried attaching the results here but json format wasn't acceptable, so created a folder under Shared Drive--> Biomedical Data Translator: Development --> Weekly_Performace_Testing. Please take a look and let me know if you have any questions?

[edeutsch]

Hi @ShervinAbd92 interesting thanks. Can you make it available somewhere programmatically accessible with minimal fuss?

Also, I'm wondering how to interpret the results. What does this mean?

          "PUBCHEM.COMPOUND:54684141": [
            [
              "ara-bte",
              96.4
            ],
            [
              "ara-improving",
              81.35593220338983
            ],
            [
              "kp-molecular",
              0
            ],
            [
              "kp-molecular",
              0
            ],
            [
              "kp-molecular",
              0
            ],
            [
              "kp-molecular",
              0
            ],
            [
              "kp-molecular",
              0
            ],
            [
              "ara-arax",
              100.0
            ]
          ],

[ShervinAbd92]

Hi @edeutsch sure, i will work with ITRB to make this accessible to you guys as soon as i can.
the above result means that BTE, improving, molepro and arax have returned the desired compound (PUBCHEM.COMPOUND:54684141) and the numbers are their normalized score. now for molepro showing up this drug multiple times, Vlado did let me know they are working on it to fix it so it will show once.

[sstemann]

Targeting full implementation 7/25 - currently running locally, posting results files to here (https://drive.google.com/drive/folders/1FLRHoL6HxJ7LM7P8YMWVyaCv2Urg_8m4?usp=sharing) , next steps are:

1. implement that "users" can configure environment to run against
2. implement and configure API Keys
3. deploy to ITRB (roughly two weeks due to their processes)

[sierra-moxon]

per @ShervinAbd92 - this ticket can be closed, the report-runner is hosted and is planned to be run weekly: https://utility.ci.transltr.io/arstest/api