x-bte operations: replace BioThings list_filter with jmespath

[colleenXu]

I've heard that jmespath should be used in replacement of list_filter for BioThings API querying (internal lab Slack thread). At the moment, list_filter behavior is still working and @newgene said a switch-over was not urgent.

However, to do this replacement / use jmespath moving forward, I'd need to review the documentation. I have some links here (some are internal lab slack threads):

• biothings/biothings.api@31898fa
• creating additional semmeddb api endpoints with custom query filters pending.api#139
• Slack discussions here and here

I'm not sure if there is other documentation / info out there, or other info I'd need to adjust existing x-bte annotation and use this going forward...

---

list_filter was used for biothings/pending.api#101, and was considered part of biothings/pending.api#100 and biothings/mychem.info#153.

I had asked about an enhancement / update to list_filter: biothings/pending.api#101 (comment) and this internal lab slack thread and was told that jmespath could potentially address this...

[colleenXu]

Note: saw post on jmespath design in the JQ stuff, the info may be helpful here. In particular, there's a link to a function's doc string (which may be helpful for understanding what to do for jmespath): https://github.com/biothings/biothings.api/blob/7b83617c40c2f12513a7db6842e5d14adf694039/biothings/web/query/formatter.py#L400

[colleenXu]

Rohan mentioned in this post (a reply to this comment) and starting here that JQ may be able to do similar behavior.

My impression is that:

• JQ can be used for non-BioThings APIs
• BioThings jmespath may be better to use for BioThings APIs?

But we'd need more discussion before deciding how to proceed + what we want to cover with this issue

[newgene]

@colleenXu that sounds like a good plan to proceed.

[colleenXu]

Narrowing the scope of this issue to just replacing list_filter. Using jmespath to write new issues can be >=1 new issues. Same with using JQ to write new operations.

[colleenXu]

Note: jmespath seems to be like list_filter, where the fields being acted upon must be included in the initial response (using the parameters.fields). See internal Slack discussion here

because list_filter is a post-query action, the field we filter on MUST be in the response. So yes, this is required

[colleenXu]

There's only 1 API/situation where we are using list_filter: MyChem drugcentral.bioactivity chem -> gene operations (they were a first use-case).

Progress so far:

• Should be able to update all the current operations. I could make queries with both filter and jmespath to retrieve only the parts of records where the action_type is the desired value. Ex: NCATS-Tangerine/translator-api-registry@efd09ee
• Have a problem with creating equivalent reverse operations. I can create a query with jmespath set to the desired action_type (drugcentral.bioactivity|[?action_type==`ACTIVATOR`]) OR to the desired ID (drugcentral.bioactivity.uniprot|[?contains(uniprot_id, `O75343`)]) but not both at the same time.

Example query without the jmespath param set

curl --location 'https://mychem.info/v1/query?size=20&fields=drugcentral.bioactivity%2Cdrugcentral.xrefs&with_total=true&filter=drugcentral.bioactivity.action_type%3AACTIVATOR&jmespath_exclude_empty=true' \
--header 'Content-Type: application/json' \
--data '{
    "q": ["O75343"],
    "scopes": "drugcentral.bioactivity.uniprot.uniprot_id"
}'

[colleenXu]

Done updating all current operations to use filter/jmespath. (1 operation, rest of operations, merging/adding new chem -> gene operations)

Going to open a different issue to track writing reverse operations (have a possible way to do it). There's also a separate issue for chem-gene data that doesn't have the action_type field #904