Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use JQ #38

Merged
merged 59 commits into from
Oct 24, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
b7dc96e
use jq for basic transformer
rjawesome Aug 31, 2022
fb67cae
simplify opentarget transformer
rjawesome Sep 2, 2022
5739889
Create JQ Transfomer for BioLink
rjawesome Sep 21, 2022
809d73e
add comments for transfomers
rjawesome Oct 11, 2022
63f2cf6
remove extra console log
rjawesome Oct 20, 2022
56a4dcc
Merge branch 'use-jmes-path' of https://github.com/biothings/api-resp…
rjawesome Oct 20, 2022
c881db0
use ebi api
rjawesome Oct 21, 2022
452b97a
Refactor JQ for cleaner filter strings & functions
rjawesome Oct 22, 2022
e32efd5
Fix biolink & add tests for other transformers
rjawesome Oct 28, 2022
c6d23d4
Fix EBI Transfomer
rjawesome Oct 29, 2022
c37e7c7
make jq strings cleaner
rjawesome Oct 29, 2022
ca62d2d
allow custom JQ strings from the query_operation
rjawesome Nov 8, 2022
e6a12cc
add edge variable to be used in filter strings
rjawesome Nov 15, 2022
e6bc9a3
clean up
rjawesome Nov 15, 2022
6693fe2
start on a semmedb transformer
rjawesome Nov 22, 2022
9b5070b
create jq string for pairCurieWithAPIResponse
rjawesome Nov 23, 2022
90bb344
Use jq_wrap parameter
rjawesome Nov 29, 2022
ce17e57
making transformer into object
rjawesome Nov 29, 2022
ffaccc6
add generate curie function
rjawesome Nov 29, 2022
ac47782
allow pair_jq option
rjawesome Nov 29, 2022
27f631e
fix checking for transformer
rjawesome Dec 3, 2022
e7962bf
Merge branch 'main' into use-jmes-path
rjawesome Dec 6, 2022
c897fd0
More merge changes from main
rjawesome Dec 6, 2022
1c63e48
Add jq filter string for pairCurie biothings
rjawesome Dec 6, 2022
7e67e82
Merge branch 'main' into use-jmes-path
rjawesome Dec 22, 2022
0ce7b80
initial list filter function (untested)
rjawesome Jan 6, 2023
d5e985e
Fix list filter all/any
rjawesome Jan 7, 2023
c00c5af
nesting
rjawesome Jan 7, 2023
2bace62
add node-jq dependency
rjawesome Jan 7, 2023
9e541c6
Multiple allowed values for list_filter_all
rjawesome Jan 13, 2023
78aecaf
Fix ctd https://github.com/biothings/BioThings_Explorer_TRAPI/issues/583
rjawesome Mar 15, 2023
a59b35a
handle direct evidence split by |
rjawesome Mar 15, 2023
3df9e40
biothings support
rjawesome Mar 23, 2023
4fd7e31
add ctd pair
rjawesome Mar 23, 2023
934eb36
spelling fixes
rjawesome Mar 24, 2023
ee9fdd9
Merge branch 'main' of https://github.com/biothings/api-respone-trans…
tokebe Mar 28, 2023
8a6ac89
remove disease id parsing from ctd
rjawesome Mar 29, 2023
9fbcb38
test fix
rjawesome Mar 29, 2023
ea9db14
fix https://github.com/biothings/BioThings_Explorer_TRAPI/issues/592
rjawesome Mar 30, 2023
f49766c
fix generateCurieWithInputs to handle array
rjawesome Aug 2, 2023
55b48c9
migrate biothings transformer test to jq
rjawesome Aug 2, 2023
f2fc0d6
update biothings test more for jq
rjawesome Aug 2, 2023
7e3b524
add back semmed transformer
rjawesome Aug 2, 2023
76b52e2
update biothings jq
rjawesome Aug 2, 2023
95743bf
Merge branch 'main' of https://github.com/biothings/api-respone-trans…
rjawesome Aug 4, 2023
77f6c7f
clean up files
rjawesome Sep 15, 2023
2375245
fix: typo, mv jq string to data/, ts formatting
tokebe Sep 18, 2023
4404147
Merge branch 'use-jmes-path' of https://github.com/biothings/api-resp…
rjawesome Sep 18, 2023
f9a9689
Fix handling of biothings JQ, biothings/biothings_explorer#489
rjawesome Sep 19, 2023
e729a8a
better typing
rjawesome Sep 29, 2023
7ab4e74
fix most tests
rjawesome Sep 29, 2023
1e3844d
chore: move jq utils string to data/
tokebe Oct 3, 2023
7df862e
fix: ctd jq pair function
tokebe Oct 3, 2023
14c50b0
Merge branch 'main' of https://github.com/biothings/api-respone-trans…
tokebe Oct 4, 2023
00f7c6e
style: indent 2
tokebe Oct 12, 2023
8d7b437
fix: CTD pair function batch support
tokebe Oct 12, 2023
549e415
fix: obtain jq wrap/pair from annotation
tokebe Oct 13, 2023
5215047
style: prettier formatting
tokebe Oct 23, 2023
17496d8
Merge branch 'main' of https://github.com/biothings/api-respone-trans…
tokebe Oct 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import OpenTargetTransformer from "./transformers/opentarget_transformer";
import BaseTransformer from "./transformers/transformer";
import TRAPITransformer from './transformers/trapi_transformer';
import EBIProteinTransformer from './transformers/ebi_protein_transformer'
import JQTransformer from "./transformers/jq_transfomer";
import { BTEQueryObject } from "./types";
const debug = require("debug")("bte:api-response-transform:index");
export { Record } from "./record";
Expand All @@ -31,7 +32,8 @@ export class Transformer {
} else if (api.startsWith('SEMMED')) {
this.tf = new SemmedTransformer(this.data, this.config);
} else if (api === 'BioLink API') {
this.tf = new BiolinkTransformer(this.data, this.config);
// this.tf = new BiolinkTransformer(this.data, this.config);
this.tf = new JQTransformer(this.data, { ...this.config, type: "biolink" })
} else if (api === 'EBI Proteins API') {
this.tf = new EBIProteinTransformer(this.data, this.config)
} else if (tags.includes("biothings")) {
Expand Down
18 changes: 18 additions & 0 deletions src/transformers/jq_transfomer.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import BaseTransformer from "./transformer";

const jq = require('node-jq');

const filterStrings = {
'ebi': '(select(.comments != null) | .comments | .[] | select(.reaction != null) | .reaction.dbReferences | .[]) |= select(.type == "Rhea")',
'ctd': '((.[] | select(.PubMedIDs != null) | .PubMedIDs) |= split("|")) | ((.[] | select(.DiseaseID != null) | .DiseaseID) |= (split(":") | last)) | {data: .}',
'opentarget': '(.data | .[] | select(.drug.id != null) | select(.drug.id | contains("CHEMBL")) | .drug.id) |= (split("/") | last)',
'biolink': '[del(.associations), {associations: [select(.associations != null) | .associations | .[] | select(.object.id != null) | (.object.id | split(":") | first) as $pref | .object[$pref] = (if $pref == "HGNC" or $pref == "NCBIGene" or $pref == "REACT" then (.object.id | split(":") | last) else .object.id end) | .publications = [foreach (.publications | .[]? | .id) as $pub ([]; []; if ($pub == null or ($pub | startswith("PMID") | not)) then empty else {id: ($pub | split(":") | last)} end)] ]}] | add'
newgene marked this conversation as resolved.
Show resolved Hide resolved
}

export default class JQTransformer extends BaseTransformer {
async wrap (res) {
res = JSON.parse(await jq.run(filterStrings[this.config.type], res, { input: 'json' }));
console.log("THIS IS WHAT WILL BE RETURNED", res.associations[1] )
return res;
}
}
20 changes: 11 additions & 9 deletions src/transformers/opentarget_transformer.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
import BaseTransformer from "./transformer";

import jq from "node-jq";

export default class OpenTargetTransformer extends BaseTransformer {
wrap(res) {
res.data = res.data.map(item => {
if ('drug' in item && "id" in item.drug && typeof item.drug.id === "string" && item.drug.id.includes("CHEMBL")) {
item.drug.id = item.drug.id.split('/').slice(-1)[0];
}
return item;
});
return res;
async wrap(res) {
// res.data = res.data.map(item => {
// if ('drug' in item && "id" in item.drug && typeof item.drug.id === "string" && item.drug.id.includes("CHEMBL")) {
// item.drug.id = item.drug.id.split('/').slice(-1)[0];
// }
// return item;
// });
// return res;
const filterString = '(.data | .[] | select(.drug.id != null) | select(.drug.id | contains("CHEMBL")) | .drug.id) |= (split("/") | last)';
return await jq.run(filterString, res);
}
}
2 changes: 1 addition & 1 deletion src/transformers/transformer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ export default class BaseTransformer {
await async.eachSeries(Object.entries(responses), async ([curie, curieResponses]) => {
if (Array.isArray(curieResponses) && curieResponses.length > 0) {
await async.eachSeries(curieResponses, async response => {
const predicateResponse = this.jsonTransform(this.wrap(response));
const predicateResponse = this.jsonTransform(await this.wrap(response));
await async.eachSeries(Object.entries(predicateResponse), async ([predicate, mappedResponses]) => {
if (Array.isArray(mappedResponses) && mappedResponses.length > 0) {
await async.eachSeries(mappedResponses, async (mappedResponse: any[]) => {
Expand Down