Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use JQ #38

Merged
merged 59 commits into from
Oct 24, 2023
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
b7dc96e
use jq for basic transformer
rjawesome Aug 31, 2022
fb67cae
simplify opentarget transformer
rjawesome Sep 2, 2022
5739889
Create JQ Transfomer for BioLink
rjawesome Sep 21, 2022
809d73e
add comments for transfomers
rjawesome Oct 11, 2022
63f2cf6
remove extra console log
rjawesome Oct 20, 2022
56a4dcc
Merge branch 'use-jmes-path' of https://github.com/biothings/api-resp…
rjawesome Oct 20, 2022
c881db0
use ebi api
rjawesome Oct 21, 2022
452b97a
Refactor JQ for cleaner filter strings & functions
rjawesome Oct 22, 2022
e32efd5
Fix biolink & add tests for other transformers
rjawesome Oct 28, 2022
c6d23d4
Fix EBI Transfomer
rjawesome Oct 29, 2022
c37e7c7
make jq strings cleaner
rjawesome Oct 29, 2022
ca62d2d
allow custom JQ strings from the query_operation
rjawesome Nov 8, 2022
e6a12cc
add edge variable to be used in filter strings
rjawesome Nov 15, 2022
e6bc9a3
clean up
rjawesome Nov 15, 2022
6693fe2
start on a semmedb transformer
rjawesome Nov 22, 2022
9b5070b
create jq string for pairCurieWithAPIResponse
rjawesome Nov 23, 2022
90bb344
Use jq_wrap parameter
rjawesome Nov 29, 2022
ce17e57
making transformer into object
rjawesome Nov 29, 2022
ffaccc6
add generate curie function
rjawesome Nov 29, 2022
ac47782
allow pair_jq option
rjawesome Nov 29, 2022
27f631e
fix checking for transformer
rjawesome Dec 3, 2022
e7962bf
Merge branch 'main' into use-jmes-path
rjawesome Dec 6, 2022
c897fd0
More merge changes from main
rjawesome Dec 6, 2022
1c63e48
Add jq filter string for pairCurie biothings
rjawesome Dec 6, 2022
7e67e82
Merge branch 'main' into use-jmes-path
rjawesome Dec 22, 2022
0ce7b80
initial list filter function (untested)
rjawesome Jan 6, 2023
d5e985e
Fix list filter all/any
rjawesome Jan 7, 2023
c00c5af
nesting
rjawesome Jan 7, 2023
2bace62
add node-jq dependency
rjawesome Jan 7, 2023
9e541c6
Multiple allowed values for list_filter_all
rjawesome Jan 13, 2023
78aecaf
Fix ctd https://github.com/biothings/BioThings_Explorer_TRAPI/issues/583
rjawesome Mar 15, 2023
a59b35a
handle direct evidence split by |
rjawesome Mar 15, 2023
3df9e40
biothings support
rjawesome Mar 23, 2023
4fd7e31
add ctd pair
rjawesome Mar 23, 2023
934eb36
spelling fixes
rjawesome Mar 24, 2023
ee9fdd9
Merge branch 'main' of https://github.com/biothings/api-respone-trans…
tokebe Mar 28, 2023
8a6ac89
remove disease id parsing from ctd
rjawesome Mar 29, 2023
9fbcb38
test fix
rjawesome Mar 29, 2023
ea9db14
fix https://github.com/biothings/BioThings_Explorer_TRAPI/issues/592
rjawesome Mar 30, 2023
f49766c
fix generateCurieWithInputs to handle array
rjawesome Aug 2, 2023
55b48c9
migrate biothings transformer test to jq
rjawesome Aug 2, 2023
f2fc0d6
update biothings test more for jq
rjawesome Aug 2, 2023
7e3b524
add back semmed transformer
rjawesome Aug 2, 2023
76b52e2
update biothings jq
rjawesome Aug 2, 2023
95743bf
Merge branch 'main' of https://github.com/biothings/api-respone-trans…
rjawesome Aug 4, 2023
77f6c7f
clean up files
rjawesome Sep 15, 2023
2375245
fix: typo, mv jq string to data/, ts formatting
tokebe Sep 18, 2023
4404147
Merge branch 'use-jmes-path' of https://github.com/biothings/api-resp…
rjawesome Sep 18, 2023
f9a9689
Fix handling of biothings JQ, biothings/biothings_explorer#489
rjawesome Sep 19, 2023
e729a8a
better typing
rjawesome Sep 29, 2023
7ab4e74
fix most tests
rjawesome Sep 29, 2023
1e3844d
chore: move jq utils string to data/
tokebe Oct 3, 2023
7df862e
fix: ctd jq pair function
tokebe Oct 3, 2023
14c50b0
Merge branch 'main' of https://github.com/biothings/api-respone-trans…
tokebe Oct 4, 2023
00f7c6e
style: indent 2
tokebe Oct 12, 2023
8d7b437
fix: CTD pair function batch support
tokebe Oct 12, 2023
549e415
fix: obtain jq wrap/pair from annotation
tokebe Oct 13, 2023
5215047
style: prettier formatting
tokebe Oct 23, 2023
17496d8
Merge branch 'main' of https://github.com/biothings/api-respone-trans…
tokebe Oct 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions __test__/base_transformer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ describe("test base transformer", () => {
}
})

test("Test pairInputWithAPIResponse function", () => {
test("Test pairInputWithAPIResponse function", async () => {
const tf = new base_tf(input, {});
const res = tf.pairCurieWithAPIResponse();
const res = await tf.pairCurieWithAPIResponse();
expect(res).toHaveProperty("DOID:9562");
expect(res["DOID:9562"]).toHaveLength(1);
})
Expand Down
38 changes: 19 additions & 19 deletions __test__/biolink_transformers.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import {describe, expect, test} from '@jest/globals';

import biolink_tf from "../src/transformers/biolink_transformer";
import jq_tf from "../built/transformers/jq_transfomer";
import fs from "fs";
import path from "path";
import { JSONDoc } from '../src/json_transform/types';
Expand All @@ -21,24 +21,24 @@ describe("test biolink transformer", () => {
}
})

test("test biolink wrapper", () => {
let tf = new biolink_tf(input, {});
let res = tf.wrap(response);
test("test biolink wrapper", async () => {
let tf = new jq_tf(input, { type: "biolink" });
let res = await tf.wrap(response);
expect(res.associations[0].object.HGNC).toBe("10956");
expect(res.associations[0].publications[0].id).toBe("21685912");
expect(res.associations[1]).not.toHaveProperty("publications");
expect(res.associations[1]).not.toHaveProperty("provided_by");
});

test("test biolink wrapper if no association field as root key", () => {
let tf = new biolink_tf(input, {});
let res = tf.wrap({ "data": [] });
test("test biolink wrapper if no association field as root key", async () => {
let tf = new jq_tf(input, { type: "biolink" });
let res = await tf.wrap({ "data": [] });
expect(res).toEqual({ "data": [] })
});

test("test biolink wrapper if no object id should be prefixed", () => {
const tf = new biolink_tf(input, {});
const res = tf.wrap(
test("test biolink wrapper if no object id should be prefixed", async () => {
const tf = new jq_tf(input, { type: "biolink" });
const res = await tf.wrap(
{
associations: [
{
Expand All @@ -51,8 +51,8 @@ describe("test biolink transformer", () => {
expect(res.associations[0].object.MONDO).toEqual("MONDO:12345")
});

test("test biolink wrapper if no object field present", () => {
const tf = new biolink_tf(input, {});
test("test biolink wrapper if no object field present", async () => {
const tf = new jq_tf(input, { type: "biolink" });
const fake_response = {
associations: [
{
Expand All @@ -62,12 +62,12 @@ describe("test biolink transformer", () => {
}
]
}
const res = tf.wrap(fake_response);
const res = await tf.wrap(fake_response);
expect(res).toEqual(fake_response)
});

test("test biolink wrapper if no object.id field present", () => {
const tf = new biolink_tf(input, {});
test("test biolink wrapper if no object.id field present", async () => {
const tf = new jq_tf(input, { type: "biolink" });
const fake_response = {
associations: [
{
Expand All @@ -77,13 +77,13 @@ describe("test biolink transformer", () => {
}
]
}
const res = tf.wrap(fake_response);
const res = await tf.wrap(fake_response);
expect(res).toEqual(fake_response)
});

test("test biolink jsonTransform function", () => {
let tf = new biolink_tf(input, {});
const wrapped_response = tf.wrap(response);
test("test biolink jsonTransform function", async () => {
let tf = new jq_tf(input, { type: "biolink" });
const wrapped_response = await tf.wrap(response);
let res: JSONDoc = tf.jsonTransform(wrapped_response);
expect(res).toHaveProperty("related_to");
expect(res.related_to[0].HGNC).toEqual("10956");
Expand Down
12 changes: 6 additions & 6 deletions __test__/biothings_transformer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ describe("test biothings transformer", () => {
}
})

test("test biothings wrapper", () => {
test("test biothings wrapper", async () => {
let tf = new biothings_tf(input, {});
let res = tf.pairCurieWithAPIResponse();
let res = await tf.pairCurieWithAPIResponse();
expect(Object.keys(res)).toHaveLength(2);
expect(res).toHaveProperty("DRUGBANK:DB00188");
expect(res["DRUGBANK:DB00188"]).toHaveLength(2);
Expand All @@ -46,9 +46,9 @@ describe("test biothings transformer", () => {
}
})

test("test biothings wrapper", () => {
test("test biothings wrapper", async () => {
let tf = new biothings_tf(input, {});
let res = tf.pairCurieWithAPIResponse();
let res = await tf.pairCurieWithAPIResponse();
expect(Object.keys(res)).toHaveLength(1);
expect(res).toHaveProperty("NCBIGene:1017");
expect(res["NCBIGene:1017"]).toHaveLength(1);
Expand Down Expand Up @@ -78,9 +78,9 @@ describe("test biothings transformer", () => {
}
})

test("test biothings wrapper", () => {
test("test biothings wrapper", async () => {
let tf = new biothings_tf(input, {});
let res = tf.pairCurieWithAPIResponse();
let res = await tf.pairCurieWithAPIResponse();
expect(Object.keys(res)).toHaveLength(1);
expect(res).toHaveProperty("PUBCHEM:11373846");
expect(res["PUBCHEM:11373846"]).toHaveLength(1);
Expand Down
21 changes: 10 additions & 11 deletions __test__/ctd_transformer.test.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
import {describe, expect, test} from '@jest/globals';
import fs from "fs";
import path from "path";

import ctd_tf from "../src/transformers/ctd_transformer";
import jq_tf from "../built/transformers/jq_transfomer";

describe("test ctd transformer", () => {

Expand All @@ -20,36 +19,36 @@ describe("test ctd transformer", () => {
}
})

test("test ctd wrapper", () => {
let tf = new ctd_tf(input, {});
let res = tf.wrap(response);
test("test ctd wrapper", async () => {
let tf = new jq_tf(input, { type: "ctd" });
let res = await tf.wrap(response);
expect(res).toHaveProperty("data");
expect(res.data).toHaveLength(2);
expect(res.data[0].PubMedIDs).toEqual(["21559390"]);
expect(res.data[0].DiseaseID).toEqual("D008545");
})

test("test ctd wrapper if pubmed id field is not string", () => {
const tf = new ctd_tf(input, {});
test("test ctd wrapper if pubmed id field is not string", async () => {
const tf = new jq_tf(input, { type: "ctd" });
const fake = [
{
DiseaseID: "MESH:D008545"
}
]
const res = tf.wrap(fake);
const res = await tf.wrap(fake);
expect(res).toHaveProperty("data");
expect(res.data).toHaveLength(1);
expect(res.data[0].PubMedIDs).toBeUndefined;
})

test("test ctd wrapper if disease id field is not string", () => {
const tf = new ctd_tf(input, {});
test("test ctd wrapper if disease id field is not string", async () => {
const tf = new jq_tf(input, { type: "ctd" });
const fake = [
{
PubMedID: "12345"
}
]
const res = tf.wrap(fake);
const res = await tf.wrap(fake);
expect(res).toHaveProperty("data");
expect(res.data).toHaveLength(1);
expect(res.data[0].DiseaseIDs).toBeUndefined;
Expand Down
8 changes: 4 additions & 4 deletions __test__/ebi_protein_transformer.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { describe, expect, test } from "@jest/globals";
import fs from "fs";
import path from "path";
import ebi_tf from "../src/transformers/ebi_protein_transformer";
import jq_tf from "../src/transformers/jq_transfomer";

describe("test EBI Protein transformer", () => {
let response;
Expand All @@ -18,9 +18,9 @@ describe("test EBI Protein transformer", () => {
};
});

test("test ebi wrapper", () => {
let tf = new ebi_tf(input, {});
let res = tf.wrap(response);
test("test ebi wrapper", async () => {
let tf = new jq_tf(input, {type: "ebi"});
let res = await tf.wrap(response);
expect(res.comments[0].reaction.dbReferences).toHaveLength(1);
});
});
14 changes: 7 additions & 7 deletions __test__/opentarget_transformer.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import {describe, expect, test} from '@jest/globals';
import jq_tf from "../built/transformers/jq_transfomer";
import fs from "fs";
import path from "path";
import opentarget_tf from "../src/transformers/opentarget_transformer";

describe("test opentarget transformer", () => {

Expand All @@ -20,15 +20,15 @@ describe("test opentarget transformer", () => {
})

// skip these tests since we're not ingesting opentargets right now
test.skip("test opentarget wrapper", () => {
let tf = new opentarget_tf(input, {});
let res = tf.wrap(response);
test.skip("test opentarget wrapper", async () => {
let tf = new jq_tf(input, { type: "opentarget" });
let res = await tf.wrap(response);
expect(res).toHaveProperty("data");
expect(res.data[0].drug.id).toEqual("CHEMBL220492");
})

test.skip("test opentarget wrapper if id field is not chembl", () => {
let tf = new opentarget_tf(input, {});
test.skip("test opentarget wrapper if id field is not chembl", async () => {
let tf = new jq_tf(input, { type: "opentarget" });
const fake = {
data: [
{
Expand All @@ -38,7 +38,7 @@ describe("test opentarget transformer", () => {
}
]
}
let res = tf.wrap(fake);
let res = await tf.wrap(fake);
expect(res).toHaveProperty("data");
expect(res.data[0].drug.id).toEqual("http://identifiers.org/drugbank/DB0001");
})
Expand Down
6 changes: 3 additions & 3 deletions __test__/semmed_transformer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,9 @@ describe("test semmed transformer", () => {
}
});

test("test semmed pairCurieWithAPIResponse", () => {
test("test semmed pairCurieWithAPIResponse", async () => {
let tf = new semmed_tf(input, {});
let res = tf.pairCurieWithAPIResponse();
let res = await tf.pairCurieWithAPIResponse();
expect(res["UMLS:C1332823"][0]['umls']).toBe("C1332823");
expect(res).toHaveProperty('UMLS:C1332823');
expect(res["UMLS:123"]).toBeUndefined();
Expand All @@ -98,7 +98,7 @@ describe("test semmed transformer", () => {

test("add edge info", async () => {
let tf = new semmed_tf(input, {});
let res = tf.pairCurieWithAPIResponse();
let res = await tf.pairCurieWithAPIResponse();
let rec = res["UMLS:C1332823"][0];
rec = tf.wrap(rec);
let result = await tf.formatRecords("UMLS:C1332823", rec["positively_regulates"][0]);
Expand Down
4 changes: 2 additions & 2 deletions __test__/transformers.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,9 @@ describe("test biothings transformer", () => {
}
});

test("test biothings pairCurieWithAPIResponse", () => {
test("test biothings pairCurieWithAPIResponse", async () => {
let tf = new biothings_tf(input, {});
let res = tf.pairCurieWithAPIResponse();
let res = await tf.pairCurieWithAPIResponse();
expect(res["UMLS:C1332823"][0]['umls']).toBe("C1332823");
expect(res).toHaveProperty('UMLS:C1332823');
expect(res["123"]).toBeUndefined();
Expand Down
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@
"common-path-prefix": "^3.0.0",
"husky": "^4.3.8",
"jsonata": "^1.8.4",
"lodash": "^4.17.21"
"lodash": "^4.17.21",
"node-jq": "^2.3.4"
},
"husky": {
"hooks": {
Expand Down
27 changes: 20 additions & 7 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import OpenTargetTransformer from "./transformers/opentarget_transformer";
import BaseTransformer from "./transformers/transformer";
import TRAPITransformer from './transformers/trapi_transformer';
import EBIProteinTransformer from './transformers/ebi_protein_transformer'
import JQTransformer from "./transformers/jq_transfomer";
import { BTEQueryObject } from "./types";
const debug = require("debug")("bte:api-response-transform:index");
export { Record } from "./record";
Expand All @@ -26,18 +27,30 @@ export class Transformer {
debug(`api name ${api}`);
let tags = this.data.edge.query_operation.tags;
debug(`api tags: ${tags}`);
if (tags.includes('bte-trapi')) {

// if (!this.data.edge.query_operation.transformer) {
// console.log(`WE DONT DO THE OP ${api}, ${this.data.edge.query_operation}`)
// }

if (this.data.edge.query_operation?.transformer?.pair_jq || this.data.edge.query_operation?.transformer?.wrap_jq) {
// console.log("WE DO THE OP", this.data.edge.query_operation.transformer)
this.tf = new JQTransformer(this.data, {
...this.config,
wrap: this.data.edge.query_operation.transformer.wrap_jq,
pair: this.data.edge.query_operation.transformer.pair_jq
tokebe marked this conversation as resolved.
Show resolved Hide resolved
})
} else if (tags.includes('bte-trapi')) {
this.tf = new TRAPITransformer(this.data, this.config);
} else if (api.startsWith('SEMMED')) {
this.tf = new SemmedTransformer(this.data, this.config);
} else if (api === 'BioLink API') {
this.tf = new BiolinkTransformer(this.data, this.config);
// this.tf = new BiolinkTransformer(this.data, this.config);
this.tf = new JQTransformer(this.data, { ...this.config, type: "biolink" })
} else if (api === 'EBI Proteins API') {
this.tf = new EBIProteinTransformer(this.data, this.config)
// this.tf = new EBIProteinTransformer(this.data, this.config)
this.tf = new JQTransformer(this.data, { ...this.config, type: "ebi" })
} else if (tags.includes("biothings")) {
this.tf = new BioThingsTransformer(this.data, this.config);
this.tf = new JQTransformer(this.data, { ...this.config, type: "biothings" })
} else if (tags.includes("ctd")) {
this.tf = new CTDTransformer(this.data, this.config);
this.tf = new JQTransformer(this.data, { ...this.config, type: "ctd" });
} else if (tags.includes("opentarget")) {
this.tf = new OpenTargetTransformer(this.data, this.config)
} else {
Expand Down
Loading