Update JSON output to JSONLines

[julesjacobsen]

JSON Lines would help users parsing the Exomiser JSON output as each gene result would be written a single line, enabling easier searching and return of results.

https://jsonlines.org/

This can be achieved using jq:

jq -c .[] Pfeiffer-hiphive-exome-PASS_ONLY.json > Pfeiffer-hiphive-exome-PASS_ONLY.jsonl

or better for really large files you can using streaming:

jq --stream -cn 'fromstream(1
  | truncate_stream(inputs))
  | select(length > 0)
  | select(.)' Pfeiffer-hiphive-exome-PASS_ONLY.json > Pfeiffer-hiphive-exome-PASS_ONLY.jsonl

But would likely be more useful as native output:

https://cowtowncoder.medium.com/line-delimited-json-with-jackson-69c9e4cb6c00

[damiansm]

This sounds like it would help a lot. Some of my analysis scripts take forever to run or even fail as they are trying to read the whole JSON output into a single object.

…

On Wed, Apr 24, 2024 at 9:47 AM Jules Jacobsen ***@***.***> wrote: JSON Lines would help users parsing the Exomiser JSON output as each gene result would be written a single line, enabling easier searching and return of results. https://jsonlines.org/ — Reply to this email directly, view it on GitHub <#558>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHO4PA7ENLJTVGJ7Y27LMDY65WTPAVCNFSM6AAAAABGWPFEM2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DANZVGI2TCMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jamesamcl]

JSON Lines also works very well with Pyspark.