Large datasets: can I use CRAM and BAM as input? Can I compress the .pairs output?

[amvarani]

Dear pairtools Team,

First and foremost, I would like to extend my gratitude to the development team for creating such an incredible tool. I am currently working with very large files generated by bwa. For example, my generated BAM file is approximately 600GB in size. Additionally, the file parsed using pairtools parse is exceeding 1.6TB.

It would be highly beneficial if pairtools could support lossless CRAM files as input. Furthermore, the ability to compact .pairs files on the fly for disk space conservation would be immensely valuable. Is this functionality feasible?

Thank you for your consideration.

[Phlya]

CRAM is a good idea, thank you.

Compression of pairs of fully supported. Simply giving the output name a .gz extension will compress the file (and .gz files are automatically uncompressed).

[Phlya]

Actually.. Since we now use pysam for parsing alignment files, possible CRAM would just work already? Have you tried?

@agalitsyna I think you implemented the pysam parsing code, do you know whether CRAM files would also be parsed?

[golobor]

hi! Indeed, outputs are automatically compressed, based on the extension. You could also output into stdout and pipe into a compression tool of your choice!
Similarly, you could input CRAM by converting CRAM into .sam, which can be fed into stdout and, from there piped into pairtools parse. Would that work? It would a bit suboptimal CPU-wise - reading from CRAM via pysam would be faster, but, it should not be drastically worse.

[Phlya]

Ah for cram pysam needs a different reading mode "rc" instead of "rb" for bam. But is that the only change we would need to implement?

[amvarani]

Thanks for the reply. I'm now using BAM and CRAM files, which I am piping. I'm also outputting the files as .gz to save disk space!