Other ways of creating db backends or obviating the need for a db

[shntnu]

For our notes, there are two alternate paths that either replicate cytominer-database functionality or obviate its need

1. @gwaygenomics works directly with CSVs in pooled Cell Painting experiments link
2. @bethac07 has a script for ingesting CSVs into a SQLIte file, which can be loaded in CellProfiler Analyst link

@bethac07 @gwaygenomics could you please edit this comment to add the links / details above? 🙏

[bethac07]

Where would you like mine to live? My generic repo, one of our profiling repos?

[shntnu]

Where would you like mine to live? My generic repo, one of our profiling repos?

I think the google docs is perfectly fine, or at least, it doesn't seem worth the effort of transferring 80+ pages into a markdown document (unless you find that easier to maintain in the longer run) sorry I misread the topic :D

[bethac07]

Sorry, clarifying- my python script that you can use in lieu of cytominer-database. It's just been living in my random Broad project repo, but if you think it should live somewhere slightly more official now is a good time for it!

[shntnu]

but if you think it should live somewhere slightly more official now is a good time for it!

A repo that has frequently-used scripts by assay devs would be great, but this might be an overhead for you to maintain, so its your call.

Either way no harm in creating a repo image-analysis-scripts or similar in github.com/broadinstitute, even if this is the only script that lives in it.

[diskontinuum]

For our notes, there are two alternate paths that either replicate cytominer-database functionality or obviate its need

1. @gwaygenomics works directly with CSVs in pooled Cell Painting experiments [link]
2. @bethac07 has a script for ingesting CSVs into a SQLIte file, which can be loaded in CellProfiler Analyst link

@bethac07 @gwaygenomics could you please edit this comment to add the links / details above? 🙏

The link is broken?

[bethac07]

The link is not broken, just only available inside the BroadInstitute, sorry!

[gwaybio]

This completely slipped through my TODOs - I updated my link above (also here it is). Beth's link is in a private repo that it looks like you don't have access to.

One of the reasons I decided to sidestep the database step is b/c in the pooled cell painting project we will likely work with single cell data. It is a pain to manipulate/extract single cell files from the standard sqlite format (especially since we need to merge across tables each and every time) - these processes take way too long.