Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stats for prod119 release #187

Merged
merged 2 commits into from
Oct 9, 2023
Merged

Add stats for prod119 release #187

merged 2 commits into from
Oct 9, 2023

Conversation

dominikl
Copy link
Member

@dominikl dominikl commented Oct 4, 2023

See title. Also added the cell studies type for idr0139 and idr0149, for some reason that was missed last time.

@dominikl
Copy link
Member Author

dominikl commented Oct 5, 2023

Something's a little bit strange. I ran the stats for idr0143 with the --disable-fsusage flag. But still this number of 29Tb showed up for disk space usage, but the number of files were missing. And when I run du on the filesystem I get 19Tb. That's quite a difference. Any idea @sbesson ?

Trying again:

(venv3) [dlindner@prod119-omeroreadwrite metadata]$ python idr-utils/scripts/stats.py --release prod119 --disable-fsusage idr0143-herbst-coculture
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
idr0143-herbst-coculture	screenA	prod119	3452	225	86400			112	259200	3208320	29.937475584	29937475584000			2160 x 2160 x 5 x 2 x 1	cell

And with du:
MIP plates

(venv3) [dlindner@prod119-omeroreadwrite 20221222-ftp]$ du --max-depth=1
...
2687536940

Raw plates

(venv3) [dlindner@prod119-omeroreadwrite 20220822-ftp]$ du --max-depth=1
...
15864049064

@sbesson
Copy link
Member

sbesson commented Oct 5, 2023

Pretty sure if you disable --fs-usage, the script attempt to guess the volume of data based on the pixel dimensions & type i.e. the number of bytes.
This obviously ignore several considerations including the presence of pyramidal levels, the underlying file format & the usage of compression.

@dominikl
Copy link
Member Author

dominikl commented Oct 5, 2023

So it would be better to use the size from OS du right?

@sbesson
Copy link
Member

sbesson commented Oct 5, 2023

So it would be better to use the size from OS du right?

Yes, there are different interpretations of what "size" is expected to capture but I think the file size captured by du is one that is easy to interpret. For people interested in downloading the raw data, this also an informative metrics.

@sbesson sbesson merged commit 9c8fb39 into IDR:master Oct 9, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants