Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Zernike calculation for larger images without tons of memory #106

Open
bethac07 opened this issue Jan 12, 2022 · 0 comments
Open

Comments

@bethac07
Copy link
Member

Is your feature request related to a problem? Please describe.
Centrosome's Zernike calculation ends up allocating several complex arrays of image size x 30 (the number of polynomials we calculate), which means for some not-too-huge images (5120^2), we can end up running out of memory to run Zernike calculation even on a 64GB-memory machine. (See example stack trace below).

I'd need to do a deep dive on that code to see if we can in any way sparsify the array, if we NEED the complex data type, if we can maybe calculate each of the 30 polynomials independently, and/or calculate each object in only its neighborhood independently; I imagine at least one if not many of these might be doable. My guess as to why it wasn't already is that it would probably cause a net slowdown on smaller images, but for large image support it might be worth it (and/or we could consider a code path to split based on image size).

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/cellprofiler_core/pipeline/_pipeline.py", line 976, in run_with_yield
    self.run_module(module, workspace)
  File "/usr/local/lib/python3.8/dist-packages/cellprofiler_core/pipeline/_pipeline.py", line 1298, in run_module
    module.run(workspace)
  File "/usr/local/src/CellProfiler/cellprofiler/modules/measureobjectsizeshape.py", line 524, in run
    self.run_on_objects(object_name, workspace)
  File "/usr/local/src/CellProfiler/cellprofiler/modules/measureobjectsizeshape.py", line 586, in run_on_objects
    features_to_record = self.analyze_objects(objects, desired_properties)
  File "/usr/local/src/CellProfiler/cellprofiler/modules/measureobjectsizeshape.py", line 652, in analyze_objects
    zf_l = centrosome.zernike.zernike(
  File "/usr/local/lib/python3.8/dist-packages/centrosome/zernike.py", line 195, in zernike
    zf = construct_zernike_polynomials(x, y, zernike_indexes, mask)
  File "/usr/local/lib/python3.8/dist-packages/centrosome/zernike.py", line 101, in construct_zernike_polynomials
    result = np.zeros(mask.shape + (nzernikes,), np.complex)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 11.7 GiB for an array with shape (5120, 5120, 30) and data type complex128
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant