Find centre of rotation (tentative 2) (#30)

* Squashed commit of the following:

commit 16734d0
Author: lauraporta <[email protected]>
Date:   Fri Oct 25 13:59:25 2024 +0100

    Add docstrings and description on the very long test integration module

commit 89f8a77
Author: lauraporta <[email protected]>
Date:   Fri Oct 25 11:55:19 2024 +0100

    Small refactor conftests

commit eb60b72
Author: lauraporta <[email protected]>
Date:   Thu Oct 24 18:36:25 2024 +0100

    Move out assertions handling in a separate function, handle paths differently and reorganize part of the fixtures

commit 758a498
Author: lauraporta <[email protected]>
Date:   Thu Oct 24 17:31:21 2024 +0100

    Add spaces to improve table formatting

commit 141daa4
Author: lauraporta <[email protected]>
Date:   Thu Oct 24 17:23:26 2024 +0100

    Add readme to describe the way regression tests work

commit 02fbadd
Author: lauraporta <[email protected]>
Date:   Thu Oct 24 17:02:45 2024 +0100

    Add more comments, docstrings and typing in regression tests files to regenerate target images

commit 5a83168
Author: lauraporta <[email protected]>
Date:   Thu Oct 24 16:07:13 2024 +0100

    Remove repeated functions

commit fc2d9a7
Author: lauraporta <[email protected]>
Date:   Thu Oct 24 15:34:23 2024 +0100

    Move in separate files the logic to recreate the images for regression tests

commit 5137eb2
Author: lauraporta <[email protected]>
Date:   Thu Oct 17 21:21:07 2024 +0100

    Apply suggestions from PR review

commit cf3998d
Author: lauraporta <[email protected]>
Date:   Mon Oct 14 14:16:54 2024 +0100

    Simplify filter

commit 3f631bf
Author: lauraporta <[email protected]>
Date:   Fri Oct 4 11:32:45 2024 +0100

    Move the plots of the integration script into the integration test

commit aa37d4e
Author: lauraporta <[email protected]>
Date:   Fri Oct 4 11:16:48 2024 +0100

    Fix mypy error messages

commit ff55b72
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 18:30:46 2024 +0100

    WIP docs

commit 0737e65
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 16:50:59 2024 +0100

    Update workflows and add dependabot yml

commit 81a72c8
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 16:35:39 2024 +0100

    Increase the tolerance and improve error message

commit e45ff2c
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 16:16:34 2024 +0100

    Let's accept little differences in the derotation

commit a941f41
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 16:00:01 2024 +0100

    Also do't need the folder

commit 1f8ded8
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 15:57:42 2024 +0100

    Don't need to save plots in tests

commit ef186f9
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 15:52:45 2024 +0100

    Add integration tests on the integration of the two pipelines to find the center of rotation

commit 10ec8e8
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 15:16:13 2024 +0100

    Move logic to get the largest blob in a separate method

commit a211832
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 14:33:17 2024 +0100

    Move the plots out of the main function in integration script

commit 7a00c37
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 14:13:09 2024 +0100

    Fix pytest error (missing required property)

commit d69ec18
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 14:12:43 2024 +0100

    Uncomment part of plotting hooks

commit dddbaf3
Author: lauraporta <[email protected]>
Date:   Thu Oct 3 14:12:30 2024 +0100

    Reorganize examples

commit 48c8b54
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 17:10:45 2024 +0100

    Add another debugging plot

commit 3d5c1ce
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 17:10:14 2024 +0100

    Use the script to test with a different center of rotation, works when checking for correct order of x-y coordinates

commit 52be41c
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 15:51:07 2024 +0100

    Use blank pixel

commit 8963f5f
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 15:43:58 2024 +0100

    Comment one hook

commit a4dc480
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 15:37:18 2024 +0100

    WIP: script to test the integration of the two pipelines to find the center of rotation and then derotate

commit d783005
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 15:35:33 2024 +0100

    WIP: finding the center of rotation via ellipse fitting

commit fde0592
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 15:31:33 2024 +0100

    Fix angle indices bug in Rotator

commit ca6cc03
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 15:28:47 2024 +0100

    Plotting bug fixed

commit 0dd3c6d
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 15:28:06 2024 +0100

    Change variable names in plotting hooks

commit bfa4443
Author: lauraporta <[email protected]>
Date:   Mon Sep 30 15:25:35 2024 +0100

    Remove hooks from pipeline

commit 2779819
Author: lauraporta <[email protected]>
Date:   Fri Sep 27 14:37:27 2024 +0100

    Clean up and refactor the main pipeline around the center of rotation feature - also make separate module for hooks

commit edee980
Author: lauraporta <[email protected]>
Date:   Fri Sep 27 14:10:04 2024 +0100

    Add wrongly deleted hook

commit c4f9d14
Author: lauraporta <[email protected]>
Date:   Fri Sep 27 14:09:21 2024 +0100

    Add test for derotation with different centre

commit 0721e5d
Author: lauraporta <[email protected]>
Date:   Fri Sep 27 13:53:58 2024 +0100

    Add new generated derotated squares: faster derotation by line leads to worse performance...

commit df141be
Author: lauraporta <[email protected]>
Date:   Fri Sep 27 13:19:23 2024 +0100

    Update tests with new derotation version

commit 765c4de
Author: lauraporta <[email protected]>
Date:   Fri Sep 27 13:14:36 2024 +0100

    Update tests rotator with different center

commit c0423a0
Author: lauraporta <[email protected]>
Date:   Fri Sep 27 11:49:39 2024 +0100

    Remove linear interpolation and recompute test images - now it's a square of different gray lines

commit 1412348
Author: lauraporta <[email protected]>
Date:   Thu Sep 26 16:21:46 2024 +0100

    Resolve naming (in many cases with "rotation" I meant derotation)

commit 860a2fa
Author: lauraporta <[email protected]>
Date:   Thu Sep 26 16:08:21 2024 +0100

    WIP: changing the main derotation by line algorithm

commit e5cac06
Author: lauraporta <[email protected]>
Date:   Thu Sep 26 15:50:05 2024 +0100

    Update basic rotator to include different center of rotation

commit 35a0b7f
Merge: a0c6017 c825a82
Author: lauraporta <[email protected]>
Date:   Thu Sep 26 15:06:45 2024 +0100

    Merge commit 'c825a828116dd382d8e9f8e1695c46adcdb94ecc' into feature/finding-centre-of-rotation

commit c825a82
Author: lauraporta <[email protected]>
Date:   Thu Sep 26 14:59:10 2024 +0100

    Delete duplicate file

commit c3cf60e
Author: lauraporta <[email protected]>
Date:   Thu Sep 26 14:42:53 2024 +0100

    Add tests

commit 55b080e
Author: lauraporta <[email protected]>
Date:   Thu Sep 26 14:41:05 2024 +0100

    Fix Rotator and join it with the derotator in an example

commit 4832672
Author: lauraporta <[email protected]>
Date:   Thu Sep 26 14:39:37 2024 +0100

    Make function name consistent

commit c913a06
Author: lauraporta <[email protected]>
Date:   Wed Sep 25 17:40:25 2024 +0100

    Add basic tests

commit dfe659a
Author: lauraporta <[email protected]>
Date:   Wed Sep 25 17:15:19 2024 +0100

    Add docstrings

commit 29e9d27
Author: lauraporta <[email protected]>
Date:   Wed Sep 25 17:11:07 2024 +0100

    Add basic rotator and a simple usage of it

commit a0c6017
Author: lauraporta <[email protected]>
Date:   Wed Sep 25 16:28:37 2024 +0100

    WIP 🏗️: draft pipeline for adapting center of rotation

commit 6000609
Author: lauraporta <[email protected]>
Date:   Wed Sep 25 16:24:55 2024 +0100

    Improve debug plots

commit 9bf17eb
Author: lauraporta <[email protected]>
Date:   Wed Sep 25 16:24:26 2024 +0100

    Add plotting hooks and further params to main derotate by line function

commit aae7c67
Author: lauraporta <[email protected]>
Date:   Mon Sep 23 16:29:24 2024 +0100

    Fixes debugging plots and logging usage

commit 174fd9e
Author: lauraporta <[email protected]>
Date:   Thu Sep 19 16:48:00 2024 +0100

    Include usage of center of rotation in main pipeline

commit f6adedf
Author: lauraporta <[email protected]>
Date:   Thu Sep 19 16:35:41 2024 +0100

    Add working methods to find center of rotation 🎉

commit b76a30a
Author: lauraporta <[email protected]>
Date:   Mon Sep 16 11:36:50 2024 +0100

    Reorganize methods order

commit 168100a
Author: lauraporta <[email protected]>
Date:   Mon Sep 16 11:26:07 2024 +0100

    Add empty method

* Update manifest

* Recalculate images

* Fix tuple bug

* Fix typo

* Add explanation on image offset

* Simplify handling of center

* Expand description of one of the plotting hooks

* Delete unused line

* Typo

* Create debug folder by default

* Remove hardcoded value

* Fix typo

* Add rtol usage

* Remove unuseful path

* Be more explicit about how to recreate certain images

* Improve path handling

* Rename image

* Remove images that shouldn't be here! 🙅🏻‍♀️

* Change imports

* Remove some magic numbers

* Unique function for center descriptors

MANIFEST.in

@@ -3,11 +3,10 @@ include README.md
 include *.yaml
 recursive-include * *.yaml
 recursive-include * *.yml
-recursive-include tests *.py
 recursive-include examples *.py
-recursive-include tests *.png
 recursive-include images *.png
 
 recursive-exclude * __pycache__
 recursive-exclude * *.py[co]
 recursive-exclude docs *
+recursive-exclude tests *

derotation/analysis/full_derotation_pipeline.py

@@ -168,14 +168,21 @@ def load_data(self):
         self.debugging_plots = self.config["debugging_plots"]
 
         if self.debugging_plots:
-            self.debug_plots_folder = self.config["paths_write"][
-                "debug_plots_folder"
-            ]
-            Path(self.debug_plots_folder).mkdir(parents=True, exist_ok=True)
+            self.debug_plots_folder = Path(
+                self.config["paths_write"]["debug_plots_folder"]
+            )
+            self.debug_plots_folder.mkdir(parents=True, exist_ok=True)
 
         logging.info(f"Dataset {self.filename_raw} loaded")
         logging.info(f"Filename: {self.filename}")
 
+        #  by default the center of rotation is the center of the image
+        self.center_of_rotation = (
+            self.num_lines_per_frame // 2,
+            self.num_lines_per_frame // 2,
+        )
+        self.hooks = {}
+
     ### ----------------- Analog signals processing pipeline ------------- ###
     def process_analog_signals(self):
         """From the analog signals (frame clock, line clock, full rotation,
@@ -790,26 +797,39 @@ def plot_rotation_angles(self):
         plt.savefig(self.debug_plots_folder / "rotation_angles.png")
 
     ### ----------------- Derotation ----------------- ###
+
+    def plot_max_projection_with_center(self):
+        """Plots the maximum projection of the image stack with the center
+        of rotation.
+        This plot will be saved in the debug_plots folder.
+        Please inspect it to check that the center of rotation is correctly
+        placed.
+        """
+        logging.info("Plotting max projection with center...")
+
+        max_projection = np.max(self.image_stack, axis=0)
+
+        fig, ax = plt.subplots(1, 1, figsize=(5, 5))
+
+        ax.imshow(max_projection, cmap="gray")
+        ax.scatter(
+            self.center_of_rotation[0],
+            self.center_of_rotation[1],
+            color="red",
+            marker="x",
+        )
+
+        ax.spines["top"].set_visible(False)
+        ax.spines["right"].set_visible(False)
+
+        ax.axis("off")
+
+        plt.savefig(self.debug_plots_folder / "max_projection_with_center.png")
+
     def derotate_frames_line_by_line(self) -> np.ndarray:
-        """Rotates the image stack line by line, using the rotation angles
-        by line calculated from the analog signals.
-
-        Description of the algorithm:
-        - takes one line from the image stack
-        - creates a new image with only that line
-        - rotates the line by the given angle
-        - substitutes the line in the new image
-        - adds the new image to the rotated image stack
-
-        Edge cases and how they are handled:
-        - the rotation starts in the middle of the image -> the previous lines
-        are copied from the first frame
-        - the rotation ends in the middle of the image -> the remaining lines
-        are copied from the last frame
-
-        Before derotation, it finds the image offset, which is the peak of
-        the gaussian mixture model fitted to the image histogram. It is
-        useful to fill in the blank pixels that appear during the derotation.
+        """Wrapper for the function `derotate_an_image_array_line_by_line`.
+        Before calling the function, it finds the F0 image offset with
+        `find_image_offset`.
 
         Returns
         -------
@@ -818,12 +838,22 @@ def derotate_frames_line_by_line(self) -> np.ndarray:
         """
         logging.info("Starting derotation by line...")
 
+        if self.debugging_plots:
+            self.plot_max_projection_with_center()
+
         offset = self.find_image_offset(self.image_stack[0])
 
         rotated_image_stack = derotate_an_image_array_line_by_line(
             self.image_stack,
             self.rot_deg_line,
             blank_pixels_value=offset,
+            center=self.center_of_rotation,
+            plotting_hook_line_addition=self.hooks.get(
+                "plotting_hook_line_addition"
+            ),
+            plotting_hook_image_completed=self.hooks.get(
+                "plotting_hook_image_completed"
+            ),
         )
 
         logging.info("✨ Image stack rotated ✨")
@@ -833,6 +863,19 @@ def derotate_frames_line_by_line(self) -> np.ndarray:
     def find_image_offset(img):
         """Find the "F0", also called "image offset" for a given image.
 
+        Explanations
+        ------------
+        What is the image offset?
+        The PMT (photo-multiplier tube) adds an arbitrary offset to the
+        image that corresponds to 0 photons received. We can use a Gaussian
+        Mixture Model to find this offset by assuming that it will be the
+        smallest mean of the Gaussian components.
+
+        Why do we need to find it?
+        When we rotate the image, the pixels of the image that are not
+        sampled will be filled with the offset is order to correctly express
+        "0 photons received".
+
         Parameters
         ----------
         img : np.ndarray