General Cleanup and Following the Better Tracing Merger #976

MaxGamill-Sheffield · 2024-10-21T12:13:18Z

This PR:

Should prevent the logging warning about unclosed files popping up - this was due to the same logger trying to be setup multiple times.
Close Summary plots are broken #975 by re-enabling the stats plotting with the for contour length and average_end_to_end distance as these changed.
Actually removes skeletons < min_skel_size from being processed instead of just telling you and doing nothing about it, this prevents a Skan error being thrown.
Close [feature] : Reduce command line output for info logging level #929 by switching the logging level mostly to debug. Now there is only the following output:

config + loading:

(topo_test) Max:TopoStats Maxgamill$ topostats process -c minicircle_test/mc_config.yaml 
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] The YAML configuration file is valid.
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] The YAML plotting configuration file is valid.
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Configuration run options are consistent, processing can proceed.
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Configuration file loaded from      : minicircle_test/mc_config.yaml
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Scanning for images in              : tests/resources
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Output directory                    : minicircle_test
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Looking for images with extension   : .spm
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Images with extension .spm in tests/resources : 1
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Thresholding method (Filtering)     : std_dev
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Thresholding method (Grains)        : std_dev
[Sat, 19 Oct 2024 13:02:52] [INFO    ] [topostats] Extracting image from tests/resources/minicircle.spm

The running of the pipeline:

Processing images from tests/resources, results are under minicircle_test:   0%|                                                                               | 0/1 [00:00<?, ?it/s][Sat, 19 Oct 2024 13:02:56] [INFO    ] [topostats] Processing : minicircle
[Sat, 19 Oct 2024 13:02:56] [INFO    ] [topostats] [minicircle] : *** Filtering ***
[Sat, 19 Oct 2024 13:02:58] [INFO    ] [topostats] [minicircle] : Plotting Filtering Images
[Sat, 19 Oct 2024 13:02:59] [INFO    ] [topostats] [minicircle] : Filters stage completed successfully.
[Sat, 19 Oct 2024 13:02:59] [INFO    ] [topostats] [minicircle] : *** Grain Finding ***
[Sat, 19 Oct 2024 13:02:59] [INFO    ] [topostats] [minicircle] : Grains found for direction above : 21
[Sat, 19 Oct 2024 13:02:59] [INFO    ] [topostats] [minicircle] : Plotting Grain Finding Images
[Sat, 19 Oct 2024 13:02:59] [INFO    ] [topostats] [minicircle] : Grain Finding stage completed successfully.
[Sat, 19 Oct 2024 13:02:59] [INFO    ] [topostats] [minicircle] : *** Grain Statistics ***
[Sat, 19 Oct 2024 13:03:00] [INFO    ] [topostats] [minicircle] : Calculated grainstats for 21 grains.
[Sat, 19 Oct 2024 13:03:00] [INFO    ] [topostats] [minicircle] : Grainstats stage completed successfully.
[Sat, 19 Oct 2024 13:03:00] [INFO    ] [topostats] [minicircle] : *** Disordered Tracing ***
[Sat, 19 Oct 2024 13:03:00] [INFO    ] [topostats] [minicircle] : Calculating Disordered Tracing statistics for 21 grains...
[Sat, 19 Oct 2024 13:03:07] [INFO    ] [topostats] [minicircle] : Disordered Tracing stage completed successfully.
[Sat, 19 Oct 2024 13:03:07] [INFO    ] [topostats] [minicircle] : *** Nodestats ***
[Sat, 19 Oct 2024 13:03:07] [INFO    ] [topostats] [minicircle] : Calculating NodeStats statistics for 21 grains...
[Sat, 19 Oct 2024 13:03:07] [INFO    ] [topostats] [minicircle] : NodeStats stage completed successfully.
[Sat, 19 Oct 2024 13:03:07] [INFO    ] [topostats] [minicircle] : *** Ordered Tracing ***
[Sat, 19 Oct 2024 13:03:07] [INFO    ] [topostats] [minicircle] : Calculating Ordered Traces and statistics for 21 grains...
[Sat, 19 Oct 2024 13:03:09] [INFO    ] [topostats] [minicircle] : Ordered Tracing stage completed successfully.
[Sat, 19 Oct 2024 13:03:09] [INFO    ] [topostats] [minicircle] : *** Splining ***
[Sat, 19 Oct 2024 13:03:09] [INFO    ] [topostats] [minicircle] : Calculating Splining statistics for 25 molecules...
[Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] [minicircle] : Splining stage completed successfully.
[Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] [minicircle] : *** Image Statistics ***
[Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] [minicircle] : Saving image to .topostats file
Processing images from tests/resources, results are under minicircle_test: 100%|███████████████████████████████████████████████████████████████████████| 1/1 [00:17<00:00, 17.50s/it][Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] [minicircle] Processing completed.

and the output message:

[Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] Saving image stats to : minicircle_test/image_stats.csv.
[Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] Saving all height profiles to minicircle_test/height_profiles.json
[Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] The YAML summarisation config is valid.
[Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] [plotting] Default variable to labels mapping loaded.
[Sat, 19 Oct 2024 13:03:10] [INFO    ] [topostats] Summary plots and statistics will be saved to : minicircle_test/summary_distributions
[Sat, 19 Oct 2024 13:03:12] [INFO    ] [topostats] Folder-wise statistics saved to: minicircle_test/folder_grain_stats.csv
[Sat, 19 Oct 2024 13:03:12] [INFO    ] [topostats] Folder-wise statistics saved to: minicircle_test/folder_disordered_trace_stats.csv
[Sat, 19 Oct 2024 13:03:12] [INFO    ] [topostats] Folder-wise statistics saved to: minicircle_test/folder_mol_stats.csv
[Sat, 19 Oct 2024 13:03:12] [INFO    ] [topostats] 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ COMPLETE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  TopoStats Version           : 2.2.2.dev904+g68868e280.d20241018
  Base Directory              : tests/resources
  File Extension              : .spm
  Files Found                 : 1
  Successfully Processed^1    : 1 (100.0%)
  All statistics              : minicircle_test/all_statistics.csv
  Distribution Plots          : minicircle_test/summary_distributions

  Configuration               : minicircle_test/config.yaml

  Email                       : [email protected]
  Documentation               : https://afm-spm.github.io/topostats/
  Source Code                 : https://github.com/AFM-SPM/TopoStats/
  Bug Reports/Feature Request : https://github.com/AFM-SPM/TopoStats/issues/new/choose
  Citation File Format        : https://github.com/AFM-SPM/TopoStats/blob/main/CITATION.cff

  ^1 Successful processing of an image is detection of grains and calculation of at least
     grain statistics. If these have been disabled the percentage will be 0.

  If you encounter bugs/issues or have feature requests please report them at the above URL
  or email us.

  If you have found TopoStats useful please consider citing it. A Citation File Format is
  linked above and available from the Source Code page.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ns-rse · 2024-10-21T13:23:02Z

Neat, what does minicircle_test/mc_config.yaml look like now? I've tried this and still get verbose INFO, tried switching log_level: warning in a custom output but had very little output beyond the tqdm process bar.

…e, and addresses errors

MaxGamill-Sheffield · 2024-10-21T17:22:47Z

Neat, what does minicircle_test/mc_config.yaml look like now? I've tried this and still get verbose INFO, tried switching log_level: warning in a custom output but had very little output beyond the tqdm process bar.

I jumped the gun with the text because I had it all ready to go but wanted to make commits which shouldn't break the tests at each commit.

I found the source of the issue I mentioned this morning and now this should now be ready for review (maybe a few .topostats or .csv test files need to be added first but that should be ok as I've got them on my system).

…com/AFM-SPM/TopoStats into maxgamill-sheffield/general-tidying

…mill-sheffield/general-tidying

…e, and addresses errors

ns-rse

With the topostats/default_config.yaml I still get a lot of LOGGER.info() output...

❱ topostats process -c test_logging_config.yaml
2024-10-22 10:47:45.456635: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-10-22 10:47:45.459708: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-10-22 10:47:45.467540: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-10-22 10:47:45.482635: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-10-22 10:47:45.486386: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-22 10:47:45.497009: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-22 10:47:46.318670: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] The YAML configuration file is valid.
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] The YAML plotting configuration file is valid.
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Configuration run options are consistent, processing can proceed.
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[mask_cmap] : blue_purple_green > blue_purple_green 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 200 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 200 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 600 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Updated config config[savefig_dpi] : 300 > 100 
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Configuration file loaded from      : test_logging_config.yaml
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Scanning for images in              : /home/neil/work/git/hub/AFM-SPM/TopoStats
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Output directory                    : output
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Looking for images with extension   : .spm
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Images with extension .spm in /home/neil/work/git/hub/AFM-SPM/TopoStats : 1
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Thresholding method (Filtering)     : std_dev
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Thresholding method (Grains)        : std_dev
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Extracting image from /home/neil/work/git/hub/AFM-SPM/TopoStats/tests/resources/minicircle.spm
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Loading image from : /home/neil/work/git/hub/AFM-SPM/TopoStats/tests/resources/minicircle.spm
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : Loaded image from : /home/neil/work/git/hub/AFM-SPM/TopoStats/tests/resources/minicircle.spm
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : Extracted channel Height
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : Pixel to nm scaling : 0.4940029296875
Exception ignored in: <_io.FileIO name='/home/neil/work/git/hub/AFM-SPM/TopoStats/tests/resources/minicircle.spm' mode='rb' closefd=True>
Traceback (most recent call last):
  File "/home/neil/work/git/hub/AFM-SPM/TopoStats/topostats/io.py", line 1080, in get_data
    self.image, self.pixel_to_nm_scaling = suffix_to_loader[suffix]()
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: unclosed file <_io.BufferedReader name='/home/neil/work/git/hub/AFM-SPM/TopoStats/tests/resources/minicircle.spm'>
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] Image added to processing.
Processing images from /home/neil/work/git/hub/AFM-SPM/TopoStats, results are under output:   0%|                    | 0/1 [00:00<?, ?it/s][Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] Processing : minicircle
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] Image dimensions: (1024, 1024)
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : *** Filtering ***
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : Median flattening without mask
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : Plane tilt removal without mask
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : x-polyfit 1st order: [ 0.00319595 -1.71575752]
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : y-polyfit 1st order: [ 3.50284854e-19 -1.67027638e-16]
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : Removing x plane tilt
[Tue, 22 Oct 2024 10:47:47] [INFO    ] [topostats] [minicircle] : removing y plane tilt
[Tue, 22 Oct 2024 10:47:48] [INFO    ] [topostats] [minicircle] : Remove quadratic bow without mask
[Tue, 22 Oct 2024 10:47:48] [INFO    ] [topostats] [minicircle] : x polyfit 2nd order: [-7.84571521e-09  8.02616666e-06 -1.71712464e+00]
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : Nonlinear polynomial removal optimal params: const: -1.5584991680323195 xy: 2.8458269927179204e-07 x: 0.00011888637265488689 y: 9.815017834642102e-05
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : Skipping scar removal as requested from config
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : Zero averaging background : 2.1836820546239366e-12 nm
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : Masking (above) Threshold: 0.6681016965839295
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : Masking (below) Threshold: -6.681016965839294
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : Median flattening with mask
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : Plane tilt removal with mask
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : x-polyfit 1st order: [ 1.08937631e-05 -5.59554373e-03]
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : y-polyfit 1st order: [ 8.48377925e-21 -4.01419243e-18]
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : Removing x plane tilt
[Tue, 22 Oct 2024 10:47:49] [INFO    ] [topostats] [minicircle] : removing y plane tilt
[Tue, 22 Oct 2024 10:47:50] [INFO    ] [topostats] [minicircle] : Remove quadratic bow with mask
[Tue, 22 Oct 2024 10:47:50] [INFO    ] [topostats] [minicircle] : x polyfit 2nd order: [-6.42672798e-08  6.57454273e-05 -1.67941815e-02]
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Nonlinear polynomial removal optimal params: const: -0.0067495028844008174 xy: 3.859109115328997e-08 x: 1.9951742103531437e-05 y: 5.147466173401059e-06
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Skipping scar removal as requested from config
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Zero averaging background : 2.308883098657504e-09 nm
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Applying Gaussian filter (mode : nearest; Gaussian blur (px) : 1.0121397464510862).
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting Filtering Images
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : *** Grain Finding ***
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Thresholding method (grains) : std_dev
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Finding above grains, threshold: (0.7846683886788833)
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Masking (above) Threshold: 0.7846683886788833
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Tidying borders
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Removing noise (above)
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Area thresholding grains | Thresholds: L: 204.89,U: 1048576.00 px^2, L: 50.00, U: 255893.33 nm^2.
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Removing small / large grains (above)
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Area thresholding grains | Thresholds: L: 1229.31,U: 12293.12 px^2, L: 300.00, U: 3000.00 nm^2.
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Region properties calculated (above)
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Coloured regions
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Extracted bounding boxes (above)
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Grains found for direction above : 21
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting Grain Finding Images
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting above Grain Finding Images
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting mask_grains image
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting labelled_regions_01 image
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting tidied_border image
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting removed_noise image
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting removed_small_objects image
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting removed_objects_too_small_to_process image
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting labelled_regions_02 image
[Tue, 22 Oct 2024 10:47:51] [INFO    ] [topostats] [minicircle] : Plotting coloured_regions image
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : *** Grain Statistics ***
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Full Mask dimensions: (1024, 1024, 2)
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : DNA Mask dimensions: (1024, 1024)
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 0
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 1
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 2
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 3
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 4
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 5
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 6
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 7
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:52] [INFO    ] [topostats] [minicircle] : Processing grain: 8
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 9
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 10
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 11
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 12
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 13
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 14
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 15
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 16
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 17
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 18
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:53] [INFO    ] [topostats] [minicircle] : Processing grain: 19
[Tue, 22 Oct 2024 10:47:54] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:54] [INFO    ] [topostats] [minicircle] : Processing grain: 20
[Tue, 22 Oct 2024 10:47:54] [INFO    ] [topostats] [minicircle] : Height profiles extracted.
[Tue, 22 Oct 2024 10:47:54] [INFO    ] [topostats] [minicircle] : *** Disordered Tracing ***
[Tue, 22 Oct 2024 10:47:54] [INFO    ] [topostats] [minicircle] : Calculating Disordered Tracing statistics for 21 grains...
[Tue, 22 Oct 2024 10:48:04] [INFO    ] [topostats] [minicircle] : *** Nodestats ***
[Tue, 22 Oct 2024 10:48:04] [INFO    ] [topostats] [minicircle] : Calculating NodeStats statistics for 21 grains...
[Tue, 22 Oct 2024 10:48:05] [INFO    ] [topostats] [minicircle] : Finished Plotting NodeStats Images
[Tue, 22 Oct 2024 10:48:05] [INFO    ] [topostats] [minicircle] : *** Ordered Tracing ***
[Tue, 22 Oct 2024 10:48:05] [INFO    ] [topostats] [minicircle] : Calculating Ordered Traces and statistics for 21 grains...
[Tue, 22 Oct 2024 10:48:08] [INFO    ] [topostats] [minicircle] : Finished Plotting Ordered Tracing Images
[Tue, 22 Oct 2024 10:48:08] [INFO    ] [topostats] [minicircle] : *** Splining ***
[Tue, 22 Oct 2024 10:48:08] [INFO    ] [topostats] [minicircle] : Calculating Splining statistics for 25 molecules...
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [minicircle] : Finished Plotting Splining Images
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [minicircle] : *** Image Statistics ***
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [minicircle] : Saving image to .topostats file
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'topology_flip' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'topology_flip' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'confidence' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'confidence' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'confidence' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'confidence' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'confidence' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'confidence' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Item 'confidence' is None. Skipping.
[Tue, 22 Oct 2024 10:48:09] [WARNING ] [topostats] Cannot save key 'confidence' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
Processing images from /home/neil/work/git/hub/AFM-SPM/TopoStats, results are under output: 100%|████████████| 1/1 [00:21<00:00, 21.99s/it][Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [minicircle] Processing completed.
Processing images from /home/neil/work/git/hub/AFM-SPM/TopoStats, results are under output: 100%|████████████| 1/1 [00:21<00:00, 21.99s/it]
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] Saving image stats to : output/image_stats.csv.
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] Saving all height profiles to output/height_profiles.json
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] The YAML summarisation config is valid.
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [plotting] Default variable to labels mapping loaded.
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] Summary plots and statistics will be saved to : output/summary_distributions
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [plotting] Data has been melted to long format for plotting.
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [plotting] Seaborn color palette : bright
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [plotting] Setting x-axis range       : 6.157101309239989e-16 - 7.218670500488263e-16
[Tue, 22 Oct 2024 10:48:09] [INFO    ] [topostats] [plotting] Plotted area to : output/summary_distributions/area_hist_kde.png
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Plotted area to : output/summary_distributions/area_violin.png
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Data has been melted to long format for plotting.
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Seaborn color palette : bright
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Setting x-axis range       : 1.1374652874501621e-15 - 1.7702581409919492e-15
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Plotted area_cartesian_bbox to : output/summary_distributions/area_cartesian_bbox_hist_kde.png
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Plotted area_cartesian_bbox to : output/summary_distributions/area_cartesian_bbox_violin.png
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Data has been melted to long format for plotting.
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Seaborn color palette : bright
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Setting x-axis range       : 4.7604273871779595e-08 - 9.805659339106248e-08
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Plotted total_contour_length to : output/summary_distributions/total_contour_length_hist_kde.png
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Plotted total_contour_length to : output/summary_distributions/total_contour_length_violin.png
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Statistic is not in dataframe : avg_end_to_end_distance
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Data has been melted to long format for plotting.
[Tue, 22 Oct 2024 10:48:10] [INFO    ] [topostats] [plotting] Seaborn color palette : bright
[Tue, 22 Oct 2024 10:48:11] [INFO    ] [topostats] [plotting] Setting x-axis range       : 1.1899258473036381e-24 - 1.4439294258957144e-24
[Tue, 22 Oct 2024 10:48:11] [INFO    ] [topostats] [plotting] Plotted volume to : output/summary_distributions/volume_hist_kde.png
[Tue, 22 Oct 2024 10:48:11] [INFO    ] [topostats] [plotting] Plotted volume to : output/summary_distributions/volume_violin.png
[Tue, 22 Oct 2024 10:48:11] [INFO    ] [topostats] Folder-wise statistics saved to: output/tests/resources/folder_grain_stats.csv
[Tue, 22 Oct 2024 10:48:11] [INFO    ] [topostats] Folder-wise statistics saved to: output/tests/resources/folder_disordered_trace_stats.csv
[Tue, 22 Oct 2024 10:48:11] [INFO    ] [topostats] Folder-wise statistics saved to: output/tests/resources/folder_mol_stats.csv
[Tue, 22 Oct 2024 10:48:11] [INFO    ] [topostats] 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ COMPLETE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  TopoStats Version           : 2.2.2.dev906+g693e4008c
  Base Directory              : /home/neil/work/git/hub/AFM-SPM/TopoStats
  File Extension              : .spm
  Files Found                 : 1
  Successfully Processed^1    : 1 (100.0%)
  All statistics              : output/all_statistics.csv
  Distribution Plots          : output/summary_distributions

  Configuration               : output/config.yaml

  Email                       : [email protected]
  Documentation               : https://afm-spm.github.io/topostats/
  Source Code                 : https://github.com/AFM-SPM/TopoStats/
  Bug Reports/Feature Request : https://github.com/AFM-SPM/TopoStats/issues/new/choose
  Citation File Format        : https://github.com/AFM-SPM/TopoStats/blob/main/CITATION.cff

  ^1 Successful processing of an image is detection of grains and calculation of at least
     grain statistics. If these have been disabled the percentage will be 0.

  If you encounter bugs/issues or have feature requests please report them at the above URL
  or email us.

  If you have found TopoStats useful please consider citing it. A Citation File Format is
  linked above and available from the Source Code page.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Do we want to consider setting the default log level to WARNING so that only warnings are reported perhaps?

CUDA Warnings

Not everyone has nVidia grahpics cards (or dedicated graphics cards for that matter). My system does but I don't have the CUDA drivers installed so see this error. This might be annoying to users without and we should look to capture and suppress or replace this message with a simpler one.

`topostats.utils.update_config()`

Line 123 still has a LOGGER.info() which I think is responsible for the first big chunk about savefig_dpi being updated. In part this comes from appending the plotting_config.yaml to the config.yaml (as dictionaries).

Do we want to tell people when configuration has been updated or not? I would guess that people will know when they are using command line option flags to update parameters and these are all written to the resulting .yaml that accompanies output anyway.

I'd be inclined to suppress these.

`filters.py`

Still has a lot of LOGGER.info() messages in. Will users for example be interested or nead to know if Median flattening with[out] mask has been performed? They would notionally know if this has been modified from the default if they have changed it in the configuration file.

Ditto for many other actions such as Plane tilt removal with[out] mask.

Perhaps filters.py was missed in the find/replace LOGGER.info > LOGGER.debug | LOGGER.warning | LOGGER.error.

`grainstats.py`

Not sure we need to report that Processing of grain : # and definitely reporting that Height profiles extracted. seems excessive.

Plotting

Do we need to report the plotting of all figures and changes to how plots are made? Probably not I feel.

General Comment

I know I'm the one who started liberally sprinkling LOGGER.info() everywhere early on in the refactoring so I apologise for that.

Broadly I think longer term it would be useful, as I've mentioned previously, to switch to loguru throughout. One useful feature, probably not unique though, is the ability to easily format the log output to indicate which module messages are coming from (which is really useful when debugging). It also only requires a single handler to be added.

P.S. - Use of keyword Close won't be picked up because its nested within unbullteted lists.

topostats/processing.py

topostats/tracing/disordered_tracing.py

topostats/tracing/ordered_tracing.py

MaxGamill-Sheffield · 2024-10-22T14:40:04Z

With the topostats/default_config.yaml I still get a lot of LOGGER.info() output...

This is very unusual as I only see the following when running a single file:

(topo_test) Max:TopoStats Maxgamill$ topostats process -c workshop/ws_config.yaml
[Tue, 22 Oct 2024 15:24:40] [INFO    ] [topostats] The YAML configuration file is valid.
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] The YAML plotting configuration file is valid.
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Configuration run options are consistent, processing can proceed.
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Configuration file loaded from      : workshop/ws_config.yaml
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Scanning for images in              : workshop
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Output directory                    : workshop
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Looking for images with extension   : .spm
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Images with extension .spm in workshop : 1
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Thresholding method (Filtering)     : std_dev
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Thresholding method (Grains)        : std_dev
[Tue, 22 Oct 2024 15:24:41] [INFO    ] [topostats] Extracting image from workshop/20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022.spm
Processing images from workshop, results are under workshop:   0%|                                                                                             | 0/1 [00:00<?, ?it/s][Tue, 22 Oct 2024 15:24:44] [INFO    ] [topostats] Processing : 20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022
[Tue, 22 Oct 2024 15:24:44] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : *** Filtering ***
[Tue, 22 Oct 2024 15:24:44] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Plotting Filtering Images
[Tue, 22 Oct 2024 15:24:45] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Filters stage completed successfully.
[Tue, 22 Oct 2024 15:24:45] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : *** Grain Finding ***
[Tue, 22 Oct 2024 15:24:45] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Grains found for direction above : 2
[Tue, 22 Oct 2024 15:24:45] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Plotting Grain Finding Images
[Tue, 22 Oct 2024 15:24:46] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Grain Finding stage completed successfully.
[Tue, 22 Oct 2024 15:24:46] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : *** Grain Statistics ***
[Tue, 22 Oct 2024 15:24:46] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Calculated grainstats for 2 grains.
[Tue, 22 Oct 2024 15:24:46] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Grainstats stage completed successfully.
[Tue, 22 Oct 2024 15:24:46] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : *** Disordered Tracing ***
[Tue, 22 Oct 2024 15:24:46] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Calculating Disordered Tracing statistics for 2 grains...
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Disordered Tracing stage completed successfully.
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : *** Nodestats ***
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Calculating NodeStats statistics for 2 grains...
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : NodeStats stage completed successfully.
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : *** Ordered Tracing ***
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Calculating Ordered Traces and statistics for 2 grains...
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Ordered Tracing stage completed successfully.
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : *** Splining ***
[Tue, 22 Oct 2024 15:24:48] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Calculating Splining statistics for 2 molecules...
[Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Splining stage completed successfully.
[Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : *** Image Statistics ***
[Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] : Saving image to .topostats file
Processing images from workshop, results are under workshop: 100%|█████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:08<00:00,  8.04s/it][Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] [20230823_TAF_IR_minicircle_1.5ng_CaptureFile.0_00022] Processing completed.
Processing images from workshop, results are under workshop: 100%|█████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:08<00:00,  8.04s/it]
[Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] Saving image stats to : workshop/image_stats.csv.
[Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] Saving all height profiles to workshop/height_profiles.json
[Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] The YAML summarisation config is valid.
[Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] [plotting] Default variable to labels mapping loaded.
[Tue, 22 Oct 2024 15:24:49] [INFO    ] [topostats] Summary plots and statistics will be saved to : workshop/summary_distributions
[Tue, 22 Oct 2024 15:24:50] [INFO    ] [topostats] Folder-wise statistics saved to: workshop/folder_grain_stats.csv
[Tue, 22 Oct 2024 15:24:50] [INFO    ] [topostats] Folder-wise statistics saved to: workshop/folder_disordered_trace_stats.csv
[Tue, 22 Oct 2024 15:24:51] [INFO    ] [topostats] Folder-wise statistics saved to: workshop/folder_mol_stats.csv
[Tue, 22 Oct 2024 15:24:51] [INFO    ] [topostats] 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ COMPLETE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  TopoStats Version           : 2.2.2.dev923+gd3444cdc8
  Base Directory              : workshop
  File Extension              : .spm
  Files Found                 : 1
  Successfully Processed^1    : 1 (100.0%)
  All statistics              : workshop/all_statistics.csv
  Distribution Plots          : workshop/summary_distributions

  Configuration               : workshop/config.yaml

  Email                       : [email protected]
  Documentation               : https://afm-spm.github.io/topostats/
  Source Code                 : https://github.com/AFM-SPM/TopoStats/
  Bug Reports/Feature Request : https://github.com/AFM-SPM/TopoStats/issues/new/choose
  Citation File Format        : https://github.com/AFM-SPM/TopoStats/blob/main/CITATION.cff

  ^1 Successful processing of an image is detection of grains and calculation of at least
     grain statistics. If these have been disabled the percentage will be 0.

  If you encounter bugs/issues or have feature requests please report them at the above URL
  or email us.

  If you have found TopoStats useful please consider citing it. A Citation File Format is
  linked above and available from the Source Code page.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


(topo_test) Max:TopoStats Maxgamill$

Do we want to consider setting the default log level to WARNING so that only warnings are reported perhaps?

I'd want to avoid this so that the users know that TopoStats IS working and not just stalling. I believe that seeing some progress is better than no progress. Are you able to retry this branch and double check the output for you is not the same as I see?

CUDA Warnings

Not everyone has nVidia grahpics cards (or dedicated graphics cards for that matter). My system does but I don't have the CUDA drivers installed so see this error. This might be annoying to users without and we should look to capture and suppress or replace this message with a simpler one.

No idea what this is and haven't seen it before. I looks like it might be from a Tensorflow dependancy? I could take a look into it but I don't think its from anything that has been added in this PR or the better tracing ones - this might be from the unets?

We could disable all warnings with the following but is this wise?

import warnings
warnings.filterwarnings("ignore")

topostats.utils.update_config()

Line 123 still has a LOGGER.info() which I think is responsible for the first big chunk about savefig_dpi being updated. In part this comes from appending the plotting_config.yaml to the config.yaml (as dictionaries).
Do we want to tell people when configuration has been updated or not? I would guess that people will know when they are using command line option flags to update parameters and these are all written to the resulting .yaml that accompanies output anyway.

filters.py

Still has a lot of LOGGER.info() messages in. Will users for example be interested or nead to know if Median flattening with[out] mask has been performed? They would notionally know if this has been modified from the default if they have changed it in the configuration file.

Ditto for many other actions such as Plane tilt removal with[out] mask.

Perhaps filters.py was missed in the find/replace LOGGER.info > LOGGER.debug | LOGGER.warning | LOGGER.error.

grainstats.py

Not sure we need to report that Processing of grain : # and definitely reporting that Height profiles extracted. seems excessive.

Plotting

Do we need to report the plotting of all figures and changes to how plots are made? Probably not I feel.

General Comment

I know I'm the one who started liberally sprinkling LOGGER.info() everywhere early on in the refactoring so I apologise for that.

Please could you re-run these? I've checked the utils.py (at the least) and line 123 (along with all the other logging and caplog updates was updated the other day and so should have the affect of all of the above.

Broadly I think longer term it would be useful, as I've mentioned previously, to switch to loguru throughout. One useful feature, probably not unique though, is the ability to easily format the log output to indicate which module messages are coming from (which is really useful when debugging). It also only requires a single handler to be added.

Aye, Sylvia mentioned this but I wanted to provide something quick to tidy up the terminal outputs and errors and I didn't know how loguru would implement its own handlers or be linked to capslog so I thought I should leave it for now :)

P.S. - Use of keyword Close won't be picked up because its nested within unbullteted lists.

Ah nightmare, but cool, I'll tag the issues on this

ns-rse · 2024-10-23T12:24:48Z

No idea what this is and haven't seen it before. I looks like it might be from a Tensorflow dependancy? I could take a look into it but I don't think its from anything that has been added in this PR or the better tracing ones - this might be from the unets?

Yes, this is due to Tensorflow.

We could disable all warnings with the following but is this wise?

But then users wouldn't see TopoStats warnings so I'm not sure that is what is required.

A quick search led me to this solution

Putting the following in __init__.py would suppress the warnings

import os

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

Incoming PR (with small "bonus").

Are you able to retry this branch and double check the output for you is not the same as I see?

Seems I hadn't git pull'd the branch, my apologies. Much quieter output now, once the TensorFlow message supression is merged this is good to go.

`loguru`

I wasn't proposing overhauling things with regards to loguru here and have made some simple forays into switching on Monday but its a bit more involved so I've shelved it in favour of focusing on #517 as a priority.

I understand the need to always do things "quickly" but sometimes its a false economy spending time on a quick fix and then (maybe) going back and doing the larger piece of work that should have been done in the first place (this is in essence what accumulates as "technical debt"). We don't always get things right first time, and I know I'm continually learning better ways of doing things. Taking ownership of work and doing it to the best of our ability helps minimise the accumulation of technical debt.

MaxGamill-Sheffield · 2024-10-23T13:04:44Z

Putting the following in __init__.py would suppress the warnings
import os

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

Cool I've added this now! @llwiggins have you tested this yet and is it good to go?

Prints "TopoStast" using the [art]() package and the "twisted" font which I felt was appropriate for DNA!

ns-rse · 2024-10-23T14:20:46Z

It was in the PR I created for this branch, I've just fixed the failing test (I tried to keep output clean by switching from LOGGER.info() to print in the completion message but the tests look for lines in log so I've reverted it).

feature: Print 'art' above completion message

I should have added all the changes and I'm not sure why this is still "requested changes"

ns-rse · 2024-10-24T09:49:36Z

There is now duplicate lines for disabling Tensor Flow.

I will remove one.

MaxGamill-Sheffield added 2 commits October 18, 2024 14:39

get not setup logger in non-init.py files

494471c

renames summary plotting variables to match new output names

693e400

MaxGamill-Sheffield added 3 commits October 21, 2024 17:57

renames summary plotting variables to match new output names

3cf54d9

fixes skeleton removal below min_skel_size, and small simplified trac…

60031ac

…e, and addresses errors

minimalises terminal output and adjusts caplog output tests

5593704

MaxGamill-Sheffield marked this pull request as ready for review October 21, 2024 17:19

MaxGamill-Sheffield requested review from ns-rse, SylviaWhittle and llwiggins October 21, 2024 17:19

MaxGamill-Sheffield and others added 10 commits October 21, 2024 18:38

Merge branch 'maxgamill-sheffield/general-tidying' of https://github.…

dc8a23f

…com/AFM-SPM/TopoStats into maxgamill-sheffield/general-tidying

Merge branch 'maxgamill-sheffield/969-concat-issue-hotfix' into maxga…

acb2709

…mill-sheffield/general-tidying

get not setup logger in non-init.py files

6ac530e

renames summary plotting variables to match new output names

1dd564f

fixes skeleton removal below min_skel_size, and small simplified trac…

79b01e0

…e, and addresses errors

minimalises terminal output and adjusts caplog output tests

5e91435

renames summary plotting variables to match new output names

a208747

merge conflicts?

1b7a583

[pre-commit.ci] Fixing issues with pre-commit

3496939

removed unwanted files

7d4bfa3

ns-rse previously requested changes Oct 22, 2024

View reviewed changes

adjusted typehints + descriptions around the df args

d3444cd

This was linked to issues Oct 22, 2024

Summary plots are broken #975

Closed

[feature] : Reduce command line output for info logging level #929

Closed

remove superflous '+' from logging statement

a37ac7f

MaxGamill-Sheffield requested a review from ns-rse October 23, 2024 05:10

prevents non-OSX / GPU machines throwing a TF error

fea60f2

Base automatically changed from maxgamill-sheffield/969-concat-issue-hotfix to main October 23, 2024 14:11

ns-rse added 2 commits October 23, 2024 15:18

feature: Print 'art' above completion message

0dd9998

Prints "TopoStast" using the [art]() package and the "twisted" font which I felt was appropriate for DNA!

chore: Suppress Tensor Flow warnings

765ab8d

llwiggins enabled auto-merge October 23, 2024 14:40

ns-rse and others added 2 commits October 23, 2024 15:46

Merge branch 'main' into maxgamill-sheffield/general-tidying

83048c9

Merge pull request #980 from AFM-SPM/ns-rse/tweak-final-message

4413177

feature: Print 'art' above completion message

llwiggins approved these changes Oct 24, 2024

View reviewed changes

llwiggins added this pull request to the merge queue Oct 24, 2024

Merged via the queue into main with commit 5872958 Oct 24, 2024
11 checks passed

llwiggins deleted the maxgamill-sheffield/general-tidying branch October 24, 2024 06:12

ns-rse mentioned this pull request Oct 24, 2024

Improve documentation and perhaps warn users about GPU requirements #944

Open

MaxGamill-Sheffield mentioned this pull request Nov 13, 2024

ResourceWarning: unclosed file <_io.TextIOWrapper name='./TopoStats-2024-07-04-18-25-53.log' mode='a' encoding='UTF-8'> #870

Closed

ns-rse mentioned this pull request Nov 19, 2024

Remove broad Exception catch surrounding dnatracing #699

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

General Cleanup and Following the Better Tracing Merger #976

General Cleanup and Following the Better Tracing Merger #976

MaxGamill-Sheffield commented Oct 21, 2024 •

edited

Loading

ns-rse commented Oct 21, 2024

MaxGamill-Sheffield commented Oct 21, 2024

ns-rse left a comment •

edited

Loading

MaxGamill-Sheffield commented Oct 22, 2024

CUDA Warnings

`topostats.utils.update_config()`

`filters.py`

`grainstats.py`

Plotting

General Comment

ns-rse commented Oct 23, 2024

MaxGamill-Sheffield commented Oct 23, 2024

ns-rse commented Oct 23, 2024

ns-rse commented Oct 24, 2024

General Cleanup and Following the Better Tracing Merger #976

General Cleanup and Following the Better Tracing Merger #976

Conversation

MaxGamill-Sheffield commented Oct 21, 2024 • edited Loading

ns-rse commented Oct 21, 2024

MaxGamill-Sheffield commented Oct 21, 2024

ns-rse left a comment • edited Loading

Choose a reason for hiding this comment

CUDA Warnings

topostats.utils.update_config()

filters.py

grainstats.py

Plotting

General Comment

MaxGamill-Sheffield commented Oct 22, 2024

CUDA Warnings

topostats.utils.update_config()

filters.py

grainstats.py

Plotting

General Comment

ns-rse commented Oct 23, 2024

loguru

MaxGamill-Sheffield commented Oct 23, 2024

ns-rse commented Oct 23, 2024

ns-rse commented Oct 24, 2024

MaxGamill-Sheffield commented Oct 21, 2024 •

edited

Loading

ns-rse left a comment •

edited

Loading

`topostats.utils.update_config()`

`filters.py`

`grainstats.py`

`topostats.utils.update_config()`

`filters.py`

`grainstats.py`

`loguru`