[Bug]: "Curvature" image not generating per file in the processed folder

[tcatley]

Checklist

• Find the offending file in the output. If processing halts, re-run analysis with topostats --core 1 process.
• Describe the bug.
• Include the configuration file.
• Copy of the output.
• The exact command that failed. This is what you typed at the command line, including any options.
• TopoStats version, this is reported by topostats --version
• Operating System and Python Version

Describe the bug

When running an updated version of main (2.2.2.dev1282+gdf4cb02e7), the processed output only produces a single curvature image in the folder, rather than one for every .spm file processed.

Copy of the output

Include the configuration file

# Configuration from TopoStats run completed : 2024-12-11 15:57:48
# For more information on configuration and how to use it:
# https://afm-spm.github.io/TopoStats/main/configuration.html
base_dir: /Volumes/shared/pyne_group/Shared/AFM_Data/Top2_HSF1/test
cores: 6
curvature:
  colourmap_normalisation_bounds: [-0.5, 0.5]
  run: true
disordered_tracing:
  mask_smoothing_params:
    dilation_iterations: 2
    gaussian_sigma: 2
    holearea_min_max: [0, null]
  min_skeleton_size: 10
  pad_width: 1
  pruning_params: {height_threshold: null, max_length: 10.0, method: topostats, method_outlier: mean_abs,
    method_values: mid}
  run: true
  skeletonisation_params: {height_bias: 0.5, method: topostats}
file_ext: .spm
filter:
  gaussian_mode: nearest
  gaussian_size: 1.0121397464510862
  otsu_threshold_multiplier: 1.0
  remove_scars: {max_scar_width: 4, min_scar_length: 16, removal_iterations: 2, run: false,
    threshold_high: 0.666, threshold_low: 0.25}
  row_alignment_quantile: 0.5
  run: true
  threshold_absolute: {above: 0.6, below: -1.0}
  threshold_method: std_dev
  threshold_std_dev: {above: 1.0, below: 10.0}
grains:
  absolute_area_threshold:
    above: [5000, 20000]
    below: [null, null]
  direction: above
  otsu_threshold_multiplier: 1.0
  remove_edge_intersecting_grains: true
  run: true
  smallest_grain_size_nm2: 50
  threshold_absolute: {above: 0.6, below: -1.0}
  threshold_method: absolute
  threshold_std_dev: {above: 1.0, below: 10.0}
  unet_config: {grain_crop_padding: 2, lower_norm_bound: -1.0, model_path: null, upper_norm_bound: 5.0}
  vetting: {class_connection_point_thresholds: null, class_conversion_size_thresholds: null,
    class_region_number_thresholds: null, class_size_thresholds: null, class_touching_threshold: 5,
    keep_largest_labelled_regions_classes: null, nearby_conversion_classes_to_convert: null}
grainstats: {cropped_size: -1, edge_detection_method: binary_erosion, extract_height_profile: true,
  run: true}
loading: {channel: Height}
log_level: info
nodestats: {branch_pairing_length: 20.0, node_extend_dist: 14.0, node_joining_length: 7.0,
  pad_width: 1, pair_odd_branches: false, run: true}
ordered_tracing: {ordering_method: nodestats, pad_width: 1, run: true}
output_dir: /Volumes/shared/pyne_group/Shared/AFM_Data/Top2_HSF1/test
plotting:
  axes: true
  cmap: null
  colorbar: true
  histogram_log_axis: false
  image_set: core
  mask_cmap: blue_purple_green
  num_ticks: [null, null]
  pixel_interpolation: null
  run: true
  savefig_dpi: 600
  savefig_format: null
  style: topostats.mplstyle
  zrange: [-3, 4]
splining: {method: rolling_window, rolling_window_size: 2e-08, run: true, spline_circular_smoothing: 5.0,
  spline_degree: 3, spline_linear_smoothing: 5.0, spline_step_size: 7e-09}
summary_stats: {config: null, run: false}

To Reproduce

No response

TopoStats Version

Git main branch

Python Version

3.12

Operating System

MacOS M1/M2 (post-2021)

Python Packages

No response

[ns-rse]

Thanks for the report @tcatley

If you've time could you re-run using topostats --log-level debug process and paste the output (or attach the log-file) please?

I doubt its an issue with the Python packages so no need to post those.

[tcatley]

Hi @ns-rse, here is the log file from the debug run:

TopoStats-2024-12-17-09-28-45.log

[ns-rse]

Brilliant, thanks @tcatley 🙏

[ns-rse]

Although I've just looked and this is the newer condensed output as its only 26 lines long...

[Tue, 17 Dec 2024 09:28:48] [INFO    ] [topostats] [processing.py] [1027] Processing : minicircle
[Tue, 17 Dec 2024 09:28:48] [INFO    ] [topostats] [processing.py] [87] [minicircle] : *** Filtering ***
[Tue, 17 Dec 2024 09:28:49] [INFO    ] [topostats] [processing.py] [98] [minicircle] : Plotting Filtering Images
[Tue, 17 Dec 2024 09:28:50] [INFO    ] [topostats] [processing.py] [124] [minicircle] : Filters stage completed successfully.
[Tue, 17 Dec 2024 09:28:50] [INFO    ] [topostats] [processing.py] [173] [minicircle] : *** Grain Finding ***
[Tue, 17 Dec 2024 09:28:50] [INFO    ] [topostats] [processing.py] [182] [minicircle] : Grains found for direction above : 21
[Tue, 17 Dec 2024 09:28:50] [INFO    ] [topostats] [processing.py] [198] [minicircle] : Plotting Grain Finding Images
[Tue, 17 Dec 2024 09:28:51] [INFO    ] [topostats] [processing.py] [247] [minicircle] : Grain Finding stage completed successfully.
[Tue, 17 Dec 2024 09:28:51] [INFO    ] [topostats] [processing.py] [299] [minicircle] : *** Grain Statistics ***
[Tue, 17 Dec 2024 09:28:52] [INFO    ] [topostats] [processing.py] [364] [minicircle] : Calculated grainstats for 21 grains.
[Tue, 17 Dec 2024 09:28:52] [INFO    ] [topostats] [processing.py] [365] [minicircle] : Grainstats stage completed successfully.
[Tue, 17 Dec 2024 09:28:52] [INFO    ] [topostats] [processing.py] [425] [minicircle] : *** Disordered Tracing ***
[Tue, 17 Dec 2024 09:28:52] [INFO    ] [topostats] [disordered_tracing.py] [347] [minicircle] : Calculating Disordered Tracing statistics for 21 grains...
[Tue, 17 Dec 2024 09:28:57] [INFO    ] [topostats] [processing.py] [488] [minicircle] : Disordered Tracing stage completed successfully.
[Tue, 17 Dec 2024 09:28:57] [INFO    ] [topostats] [processing.py] [549] [minicircle] : *** Nodestats ***
[Tue, 17 Dec 2024 09:28:57] [INFO    ] [topostats] [nodestats.py] [1885] [minicircle] : Calculating NodeStats statistics for 21 grains...
[Tue, 17 Dec 2024 09:28:58] [INFO    ] [topostats] [processing.py] [627] [minicircle] : NodeStats stage completed successfully.
[Tue, 17 Dec 2024 09:28:58] [INFO    ] [topostats] [processing.py] [694] [minicircle] : *** Ordered Tracing ***
[Tue, 17 Dec 2024 09:28:58] [INFO    ] [topostats] [ordered_tracing.py] [902] [minicircle] : Calculating Ordered Traces and statistics for 21 grains...
[Tue, 17 Dec 2024 09:29:00] [INFO    ] [topostats] [processing.py] [757] [minicircle] : Ordered Tracing stage completed successfully.
[Tue, 17 Dec 2024 09:29:00] [INFO    ] [topostats] [processing.py] [826] [minicircle] : *** Splining ***
[Tue, 17 Dec 2024 09:29:00] [INFO    ] [topostats] [splining.py] [576] [minicircle] : Calculating Splining statistics for 25 molecules...
[Tue, 17 Dec 2024 09:29:00] [INFO    ] [topostats] [processing.py] [894] [minicircle] : Splining stage completed successfully.
[Tue, 17 Dec 2024 09:29:00] [INFO    ] [topostats] [processing.py] [960] [minicircle] : *** Curvature Stats ***
[Tue, 17 Dec 2024 09:29:01] [INFO    ] [topostats] [processing.py] [1232] [minicircle] : *** Image Statistics ***
[Tue, 17 Dec 2024 09:29:01] [INFO    ] [topostats] [io.py] [951] [minicircle] : Saving image to .topostats file

Which might suggest the topostats --log-level debug process doesn't work for you. 🤔

I've just checked at my end and I get much more verbose output.

❱ head TopoStats-2024-12-17-09-40-31.log -n 60
[Tue, 17 Dec 2024 09:40:33] [INFO    ] [topostats] [validation.py] [32] The YAML configuration file is valid.
[Tue, 17 Dec 2024 09:40:33] [INFO    ] [topostats] [validation.py] [32] The YAML plotting configuration file is valid.
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [processing.py] [1292] filter_run=True
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [processing.py] [1293] grains_run=True
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [processing.py] [1294] grainstats_run=True
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [processing.py] [1295] disordered_tracing_run=True
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [processing.py] [1296] nodestats_run=True
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [processing.py] [1297] ordered_tracing_run=True
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [processing.py] [1298] splining_run=True
[Tue, 17 Dec 2024 09:40:33] [INFO    ] [topostats] [processing.py] [1313] Configuration run options are consistent, processing can proceed.
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [150] Main plotting options that need updating/adding to plotting dict :
{   'axes': True,
    'cmap': None,
    'colorbar': True,
    'histogram_log_axis': False,
    'image_set': 'core',
    'mask_cmap': 'blue_purple_green',
    'num_ticks': [None, None],
    'pixel_interpolation': None,
    'savefig_dpi': 100,
    'savefig_format': None,
    'style': 'topostats.mplstyle',
    'zrange': [None, None]}
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [155] Dictionary for image : extracted_channel
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [156] {   'core_set': False,
    'filename': '00-raw_heightmap',
    'image_type': 'non-binary',
    'savefig_dpi': 100,
    'title': 'Raw Height'}
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [123] Updated config config[savefig_dpi] : 100 > 100 
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [165] Updated values :
{'core_set': False,
 'filename': '00-raw_heightmap',
 'image_type': 'non-binary',
 'savefig_dpi': 100,
 'title': 'Raw Height'}
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [170] After adding missing configuration options :
{'axes': True,
 'cmap': None,
 'colorbar': True,
 'core_set': False,
 'filename': '00-raw_heightmap',
 'histogram_log_axis': False,
 'image_set': 'core',
 'image_type': 'non-binary',
 'mask_cmap': 'blue_purple_green',
 'num_ticks': [None, None],
 'pixel_interpolation': None,
 'savefig_dpi': 100,
 'savefig_format': None,
 'style': 'topostats.mplstyle',
 'title': 'Raw Height',
 'zrange': [None, None]}
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [155] Dictionary for image : pixels
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [156] {   'core_set': False,
    'filename': '01-pixels',
    'image_type': 'non-binary',
    'savefig_dpi': 100,
    'title': 'Pixels'}
[Tue, 17 Dec 2024 09:40:33] [DEBUG   ] [topostats] [utils.py] [123] Updated config config[savefig_dpi] : 100 > 1
...