[Bug]: Flattening anomoly

[tcatley]

Checklist

• Re-run analysis with topostats process --core 1.
• Describe the bug.
• Include the configuration file.
• Copy of the output.
• The exact command that failed. This is what you typed at the command line, including any options.
• TopoStats version, this is reported by topostats --version
• Operating System and Python Version

Describe the bug

Issue with flattening data that has an artefact which causes non-linear background in x

Copy of the output

n/a

Include the configuration file

# Configuration from TopoStats run completed : 2024-02-05 10:39:48
# For more information on configuration and how to use it:
# https://afm-spm.github.io/TopoStats/main/configuration.html
base_dir: /Users/work/Library/CloudStorage/[email protected]/Shared
  drives/Nanocharacterisation lab/3_Work request/External work data/ROYCE access schemes/AFM
  Data/Joseph Davy/20240125/Data
output_dir: /Users/work/Library/CloudStorage/[email protected]/Shared
  drives/Nanocharacterisation lab/3_Work request/External work data/ROYCE access schemes/AFM
  Data/Joseph Davy/20240125/Processed
log_level: info
cores: 6
file_ext: .spm
loading:
  channel: Height
filter:
  run: true
  row_alignment_quantile: 0.5
  threshold_method: std_dev
  otsu_threshold_multiplier: 1.0
  threshold_std_dev:
    below: 10.0
    above: 1.0
  threshold_absolute:
    below: -1.0
    above: 1.0
  gaussian_size: 1.01
  gaussian_mode: nearest
  remove_scars:
    run: false
    removal_iterations: 2
    threshold_low: 0.25
    threshold_high: 0.666
    max_scar_width: 4
    min_scar_length: 100
grains:
  run: true
  threshold_method: absolute
  otsu_threshold_multiplier: 1.0
  threshold_std_dev:
    below: 10.0
    above: 1.0
  threshold_absolute:
    below: -1.0
    above: 0.95
  direction: above
  smallest_grain_size_nm2: 2000
  absolute_area_threshold:
    above:
    - 4000
    - 25000
    below:
    - 
    - 
  remove_edge_intersecting_grains: true
grainstats:
  run: true
  edge_detection_method: binary_erosion
  cropped_size: 300
dnatracing:
  run: false
  min_skeleton_size: 10
  skeletonisation_method: topostats
  pad_width: 1
plotting:
  run: true
  save_format: png
  pixel_interpolation:
  image_set: core
  zrange:
  - -3
  - 4
  colorbar: true
  axes: true
  cmap: nanoscope
  mask_cmap: blu
  histogram_log_axis: false
  histogram_bins: 200
  dpi: 150
summary_stats:
  run: false
  config: /Users/work/Documents/TopoStats_GM/TopoStats/topostats/summary_config.yaml

To Reproduce

run topostats using above config on the file (contact me for file)

TopoStats Version

Git main branch

Python Version

3.11

Operating System

MacOS M1/M2 (post-2021)

Python Packages

No response

[ns-rse]

Thanks for the report @tcatley

Just to check which commit your own would you be able to paste in the output of...

pip show topostats
pip freeze

...and...

git log --graph --pretty="%Cred%h%Creset -%C(auto)%d%Creset %s %Cgreen(%ad) %C(bold blue)<%an>%Creset"

(Only need the top 20-30 lines of this, no need to scroll through all the history).

[tcatley]

$ pip show topostats
Name: topostats
Version: 2.1.1.dev247+g111dc5b
Summary: Automated Analysis for Atomic Force Microscopy Images of Biomolecules
Home-page: 
Author: 
Author-email: TopoStats Team <[email protected]>, Alice Pyne <[email protected]>, Sylvia Whittle <[email protected]>, Neil Shephard <[email protected]>, Max Gamill <[email protected]>, Jean Du <[email protected]>
License: GNU Lesser GPLv3 only
Location: /Users/work/opt/anaconda3/envs/topostats_git_main/lib/python3.10/site-packages
Requires: h5py, igor, matplotlib, numpy, pandas, pyfiglet, pySPM, pyyaml, ruamel.yaml, schema, scikit-image, scipy, seaborn, tifffile, tqdm
Required-by:

$ pip freeze
contextlib2==21.6.0
contourpy==1.1.0
cycler==0.11.0
fonttools==4.40.0
h5py==3.9.0
igor==0.3
imageio==2.31.1
joblib==1.2.0
kiwisolver==1.4.4
lazy_loader==0.2
matplotlib==3.7.1
networkx==3.1
numpy==1.23.4
packaging==23.1
pandas==2.0.2
Pillow==9.5.0
pyfiglet==0.8.post1
pyparsing==3.1.0
pySPM==0.2.23
python-dateutil==2.8.2
pytz==2023.3
PyWavelets==1.4.1
PyYAML==6.0
ruamel.yaml==0.17.32
ruamel.yaml.clib==0.2.7
schema==0.7.5
scikit-image==0.21.0
scikit-learn==1.2.2
scipy==1.11.0
seaborn==0.12.2
six==1.16.0
threadpoolctl==3.1.0
tifffile==2023.4.12
topostats @ file:///Users/work/Documents/TopoStats_GM/TopoStats
tqdm==4.65.0
tzdata==2023.3

*   42d1d19a - (HEAD -> main, origin/main, origin/HEAD) Merge pull request #749 from AFM-SPM/ns-rse/snoop (Fri Dec 8 00:16:06 2023 +0000) <Neil Shephard>
|\  
| * cc353b9a - Moving snoop dependency to main (Tue Dec 5 17:41:55 2023 +0000) <Neil Shephard>
| * 284ec842 - Add snoop as a dependency but disable by default (Tue Dec 5 16:59:05 2023 +0000) <Neil Shephard>
* |   7d7546f0 - Merge pull request #747 from AFM-SPM/ns-rse/741-filter-entry-point (Fri Dec 8 00:14:10 2023 +0000) <Neil Shephard>
|\ \  
| |/  
|/|   
| * a18eeb31 - Adds subparsers for further modularisation (Thu Nov 30 17:21:36 2023 +0000) <Neil Shephard>
|/  
*   5a6a7609 - Merge pull request #739 from AFM-SPM/ns-rse/update-badges (Tue Nov 21 17:33:50 2023 +0000) <Neil Shephard>
|\  
| * f29b0eb2 - Restore Ruff badge (Tue Nov 21 17:33:00 2023 +0000) <Neil Shephard>
* | ebfb2d6a - Merge pull request #738 from AFM-SPM/ns-rse/update-badges (Tue Nov 21 17:03:54 2023 +0000) <Neil Shephard>
|\| 
:

[ns-rse]

Thanks @tcatley that is really useful.

Not 100% clear but it looks like you have installed from the cloned repository locally (on 2023-12-08) but that would have been after v2.1.2 tag was applied but pip show topostats is reporting the tagged version as v2.1.1.

I don't think its likely to make any difference to the flattening but would you mind re-running with the latest released version of TopoStats please?

The following should work to create a new conda environment, activate it and install TopoStats v2.2.post0 from PyPI.

conda create -n topostats220
conda activate topostats220
pip install topostats       # This will install the latest version of TopoStats from PyPI

Ping me on Slack if you have any problems setting up the Conda environment.

Thanks,

@ns-rse

[tcatley]

For some reason pip install topostats isn't working for me right now? I'm not sure if I've done something wrong here.

(topostats220) work@Toms-MacBook-Pro ~ % pip install topostats
zsh: command not found: pip

[ns-rse]

Looks like Conda didn't install pip you can do so with...

conda install pip

A useful site to bookmark for working with Conda environments (aside from their official documentation) is Conda environments for effective and reproducible research. Chapter 3 goes into detail about Conda packages, channels and pip/PyPI (the Python Package Index).

[tcatley]

Have now re-run on the the latest PyPI release of topostats and having the same outcome. I don't know if this is a bug as such, as the artefact is coming from the AFM - more so that an increase in the flattening order would be useful to see if this fixes it. This was achieved in gwyddion with higher order polynomial flattening -

[ns-rse]

Thanks for checking @tcatley

Perhaps a solution might be to make the polynomial flattening configurable in some manner.

Like most of the complex tasks its something @SylviaWhittle implemented and I'm not too familiar with the maths/methods involved.

There is the Filter method remove_nonlinear_polynomial() and also remove_quadratic().

But perhaps a new method will be required. 🤔