How to deprecate old codes in Xmipp?

[albertmena]

Deprecating old code!

We have a lot of old code with bugs that makes impossible to run it, it has no use and is difficult to maintenance,

But how we decide which is deprecable code and which not?
Some ideas:

• Ranking protocols by use
• Experience of users of our team
• Protocols never worked and has no test

[albertmena]

Protocols failing: (programs called are candidates to deprecate)
protocol_deep_align.py
protocol_ctf_defocus_group.py
protocol_enrich.py
protocol_classify_kmeans2d.py
protocol_reconstruct_heterogeneous.py
protocol_reconstruct_significant.py
protocol_split_volume_hierarchical_cluster.py
protocol_split_volume_hierarchical_cluster

[albertmena]

Move to deprecated folder protocols without maintenance and use

[DStrelak]

We should start by listing which xmipp programs are called from Scipion protocols. Those NOT invoked are clearly candidates, though some of them are still useful (*phantom* etc).
Other option is to list all programs xmipp provides and see if there are some overlaps. I can't remember exact names, but those programs responsible for image handling have some overlap for sure.
We can also try to run some static analysis on the code base, to see what is not internally invoked.

In any case, we don't need to purge that code straight away. We can simply move it to a legacy folder, not compile it, and if somebody complains, we bring it back.

[DStrelak]

Unique list is then:
xmipp_metadata_utilities
xmipp_image_ssnr
xmipp_metadata_split
xmipp_transform_window
xmipp_transform_randomize_phases
xmipp_image_operate
xmipp_transform_threshold
xmipp_resolution_fsc
xmipp_transform_filter
xmipp_image_resize
xmipp_image_convert
xmipp_transform_mask
xmipp_ctf_correct_phase
xmipp_ctf_group
xmipp_angular_distance
xmipp_image_residuals
xmipp_image_eliminate_byEnergy
xmipp_transform_symmetrize
xmipp_volume_align
xmipp_image_header
xmipp_phantom_create
xmipp_transform_morphology
xmipp_reconstruct_significant
xmipp_transform_adjust_volume_grey_levels
xmipp_volume_segment
xmipp_volume_homogenizer
xmipp_ctf_correct_wiener2d
xmipp_volume_local_sharpening
xmipp_angular_project_library
xmipp_transform_add_noise
xmipp_angular_projection_matching
xmipp_transform_downsample
xmipp_ctf_sort_psds
xmipp_transform_geometry
xmipp_phantom_project
xmipp_volume_find_symmetry
xmipp_classify_CL2D
xmipp_transform_dimred
xmipp_reconstruct_fourier_accel
xmipp_image_statistics
xmipp_classify_evaluate_classes
xmipp_classify_compare_classes
xmipp_cuda_reconstruct_fourier
xmipp_transform_mirror
xmipp_ctf_phase_flip
xmipp_transform_normalize
xmipp_write_test
xmipp_classify_significant

[albertmena]

thanks

[albertmena]

List of programs called (runJob) from protocols/*

[DStrelak]

Which leaves this list of unused programs. @albertmena, please double check your ('used') list, coz we have 4 times more programs than we actually use in protocols, ignoring MPI versions and tests ...

xmipp_angular_accuracy_pca
xmipp_angular_assignment_mag
xmipp_angular_break_symmetry
xmipp_angular_commonline
xmipp_angular_continuous_assign
xmipp_angular_continuous_assign2
xmipp_angular_discrete_assign
xmipp_angular_distribution_show
xmipp_angular_estimate_tilt_axis
xmipp_angular_neighbourhood
xmipp_angular_rotate
xmipp_angular_sph_alignment
xmipp_apropos
xmipp_art_zernike3d
xmipp_classify_analyze_cluster
xmipp_classify_extract_features
xmipp_classify_first_split
xmipp_classify_first_split3
xmipp_classify_kerdensom
xmipp_classify_kmeans_2d
xmipp_compile
xmipp_cone_deepalign
xmipp_cone_deepalign_predict
xmipp_coordinates_consensus
xmipp_coordinates_noisy_zones_filter
xmipp_ctf_correct_idr
xmipp_ctf_correct_wiener3d
xmipp_ctf_create_ctfdat
xmipp_ctf_enhance_psd
xmipp_ctf_estimate_from_micrograph
xmipp_ctf_estimate_from_psd
xmipp_ctf_estimate_from_psd_fast
xmipp_ctf_show
xmipp_cuda_align_significant
xmipp_cuda_angular_sph_alignment
xmipp_cuda_correlation
xmipp_cuda_movie_alignment_correlation
xmipp_cuda_volume_deform_sph
xmipp_cuda_volume_halves_restoration
xmipp_deep_consensus
xmipp_deep_denoising
xmipp_deep_hand
xmipp_deep_micrograph_cleaner
xmipp_deepRes_resolution
xmipp_deep_volume_postprocessing
xmipp_evaluate_coordinates
xmipp_extract_particles
xmipp_extract_subset
xmipp_flexible_alignment
xmipp_idr_xray_tomo
xmipp_image_align
xmipp_image_align_tilt_pairs
xmipp_image_assignment_tilt_pair
xmipp_image_common_lines
xmipp_image_eliminate_empty_particles
xmipp_image_find_center
xmipp_image_histogram
xmipp_image_odd_even
xmipp_image_rotational_pca
xmipp_image_rotational_spectra
xmipp_image_separate_objects
xmipp_image_sort_by_statistics
xmipp_image_vectorize
xmipp_matrix_dimred
xmipp_metadata_convert_to_spider
xmipp_metadata_histogram
xmipp_metadata_import
xmipp_metadata_plot
xmipp_metadata_selfile_create
xmipp_metadata_split_3D
xmipp_metadata_xml
xmipp_micrograph_automatic_picking
xmipp_micrograph_scissor
xmipp_ml_align2d
xmipp_mlf_align2d
xmipp_mlf_refine3d
xmipp_ml_refine3d
xmipp_ml_tomo
xmipp_movie_alignment_correlation
xmipp_movie_estimate_gain
xmipp_movie_filter_dose
xmipp_movie_optical_alignment_cpu
xmipp_mrc_create_metadata
xmipp_multireference_aligneability
xmipp_nma_alignment
xmipp_nma_alignment_vol
xmipp_particle_boxsize
xmipp_pdb_analysis
xmipp_pdb_center
xmipp_pdb_construct_dictionary
xmipp_pdb_label_from_volume
xmipp_pdb_nma_deform
xmipp_pdb_reduce_pseudoatoms
xmipp_pdb_restore_with_dictionary
xmipp_pdb_select
xmipp_pdb_sph_deform
xmipp_phantom_movie
xmipp_phantom_simulate_microscope
xmipp_phantom_transform
xmipp_pick_noise
xmipp_preprocess_mics
xmipp_reconstruct_admm
xmipp_reconstruct_art
xmipp_reconstruct_art_pseudo
xmipp_reconstruct_art_xray
xmipp_reconstruct_fourier
xmipp_reconstruct_wbp
xmipp_resolution_directional
xmipp_resolution_fso
xmipp_resolution_ibw
xmipp_resolution_localfilter
xmipp_resolution_monogenic_signal
xmipp_resolution_monotomo
xmipp_resolution_pdb_bfactor
xmipp_resolution_ssnr
xmipp_score_micrograph
xmipp_showj
xmipp_starpu_reconstruct_fourier
xmipp_subtract_projection
xmipp_sync_data
xmipp_tomo_align_dual_tilt_series
xmipp_tomo_align_refinement
xmipp_tomo_align_tilt_series
xmipp_tomo_detect_missing_wedge
xmipp_tomo_extract_subvolume
xmipp_tomo_filter_coordinates
xmipp_tomo_map_back
xmipp_tomo_project
xmipp_tomo_remove_fluctuations
xmipp_transform_adjust_image_grey_levels
xmipp_transform_center_image
xmipp_transform_range_adjust
xmipp_validation_nontilt
xmipp_validation_tilt_pairs
xmipp_version
xmipp_volume_align_prog
xmipp_volume_apply_deform_sph
xmipp_volume_center
xmipp_volume_consensus
xmipp_volume_correct_bfactor
xmipp_volume_deform_sph
xmipp_volume_enhance_contrast
xmipp_volume_from_pdb
xmipp_volume_halves_restoration
xmipp_volume_initial_simulated_annealing
xmipp_volume_pca
xmipp_volume_reslice
xmipp_volumeset_align
xmipp_volume_structure_factor
xmipp_volume_subtraction
xmipp_volume_to_pseudoatoms
xmipp_volume_to_web
xmipp_volume_validate_pca
xmipp_work_test
xmipp_xray_import
xmipp_xray_project
xmipp_xray_psf_create

[DStrelak]

Ok, so I've:

1. git grep -F -- 'xmipp_' > used.txt
2. removed everything before 'xmipp_.*' and after the word
3. ls -1 build/bin/ > all.txt
4. grep -Fxv -f used.txt all.txt > unused.txt
   I got the following list:
   xmipp_angular_continuous_assign
   xmipp_angular_discrete_assign
   xmipp_angular_distribution_show
   xmipp_angular_rotate
   xmipp_apropos
   xmipp_art_zernike3d
   xmipp_classify_analyze_cluster
   xmipp_classify_extract_features
   xmipp_compile
   xmipp_ctf_correct_idr
   xmipp_ctf_correct_wiener3d
   xmipp_ctf_create_ctfdat
   xmipp_ctf_enhance_psd
   xmipp_ctf_estimate_from_psd
   xmipp_ctf_show
   xmipp_evaluate_coordinates
   xmipp_extract_subset
   xmipp_flexible_alignment
   xmipp_idr_xray_tomo
   xmipp_image_common_lines
   xmipp_image_separate_objects
   xmipp_matrix_dimred
   xmipp_metadata_convert_to_spider
   xmipp_metadata_histogram
   xmipp_metadata_import
   xmipp_metadata_plot
   xmipp_metadata_selfile_create
   xmipp_metadata_split_3D
   xmipp_metadata_xml
   xmipp_mlf_refine3d
   xmipp_ml_refine3d
   xmipp_movie_filter_dose
   xmipp_mrc_create_metadata
   xmipp_nma_alignment
   xmipp_pdb_analysis
   xmipp_pdb_center
   xmipp_pdb_construct_dictionary
   xmipp_pdb_nma_deform
   xmipp_pdb_reduce_pseudoatoms
   xmipp_pdb_restore_with_dictionary
   xmipp_pdb_select
   xmipp_phantom_movie
   xmipp_phantom_simulate_microscope
   xmipp_phantom_transform
   xmipp_reconstruct_admm
   xmipp_reconstruct_art_pseudo
   xmipp_reconstruct_art_xray
   xmipp_resolution_ibw
   xmipp_resolution_localfilter
   xmipp_resolution_monotomo
   xmipp_resolution_ssnr
   xmipp_score_micrograph
   xmipp_starpu_reconstruct_fourier
   xmipp_tomo_align_dual_tilt_series
   xmipp_tomo_align_refinement
   xmipp_tomo_align_tilt_series
   xmipp_tomo_detect_missing_wedge
   xmipp_tomo_extract_subvolume
   xmipp_tomo_filter_coordinates
   xmipp_tomo_map_back
   xmipp_tomo_project
   xmipp_tomo_remove_fluctuations
   xmipp_transform_range_adjust
   xmipp_validation_tilt_pairs
   xmipp_version
   xmipp_volume_align_prog
   xmipp_volume_center
   xmipp_volume_enhance_contrast
   xmipp_volume_initial_simulated_annealing
   xmipp_volume_local_sharpening
   xmipp_volume_pca
   xmipp_volume_reslice
   xmipp_volume_to_pseudoatoms
   xmipp_volume_to_web
   xmipp_volume_validate_pca
   xmipp_work_test
   xmipp_xray_import
   xmipp_xray_project
   xmipp_xray_psf_create

[albertmena]

Thanks!!!, are many!

[albertmena]

#681