CASMTRIAGE-7272: update-cfs-config and prepare-images stage is failing after hanging during CSM only upgrade using IUF

[annapoorna-s-alt]

Summary and Scope

update-cfs-config and prepare-images stage is failing after hanging during CSM only upgrade using IUF
It hangs up when its executing the command "git ls-remote https://[email protected]/vcs/cray/uss-config-management.git "
Tried fetching the credentials from container if present else kubernetes which doesn't get hungs up for long time

Issues and Related PRs

Resolves CASMTRIAGE-7272

• Fix from csm-api-client: CASMTRIAGE-7272: update-cfs-config and prepare-images stage is failing after hanging during CSM only upgrade using IUF python-csm-api-client#40

Testing

List the environments in which these changes were tested.

Tested on:

fanta

Test description:

Tested with IUF which is working without hangs up

Risks and Mitigations

Minimal

Pull Request Checklist

• Version number(s) incremented, if applicable
• Copyrights updated
• License file intact
• Target branch correct
• CHANGELOG.md updated
• Testing is appropriate and complete, if applicable
• HPC Product Announcement prepared, if applicable

[annapoorna-s-alt]

Tested from SAT

ncn-m001:/sann # sat --version
sat 3.32.2
ncn-m001:/sann # ls
shasta-utils.sh
ncn-m001:~/sann # cd /etc/cray/upgrade/csm/media/uss.1.2.0-53
ncn-m001:/etc/cray/upgrade/csm/media/uss.1.2.0-53 # sat bootprep run --limit configurations --overwrite-configs --vars-file "session_vars.yaml" --format json --bos-version v2 .bootprep-casmtriage-8443/compute-and-uan-bootprep.yaml
INFO: Validating given input file .bootprep-casmtriage-8443/compute-and-uan-bootprep.yaml
INFO: Input file successfully validated against schema
INFO: 1 CFS configuration already exists with the name compute-1.0.0-8443 and will be overwritten.
INFO: 1 CFS configuration already exists with the name lnet-1.0.0-8443 and will be overwritten.
INFO: 1 CFS configuration already exists with the name uan-1.0.0-8443 and will be overwritten.
INFO: Creating 3 CFS configurations
INFO: Creating CFS configuration at index 0 with name=compute-1.0.0-8443
INFO: Successfully created CFS configuration at index 0 with name=compute-1.0.0-8443
INFO: Creating CFS configuration at index 1 with name=lnet-1.0.0-8443
INFO: Successfully created CFS configuration at index 1 with name=lnet-1.0.0-8443
INFO: Creating CFS configuration at index 2 with name=uan-1.0.0-8443
INFO: Successfully created CFS configuration at index 2 with name=uan-1.0.0-8443
INFO: Skipping creation of IMS images based on value of --limit option.
INFO: Skipping creation of BOS session templates based on value of --limit option.
{
"configurations": [
{
"name": "compute-1.0.0-8443"
},
{
"name": "lnet-1.0.0-8443"
},
{
"name": "uan-1.0.0-8443"
}
]
}

[annapoorna-s-alt]

Testing output from IUF

ncn-m001:/etc/cray/upgrade/csm/iuf-blr # iuf -a casmtriage-8443 -m /etc/cray/upgrade/csm/media/uss.1.2.0-53 run -sv /etc/cray/upgrade/csm/media/uss.1.2.0-53/site_vars-8443.yaml -mrp csm -rv /etc/cray/upgrade/csm/media/uss.1.2.0-53/product_vars.yaml -bc /etc/cray/upgrade/csm/media/uss.1.2.0-53/compute-and-uan-bootprep.yaml -mrs reboot -r update-cfs-config
INFO All logs will be stored in /etc/cray/upgrade/csm/iuf/casmtriage-8443/log/20240927072356
WARN --bootprep-config-managed was specified without --bootprep-config-management. The management images will not be built.
INFO [ACTIVITY: casmtriage-8443 ] BEG Install started at 2024-09-27 07:23:56.287326
INFO [IUF SESSION: casmtriage-8443-utdj1 ] BEG Started at 2024-09-27 07:24:13.153442
INFO [STAGE: update-cfs-config ] BEG Argo workflow: casmtriage-8443-utdj1-update-cfs-config-wwgpj
INFO [update-management-cfs-config ] BEG update-management-cfs-config
INFO [update-management-cfs-config ] BEG start-operation
INFO [update-management-cfs-config ] END start-operation [Succeeded]
INFO [update-management-cfs-config ] BEG sat-bootprep-run
INFO [update-management-cfs-config ] Configuration file "/root/.config/sat/sat.toml" generated.
INFO [update-management-cfs-config ] Cannot access bootprep config file (Input parameter '-bm' not provided). Skipping the 'update-management-cfs-config' operation
INFO [update-management-cfs-config ] END sat-bootprep-run [Succeeded]
INFO [update-management-cfs-config ] BEG end-operation
INFO [update-management-cfs-config ] END end-operation [Succeeded]
INFO [update-management-cfs-config ] BEG prom-metrics
INFO [update-management-cfs-config ] END update-management-cfs-config [Succeeded]
INFO [update-management-cfs-config ] END prom-metrics [Succeeded]
INFO [update-managed-cfs-config ] BEG update-managed-cfs-config
INFO [update-managed-cfs-config ] BEG start-operation
INFO [update-managed-cfs-config ] END start-operation [Succeeded]
INFO [update-managed-cfs-config ] BEG sat-bootprep-run
INFO [update-managed-cfs-config ] Configuration file "/root/.config/sat/sat.toml" generated.
INFO [update-managed-cfs-config ] Using bootprep config file .bootprep-casmtriage-8443/compute-and-uan-bootprep.yaml for updating CFS config on managed nodes
INFO [update-managed-cfs-config ] Validating given input file /etc/cray/upgrade/csm/media/uss.1.2.0-53/.bootprep-casmtriage-8443/compute-and-uan-bootprep.yaml
INFO [update-managed-cfs-config ] Input file successfully validated against schema
INFO [update-managed-cfs-config ] 1 CFS configuration already exists with the name compute-1.0.0-8443 and will be overwritten.
INFO [update-managed-cfs-config ] 1 CFS configuration already exists with the name lnet-1.0.0-8443 and will be overwritten.
INFO [update-managed-cfs-config ] 1 CFS configuration already exists with the name uan-1.0.0-8443 and will be overwritten.
INFO [update-managed-cfs-config ] Creating 3 CFS configurations
INFO [update-managed-cfs-config ] Creating CFS configuration at index 0 with name=compute-1.0.0-8443
INFO [update-managed-cfs-config ] Successfully created CFS configuration at index 0 with name=compute-1.0.0-8443
INFO [update-managed-cfs-config ] Creating CFS configuration at index 1 with name=lnet-1.0.0-8443
INFO [update-managed-cfs-config ] Successfully created CFS configuration at index 1 with name=lnet-1.0.0-8443
INFO [update-managed-cfs-config ] Creating CFS configuration at index 2 with name=uan-1.0.0-8443
INFO [update-managed-cfs-config ] Successfully created CFS configuration at index 2 with name=uan-1.0.0-8443
INFO [update-managed-cfs-config ] Skipping creation of IMS images based on value of --limit option.
INFO [update-managed-cfs-config ] Skipping creation of BOS session templates based on value of --limit option.
INFO [update-managed-cfs-config ] END sat-bootprep-run [Succeeded]
INFO [update-managed-cfs-config ] BEG end-operation
INFO [update-managed-cfs-config ] END end-operation [Succeeded]
INFO [update-managed-cfs-config ] BEG prom-metrics
INFO [update-managed-cfs-config ] END update-managed-cfs-config [Succeeded]
INFO [update-managed-cfs-config ] END prom-metrics [Succeeded]
INFO [STAGE: update-cfs-config ] END Succeeded in 0:01:39
INFO [IUF SESSION: casmtriage-8443-utdj1 ] END Completed at 2024-09-27 07:25:55.484309
INFO [ACTIVITY: casmtriage-8443 ] END Completed in 0:01:59

Install Summary
command line: iuf -a casmtriage-8443 -m /etc/cray/upgrade/csm/media/uss.1.2.0-53 run -sv /etc/cray/upgrade/csm/media/uss.1.2.0-53/site_vars-8443.yaml -mrp csm -rv /etc/cray/upgrade/csm/media/uss.1.2.0-53/product_vars.yaml -bc /etc/cray/upgrade/csm/media/uss.1.2.0-53/compute-and-uan-bootprep.yaml -mrs reboot -r update-cfs-config
activity: casmtriage-8443
media dir: /etc/cray/upgrade/csm/media/uss.1.2.0-53
state dir: /etc/cray/upgrade/csm/iuf/casmtriage-8443/state
log dir: /etc/cray/upgrade/csm/iuf/casmtriage-8443/log/20240927072356
site vars: /etc/cray/upgrade/csm/iuf/casmtriage-8443/state/session_vars.yaml

sat can be run manually with the following commands:
cd /etc/cray/upgrade/csm/media/uss.1.2.0-53
sat bootprep run --limit configurations --overwrite-configs --vars-file
"session_vars.yaml" --format json --bos-version v2
.bootprep-casmtriage-8443/compute-and-uan-bootprep.yaml