Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CRAYSAT-1706: To stop waiting if BOS session is deleted #255

Merged
merged 1 commit into from
Aug 20, 2024
Merged

CRAYSAT-1706: To stop waiting if BOS session is deleted #255

merged 1 commit into from
Aug 20, 2024

Conversation

annapoorna-s-alt
Copy link
Contributor

@annapoorna-s-alt annapoorna-s-alt commented Aug 12, 2024 •
edited
Loading

Summary and Scope

Stop waiting if BOS session is deleted
If a BOS session is deleted when sat bootsys is running, then the bootsys session should end early instead of repeatedly showing a 404 error message.

Issues and Related PRs

CRAYSAT-1706

Testing

List the environments in which these changes were tested.

Tested on:

drax

Test description:

Start the BOS session with sat bootsys boot/shutdown/reboot --stage bos-operations and then delete the BOS session while it's in progress.
It should exit early instead of repeatedly showing a 404 error message.

Risks and Mitigations

Low risk

Pull Request Checklist

  • Version number(s) incremented, if applicable
  • Copyrights updated
  • License file intact
  • Target branch correct
  • CHANGELOG.md updated
  • Testing is appropriate and complete, if applicable
  • HPC Product Announcement prepared, if applicable

@annapoorna-s-alt
Copy link
Contributor Author

Here is the latest output

@annapoorna-s-alt annapoorna-s-alt force-pushed the CRAYSAT-1706-stop-waiting-for-deleted-bos-session branch 2 times, most recently from b924862 to 55bf1c1 Compare August 12, 2024 11:14
haasken-hpe
haasken-hpe reviewed Aug 12, 2024
View reviewed changes
sat/cli/bootsys/bos.py Outdated Show resolved Hide resolved
sat/cli/bootsys/bos.py Outdated Show resolved Hide resolved
@annapoorna-s-alt annapoorna-s-alt force-pushed the CRAYSAT-1706-stop-waiting-for-deleted-bos-session branch 2 times, most recently from 366b4f7 to 13ac003 Compare August 13, 2024 07:38
@annapoorna-s-alt
Copy link
Contributor Author

Final output on drax

haasken-hpe
haasken-hpe approved these changes Aug 13, 2024
View reviewed changes
Copy link
Contributor

@haasken-hpe haasken-hpe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with minor suggestions for improvement for you to consider.

sat/cli/bootsys/bos.py Outdated Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
@haasken-hpe
Copy link
Contributor

Testing output looks good, but there appears to be some strange blank spaces introduced in the output. I assume that is due to a copy-paste issue.

If you make further changes, I think it's acceptable to test with just one type of action (e.g. shutdown, boot, or reboot) rather than all three to save some time. I think the code path is essentially the same in SAT. It creates a session in BOS and then waits on the session to be done.

shivaprasad-metimath
shivaprasad-metimath approved these changes Aug 14, 2024
View reviewed changes
Copy link
Contributor

@shivaprasad-metimath shivaprasad-metimath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, can address the suggested changes

@annapoorna-s-alt annapoorna-s-alt force-pushed the CRAYSAT-1706-stop-waiting-for-deleted-bos-session branch 7 times, most recently from c8b0a47 to 981cca1 Compare August 19, 2024 09:18
@annapoorna-s-alt
Copy link
Contributor Author

Tested on drax with shutdown of managed nodes

ncn-m001:~/sann # sat bootsys shutdown --stage bos-operations --bos-shutdown-timeout 1800 --bos-templates $SESSION_TEMPLATES
Proceed with shutdown of nodes using BOS? [yes,no] yes
Proceeding with shutdown of nodes using BOS.
INFO: Using session templates provided by --bos-templates/bos_templates option: ['compute-24.3.1.x86_64-csm-1.5.2-final', 'uan-24.3.1.x86_64-csm-1.5.2-final']
INFO: Starting shutdown operation on BOS session templates: compute-24.3.1.x86_64-csm-1.5.2-final, uan-24.3.1.x86_64-csm-1.5.2-final.
INFO: Waiting up to 1800 seconds for sessions to complete.
INFO: Waiting for BOS session 2d79d94d-c6f3-4783-ba55-585f4bee7228 to reach target state complete. Session template: uan-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session 2d79d94d-c6f3-4783-ba55-585f4bee7228 to reach target state complete. Session template: uan-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
ERROR: Could not wait for condition "session 2d79d94d-c6f3-4783-ba55-585f4bee7228 reached target state complete": Failed to query session status: Session 2d79d94d-c6f3-4783-ba55-585f4bee7228 does not exist.
ERROR: Operation 'shutdown' failed on BOS session template 'uan-24.3.1.x86_64-csm-1.5.2-final': BOS session with id 2d79d94d-c6f3-4783-ba55-585f4bee7228 and session template uan-24.3.1.x86_64-csm-1.5.2-final failed.
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Still waiting on session(s) for template(s): compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Session faaa9c2a-7574-488a-adbb-d537dfc42dc1: 100.00% components succeeded, 0.00% components failed
INFO: Shutdown with BOS session template compute-24.3.1.x86_64-csm-1.5.2-final completed.
INFO: Session faaa9c2a-7574-488a-adbb-d537dfc42dc1: 100.00% components succeeded, 0.00% components failed
INFO: Session 2d79d94d-c6f3-4783-ba55-585f4bee7228: 0.00% components succeeded, 0.00% components failed
ERROR: Shutdown failed or timed out for session template: uan-24.3.1.x86_64-csm-1.5.2-final

haasken-hpe
haasken-hpe reviewed Aug 20, 2024
View reviewed changes
sat/apiclient/bos.py Outdated Show resolved Hide resolved
haasken-hpe
haasken-hpe reviewed Aug 20, 2024
View reviewed changes
sat/cli/bootsys/bos.py Outdated Show resolved Hide resolved
haasken-hpe
haasken-hpe approved these changes Aug 20, 2024
View reviewed changes
Copy link
Contributor

@haasken-hpe haasken-hpe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with minor fixes suggested.

@annapoorna-s-alt annapoorna-s-alt force-pushed the CRAYSAT-1706-stop-waiting-for-deleted-bos-session branch from 981cca1 to a8846c9 Compare August 20, 2024 13:41
@annapoorna-s-alt annapoorna-s-alt merged commit 0a301f5 into main Aug 20, 2024
3 checks passed
@annapoorna-s-alt annapoorna-s-alt deleted the CRAYSAT-1706-stop-waiting-for-deleted-bos-session branch August 20, 2024 14:02
@annapoorna-s-alt annapoorna-s-alt mentioned this pull request Aug 21, 2024
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shivaprasad-metimath shivaprasad-metimath shivaprasad-metimath approved these changes

@haasken-hpe haasken-hpe haasken-hpe approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@annapoorna-s-alt @haasken-hpe @shivaprasad-metimath