-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate cray-product-catalog ConfigMap to multiple product ConfigMaps #292
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We will write unit test cases post that.
@lathanm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few cases we need to add
- empty main config map -> exit
- main config map read failed -> exist
- Already splited cms i.e config version not present in main -> exit
shall complete all cases.
Others please look @lathanm @krabhi-hpe @paulose-kuriakose @anoop1402
Added revoke and grant update permission to main ConfigMap
Added all these test cases |
merged the migration job into the orginal helm chart
… to multiple ConfigMaps as pre-upgrade process Also includes: * CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the cray-product-catalog Kubernetes ConfigMap is split into multiple smaller ConfigMaps with each product's 'component_versions' data in its own ConfigMap. Modify the catalog_update.py script to update data across these ConfigMaps. * CASM-4427: Implement a prototype to have granular query from main and sub ConfigMaps * CASM-4368: Delete Cray Product Catalog details from main and sub ConfigMaps for a particular product version. * Added new argument max_attempts to modify_config_map function in catalog_delete.py, because we need not retry 100 times when read ConfigMap fails for a product ConfigMap. * CASM-4504: Added label 'type=cray-product-catalog' to all cray-product-catalog related ConfigMaps * Implemented migration of cray-product-catalog ConfigMap to multiple ConfigMaps as part of pre-upgrade steps * Added migration job to the configmap-hook 7e617ee CASM-4427: Combine ConfigMap data in the query output and CASM-3981: Schema update 4c46725 Replaced the data with items while mocking the data 2ccae06 modified the mock data 93a4ac5 modified the mock data fd4d6c7 Reverted prev change ebd0b6c Added one more mock data c8d9b42 Resolved error 684d867 Resolved error cbf8e04 Resolved failure bdb2bab Changed message in test_query.py as per query.py 52cc275 Resolved failure 062ee3e Removed mocking mock_loadConfigMapData output for 2 funcs ba9d0c3 Removed mocking mock_loadConfigMapData output for 2 funcs bc71071 Added mock_invalid_data 7192d1a syntax error b46bb82 message formating corrected b72c1a1 Aded one more invalid data b40b341 created mock obj with invalid yaml ca51fa5 created mock obj with invalid yaml 300f387 created mock obj with duplicate value df6487b Commented test case test_create_product_catalog_invalid_product_data ad09463 pycodestyle correction 5175e2f Fixed invalid yaml testcase and uncommented it c99ee1e Fixed pytest_mod issues 352223c Fixed typo error dd7f529 Addressed review comments c9ebd76 Fixed pymod test errors 46049f8 Merging PR 269 215cd0c Fixed pycodestyle 65b492c CASM-4368: Delete cray-product-catalog details from main and sub cm for a particular product version c1a3c9a fixed pycodestyle error 224530f CASM-4350: Changes in Update code path to accommodate Split config maps. 4bbde55 Modified message in CHANGELOG.md 465e84e Addressed review comments fee6117 Added try catch block for call create_namespaced_config_map 8e39b49 (origin/feature/catalog-delete) Addressed review comments a1fc575 (origin/casm-4427) Addressed all except review comments related to test case f9f3158 (origin/catalog-delete-ut) adding unit test cases for delete logic+making the code more testable 4029e41 Addressing the pr comments by mharding,21-07-2023 b81be8b Addressing the pr comments by abhishek,24-07-2023 542ea63 - Added positive and negative testcases for docker, s3, helm, manifests - Modified query to look for configmaps starting with the ConfigMap name passed as argument. If nothing is passed then ConfigMap name will be set to cray-product-catalog 2c0eb39 Fixed pymodd test errors 9fa89c0 Removed trailing space 09ef4cd Added condition to come out of infiinte loop while updating the ConfigMap data dc97b11 Reverting ConfigMap doesn't exist logic as per Ryan's comment c5e584d Resolved merge conflict by incorporating both suggestions. 1fd17f3 Merge pull request #273 from Cray-HPE/catalog-delete-ut 05c6214 Added test cases for catalog_update.py Modified test cases of test_data_catalog_helper.py to use unittest 153c6cb Fixed pymodtest errors 9285943 Fixed pycodestyle error 1c19cf2 Fixed pycodestyle error b28eda2 Merge branch 'develop' into feature/catalog-delete ee34158 Update cray_product_catalog/catalog_delete.py efac9bd Addressed review comments 7283217 Resolved merge conflict by incorporating both suggestions. 3edbaa6 Minor change in Changelog 5606651 Addressed some of Ryaan's review comments a5043ea Addressed Ryaan's review comments: - Added log message if attempts exceeds max_attempts(retries) - Added test case to test max retries - Added logic to exit if create_ config_map fails a266279 Fixed pycode style error c0ca0b7 Resolved merge conflict by incorporating both suggestions. cd4b443 Merge pull request #271 from Cray-HPE/feature/catalog-delete 8935515 Resolved merge conflict by incorporating both suggestions. 8c5908b Merge pull request #261 from Cray-HPE/casm-4427 6572b3d Resolved merge conflict by incorporating both suggestions. 2e61364 Merge pull request #272 from Cray-HPE/feature/casm-4350 3f0d940 CASM-4504: Add label type=cray-product-catalog for all cray-product-catalog related ConfigMaps a4aa7d2 - Addressed review comment - Fixed unit test failure due to addition of label_serlector argument to list_namespaced_config_map function b28ec2f Fixed pycodestyle error 1b9d965 (origin/CASM-4504-label) Fixed license error ddafa16 (origin/feature/split-cm-migration-dockerfile) migration init 57ebc86 Addressed review comments ae26488 Removed duplicate line ec8d935 Merge pull request #278 from Cray-HPE/CASM-4504-label 38a79df init roll back 652d1df migration init cd365f4 rebase changes 2638d1f merge from epic feature branch e5adc31 rollback and exit handler aa41547 binary cleanup 8184a2d CASM-4279: Add Kubernetes Api for create/delete/read/etc bb68076 Optimisation bab6191 Initial change to handle config map data split into multiple configmaps b1475d9 Merge pull request #290 from Cray-HPE/casm-4529-migration-api 9f8cf3c Merge branch 'feature/split-cm-migration' into migration/data-manager e0478aa Merge pull request #291 from Cray-HPE/migration/data-manager ade53cf Merge branch 'feature/split-cm-migration' into feature/split-cm-migration-rollback-exit e3cc918 Merge pull request #289 from Cray-HPE/feature/split-cm-migration-rollback-exit 02a21a8 Added rollback operation and modified rename_config_map function 1c94ba7 Fixed few corner case logic 6445ff0 Added unit test cases 3ebaa20 added rename_config_map test case- not working 73645c8 Added few more test cases to configmap data handler f343b23 ut fixes f61d384 Added unit test cases related to main.py 67bc931 Fixing pycode-style test error d4e704d Addressed review comments Added revoke and grant update permission to main ConfigMap d53c411 Updated Copyright license for setup.py e171abe Fixed UT issue cdf51bb Modified code to accept env var for name and namespace 47239ec Added check against resource_version 36c63ed Fixed pycode style error 35c6496 Modified backup restore to use kubectl create command instead of kubectl apply 4867f7e Corrected license a7fb650 merged the migration job into the orginal helm chart 4c0a014 entry in changelog and updated license in values d9b8b23 changed to get the latest docker image 9b552ad modified to use the updated docker image 9056d1f (origin/add_rollback) Merge pull request #297 from Cray-HPE/add_rollback-pending 9ec7920 (origin/feature/split-cm-migration) Merge pull request #292 from Cray-HPE/add_rollback Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Mitch Harding <[email protected]> Co-authored-by: Mukherjee <[email protected]> Co-authored-by: U-ASIAPACIFIC\nanjundl <[email protected]> Co-authored-by: anoop1402 <[email protected]> Co-authored-by: anoop1402 <[email protected]> Co-authored-by: lathanm <[email protected]> Co-authored-by: Ryan Haasken <[email protected]>
… to multiple ConfigMaps as pre-upgrade process Also includes: * CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the cray-product-catalog Kubernetes ConfigMap is split into multiple smaller ConfigMaps with each product's 'component_versions' data in its own ConfigMap. Modify the catalog_update.py script to update data across these ConfigMaps. * CASM-4427: Implement a prototype to have granular query from main and sub ConfigMaps * CASM-4368: Delete Cray Product Catalog details from main and sub ConfigMaps for a particular product version. * Added new argument max_attempts to modify_config_map function in catalog_delete.py, because we need not retry 100 times when read ConfigMap fails for a product ConfigMap. * CASM-4504: Added label 'type=cray-product-catalog' to all cray-product-catalog related ConfigMaps * Implemented migration of cray-product-catalog ConfigMap to multiple ConfigMaps as part of pre-upgrade steps * Added migration job to the configmap-hook 7e617ee CASM-4427: Combine ConfigMap data in the query output and CASM-3981: Schema update 4c46725 Replaced the data with items while mocking the data 2ccae06 modified the mock data 93a4ac5 modified the mock data fd4d6c7 Reverted prev change ebd0b6c Added one more mock data c8d9b42 Resolved error 684d867 Resolved error cbf8e04 Resolved failure bdb2bab Changed message in test_query.py as per query.py 52cc275 Resolved failure 062ee3e Removed mocking mock_loadConfigMapData output for 2 funcs ba9d0c3 Removed mocking mock_loadConfigMapData output for 2 funcs bc71071 Added mock_invalid_data 7192d1a syntax error b46bb82 message formating corrected b72c1a1 Aded one more invalid data b40b341 created mock obj with invalid yaml ca51fa5 created mock obj with invalid yaml 300f387 created mock obj with duplicate value df6487b Commented test case test_create_product_catalog_invalid_product_data ad09463 pycodestyle correction 5175e2f Fixed invalid yaml testcase and uncommented it c99ee1e Fixed pytest_mod issues 352223c Fixed typo error dd7f529 Addressed review comments c9ebd76 Fixed pymod test errors 46049f8 Merging PR 269 215cd0c Fixed pycodestyle 65b492c CASM-4368: Delete cray-product-catalog details from main and sub cm for a particular product version c1a3c9a fixed pycodestyle error 224530f CASM-4350: Changes in Update code path to accommodate Split config maps. 4bbde55 Modified message in CHANGELOG.md 465e84e Addressed review comments fee6117 Added try catch block for call create_namespaced_config_map 8e39b49 (origin/feature/catalog-delete) Addressed review comments a1fc575 (origin/casm-4427) Addressed all except review comments related to test case f9f3158 (origin/catalog-delete-ut) adding unit test cases for delete logic+making the code more testable 4029e41 Addressing the pr comments by mharding,21-07-2023 b81be8b Addressing the pr comments by abhishek,24-07-2023 542ea63 - Added positive and negative testcases for docker, s3, helm, manifests - Modified query to look for configmaps starting with the ConfigMap name passed as argument. If nothing is passed then ConfigMap name will be set to cray-product-catalog 2c0eb39 Fixed pymodd test errors 9fa89c0 Removed trailing space 09ef4cd Added condition to come out of infiinte loop while updating the ConfigMap data dc97b11 Reverting ConfigMap doesn't exist logic as per Ryan's comment c5e584d Resolved merge conflict by incorporating both suggestions. 1fd17f3 Merge pull request #273 from Cray-HPE/catalog-delete-ut 05c6214 Added test cases for catalog_update.py Modified test cases of test_data_catalog_helper.py to use unittest 153c6cb Fixed pymodtest errors 9285943 Fixed pycodestyle error 1c19cf2 Fixed pycodestyle error b28eda2 Merge branch 'develop' into feature/catalog-delete ee34158 Update cray_product_catalog/catalog_delete.py efac9bd Addressed review comments 7283217 Resolved merge conflict by incorporating both suggestions. 3edbaa6 Minor change in Changelog 5606651 Addressed some of Ryaan's review comments a5043ea Addressed Ryaan's review comments: - Added log message if attempts exceeds max_attempts(retries) - Added test case to test max retries - Added logic to exit if create_ config_map fails a266279 Fixed pycode style error c0ca0b7 Resolved merge conflict by incorporating both suggestions. cd4b443 Merge pull request #271 from Cray-HPE/feature/catalog-delete 8935515 Resolved merge conflict by incorporating both suggestions. 8c5908b Merge pull request #261 from Cray-HPE/casm-4427 6572b3d Resolved merge conflict by incorporating both suggestions. 2e61364 Merge pull request #272 from Cray-HPE/feature/casm-4350 3f0d940 CASM-4504: Add label type=cray-product-catalog for all cray-product-catalog related ConfigMaps a4aa7d2 - Addressed review comment - Fixed unit test failure due to addition of label_serlector argument to list_namespaced_config_map function b28ec2f Fixed pycodestyle error 1b9d965 (origin/CASM-4504-label) Fixed license error ddafa16 (origin/feature/split-cm-migration-dockerfile) migration init 57ebc86 Addressed review comments ae26488 Removed duplicate line ec8d935 Merge pull request #278 from Cray-HPE/CASM-4504-label 38a79df init roll back 652d1df migration init cd365f4 rebase changes 2638d1f merge from epic feature branch e5adc31 rollback and exit handler aa41547 binary cleanup 8184a2d CASM-4279: Add Kubernetes Api for create/delete/read/etc bb68076 Optimisation bab6191 Initial change to handle config map data split into multiple configmaps b1475d9 Merge pull request #290 from Cray-HPE/casm-4529-migration-api 9f8cf3c Merge branch 'feature/split-cm-migration' into migration/data-manager e0478aa Merge pull request #291 from Cray-HPE/migration/data-manager ade53cf Merge branch 'feature/split-cm-migration' into feature/split-cm-migration-rollback-exit e3cc918 Merge pull request #289 from Cray-HPE/feature/split-cm-migration-rollback-exit 02a21a8 Added rollback operation and modified rename_config_map function 1c94ba7 Fixed few corner case logic 6445ff0 Added unit test cases 3ebaa20 added rename_config_map test case- not working 73645c8 Added few more test cases to configmap data handler f343b23 ut fixes f61d384 Added unit test cases related to main.py 67bc931 Fixing pycode-style test error d4e704d Addressed review comments Added revoke and grant update permission to main ConfigMap d53c411 Updated Copyright license for setup.py e171abe Fixed UT issue cdf51bb Modified code to accept env var for name and namespace 47239ec Added check against resource_version 36c63ed Fixed pycode style error 35c6496 Modified backup restore to use kubectl create command instead of kubectl apply 4867f7e Corrected license a7fb650 merged the migration job into the orginal helm chart 4c0a014 entry in changelog and updated license in values d9b8b23 changed to get the latest docker image 9b552ad modified to use the updated docker image 9056d1f (origin/add_rollback) Merge pull request #297 from Cray-HPE/add_rollback-pending 9ec7920 (origin/feature/split-cm-migration) Merge pull request #292 from Cray-HPE/add_rollback Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Mitch Harding <[email protected]> Co-authored-by: Mukherjee <[email protected]> Co-authored-by: U-ASIAPACIFIC\nanjundl <[email protected]> Co-authored-by: anoop1402 <[email protected]> Co-authored-by: anoop1402 <[email protected]> Co-authored-by: lathanm <[email protected]> Co-authored-by: Ryan Haasken <[email protected]>
… to multiple ConfigMaps as pre-upgrade process Also includes: * CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the cray-product-catalog Kubernetes ConfigMap is split into multiple smaller ConfigMaps with each product's 'component_versions' data in its own ConfigMap. Modify the catalog_update.py script to update data across these ConfigMaps. * CASM-4427: Implement a prototype to have granular query from main and sub ConfigMaps * CASM-4368: Delete Cray Product Catalog details from main and sub ConfigMaps for a particular product version. * Added new argument max_attempts to modify_config_map function in catalog_delete.py, because we need not retry 100 times when read ConfigMap fails for a product ConfigMap. * CASM-4504: Added label 'type=cray-product-catalog' to all cray-product-catalog related ConfigMaps * Implemented migration of cray-product-catalog ConfigMap to multiple ConfigMaps as part of pre-upgrade steps * Added migration job to the configmap-hook 7e617ee CASM-4427: Combine ConfigMap data in the query output and CASM-3981: Schema update 4c46725 Replaced the data with items while mocking the data 2ccae06 modified the mock data 93a4ac5 modified the mock data fd4d6c7 Reverted prev change ebd0b6c Added one more mock data c8d9b42 Resolved error 684d867 Resolved error cbf8e04 Resolved failure bdb2bab Changed message in test_query.py as per query.py 52cc275 Resolved failure 062ee3e Removed mocking mock_loadConfigMapData output for 2 funcs ba9d0c3 Removed mocking mock_loadConfigMapData output for 2 funcs bc71071 Added mock_invalid_data 7192d1a syntax error b46bb82 message formating corrected b72c1a1 Aded one more invalid data b40b341 created mock obj with invalid yaml ca51fa5 created mock obj with invalid yaml 300f387 created mock obj with duplicate value df6487b Commented test case test_create_product_catalog_invalid_product_data ad09463 pycodestyle correction 5175e2f Fixed invalid yaml testcase and uncommented it c99ee1e Fixed pytest_mod issues 352223c Fixed typo error dd7f529 Addressed review comments c9ebd76 Fixed pymod test errors 46049f8 Merging PR 269 215cd0c Fixed pycodestyle 65b492c CASM-4368: Delete cray-product-catalog details from main and sub cm for a particular product version c1a3c9a fixed pycodestyle error 224530f CASM-4350: Changes in Update code path to accommodate Split config maps. 4bbde55 Modified message in CHANGELOG.md 465e84e Addressed review comments fee6117 Added try catch block for call create_namespaced_config_map 8e39b49 (origin/feature/catalog-delete) Addressed review comments a1fc575 (origin/casm-4427) Addressed all except review comments related to test case f9f3158 (origin/catalog-delete-ut) adding unit test cases for delete logic+making the code more testable 4029e41 Addressing the pr comments by mharding,21-07-2023 b81be8b Addressing the pr comments by abhishek,24-07-2023 542ea63 - Added positive and negative testcases for docker, s3, helm, manifests - Modified query to look for configmaps starting with the ConfigMap name passed as argument. If nothing is passed then ConfigMap name will be set to cray-product-catalog 2c0eb39 Fixed pymodd test errors 9fa89c0 Removed trailing space 09ef4cd Added condition to come out of infiinte loop while updating the ConfigMap data dc97b11 Reverting ConfigMap doesn't exist logic as per Ryan's comment c5e584d Resolved merge conflict by incorporating both suggestions. 1fd17f3 Merge pull request #273 from Cray-HPE/catalog-delete-ut 05c6214 Added test cases for catalog_update.py Modified test cases of test_data_catalog_helper.py to use unittest 153c6cb Fixed pymodtest errors 9285943 Fixed pycodestyle error 1c19cf2 Fixed pycodestyle error b28eda2 Merge branch 'develop' into feature/catalog-delete ee34158 Update cray_product_catalog/catalog_delete.py efac9bd Addressed review comments 7283217 Resolved merge conflict by incorporating both suggestions. 3edbaa6 Minor change in Changelog 5606651 Addressed some of Ryaan's review comments a5043ea Addressed Ryaan's review comments: - Added log message if attempts exceeds max_attempts(retries) - Added test case to test max retries - Added logic to exit if create_ config_map fails a266279 Fixed pycode style error c0ca0b7 Resolved merge conflict by incorporating both suggestions. cd4b443 Merge pull request #271 from Cray-HPE/feature/catalog-delete 8935515 Resolved merge conflict by incorporating both suggestions. 8c5908b Merge pull request #261 from Cray-HPE/casm-4427 6572b3d Resolved merge conflict by incorporating both suggestions. 2e61364 Merge pull request #272 from Cray-HPE/feature/casm-4350 3f0d940 CASM-4504: Add label type=cray-product-catalog for all cray-product-catalog related ConfigMaps a4aa7d2 - Addressed review comment - Fixed unit test failure due to addition of label_serlector argument to list_namespaced_config_map function b28ec2f Fixed pycodestyle error 1b9d965 (origin/CASM-4504-label) Fixed license error ddafa16 (origin/feature/split-cm-migration-dockerfile) migration init 57ebc86 Addressed review comments ae26488 Removed duplicate line ec8d935 Merge pull request #278 from Cray-HPE/CASM-4504-label 38a79df init roll back 652d1df migration init cd365f4 rebase changes 2638d1f merge from epic feature branch e5adc31 rollback and exit handler aa41547 binary cleanup 8184a2d CASM-4279: Add Kubernetes Api for create/delete/read/etc bb68076 Optimisation bab6191 Initial change to handle config map data split into multiple configmaps b1475d9 Merge pull request #290 from Cray-HPE/casm-4529-migration-api 9f8cf3c Merge branch 'feature/split-cm-migration' into migration/data-manager e0478aa Merge pull request #291 from Cray-HPE/migration/data-manager ade53cf Merge branch 'feature/split-cm-migration' into feature/split-cm-migration-rollback-exit e3cc918 Merge pull request #289 from Cray-HPE/feature/split-cm-migration-rollback-exit 02a21a8 Added rollback operation and modified rename_config_map function 1c94ba7 Fixed few corner case logic 6445ff0 Added unit test cases 3ebaa20 added rename_config_map test case- not working 73645c8 Added few more test cases to configmap data handler f343b23 ut fixes f61d384 Added unit test cases related to main.py 67bc931 Fixing pycode-style test error d4e704d Addressed review comments Added revoke and grant update permission to main ConfigMap d53c411 Updated Copyright license for setup.py e171abe Fixed UT issue cdf51bb Modified code to accept env var for name and namespace 47239ec Added check against resource_version 36c63ed Fixed pycode style error 35c6496 Modified backup restore to use kubectl create command instead of kubectl apply 4867f7e Corrected license a7fb650 merged the migration job into the orginal helm chart 4c0a014 entry in changelog and updated license in values d9b8b23 changed to get the latest docker image 9b552ad modified to use the updated docker image 9056d1f (origin/add_rollback) Merge pull request #297 from Cray-HPE/add_rollback-pending 9ec7920 (origin/feature/split-cm-migration) Merge pull request #292 from Cray-HPE/add_rollback Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Mitch Harding <[email protected]> Co-authored-by: Mukherjee <[email protected]> Co-authored-by: U-ASIAPACIFIC\nanjundl <[email protected]> Co-authored-by: anoop1402 <[email protected]> Co-authored-by: anoop1402 <[email protected]> Co-authored-by: lathanm <[email protected]> Co-authored-by: Ryan Haasken <[email protected]>
Summary and Scope
Migrate cray-product-catalog ConfigMap to multiple product ConfigMaps before upgrading cray-product-catalog to 1.10.0 which puts the cray-product-catalog data into main and associated product ConfigMap
Issues and Related PRs
List and characterize relationship to Jira/Github issues and other pull requests. Be sure to list dependencies.
<insert PR URL here>
Testing
Tested on local system and mug system as an independent python script
Tested on:
<mug-ncn-m001.hpc.amslabs.hpecorp.net>
Test description:
How were the changes tested and success verified? If schema changes were part of this change, how were those handled in your upgrade/downgrade testing?
Risks and Mitigations
Are there known issues with these changes? Any other special considerations?
Pull Request Checklist