Use watchdog timeout to catalog properties #48
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ericvergnaud
temporarily deployed
to
account-admin
GitHub Actions
Inactive
|
This PR breaks backwards compatibility for databrickslabs/blueprint downstream. See build logs for more details. Running from downstreams #28 |
|
This PR breaks backwards compatibility for databrickslabs/ucx downstream. See build logs for more details. Running from downstreams #23 |
|
✅ 35/35 passed, 3 skipped, 7m52s total Running from acceptance #69 |
ericvergnaud
temporarily deployed
to
account-admin
GitHub Actions
Inactive
ericvergnaud
changed the title
ericvergnaud
temporarily deployed
to
account-admin
GitHub Actions
Inactive
Context: review conversations from databrickslabs/ucx#2388 |
nfx
changed the title
# Conflicts: # README.md
ericvergnaud
temporarily deployed
to
account-admin
GitHub Actions
Inactive
nfx
added a commit
that referenced
this pull request
Sep 19, 2024
* Use watchdog timeout to catalog properties ([#48](#48)). This pull request introduces a new `RemoveAfter` property for catalogs, which allows for marking them for skipping by the watchdog. This change addresses the current implementation gap, which does not explicitly indicate when catalogs are being used. The new property will specify the time from which objects can be purged. A corresponding fixture `watchdog_remove_after` has been added to the list of available fixtures, and the `make_catalog` fixture has been updated to include this new property. Additionally, a timeout mechanism for catalogs has been implemented, which improves the system's efficiency and safety by marking catalogs as in use. A test for the `make_catalog` function has been included to ensure that the `RemoveAfter` entry is correctly added to the catalog properties. However, the specific call parameters for the `catalogs.create` method cannot be accurately determined in the test. * use tags instead of name suffix for queries ([#47](#47)). This release introduces updates to the testing library for Databricks, enhancing the naming conventions for queries to improve readability and comprehension. The previous implementation used name suffixes, which have been replaced with watchdog query tags. The `watchdog_purge_suffix` fixture has been renamed to `watchdog_remove_after`, and the new `make_query` fixture has been added to the documentation. In addition, the `make_query` and `create` functions now accept an optional `tags` argument, and the query name is generated with a unique identifier. If `tags` are provided, the `RemoveAfter` tag is added. The `original_query_tag` is no longer hardcoded in the `create` function and has been removed. These changes improve the overall user experience and maintainability of the project.
Merged
nfx
added a commit
that referenced
this pull request
Sep 19, 2024
* Use watchdog timeout to catalog properties ([#48](#48)). This pull request introduces a new `RemoveAfter` property for catalogs, which allows for marking them for skipping by the watchdog. This change addresses the current implementation gap, which does not explicitly indicate when catalogs are being used. The new property will specify the time from which objects can be purged. A corresponding fixture `watchdog_remove_after` has been added to the list of available fixtures, and the `make_catalog` fixture has been updated to include this new property. Additionally, a timeout mechanism for catalogs has been implemented, which improves the system's efficiency and safety by marking catalogs as in use. A test for the `make_catalog` function has been included to ensure that the `RemoveAfter` entry is correctly added to the catalog properties. However, the specific call parameters for the `catalogs.create` method cannot be accurately determined in the test. * use tags instead of name suffix for queries ([#47](#47)). This release introduces updates to the testing library for Databricks, enhancing the naming conventions for queries to improve readability and comprehension. The previous implementation used name suffixes, which have been replaced with watchdog query tags. The `watchdog_purge_suffix` fixture has been renamed to `watchdog_remove_after`, and the new `make_query` fixture has been added to the documentation. In addition, the `make_query` and `create` functions now accept an optional `tags` argument, and the query name is generated with a unique identifier. If `tags` are provided, the `RemoveAfter` tag is added. The `original_query_tag` is no longer hardcoded in the `create` function and has been removed. These changes improve the overall user experience and maintainability of the project.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Current implementation does not mark catalogs as being in use for the watchdog to skip.
This PR fills the gap by adding a RemoveAfter entry to catalog properties
Linked issues
PR https://github.com/databrickslabs/watchdog/pull/58
Tests
Not tested