Fix `.as_uri()` and `.absolute()` implementations for `WorkspacePath` #126

asnare · 2024-07-12T16:36:49Z

This PR is stacked on #122 and fixes some things that the integration tests expect but weren't covered by unit tests:

.as_uri(), which internally depends on the __bytes__() dunder method.
.absolute(), which is supported in the trivial (no-op) case.

Both underlying implementations depend on PathLib internals that changed across python versions.

With this PR applied, some integrations still fail due to missing globbing support, which is implemented as #125.

…ore functionality. Work-in-progress: not yet complete.

… worse than the problem.

The code as-is is temporary.

This will mean it's clearer in the PR what has changed.

Instead modify behaviour via the constructor-injected mock.

Previously to produce the desired results these tests mutated the internal state of the object under test.

This is expected by the integration tests.

github-actions · 2024-07-12T17:49:24Z

❌ 16/18 passed, 2 failed, 2 skipped, 29s total

❌ test_open_text_io: AttributeError: 'object' object has no attribute 'parse_parts' (1.308s)

AttributeError: 'object' object has no attribute 'parse_parts'
17:49 DEBUG [databricks.sdk] Loaded from environment
17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication
[gw6] linux -- Python 3.10.14 /home/runner/work/blueprint/blueprint/.venv/bin/python
17:49 DEBUG [databricks.sdk] Loaded from environment
17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:49 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me
< 200 OK
< {
<   "active": true,
<   "displayName": "labs-runtime-identity",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "**REDACTED**"
<     }
<   ],
<   "externalId": "d0f9bd2c-5651-45fd-b648-12a3fc6375c4",
<   "groups": [
<     {
<       "$ref": "Groups/300667344111082",
<       "display": "labs.scope.runtime",
<       "type": "direct",
<       "value": "**REDACTED**"
<     }
<   ],
<   "id": "4643477475987733",
<   "name": {
<     "givenName": "labs-runtime-identity"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "... (1 additional elements)"
<   ],
<   "userName": "4106dc97-a963-48f0-a079-a578238959a6"
< }
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c"
> }
< 200 OK
< {}
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 200 OK
< {
<   "object_id": 3200206029686177
< }
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 400 Bad Request
< {
<   "error_code": "RESOURCE_ALREADY_EXISTS",
<   "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c/hello.txt) already exis... (3 more bytes)"
< }
17:49 DEBUG [databricks.sdk] GET /api/2.0/workspace/export?path=/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c/hello.txt&direct_download=true&format=AUTO
< 200 OK
< [raw stream]
17:49 DEBUG [databricks.sdk] Loaded from environment
17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:49 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me
< 200 OK
< {
<   "active": true,
<   "displayName": "labs-runtime-identity",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "**REDACTED**"
<     }
<   ],
<   "externalId": "d0f9bd2c-5651-45fd-b648-12a3fc6375c4",
<   "groups": [
<     {
<       "$ref": "Groups/300667344111082",
<       "display": "labs.scope.runtime",
<       "type": "direct",
<       "value": "**REDACTED**"
<     }
<   ],
<   "id": "4643477475987733",
<   "name": {
<     "givenName": "labs-runtime-identity"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "... (1 additional elements)"
<   ],
<   "userName": "4106dc97-a963-48f0-a079-a578238959a6"
< }
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c"
> }
< 200 OK
< {}
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 200 OK
< {
<   "object_id": 3200206029686177
< }
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 400 Bad Request
< {
<   "error_code": "RESOURCE_ALREADY_EXISTS",
<   "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c/hello.txt) already exis... (3 more bytes)"
< }
17:49 DEBUG [databricks.sdk] GET /api/2.0/workspace/export?path=/Users/4106dc97-a963-48f0-a079-a578238959a6/nCQeJ3P03qkoN4m4/a/b/c/hello.txt&direct_download=true&format=AUTO
< 200 OK
< [raw stream]
[gw6] linux -- Python 3.10.14 /home/runner/work/blueprint/blueprint/.venv/bin/python

❌ test_file_and_notebook_in_same_folder_with_different_suffixes: AttributeError: 'object' object has no attribute 'parse_parts' (897ms)

AttributeError: 'object' object has no attribute 'parse_parts'
17:49 DEBUG [databricks.sdk] Loaded from environment
17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication
[gw7] linux -- Python 3.10.14 /home/runner/work/blueprint/blueprint/.venv/bin/python
17:49 DEBUG [databricks.sdk] Loaded from environment
17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:49 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me
< 200 OK
< {
<   "active": true,
<   "displayName": "labs-runtime-identity",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "**REDACTED**"
<     }
<   ],
<   "externalId": "d0f9bd2c-5651-45fd-b648-12a3fc6375c4",
<   "groups": [
<     {
<       "$ref": "Groups/300667344111082",
<       "display": "labs.scope.runtime",
<       "type": "direct",
<       "value": "**REDACTED**"
<     }
<   ],
<   "id": "4643477475987733",
<   "name": {
<     "givenName": "labs-runtime-identity"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "... (1 additional elements)"
<   ],
<   "userName": "4106dc97-a963-48f0-a079-a578238959a6"
< }
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg"
> }
< 200 OK
< {}
17:49 DEBUG [databricks.labs.blueprint.tests] added directory fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 200 OK
< {
<   "object_id": 3200206029686206
< }
17:49 DEBUG [databricks.labs.blueprint.tests] added notebook fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 200 OK
< {
<   "object_id": 3200206029686208
< }
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 400 Bad Request
< {
<   "error_code": "RESOURCE_ALREADY_EXISTS",
<   "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/a.txt) already exists."
< }
17:49 DEBUG [databricks.sdk] Loaded from environment
17:49 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
17:49 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
17:49 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:49 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me
< 200 OK
< {
<   "active": true,
<   "displayName": "labs-runtime-identity",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "**REDACTED**"
<     }
<   ],
<   "externalId": "d0f9bd2c-5651-45fd-b648-12a3fc6375c4",
<   "groups": [
<     {
<       "$ref": "Groups/300667344111082",
<       "display": "labs.scope.runtime",
<       "type": "direct",
<       "value": "**REDACTED**"
<     }
<   ],
<   "id": "4643477475987733",
<   "name": {
<     "givenName": "labs-runtime-identity"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "... (1 additional elements)"
<   ],
<   "userName": "4106dc97-a963-48f0-a079-a578238959a6"
< }
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg"
> }
< 200 OK
< {}
17:49 DEBUG [databricks.labs.blueprint.tests] added directory fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 200 OK
< {
<   "object_id": 3200206029686206
< }
17:49 DEBUG [databricks.labs.blueprint.tests] added notebook fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 200 OK
< {
<   "object_id": 3200206029686208
< }
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/import
> [raw stream]
< 400 Bad Request
< {
<   "error_code": "RESOURCE_ALREADY_EXISTS",
<   "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/a.txt) already exists."
< }
17:49 DEBUG [databricks.labs.blueprint.tests] clearing 1 directory fixtures
17:49 DEBUG [databricks.labs.blueprint.tests] removing directory fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/delete
> {
>   "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg",
>   "recursive": true
> }
< 200 OK
< {}
17:49 DEBUG [databricks.labs.blueprint.tests] clearing 1 notebook fixtures
17:49 DEBUG [databricks.labs.blueprint.tests] removing notebook fixture: /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b
17:49 DEBUG [databricks.sdk] POST /api/2.0/workspace/delete
> {
>   "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b"
> }
< 404 Not Found
< {
<   "error_code": "RESOURCE_DOES_NOT_EXIST",
<   "message": "Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b) doesn't exist."
< }
17:49 DEBUG [databricks.labs.blueprint.tests] ignoring error while notebook /Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b teardown: Path (/Users/4106dc97-a963-48f0-a079-a578238959a6/sdk-bSYg/b) doesn't exist.
[gw7] linux -- Python 3.10.14 /home/runner/work/blueprint/blueprint/.venv/bin/python

_{Running from acceptance #178}

asnare · 2024-07-12T18:12:10Z

Rebased (and superseded) by #127.

…#127) This PR is a rebase that replaces #126 and fixes some things that the integration tests expect but weren't covered by unit tests: - `.as_uri()`, which internally depends on the `__bytes__()` dunder method. - `.absolute()`, which is supported in the trivial (no-op) case. Both underlying implementations depend on PathLib internals that changed across python versions.

asnare added 24 commits July 9, 2024 14:52

Future self-type annotations.

7bbe5c4

Enable type checking for the existing tests and fix problems.

261c67c

Update internals to (mostly) work across Python 3.10-3.12, and test m…

25c50c4

…ore functionality. Work-in-progress: not yet complete.

Update project metadata to reflect Python 3.12 support.

8065bc5

Fix typo.

f9fc5ab

Fix broken super() call.

4ec3ff7

Formatting.

39c80db

Adjust code to satisfy the linter.

c8fab61

Suppress linting warnings that are unavoidable, or where avoidance is…

4c7ef40

… worse than the problem.

Mark some code where a decision is pending.

df39660

The code as-is is temporary.

Factor out init logic into a method.

7a1e450

Early return.

b536d3f

Support for /-style building of paths.

af2f118

Early return.

afad4d7

Replace relative_to/is_relative_to implementation to use our internals.

f880a4e

Remove methods that only delegate to the superclass.

e66804e

Drop type annotations to reduce changes in this PR.

c89b4fa

Whitespace.

b94ad64

Revert to original .suffix implementation.

fde8a7f

Move test to site of prior test with the same name.

1f5f138

This will mean it's clearer in the PR what has changed.

Avoid monkey-patching the class under test.

5cf0b71

Instead modify behaviour via the constructor-injected mock.

Adjust text fixture to induce behaviour via constructor-injected mock.

3c970d9

Previously to produce the desired results these tests mutated the internal state of the object under test.

Document why __fspath__() isn't supported.

1f53446

Fix as_uri() implementation by ensuring __bytes__() works.

a547e40

asnare added the bug Something isn't working label Jul 12, 2024

asnare self-assigned this Jul 12, 2024

asnare changed the base branch from main to feature/python-312-wspath July 12, 2024 16:37

asnare added 2 commits July 12, 2024 19:41

Unit test for .as_uri() implementation.

a74499a

Add a partial implementation of .absolute().

29922e6

This is expected by the integration tests.

asnare changed the title ~~Fix .as_uri() implementation for WorkspacePath~~ Fix .as_uri() and .absolute() implementations for WorkspacePath Jul 12, 2024

asnare marked this pull request as ready for review July 12, 2024 17:48

asnare requested a review from nfx as a code owner July 12, 2024 17:48

asnare had a problem deploying to runtime July 12, 2024 17:48 — with GitHub Actions Failure

Base automatically changed from feature/python-312-wspath to main July 12, 2024 17:54

asnare mentioned this pull request Jul 12, 2024

Fix .as_uri() and .absolute() implementations for WorkspacePath #127

Merged

asnare closed this Jul 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `.as_uri()` and `.absolute()` implementations for `WorkspacePath` #126

Fix `.as_uri()` and `.absolute()` implementations for `WorkspacePath` #126

asnare commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024

asnare commented Jul 12, 2024

Fix .as_uri() and .absolute() implementations for WorkspacePath #126

Fix .as_uri() and .absolute() implementations for WorkspacePath #126

Conversation

asnare commented Jul 12, 2024 • edited Loading

github-actions bot commented Jul 12, 2024

asnare commented Jul 12, 2024

Fix `.as_uri()` and `.absolute()` implementations for `WorkspacePath` #126

Fix `.as_uri()` and `.absolute()` implementations for `WorkspacePath` #126

asnare commented Jul 12, 2024 •

edited

Loading