Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #57 - change from pyfilesystem2 to fsspec #66

Merged
merged 4 commits into from
May 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file removed .DS_Store
Binary file not shown.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,5 @@ dmypy.json
# Sphinx
_build
/docs/api/

.DS_Store
3 changes: 2 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
include bw_processing/examples/simple.zip
include bw_processing/examples/*.zip
include bw_processing/examples/datapackage_1/*.parquet
recursive-exclude dev *
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,10 +107,10 @@ class ExampleArrayInterface:
Serialized datapackages cannot contain executable code, both because of our chosen data formats, and for security reasons. Therefore, when loading a datapackage with an interface, that interface object needs to be reconstituted as Python code - we call this cycle dehydration and rehydration. Dehydration happens automatically when a datapackage is finalized with `finalize_serialization()`, but rehydration needs to be done manually using `rehydrate_interface()`. For example:

```python
from fs.zipfs import ZipFS
from fsspec.implementations.zip import ZipFileSystem
from bw_processing import load_datapackage

my_dp = load_datapackage(ZipFS("some-path.zip"))
my_dp = load_datapackage(ZipFileSystem("some-path.zip"))
my_dp.rehydrate_interface("some-resource-name", ExampleVectorInterface())
```

Expand All @@ -119,7 +119,7 @@ You can list the dehydrated interfaces present with `.dehydrated_interfaces()`.
You can store useful information for the interface object initialization under the resource key `config`. This can be used in instantiating an interface if you pass `initialize_with_config`:

```python
from fs.zipfs import ZipFS
from fsspec.implementations.zip import ZipFileSystem
from bw_processing import load_datapackage
import requests
import numpy as np
Expand All @@ -133,7 +133,7 @@ class MyInterface:
return np.array(requests.get(self.url).json())


my_dp = load_datapackage(ZipFS("some-path.zip"))
my_dp = load_datapackage(ZipFileSystem("some-path.zip"))
data_obj, resource_metadata = my_dp.get_resource("some-interface")
print(resource_metadata['config'])
>>> {"url": "example.com"}
Expand Down
7 changes: 1 addition & 6 deletions bw_processing/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,7 @@


from .array_creation import create_array, create_structured_array
from .constants import (
DEFAULT_LICENSES,
INDICES_DTYPE,
UNCERTAINTY_DTYPE,
MatrixSerializeFormat,
)
from .constants import DEFAULT_LICENSES, INDICES_DTYPE, UNCERTAINTY_DTYPE, MatrixSerializeFormat
from .datapackage import (
Datapackage,
DatapackageBase,
Expand Down
Loading