Skip to content

Commit

Permalink
Merge pull request #66 from brightway-lca/fsspec
Browse files Browse the repository at this point in the history
Fix #57 - change from `pyfilesystem2` to `fsspec`
  • Loading branch information
cmutel authored May 28, 2024
2 parents 3185f0c + e5f141d commit 0e3c890
Show file tree
Hide file tree
Showing 31 changed files with 252 additions and 519 deletions.
Binary file removed .DS_Store
Binary file not shown.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,5 @@ dmypy.json
# Sphinx
_build
/docs/api/

.DS_Store
3 changes: 2 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
include bw_processing/examples/simple.zip
include bw_processing/examples/*.zip
include bw_processing/examples/datapackage_1/*.parquet
recursive-exclude dev *
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,10 +107,10 @@ class ExampleArrayInterface:
Serialized datapackages cannot contain executable code, both because of our chosen data formats, and for security reasons. Therefore, when loading a datapackage with an interface, that interface object needs to be reconstituted as Python code - we call this cycle dehydration and rehydration. Dehydration happens automatically when a datapackage is finalized with `finalize_serialization()`, but rehydration needs to be done manually using `rehydrate_interface()`. For example:

```python
from fs.zipfs import ZipFS
from fsspec.implementations.zip import ZipFileSystem
from bw_processing import load_datapackage

my_dp = load_datapackage(ZipFS("some-path.zip"))
my_dp = load_datapackage(ZipFileSystem("some-path.zip"))
my_dp.rehydrate_interface("some-resource-name", ExampleVectorInterface())
```

Expand All @@ -119,7 +119,7 @@ You can list the dehydrated interfaces present with `.dehydrated_interfaces()`.
You can store useful information for the interface object initialization under the resource key `config`. This can be used in instantiating an interface if you pass `initialize_with_config`:

```python
from fs.zipfs import ZipFS
from fsspec.implementations.zip import ZipFileSystem
from bw_processing import load_datapackage
import requests
import numpy as np
Expand All @@ -133,7 +133,7 @@ class MyInterface:
return np.array(requests.get(self.url).json())


my_dp = load_datapackage(ZipFS("some-path.zip"))
my_dp = load_datapackage(ZipFileSystem("some-path.zip"))
data_obj, resource_metadata = my_dp.get_resource("some-interface")
print(resource_metadata['config'])
>>> {"url": "example.com"}
Expand Down
7 changes: 1 addition & 6 deletions bw_processing/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,7 @@


from .array_creation import create_array, create_structured_array
from .constants import (
DEFAULT_LICENSES,
INDICES_DTYPE,
UNCERTAINTY_DTYPE,
MatrixSerializeFormat,
)
from .constants import DEFAULT_LICENSES, INDICES_DTYPE, UNCERTAINTY_DTYPE, MatrixSerializeFormat
from .datapackage import (
Datapackage,
DatapackageBase,
Expand Down
Loading

0 comments on commit 0e3c890

Please sign in to comment.