diff --git a/_sources/part1/motivation.md b/_sources/part1/motivation.md index 8d031e4..88a7b08 100644 --- a/_sources/part1/motivation.md +++ b/_sources/part1/motivation.md @@ -12,7 +12,7 @@ Since August 2023, Pangeo is a [Numfocus](https://numfocus.org) sponsored projec ## Important information about Pangeo -- [Code of Conduct](https://github.com/pangeo-data/governance/blob/master/conduct/code_of_conduct.md +- [Code of Conduct](https://github.com/pangeo-data/governance/blob/master/conduct/code_of_conduct.md) - [Governance](https://github.com/pangeo-data/governance/blob/master/governance.md) - [Roadmap](https://pangeo.io/roadmap.html) diff --git a/part1/motivation.html b/part1/motivation.html index 1280b6c..ff83769 100644 --- a/part1/motivation.html +++ b/part1/motivation.html @@ -434,7 +434,7 @@

Introduction and Motivation

Important information about Pangeo#

diff --git a/part1/visualization.html b/part1/visualization.html index f91ebcd..2a0197f 100644 --- a/part1/visualization.html +++ b/part1/visualization.html @@ -971,7 +971,7 @@

Open local dataset @@ -1573,7 +1573,7 @@

Read a shapefile with the Area Of Interest (AOI) @@ -2197,11 +2197,11 @@

Visualization with HoloViews
-
+
- + @@ -607,8 +607,8 @@

Global LTS -
CPU times: user 367 ms, sys: 29 ms, total: 396 ms
-Wall time: 2.14 s
+
CPU times: user 300 ms, sys: 88.6 ms, total: 389 ms
+Wall time: 1.95 s
 

 

 

@@ -1000,9 +1000,9 @@ 
Global LTS
@@ -1081,7 +1081,7 @@ 
Global LTS
@@ -1160,7 +1160,7 @@ 
Global LTS
@@ -1239,7 +1239,7 @@ 
Global LTS
@@ -1318,7 +1318,7 @@ 
Global LTS
@@ -1397,7 +1397,7 @@ 
Global LTS
@@ -1476,7 +1476,7 @@ 
Global LTSopen_mfdataset automatically switch from Numpy Arrays to Dask Arrays as the data structure used by Xarray.

 

@@ -1926,9 +1926,9 @@ 
Using chunks within a single file
@@ -1985,7 +1985,7 @@ 
Using chunks within a single file
@@ -2042,7 +2042,7 @@ 
Using chunks within a single file
@@ -2099,7 +2099,7 @@ 
Using chunks within a single file
@@ -2156,7 +2156,7 @@ 
Using chunks within a single file
@@ -2213,7 +2213,7 @@ 
Using chunks within a single file
@@ -2270,7 +2270,7 @@ 
Using chunks within a single file
@@ -2688,7 +2688,7 @@ 
Using chunks within a single file(1121, 2240).  We will chunk it on 600x600 pieces using the following command.

 

@@ -3090,7 +3090,7 @@ 
Using chunks within a single file
@@ -3149,7 +3149,7 @@ 
Using chunks within a single file
-

+

 

 
 test.data is the backend array Python representation of Xarray’s Data Array, Dask Array when using chunking, Numpy by default.

 
We will introduce Dask arrays and Dask graphs visualization in the next section Scaling with Dask.

@@ -3224,7 +3224,7 @@ 
Zarr storage format
-
<xarray.backends.zarr.ZarrStore at 0x7f50ad9a1150>
+
<xarray.backends.zarr.ZarrStore at 0x7f1ed4cf5770>
 

 

 

@@ -3263,18 +3263,18 @@ 
Zarr storage format
 
total 48
-drwxr-xr-x 2 runner docker 4096 Nov  3 21:39 .
-drwxr-xr-x 5 runner docker 4096 Nov  3 21:39 ..
--rw-r--r-- 1 runner docker  341 Nov  3 21:39 .zarray
--rw-r--r-- 1 runner docker  304 Nov  3 21:39 .zattrs
--rw-r--r-- 1 runner docker 1482 Nov  3 21:39 0.0
--rw-r--r-- 1 runner docker 1482 Nov  3 21:39 0.1
--rw-r--r-- 1 runner docker 1482 Nov  3 21:39 0.2
--rw-r--r-- 1 runner docker 1482 Nov  3 21:39 0.3
--rw-r--r-- 1 runner docker 1482 Nov  3 21:39 1.0
--rw-r--r-- 1 runner docker 1482 Nov  3 21:39 1.1
--rw-r--r-- 1 runner docker 1482 Nov  3 21:39 1.2
--rw-r--r-- 1 runner docker 1482 Nov  3 21:39 1.3
+drwxr-xr-x 2 runner docker 4096 Nov  3 21:40 .
+drwxr-xr-x 5 runner docker 4096 Nov  3 21:40 ..
+-rw-r--r-- 1 runner docker  341 Nov  3 21:40 .zarray
+-rw-r--r-- 1 runner docker  304 Nov  3 21:40 .zattrs
+-rw-r--r-- 1 runner docker 1482 Nov  3 21:40 0.0
+-rw-r--r-- 1 runner docker 1482 Nov  3 21:40 0.1
+-rw-r--r-- 1 runner docker 1482 Nov  3 21:40 0.2
+-rw-r--r-- 1 runner docker 1482 Nov  3 21:40 0.3
+-rw-r--r-- 1 runner docker 1482 Nov  3 21:40 1.0
+-rw-r--r-- 1 runner docker 1482 Nov  3 21:40 1.1
+-rw-r--r-- 1 runner docker 1482 Nov  3 21:40 1.2
+-rw-r--r-- 1 runner docker 1482 Nov  3 21:40 1.3
 

 

 

@@ -3788,9 +3788,9 @@ 
Extract chunk information
@@ -3869,7 +3869,7 @@ 
Extract chunk information
@@ -3948,7 +3948,7 @@ 
Extract chunk information
@@ -4027,7 +4027,7 @@ 
Extract chunk information
@@ -4106,7 +4106,7 @@ 
Extract chunk information
@@ -4185,7 +4185,7 @@ 
Extract chunk information
@@ -4264,7 +4264,7 @@ 
Extract chunk information
working on  foss4g-data/CGLS_LTS_1999_2019/c_gls_NDVI-LTS_1999-2019-0721_GLOBE_VGT-PROBAV_V3.0.1.nc
 

 

-
CPU times: user 933 ms, sys: 306 ms, total: 1.24 s
-Wall time: 22.8 s
+
CPU times: user 933 ms, sys: 269 ms, total: 1.2 s
+Wall time: 23.1 s
 

 

 

@@ -4399,8 +4399,8 @@ 
We have 36 files to process, but for this chunking_introduction example, we
-
CPU times: user 27.3 ms, sys: 3.77 ms, total: 31.1 ms
-Wall time: 30.4 ms
+
CPU times: user 30.8 ms, sys: 3.24 ms, total: 34.1 ms
+Wall time: 33.1 ms
@@ -4424,8 +4424,8 @@

We have 36 files to process, but for this chunking_introduction example, we

-
CPU times: user 17.6 ms, sys: 248 µs, total: 17.9 ms
-Wall time: 236 ms
+
CPU times: user 17 ms, sys: 0 ns, total: 17 ms
+Wall time: 243 ms
 

 

 

@@ -4818,9 +4818,9 @@ 
We have 36 files to process, but for this chunking_introduction example, we
     source:               Derived from EO satellite imagery
     time_coverage_end:    2019-12-31T23:59:59Z
     time_coverage_start:  1999-01-01T00:00:00Z
-    title:                Normalized Difference Vegetation Index: Long Term S...
+ dtype='float64', name='lon', length=40320))
  • Conventions :
    CF-1.6
    archive_facility :
    VITO
    copyright :
    Copernicus Service information 2021
    history :
    2021-03-01 - Processing line NDVI LTS
    identifier :
    urn:cgls:global:ndvi_stats_all:NDVI-LTS_1999-2019-0701_GLOBE_V3.0.1
    institution :
    VITO NV
    long_name :
    Normalized Difference Vegetation Index
    orbit_type :
    LEO
    parent_identifier :
    urn:cgls:global:ndvi_stats_all
    platform :
    SPOT-4, SPOT-5, Proba-V
    processing_level :
    L4
    processing_mode :
    Offline
    product_version :
    V3.0.1
    references :
    https://land.copernicus.eu/global/products/ndvi
    sensor :
    VEGETATION-1, VEGETATION-2, VEGETATION
    source :
    Derived from EO satellite imagery
    time_coverage_end :
    2019-12-31T23:59:59Z
    time_coverage_start :
    1999-01-01T00:00:00Z
    title :
    Normalized Difference Vegetation Index: Long Term Statistics 1KM: GLOBE 1999-2019 0701

    • We can save the consolidated metadata for our dataset in a file, and reuse it later to access the dataset. We used json for next step, but we can also use parquet.

    @@ -5259,9 +5259,9 @@

    We have 36 files to process, but for this chunking_introduction example, we source: Derived from EO satellite imagery time_coverage_end: 2019-12-31T23:59:59Z time_coverage_start: 1999-01-01T00:00:00Z - title: Normalized Difference Vegetation Index: Long Term S... + dtype='float64', name='lon', length=40320))

  • Conventions :
    CF-1.6
    archive_facility :
    VITO
    copyright :
    Copernicus Service information 2021
    history :
    2021-03-01 - Processing line NDVI LTS
    identifier :
    urn:cgls:global:ndvi_stats_all:NDVI-LTS_1999-2019-0701_GLOBE_V3.0.1
    institution :
    VITO NV
    long_name :
    Normalized Difference Vegetation Index
    orbit_type :
    LEO
    parent_identifier :
    urn:cgls:global:ndvi_stats_all
    platform :
    SPOT-4, SPOT-5, Proba-V
    processing_level :
    L4
    processing_mode :
    Offline
    product_version :
    V3.0.1
    references :
    https://land.copernicus.eu/global/products/ndvi
    sensor :
    VEGETATION-1, VEGETATION-2, VEGETATION
    source :
    Derived from EO satellite imagery
    time_coverage_end :
    2019-12-31T23:59:59Z
    time_coverage_start :
    1999-01-01T00:00:00Z
    title :
    Normalized Difference Vegetation Index: Long Term Statistics 1KM: GLOBE 1999-2019 0701

    • The catalog (json file we created) can be shared on the cloud (or GitHub, etc.) and anyone can load it from there too.

    This approach allows anyone to easily access LTS data and select the Area of Interest for their own study.

    @@ -5692,11 +5692,11 @@

    We have 36 files to process, but for this chunking_introduction example, we source: Derived from EO satellite imagery time_coverage_end: 2019-12-31T23:59:59Z time_coverage_start: 1999-01-01T00:00:00Z - title: Normalized Difference Vegetation Index: Long Term S... + dtype='float64', name='time'))
  • Conventions :
    CF-1.6
    archive_facility :
    VITO
    copyright :
    Copernicus Service information 2021
    history :
    2021-03-01 - Processing line NDVI LTS
    identifier :
    urn:cgls:global:ndvi_stats_all:NDVI-LTS_1999-2019-0101_GLOBE_V3.0.1
    institution :
    VITO NV
    long_name :
    Normalized Difference Vegetation Index
    orbit_type :
    LEO
    parent_identifier :
    urn:cgls:global:ndvi_stats_all
    platform :
    SPOT-4, SPOT-5, Proba-V
    processing_level :
    L4
    processing_mode :
    Offline
    product_version :
    V3.0.1
    references :
    https://land.copernicus.eu/global/products/ndvi
    sensor :
    VEGETATION-1, VEGETATION-2, VEGETATION
    source :
    Derived from EO satellite imagery
    time_coverage_end :
    2019-12-31T23:59:59Z
    time_coverage_start :
    1999-01-01T00:00:00Z
    title :
    Normalized Difference Vegetation Index: Long Term Statistics 1KM: GLOBE 1999-2019 0101

    • The kerchunk catalogues can be placed in an intake catalogue, then loading multiple NetCDF file in the cloud can be just done in following 3 lines, chunked and fast.

    @@ -6119,11 +6119,11 @@

    We have 36 files to process, but for this chunking_introduction example, we source: Derived from EO satellite imagery time_coverage_end: 2019-12-31T23:59:59Z time_coverage_start: 1999-01-01T00:00:00Z - title: Normalized Difference Vegetation Index: Long Term S...