Skip to content

Commit

Permalink
deploy: fbf3f07
Browse files Browse the repository at this point in the history
  • Loading branch information
@tinaok
tinaok committed Oct 31, 2023
1 parent f3167e0 commit 7f32f2d
Show file tree
Hide file tree
Showing 12 changed files with 1,009 additions and 7,915 deletions.
Binary file removed _images/chunked.png
View file
Binary file not shown.
Binary file modified _images/datasize.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed _images/notchunked.png
View file
Binary file not shown.
553 changes: 404 additions & 149 deletions _sources/part3/chunking_introduction.ipynb
View file

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions _sources/part3/scaling_dask.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@
"## Context\n",
"\n",
"\n",
"We will be using [Dask](https://docs.dask.org/) with [Xarray](https://docs.xarray.dev/en/stable/) to parallelize our data analysis. The analysis is very similar to what we have done in previous episodes but this time we will use data on a global coverage that we read from a shared catalog (stored online in the Pangeo EOSC Openstack Object Storage).\n",
"We will be using [Dask](https://docs.dask.org/) with [Xarray](https://docs.xarray.dev/en/stable/) to parallelize our data analysis. We continue to use the snow index example. \n",
"\n",
"### Data\n",
"\n",
"In this episode, we will be using Global Long Term Statistics (1999-2019) product provided by the [Copernicus Global Land Service over Lombardia](https://land.copernicus.eu/global/index.html) and access them through [S3-comptabile storage](https://en.wikipedia.org/wiki/Amazon_S3) ([OpenStack Object Storage \"Swift\"](https://wiki.openstack.org/wiki/Swift)) with a data catalog we have created and made publicly available."
"In this episode, we will be using xxx"
]
},
{
Expand Down Expand Up @@ -264,19 +264,19 @@
"source": [
"### Dask Dashboard\n",
"\n",
"Dask comes with a really handy interface: the Dask Dashboard. It is a web interface that you can open in a separated tab of your browser (but not with he link above, you've got to use Jupyterlabs proxy: https://pangeo-foss4g.vm.fedcloud.eu/jupyterhub/user/_yourusername_/proxy/8787/status).\n",
"Dask comes with a really handy interface: the Dask Dashboard. It is a web interface that you can open in a separated tab of your browser (but not with he link above, you've got to use Jupyterlabs proxy: /user/todaka/proxy/8787/status).\n",
"\n",
"We will learn here how to use it through [dask jupyterlab extension](https://github.com/dask/dask-labextension). \n",
"\n",
"To use Dask Dashboard through jupyterlab extension on Pangeo EOSC FOSS4G infrastructure,\n",
"To use Dask Dashboard through jupyterlab extension on Pangeo EOSC infrastructure,\n",
"you will just need too look at the html link you have for your jupyterlab, and Dask dashboard port number, as highlighted in the figure below.\n",
"\n",
"\n",
"<img src=\"../figures/dashboardlink.png\" width=\"50%\">\n",
"<img src=\"../figures/dasklab.png\" width=\"30%\">\n",
"<img src=\"../figures/dasklab.png\" width=\"80%\">\n",
"\n",
"Then click the orange icon indicated in the above figure, and type 'your' dashboard link (normally, you just need to replace 'todaka' to 'your username'). \n",
"\n",
"<img src=\"../figures/daskclick.png\" width=\"30%\">\n",
"\n",
"\n",
"\n",
Expand Down
10 changes: 5 additions & 5 deletions part1/visualization.html
View file

Large diffs are not rendered by default.

170 changes: 85 additions & 85 deletions part1/xarray_pitfalls.html
View file

Large diffs are not rendered by default.

7,975 changes: 391 additions & 7,584 deletions part3/chunking_introduction.html
View file

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions part3/data_exploitability_pangeo.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1100,7 +1100,7 @@ <h3>Import libraries<a class="headerlink" href="#import-libraries" title="Permal
<div style="width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<h3 style="margin-bottom: 0px;">Client</h3>
<p style="color: #9D9D9D; margin-bottom: 0px;">Client-13a5a6af-7802-11ee-8e86-000d3ae50e51</p>
<p style="color: #9D9D9D; margin-bottom: 0px;">Client-edaff6c9-7807-11ee-8ef1-002248352ce8</p>
<table style="width: 100%; text-align: left;">

<tr>
Expand Down Expand Up @@ -1135,7 +1135,7 @@ <h3 style="margin-bottom: 0px;">Client</h3>
</div>
<div style="margin-left: 48px;">
<h3 style="margin-bottom: 0px; margin-top: 0px;">LocalCluster</h3>
<p style="color: #9D9D9D; margin-bottom: 0px;">34936e63</p>
<p style="color: #9D9D9D; margin-bottom: 0px;">b7c274aa</p>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
Expand Down Expand Up @@ -1172,11 +1172,11 @@ <h3 style="display: inline;">Scheduler Info</h3>
<div style="width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<h3 style="margin-bottom: 0px;">Scheduler</h3>
<p style="color: #9D9D9D; margin-bottom: 0px;">Scheduler-68489d21-2ff0-432e-af6d-08835991d0a1</p>
<p style="color: #9D9D9D; margin-bottom: 0px;">Scheduler-96c2e906-39d5-4c32-b5f3-bbe085230c15</p>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm:</strong> tcp://127.0.0.1:37291
<strong>Comm:</strong> tcp://127.0.0.1:34541
</td>
<td style="text-align: left;">
<strong>Workers:</strong> 2
Expand Down Expand Up @@ -1218,29 +1218,29 @@ <h4 style="margin-bottom: 0px; display: inline;">Worker: 0</h4>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm: </strong> tcp://127.0.0.1:43833
<strong>Comm: </strong> tcp://127.0.0.1:37157
</td>
<td style="text-align: left;">
<strong>Total threads: </strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://127.0.0.1:35453/status" target="_blank">http://127.0.0.1:35453/status</a>
<strong>Dashboard: </strong> <a href="http://127.0.0.1:36441/status" target="_blank">http://127.0.0.1:36441/status</a>
</td>
<td style="text-align: left;">
<strong>Memory: </strong> 3.38 GiB
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Nanny: </strong> tcp://127.0.0.1:32889
<strong>Nanny: </strong> tcp://127.0.0.1:45265
</td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td colspan="2" style="text-align: left;">
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-pef2v7g6
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-4rq4mjpx
</td>
</tr>

Expand All @@ -1263,29 +1263,29 @@ <h4 style="margin-bottom: 0px; display: inline;">Worker: 1</h4>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm: </strong> tcp://127.0.0.1:46063
<strong>Comm: </strong> tcp://127.0.0.1:38709
</td>
<td style="text-align: left;">
<strong>Total threads: </strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://127.0.0.1:40309/status" target="_blank">http://127.0.0.1:40309/status</a>
<strong>Dashboard: </strong> <a href="http://127.0.0.1:40257/status" target="_blank">http://127.0.0.1:40257/status</a>
</td>
<td style="text-align: left;">
<strong>Memory: </strong> 3.38 GiB
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Nanny: </strong> tcp://127.0.0.1:32869
<strong>Nanny: </strong> tcp://127.0.0.1:41953
</td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td colspan="2" style="text-align: left;">
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-omfk_kjm
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-th2368qu
</td>
</tr>

Expand Down
Loading

0 comments on commit 7f32f2d

Please sign in to comment.