notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application Package reproducibility\n",
    "\n",
    "### Personas\n",
    "\n",
    "* **Alice** developed a Water Body detection Earth Observation application and package it as an EO Application Package\n",
    "* **Bob** scripts the execution of application\n",
    "\n",
    "### Scenario\n",
    "\n",
    "Alice included in the water bodies detection Application Package software repository a Continuous Integration configuration relying on Github Actions to:\n",
    "\n",
    "* build the containers\n",
    "* push the built containers to Github container registry\n",
    "* update the Application Package with these new container references\n",
    "* push the updated Application Package to Github's artifact registry\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alice sent an email to Bob:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "from: alice@acme.io\n",
    "\n",
    "to: bob@acme.io\n",
    "\n",
    "subject: Detecting water bodies with NDWI and the Otsu threshold\n",
    "\n",
    "\n",
    "Hi Bob!\n",
    "\n",
    "checkout my new application package for detecting water bodies using NDWI and the Ostu threshold.\n",
    "\n",
    "I've ran it over our test site bounding box and prelimanry result look promising.\n",
    "\n",
    "The github repo is https://github.com/Terradue/app-package-training-bids23 and I've just released version 1.0.0.\n",
    "\n",
    "Let me know!\n",
    "\n",
    "Cheers\n",
    "\n",
    "Alice\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this information, Bob scripts the Application Execution in a Jupyter Notebook.\n",
    "\n",
    "His environment has a container engine (e.g. podman or docker) and the cwltool CWL runner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the Scenario"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import argparse\n",
    "import asyncio\n",
    "import json\n",
    "import os\n",
    "from datetime import datetime\n",
    "from io import StringIO\n",
    "\n",
    "import nest_asyncio\n",
    "import pystac\n",
    "import rasterio\n",
    "from cwltool.main import main\n",
    "from ipyleaflet import GeoJSON, Map\n",
    "from pydantic_yaml import to_yaml_str\n",
    "from pystac_client import Client\n",
    "from rasterio.features import dataset_features, sieve\n",
    "\n",
    "from helpers import Params, get_param_model_fields, get_release_assets, stage_in\n",
    "\n",
    "from shutil import which\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check the container engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if which(\"podman\"):\n",
    "    podman = True\n",
    "elif which(\"docker\"):\n",
    "    podman = False\n",
    "else:\n",
    "    raise ValueError(\"No container engine\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application Package releases\n",
    "\n",
    "Bob uses Github API to list the artifacts published by Alice in the release"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assets = get_release_assets(\n",
    "    user=\"Terradue\",\n",
    "    repo=\"app-package-training-bids23\",\n",
    "    token=os.environ[\"GH_PAT\"],\n",
    ")\n",
    "\n",
    "assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the Application Package to detect water bodies on Sentinel-2 data\n",
    "\n",
    "Alice published three Application Packages.\n",
    "\n",
    " Bob selects the one processing several Sentinel-2 acquisitions provided as STAC Items\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_package = assets[\"1.0.0\"][0]\n",
    "\n",
    "print(app_package[\"doc\"])\n",
    "\n",
    "print(app_package[\"url\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Application Package parameters are discovered and a pydantic model is created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Params.set_fields(**get_param_model_fields(cwl_obj=app_package[\"cwl\"]))\n",
    "\n",
    "Params.get_fields()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Application Package takes as inputs:\n",
    "- one or more STAC Items\n",
    "- a list of the bands for the normalized difference\n",
    "- an area of interest\n",
    "- the EPSG code used for the area of interest coordinates\n",
    "\n",
    " Bob uses a STAC API endpoint to discover Sentinel-2 acquisitions over an area of interest and time of interest "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = \"https://earth-search.aws.element84.com/v1/\"\n",
    "\n",
    "headers = []\n",
    "\n",
    "cat = Client.open(URL, headers=headers)\n",
    "cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bod defines the search parameter and get the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Collection\n",
    "collections = [\"sentinel-2-l2a\"]\n",
    "\n",
    "# Start and end dates\n",
    "start_date = datetime.fromisoformat(\"2021-07-08T00:00:00\")\n",
    "stop_date = datetime.fromisoformat(\"2021-07-08T23:59:59\")\n",
    "\n",
    "bbox = [-121.399, 39.834, -120.74, 40.472]\n",
    "\n",
    "# Other metadata\n",
    "cloud_cover = 5\n",
    "\n",
    "# Query by AOI, start and end date and other params\n",
    "query = cat.search(\n",
    "    collections=collections,\n",
    "    datetime=(start_date, stop_date),\n",
    "    bbox=bbox,\n",
    "    query={\"eo:cloud_cover\": {\"lt\": cloud_cover}},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob plots the Sentinel-2 discovered STAC Items footprint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "center = ((bbox[1] + bbox[3]) / 2, (bbox[0] + bbox[2]) / 2)\n",
    "\n",
    "m = Map(center=center, zoom=8)\n",
    "\n",
    "for item in list(query.item_collection()):\n",
    "    geo_json = GeoJSON(\n",
    "        name=item.id,\n",
    "        data=item.geometry,\n",
    "        style={\n",
    "            \"opacity\": 1,\n",
    "            \"dashArray\": \"9\",\n",
    "            \"fillOpacity\": 0.1,\n",
    "            \"weight\": 1,\n",
    "            \"color\": \"blue\",\n",
    "        },\n",
    "        hover_style={\"color\": \"white\", \"dashArray\": \"0\", \"fillOpacity\": 0.5},\n",
    "    )\n",
    "    m.add_layer(geo_json)\n",
    "\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob lists the STAC Items self link, these are the URLs to the Sentinel-2 STAC Items to process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[item.get_self_href() for item in list(query.item_collection())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And creates the parameters for running the Application Package (the epsg and bands input parameters have default values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = Params(\n",
    "    aoi=\",\".join([str(elem) for elem in bbox]),\n",
    "    stac_items=[item.self_href for item in query.item_collection()],\n",
    "    epsg=\"EPSG:4326\",\n",
    "    bands=[\"green\", \"nir\"],\n",
    ")\n",
    "\n",
    "params.dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob writes a YAML file with the parameters and their values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"params-s2.yaml\", \"w\") as file:\n",
    "    print(to_yaml_str(params), file=file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file `params.yaml` contains:\n",
    "\n",
    "```yaml\n",
    "aoi: -121.399,39.834,-120.74,40.472\n",
    "bands:\n",
    "- green\n",
    "- nir\n",
    "epsg: EPSG:4326\n",
    "stac_items:\n",
    "- https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_0_L2A\n",
    "- https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_1_L2A\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob uses the CWL runner `cwltool` Python API to script the Application Package execution "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed_args = argparse.Namespace(\n",
    "    podman=podman,\n",
    "    parallel=True,\n",
    "    debug=False,\n",
    "    outdir=\"./runs\",\n",
    "    workflow=app_package[\"url\"],\n",
    "    job_order=[\"params-s2.yaml\"],\n",
    ")\n",
    "\n",
    "stream_out = StringIO()\n",
    "stream_err = StringIO()\n",
    "\n",
    "res = main(\n",
    "    args=parsed_args,\n",
    "    stdout=stream_out,\n",
    ")\n",
    "\n",
    "assert res == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This execution generates as output a JSON file listing all files produced.\n",
    "\n",
    "The JSON contains the output defined in the CWL workflow that can be accessed with: \n",
    "\n",
    "```python\n",
    "os.path.basename(app_package[\"cwl\"].outputs[0].id)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = json.loads(stream_out.getvalue())\n",
    "\n",
    "results[os.path.basename(app_package[\"cwl\"].outputs[0].id)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob writes a simple code to find the STAC Catalog path and then list the contents of that STAC Catalog:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat = pystac.read_file(\n",
    "    [\n",
    "        listing[\"path\"]\n",
    "        for listing in results[os.path.basename(app_package[\"cwl\"].outputs[0].id)][\n",
    "            \"listing\"\n",
    "        ]\n",
    "        if \"catalog.json\" in listing[\"path\"]\n",
    "    ][0]\n",
    ")\n",
    "\n",
    "cat.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob uses the STAC Python library to open the first STAC Item produced:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item = next(cat.get_items())\n",
    "item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob gets the path of the ostu step asset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "asset_href = item.get_assets()[\"data\"].get_absolute_href()\n",
    "\n",
    "asset_href"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob applies the sieve algorithm and then vectorizes the water bodies.\n",
    "\n",
    "Finally the water bodies are added to a map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the threshold size to remove small features (in pixels)\n",
    "threshold = 100  # Adjust this threshold as needed\n",
    "connectivity = 4  # Use 4-connected pixels for the sieve operation\n",
    "\n",
    "center = ((bbox[1] + bbox[3]) / 2, (bbox[0] + bbox[2]) / 2)\n",
    "\n",
    "m = Map(center=center, zoom=8)\n",
    "\n",
    "with rasterio.open(asset_href) as src:\n",
    "    result = sieve(src, threshold, connectivity=8)\n",
    "    for geom in dataset_features(src, band=True, as_mask=True):\n",
    "        geo_json = GeoJSON(\n",
    "            name=\"\",\n",
    "            data=geom,\n",
    "            style={\n",
    "                \"opacity\": 1,\n",
    "                \"fillOpacity\": 0.1,\n",
    "                \"weight\": 1,\n",
    "                \"color\": \"red\",\n",
    "            },\n",
    "            hover_style={\"color\": \"red\", \"dashArray\": \"0\", \"fillOpacity\": 0.5},\n",
    "        )\n",
    "        m.add_layer(geo_json)\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bob sends an email to Alice\n",
    "\n",
    "<hr>\n",
    "from: bob@acme.io\n",
    "\n",
    "to: alice@acme.io\n",
    "\n",
    "subject: RE:Detecting water bodies with NDWI and the Otsu threshold\n",
    "\n",
    "\n",
    "Hi Alice!\n",
    "\n",
    "The results look promising!\n",
    "\n",
    "Cheers,\n",
    "\n",
    "Bob\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}