post-crash save

ome · Aug 24, 2023 · 78c1bff · 78c1bff
1 parent 6879dfd
commit 78c1bff
Show file tree

Hide file tree

Showing 8 changed files with 1,078 additions and 24 deletions.
diff --git a/workshop.ipynb → Cloud.ipynb b/workshop.ipynb → Cloud.ipynb
@@ -8,18 +8,48 @@
     }
    },
    "source": [
-    "# Accessing cloud-hosted image data (practical)\n",
+    "# EMBO Practical Course \"Advanced methods in bioimage analysis\"\n",
     "\n",
     "***\n",
     "\n",
-    "**Global Bioimaging Online Workshop**\n",
-    "\n",
-    "*OME-NGFF: Towards a community standard image file format for sharing big image data in the cloud*\n",
+    "Homepage: https://www.embl.org/about/info/course-and-conference-office/events/bia23-01/\n",
     "\n",
     "***\n",
     "\n",
+    "## Day 2 - Session 1: Image Data Management - 11:30 to 12:30 \"GO!\"\n",
+    "\n",
+    "<table>\n",
+    "    <tr>\n",
+    "        <td>\n",
+    "            This is an explanation of this section.\n",
+    "        </td>\n",
+    "        <td>\n",
+    "            <center>\n",
+    "                <img src=\"images/falk/adam-uploads-300dpi.png\" width=\"50%\"/>\n",
+    "            </center>        \n",
+    "        </td>\n",
+    "    </tr>\n",
+    "</table>\n",
+    "\n",
     "\n",
-    "## Summary\n",
+    "### Introduction\n",
+    "\n",
+    "from the [BAND user guide](https://docs.google.com/document/d/1TZBUsNIciGMH_g4aFj2Lu_upISxh5TV9FBMrvNDWmc8/edit#heading=h.4tth2eqswl6h):\n",
+    "\n",
+    "> **Storage space**<br/>\n",
+    "> Disk space is shared and limited and although data stored in your home directory and under /scratch is persistent after you terminate your running desktop, we will remove content that is older than 3 months in both your home directory and /scratch.\n",
+    "Here are a few pointers:\n",
+    "> * If you know you’re not going to use BAND for a while, remove your data. \n",
+    "> * If you can, use external storage such as S3 buckets or file sharing services.\n",
+    "> * If you have more than ~100 GB of data, put it under /scratch which has up to 20 TB in total.\n",
+    "\n",
+    "\n",
+    "To remember:\n",
+    "- napari doesn't work with HTTPS (needs plugin)\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Practical summary\n",
     "0. Software versions\n",
     "1. Notebook reminders\n",
     "2. Looking into an OME-Zarr\n",
@@ -66,6 +96,47 @@
     "For example, you can use standard bash commands like `pwd` to see what directory you are in. When using mybinder, this will start with `/home/jovyan`. If you are interested in _why_, see [What is a Jovyan?](https://jupyter.readthedocs.io/en/latest/community/content-community.html#what-is-a-jovyan)\n"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "##\n",
+    "## Setup & Sanity checks\n",
+    "##\n",
+    "\n",
+    "YOURNAME=$(whoami)\n",
+    "WORKDIR=/scratch/${YOURNAME}/session1/\n",
+    "test -e ${WORKDIR} || {\n",
+    "    echo Please run the first script.\n",
+    "    exit 1\n",
+    "}\n",
+    "cd ${WORKDIR}\n",
+    "\n",
+    "##\n",
+    "## The Cloud!\n",
+    "##\n",
+    "\n",
+    "echo aws s3 --help\n",
+    "\n",
+    "mc config host list embo\n",
+    "\n",
+    "mc ls embo/\n",
+    "\n",
+    "mc ls embo/bioimagecourse2023/\n",
+    "\n",
+    "test -e a.ome.zarr && rm -rf a.ome.zarr\n",
+    "\n",
+    "bioformats2raw -p a.ome.tiff a.ome.zarr\n",
+    "\n",
+    "ome_zarr view a.ome.zarr\n",
+    "\n",
+    "mc cp -r a.ome.zarr/ embo/bioimagecourse2023/session1/${YOURNAME}/a.ome.zarr/\n",
+    "\n",
+    "open https://ome.github.io/ome-ngff-validator/?source=https://s3.embl.de/bioimagecourse2023/session1/${YOURNAME}/a.ome.zarr"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 1,
@@ -77,7 +148,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "/opt/NGFF-GBI-2022-Workshop\r\n"
+      "/opt/EMBO-Workshop-2023\n"
      ]
     }
    ],
@@ -491,12 +562,12 @@
    "source": [
     "***\n",
     "\n",
-    "# 3 Converting your data to OME-NGFF\n",
+    "## 3 Converting your data to OME-NGFF\n",
     "\n",
     "The two basic commands are `bioformats2raw` and `raw2ometiff`. Together they provide a pipeline to scalably convert large images into OME-TIFF. The primary caveat is that they require **twice** the storage for the conversion.\n",
     "\n",
     "\n",
-    "## 3.1 Conversion tools\n",
+    "### 3.1 Conversion tools\n",
     "\n",
     "https://forum.image.sc/t/converting-whole-slide-images-to-ome-tiff-a-new-workflow/32110/4\n",
     "\n",
@@ -792,7 +863,7 @@
     }
    },
    "source": [
-    "## 4.1 Minio client\n",
+    "### 4.1 Minio client\n",
     "\n",
     "There are a number of different types of cloud storage and there are a number of tools that you can use to access your cloud storage, but here we're going to focus on a single one `mc`.\n",
     "\n",
@@ -874,7 +945,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 4.2 Connections\n",
+    "### 4.2 Connections\n",
     "\n",
     "The minio project provides a safe space for you to learn about S3: https://play.minio.io:9000/minio/ Here we've used the `mc` command to find the access information:\n",
     "\n",
@@ -917,7 +988,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 4.3 Using `mc` with a public S3 bucket "
+    "### 4.3 Using `mc` with a public S3 bucket "
    ]
   },
   {
@@ -1088,14 +1159,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# 6. Extras (time-permitting)"
+    "## 6. Extras (time-permitting)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 6.1 A larger example (idr0062)\n",
+    "### 6.1 A larger example (idr0062)\n",
     "\n",
     "If you are using binder, you may need to access the link directly:\n",
     "\n",
@@ -1139,7 +1210,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 6.2 Renaming\n",
+    "### 6.2 Renaming\n",
     "\n",
     "Another important distinction to filesystems is that though it looks like hello is in a directory, you should really think of the entire string after the bucket just as a \"key\"."
    ]
@@ -1167,7 +1238,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 6.3 Other resources\n",
+    "### 6.3 Other resources\n",
     "\n",
     "<table>\n",
     "    <tr>\n",
@@ -1197,7 +1268,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 6.4 Trying more with minio's play\n",
+    "### 6.4 Trying more with minio's play\n",
     "\n",
     "Note, however, that play buckets get deleted every 24 hours."
    ]
@@ -1301,7 +1372,7 @@
     }
    },
    "source": [
-    "### License\n",
+    "## License\n",
     "Copyright (C) 2023 German BioImaging. All Rights Reserved.\n",
     "This program is free software; you can redistribute it and/or modify it\n",
     "under the terms of the GNU General Public License as published by the\n",
@@ -1315,13 +1386,6 @@
     "Free Software Foundation,\n",
     "Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA."
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {