From 9e7bff6301839a7d32a1d998b1b8fc557f94af48 Mon Sep 17 00:00:00 2001
From: EOEPCA CI <ci@eoepca.org>
Date: Mon, 18 Sep 2023 15:54:18 +0000
Subject: [PATCH] Deployed 46a3940 to current with MkDocs 1.3.0 and mike 1.1.2

---
 .../cluster/cluster-prerequisites/index.html  | 153 ++++++++++++------
 current/search/search_index.json              |   2 +-
 current/sitemap.xml.gz                        | Bin 437 -> 437 bytes
 3 files changed, 106 insertions(+), 49 deletions(-)

diff --git a/current/cluster/cluster-prerequisites/index.html b/current/cluster/cluster-prerequisites/index.html
index ae3182e3..c880c4c1 100644
--- a/current/cluster/cluster-prerequisites/index.html
+++ b/current/cluster/cluster-prerequisites/index.html
@@ -615,6 +615,13 @@
     <nav class="md-nav" aria-label="MinIO Object Storage">
       <ul class="md-nav__list">
         
+          <li class="md-nav__item">
+  <a href="#minio-credentials-secret" class="md-nav__link">
+    Minio Credentials Secret
+  </a>
+  
+</li>
+        
           <li class="md-nav__item">
   <a href="#s3cmd-configuration" class="md-nav__link">
     s3cmd Configuration
@@ -867,8 +874,12 @@ <h2 id="nginx-ingress-controller">Nginx Ingress Controller<a class="headerlink"
 </span><span id="__span-0-3"><a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a>helm repo update
 </span><span id="__span-0-4"><a id="__codelineno-0-4" name="__codelineno-0-4" href="#__codelineno-0-4"></a>
 </span><span id="__span-0-5"><a id="__codelineno-0-5" name="__codelineno-0-5" href="#__codelineno-0-5"></a><span class="c1"># Install the Nginx Ingress Controller helm chart</span>
-</span><span id="__span-0-6"><a id="__codelineno-0-6" name="__codelineno-0-6" href="#__codelineno-0-6"></a>helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx --wait
+</span><span id="__span-0-6"><a id="__codelineno-0-6" name="__codelineno-0-6" href="#__codelineno-0-6"></a>helm upgrade -i --version<span class="o">=</span><span class="s1">&#39;&lt;4.5.0&#39;</span> ingress-nginx ingress-nginx/ingress-nginx --wait
 </span></code></pre></div>
+<div class="admonition note">
+<p class="admonition-title">Note</p>
+<p>For Kubernetes version 1.22 and earlier the version of the Nginx Ingress Controller must be before v4.5.0.</p>
+</div>
 <p>To target the <em>Nginx Ingress Controller</em> the <code>kubernetes.io/ingress.class: nginx</code> annotation must be applied to the Ingress resource&hellip;
 <div class="language-yaml highlight"><pre><span></span><code><span id="__span-1-1"><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">networking.k8s.io/v1</span><span class="w"></span>
 </span><span id="__span-1-2"><a id="__codelineno-1-2" name="__codelineno-1-2" href="#__codelineno-1-2"></a><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Ingress</span><span class="w"></span>
@@ -974,67 +985,113 @@ <h3 id="references">References<a class="headerlink" href="#references" title="Pe
 </ul>
 <h2 id="minio-object-storage">MinIO Object Storage<a class="headerlink" href="#minio-object-storage" title="Permanent link">⚓︎</a></h2>
 <p>Various building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service.</p>
-<p>As a workaround, in the absence of an existing object storage, it is possible to use <a href="https://min.io/">MinIO</a> to establish an object storage service within the Kubernetes cluster. We use the <a href="https://bitnami.com/stack/minio/helm">minio helm chart provided by bitnami</a>.</p>
-<div class="language-bash highlight"><pre><span></span><code><span id="__span-9-1"><a id="__codelineno-9-1" name="__codelineno-9-1" href="#__codelineno-9-1"></a><span class="c1"># Add the bitnami helm repository</span>
-</span><span id="__span-9-2"><a id="__codelineno-9-2" name="__codelineno-9-2" href="#__codelineno-9-2"></a>helm repo add bitnami https://charts.bitnami.com/bitnami
-</span><span id="__span-9-3"><a id="__codelineno-9-3" name="__codelineno-9-3" href="#__codelineno-9-3"></a>helm repo update
-</span><span id="__span-9-4"><a id="__codelineno-9-4" name="__codelineno-9-4" href="#__codelineno-9-4"></a>
-</span><span id="__span-9-5"><a id="__codelineno-9-5" name="__codelineno-9-5" href="#__codelineno-9-5"></a><span class="c1"># Install the minio helm chart</span>
-</span><span id="__span-9-6"><a id="__codelineno-9-6" name="__codelineno-9-6" href="#__codelineno-9-6"></a>helm upgrade -i minio -f minio-values.yaml bitnami/minio
+<p>As a workaround, in the absence of an existing object storage, it is possible to use <a href="https://min.io/">MinIO</a> to establish an object storage service within the Kubernetes cluster. We use the <a href="https://charts.min.io/">minio helm chart provided by the MinIO Project</a>.</p>
+<div class="language-bash highlight"><pre><span></span><code><span id="__span-9-1"><a id="__codelineno-9-1" name="__codelineno-9-1" href="#__codelineno-9-1"></a><span class="c1"># Install the minio helm chart</span>
+</span><span id="__span-9-2"><a id="__codelineno-9-2" name="__codelineno-9-2" href="#__codelineno-9-2"></a>helm upgrade -i minio -f minio-values.yaml bitnami/minio <span class="se">\</span>
+</span><span id="__span-9-3"><a id="__codelineno-9-3" name="__codelineno-9-3" href="#__codelineno-9-3"></a>  --repo https://charts.min.io/ <span class="se">\</span>
+</span><span id="__span-9-4"><a id="__codelineno-9-4" name="__codelineno-9-4" href="#__codelineno-9-4"></a>  --namespace rm --create-namespace <span class="se">\</span>
+</span><span id="__span-9-5"><a id="__codelineno-9-5" name="__codelineno-9-5" href="#__codelineno-9-5"></a>  --wait
 </span></code></pre></div>
+<div class="admonition note">
+<p class="admonition-title">Note</p>
+<p>The Kubernetes namespace <code>rm</code> is used above as an example, and can be changed according to your deployment preference.</p>
+</div>
 <p>The minio deployment is customised via the values file <code>minio-values.yaml</code>, for example&hellip;</p>
-<div class="language-yaml highlight"><pre><span></span><code><span id="__span-10-1"><a id="__codelineno-10-1" name="__codelineno-10-1" href="#__codelineno-10-1"></a><span class="nt">auth</span><span class="p">:</span><span class="w"></span>
-</span><span id="__span-10-2"><a id="__codelineno-10-2" name="__codelineno-10-2" href="#__codelineno-10-2"></a><span class="w">  </span><span class="nt">rootUser</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">eoepca</span><span class="w"></span>
-</span><span id="__span-10-3"><a id="__codelineno-10-3" name="__codelineno-10-3" href="#__codelineno-10-3"></a><span class="w">  </span><span class="nt">rootPassword</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">changeme</span><span class="w"></span>
-</span><span id="__span-10-4"><a id="__codelineno-10-4" name="__codelineno-10-4" href="#__codelineno-10-4"></a>
-</span><span id="__span-10-5"><a id="__codelineno-10-5" name="__codelineno-10-5" href="#__codelineno-10-5"></a><span class="nt">ingress</span><span class="p">:</span><span class="w"></span>
-</span><span id="__span-10-6"><a id="__codelineno-10-6" name="__codelineno-10-6" href="#__codelineno-10-6"></a><span class="w">  </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span><span class="w"></span>
-</span><span id="__span-10-7"><a id="__codelineno-10-7" name="__codelineno-10-7" href="#__codelineno-10-7"></a><span class="w">  </span><span class="nt">ingressClassName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span><span class="w"></span>
-</span><span id="__span-10-8"><a id="__codelineno-10-8" name="__codelineno-10-8" href="#__codelineno-10-8"></a><span class="w">  </span><span class="nt">hostname</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minio-console.192-168-49-2.nip.io</span><span class="w"></span>
-</span><span id="__span-10-9"><a id="__codelineno-10-9" name="__codelineno-10-9" href="#__codelineno-10-9"></a><span class="w">  </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
-</span><span id="__span-10-10"><a id="__codelineno-10-10" name="__codelineno-10-10" href="#__codelineno-10-10"></a><span class="w">    </span><span class="nt">nginx.ingress.kubernetes.io/proxy-body-size</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0m</span><span class="w"></span>
-</span><span id="__span-10-11"><a id="__codelineno-10-11" name="__codelineno-10-11" href="#__codelineno-10-11"></a>
-</span><span id="__span-10-12"><a id="__codelineno-10-12" name="__codelineno-10-12" href="#__codelineno-10-12"></a><span class="nt">apiIngress</span><span class="p">:</span><span class="w"></span>
-</span><span id="__span-10-13"><a id="__codelineno-10-13" name="__codelineno-10-13" href="#__codelineno-10-13"></a><span class="w">  </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span><span class="w"></span>
-</span><span id="__span-10-14"><a id="__codelineno-10-14" name="__codelineno-10-14" href="#__codelineno-10-14"></a><span class="w">  </span><span class="nt">ingressClassName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span><span class="w"></span>
-</span><span id="__span-10-15"><a id="__codelineno-10-15" name="__codelineno-10-15" href="#__codelineno-10-15"></a><span class="w">  </span><span class="nt">hostname</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minio.192-168-49-2.nip.io</span><span class="w"></span>
-</span><span id="__span-10-16"><a id="__codelineno-10-16" name="__codelineno-10-16" href="#__codelineno-10-16"></a><span class="w">  </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
-</span><span id="__span-10-17"><a id="__codelineno-10-17" name="__codelineno-10-17" href="#__codelineno-10-17"></a><span class="w">    </span><span class="nt">nginx.ingress.kubernetes.io/proxy-body-size</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0m</span><span class="w"></span>
-</span><span id="__span-10-18"><a id="__codelineno-10-18" name="__codelineno-10-18" href="#__codelineno-10-18"></a>
-</span><span id="__span-10-19"><a id="__codelineno-10-19" name="__codelineno-10-19" href="#__codelineno-10-19"></a><span class="nt">persistence</span><span class="p">:</span><span class="w"></span>
-</span><span id="__span-10-20"><a id="__codelineno-10-20" name="__codelineno-10-20" href="#__codelineno-10-20"></a><span class="w">  </span><span class="nt">storageClass</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">standard</span><span class="w"></span>
+<div class="language-yaml highlight"><pre><span></span><code><span id="__span-10-1"><a id="__codelineno-10-1" name="__codelineno-10-1" href="#__codelineno-10-1"></a><span class="nt">existingSecret</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minio-auth</span><span class="w"></span>
+</span><span id="__span-10-2"><a id="__codelineno-10-2" name="__codelineno-10-2" href="#__codelineno-10-2"></a><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2</span><span class="w"></span>
+</span><span id="__span-10-3"><a id="__codelineno-10-3" name="__codelineno-10-3" href="#__codelineno-10-3"></a>
+</span><span id="__span-10-4"><a id="__codelineno-10-4" name="__codelineno-10-4" href="#__codelineno-10-4"></a><span class="nt">ingress</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-5"><a id="__codelineno-10-5" name="__codelineno-10-5" href="#__codelineno-10-5"></a><span class="w">  </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span><span class="w"></span>
+</span><span id="__span-10-6"><a id="__codelineno-10-6" name="__codelineno-10-6" href="#__codelineno-10-6"></a><span class="w">  </span><span class="nt">ingressClassName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span><span class="w"></span>
+</span><span id="__span-10-7"><a id="__codelineno-10-7" name="__codelineno-10-7" href="#__codelineno-10-7"></a><span class="w">  </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-8"><a id="__codelineno-10-8" name="__codelineno-10-8" href="#__codelineno-10-8"></a><span class="w">    </span><span class="nt">cert-manager.io/cluster-issuer</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;letsencrypt&quot;</span><span class="w"></span>
+</span><span id="__span-10-9"><a id="__codelineno-10-9" name="__codelineno-10-9" href="#__codelineno-10-9"></a><span class="w">    </span><span class="nt">nginx.ingress.kubernetes.io/ssl-redirect</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;true&quot;</span><span class="w"></span>
+</span><span id="__span-10-10"><a id="__codelineno-10-10" name="__codelineno-10-10" href="#__codelineno-10-10"></a><span class="w">    </span><span class="nt">nginx.ingress.kubernetes.io/proxy-body-size</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;0&quot;</span><span class="w"></span>
+</span><span id="__span-10-11"><a id="__codelineno-10-11" name="__codelineno-10-11" href="#__codelineno-10-11"></a><span class="w">    </span><span class="nt">nginx.ingress.kubernetes.io/proxy-read-timeout</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;600&#39;</span><span class="w"></span>
+</span><span id="__span-10-12"><a id="__codelineno-10-12" name="__codelineno-10-12" href="#__codelineno-10-12"></a><span class="w">  </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/</span><span class="w"></span>
+</span><span id="__span-10-13"><a id="__codelineno-10-13" name="__codelineno-10-13" href="#__codelineno-10-13"></a><span class="w">  </span><span class="nt">hosts</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-14"><a id="__codelineno-10-14" name="__codelineno-10-14" href="#__codelineno-10-14"></a><span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minio.192-168-49-2.nip.io</span><span class="w"></span>
+</span><span id="__span-10-15"><a id="__codelineno-10-15" name="__codelineno-10-15" href="#__codelineno-10-15"></a><span class="w">  </span><span class="nt">tls</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-16"><a id="__codelineno-10-16" name="__codelineno-10-16" href="#__codelineno-10-16"></a><span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">secretName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minio-tls</span><span class="w"></span>
+</span><span id="__span-10-17"><a id="__codelineno-10-17" name="__codelineno-10-17" href="#__codelineno-10-17"></a><span class="w">      </span><span class="nt">hosts</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-18"><a id="__codelineno-10-18" name="__codelineno-10-18" href="#__codelineno-10-18"></a><span class="w">        </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minio.192-168-49-2.nip.io</span><span class="w"></span>
+</span><span id="__span-10-19"><a id="__codelineno-10-19" name="__codelineno-10-19" href="#__codelineno-10-19"></a>
+</span><span id="__span-10-20"><a id="__codelineno-10-20" name="__codelineno-10-20" href="#__codelineno-10-20"></a><span class="nt">consoleIngress</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-21"><a id="__codelineno-10-21" name="__codelineno-10-21" href="#__codelineno-10-21"></a><span class="w">  </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span><span class="w"></span>
+</span><span id="__span-10-22"><a id="__codelineno-10-22" name="__codelineno-10-22" href="#__codelineno-10-22"></a><span class="w">  </span><span class="nt">ingressClassName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span><span class="w"></span>
+</span><span id="__span-10-23"><a id="__codelineno-10-23" name="__codelineno-10-23" href="#__codelineno-10-23"></a><span class="w">  </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-24"><a id="__codelineno-10-24" name="__codelineno-10-24" href="#__codelineno-10-24"></a><span class="w">    </span><span class="nt">cert-manager.io/cluster-issuer</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;letsencrypt&quot;</span><span class="w"></span>
+</span><span id="__span-10-25"><a id="__codelineno-10-25" name="__codelineno-10-25" href="#__codelineno-10-25"></a><span class="w">    </span><span class="nt">nginx.ingress.kubernetes.io/ssl-redirect</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;true&quot;</span><span class="w"></span>
+</span><span id="__span-10-26"><a id="__codelineno-10-26" name="__codelineno-10-26" href="#__codelineno-10-26"></a><span class="w">    </span><span class="nt">nginx.ingress.kubernetes.io/proxy-body-size</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;0&quot;</span><span class="w"></span>
+</span><span id="__span-10-27"><a id="__codelineno-10-27" name="__codelineno-10-27" href="#__codelineno-10-27"></a><span class="w">    </span><span class="nt">nginx.ingress.kubernetes.io/proxy-read-timeout</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;600&#39;</span><span class="w"></span>
+</span><span id="__span-10-28"><a id="__codelineno-10-28" name="__codelineno-10-28" href="#__codelineno-10-28"></a><span class="w">  </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/</span><span class="w"></span>
+</span><span id="__span-10-29"><a id="__codelineno-10-29" name="__codelineno-10-29" href="#__codelineno-10-29"></a><span class="w">  </span><span class="nt">hosts</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-30"><a id="__codelineno-10-30" name="__codelineno-10-30" href="#__codelineno-10-30"></a><span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">console.minio.192-168-49-2.nip.io</span><span class="w"></span>
+</span><span id="__span-10-31"><a id="__codelineno-10-31" name="__codelineno-10-31" href="#__codelineno-10-31"></a><span class="w">  </span><span class="nt">tls</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-32"><a id="__codelineno-10-32" name="__codelineno-10-32" href="#__codelineno-10-32"></a><span class="w">  </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">secretName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minio-console-tls</span><span class="w"></span>
+</span><span id="__span-10-33"><a id="__codelineno-10-33" name="__codelineno-10-33" href="#__codelineno-10-33"></a><span class="w">    </span><span class="nt">hosts</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-34"><a id="__codelineno-10-34" name="__codelineno-10-34" href="#__codelineno-10-34"></a><span class="w">      </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">console.minio.192-168-49-2.nip.io</span><span class="w"></span>
+</span><span id="__span-10-35"><a id="__codelineno-10-35" name="__codelineno-10-35" href="#__codelineno-10-35"></a>
+</span><span id="__span-10-36"><a id="__codelineno-10-36" name="__codelineno-10-36" href="#__codelineno-10-36"></a><span class="nt">resources</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-37"><a id="__codelineno-10-37" name="__codelineno-10-37" href="#__codelineno-10-37"></a><span class="w">  </span><span class="nt">requests</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-38"><a id="__codelineno-10-38" name="__codelineno-10-38" href="#__codelineno-10-38"></a><span class="w">    </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1Gi</span><span class="w"></span>
+</span><span id="__span-10-39"><a id="__codelineno-10-39" name="__codelineno-10-39" href="#__codelineno-10-39"></a>
+</span><span id="__span-10-40"><a id="__codelineno-10-40" name="__codelineno-10-40" href="#__codelineno-10-40"></a><span class="nt">persistence</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-41"><a id="__codelineno-10-41" name="__codelineno-10-41" href="#__codelineno-10-41"></a><span class="w">  </span><span class="nt">storageClass</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">standard</span><span class="w"></span>
+</span><span id="__span-10-42"><a id="__codelineno-10-42" name="__codelineno-10-42" href="#__codelineno-10-42"></a>
+</span><span id="__span-10-43"><a id="__codelineno-10-43" name="__codelineno-10-43" href="#__codelineno-10-43"></a><span class="nt">buckets</span><span class="p">:</span><span class="w"></span>
+</span><span id="__span-10-44"><a id="__codelineno-10-44" name="__codelineno-10-44" href="#__codelineno-10-44"></a><span class="w">  </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">eoepca</span><span class="w"></span>
+</span><span id="__span-10-45"><a id="__codelineno-10-45" name="__codelineno-10-45" href="#__codelineno-10-45"></a><span class="w">  </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cache-bucket</span><span class="w"></span>
+</span></code></pre></div>
+<div class="admonition note">
+<p class="admonition-title">Note</p>
+<ul>
+<li>The example values assuming a TLS configuration using <code>letsencrypt</code> certificate provider</li>
+<li>The admin credentials are provided by the Kubernetes secret named <code>minio-auth</code> - see below</li>
+<li>The annotation <code>nginx.ingress.kubernetes.io/proxy-body-size</code> was found to be required to allow transfer of large files (such as data products) through the nginx proxy</li>
+</ul>
+</div>
+<h3 id="minio-credentials-secret">Minio Credentials Secret<a class="headerlink" href="#minio-credentials-secret" title="Permanent link">⚓︎</a></h3>
+<p>The Minio admin credentials are provided via a Kubernetes secret that is referenced from the Minio helm chart deployment values. For example&hellip;</p>
+<div class="language-text highlight"><pre><span></span><code><span id="__span-11-1"><a id="__codelineno-11-1" name="__codelineno-11-1" href="#__codelineno-11-1"></a>kubectl -n rm create secret generic minio-auth \
+</span><span id="__span-11-2"><a id="__codelineno-11-2" name="__codelineno-11-2" href="#__codelineno-11-2"></a>  --from-literal=rootUser=&quot;eoepca&quot; \
+</span><span id="__span-11-3"><a id="__codelineno-11-3" name="__codelineno-11-3" href="#__codelineno-11-3"></a>  --from-literal=rootPassword=&quot;changeme&quot;
 </span></code></pre></div>
-<p><em>Note that the annotation <code>nginx.ingress.kubernetes.io/proxy-body-size</code> was found to be required to allow transfer of large files (such as data products) through the nginx proxy</em></p>
+<div class="admonition note">
+<p class="admonition-title">Note</p>
+<p>The secret must be created in the same Kubernetes namespace as the Minio service deployment - e.g. <code>rm</code> namespce in the example above.</p>
+</div>
 <h3 id="s3cmd-configuration">s3cmd Configuration<a class="headerlink" href="#s3cmd-configuration" title="Permanent link">⚓︎</a></h3>
 <p>The <code>s3cmd</code> can be configured for access to the MinIO deployment. The <code>--configure</code> option can be used to prepare a suitable configuration file for <code>s3cmd</code>&hellip;</p>
-<div class="language-bash highlight"><pre><span></span><code><span id="__span-11-1"><a id="__codelineno-11-1" name="__codelineno-11-1" href="#__codelineno-11-1"></a>s3cmd -c mys3cfg --configure
+<div class="language-bash highlight"><pre><span></span><code><span id="__span-12-1"><a id="__codelineno-12-1" name="__codelineno-12-1" href="#__codelineno-12-1"></a>s3cmd -c mys3cfg --configure
 </span></code></pre></div>
 <p>In response to the prompts, the following configuration selections are applicable to the above settings&hellip;</p>
-<div class="language-text highlight"><pre><span></span><code><span id="__span-12-1"><a id="__codelineno-12-1" name="__codelineno-12-1" href="#__codelineno-12-1"></a>Access Key: eoepca
-</span><span id="__span-12-2"><a id="__codelineno-12-2" name="__codelineno-12-2" href="#__codelineno-12-2"></a>Secret Key: changeme
-</span><span id="__span-12-3"><a id="__codelineno-12-3" name="__codelineno-12-3" href="#__codelineno-12-3"></a>Default Region: us-east-1
-</span><span id="__span-12-4"><a id="__codelineno-12-4" name="__codelineno-12-4" href="#__codelineno-12-4"></a>S3 Endpoint: minio.192-168-49-2.nip.io
-</span><span id="__span-12-5"><a id="__codelineno-12-5" name="__codelineno-12-5" href="#__codelineno-12-5"></a>DNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io
-</span><span id="__span-12-6"><a id="__codelineno-12-6" name="__codelineno-12-6" href="#__codelineno-12-6"></a>Encryption password: 
-</span><span id="__span-12-7"><a id="__codelineno-12-7" name="__codelineno-12-7" href="#__codelineno-12-7"></a>Path to GPG program: /usr/bin/gpg
-</span><span id="__span-12-8"><a id="__codelineno-12-8" name="__codelineno-12-8" href="#__codelineno-12-8"></a>Use HTTPS protocol: False
-</span><span id="__span-12-9"><a id="__codelineno-12-9" name="__codelineno-12-9" href="#__codelineno-12-9"></a>HTTP Proxy server name: 
-</span><span id="__span-12-10"><a id="__codelineno-12-10" name="__codelineno-12-10" href="#__codelineno-12-10"></a>HTTP Proxy server port: 0
+<div class="language-text highlight"><pre><span></span><code><span id="__span-13-1"><a id="__codelineno-13-1" name="__codelineno-13-1" href="#__codelineno-13-1"></a>Access Key: eoepca
+</span><span id="__span-13-2"><a id="__codelineno-13-2" name="__codelineno-13-2" href="#__codelineno-13-2"></a>Secret Key: changeme
+</span><span id="__span-13-3"><a id="__codelineno-13-3" name="__codelineno-13-3" href="#__codelineno-13-3"></a>Default Region: us-east-1
+</span><span id="__span-13-4"><a id="__codelineno-13-4" name="__codelineno-13-4" href="#__codelineno-13-4"></a>S3 Endpoint: minio.192-168-49-2.nip.io
+</span><span id="__span-13-5"><a id="__codelineno-13-5" name="__codelineno-13-5" href="#__codelineno-13-5"></a>DNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io
+</span><span id="__span-13-6"><a id="__codelineno-13-6" name="__codelineno-13-6" href="#__codelineno-13-6"></a>Encryption password: 
+</span><span id="__span-13-7"><a id="__codelineno-13-7" name="__codelineno-13-7" href="#__codelineno-13-7"></a>Path to GPG program: /usr/bin/gpg
+</span><span id="__span-13-8"><a id="__codelineno-13-8" name="__codelineno-13-8" href="#__codelineno-13-8"></a>Use HTTPS protocol: True
+</span><span id="__span-13-9"><a id="__codelineno-13-9" name="__codelineno-13-9" href="#__codelineno-13-9"></a>HTTP Proxy server name: 
+</span><span id="__span-13-10"><a id="__codelineno-13-10" name="__codelineno-13-10" href="#__codelineno-13-10"></a>HTTP Proxy server port: 0
 </span></code></pre></div>
 <p>Save the configuration file, and check access to the S3 object store with&hellip;</p>
-<div class="language-bash highlight"><pre><span></span><code><span id="__span-13-1"><a id="__codelineno-13-1" name="__codelineno-13-1" href="#__codelineno-13-1"></a><span class="c1"># Create a bucket</span>
-</span><span id="__span-13-2"><a id="__codelineno-13-2" name="__codelineno-13-2" href="#__codelineno-13-2"></a>s3cmd -c mys3cfg mb s3://eoepca
-</span><span id="__span-13-3"><a id="__codelineno-13-3" name="__codelineno-13-3" href="#__codelineno-13-3"></a>
-</span><span id="__span-13-4"><a id="__codelineno-13-4" name="__codelineno-13-4" href="#__codelineno-13-4"></a><span class="c1"># List buckets</span>
-</span><span id="__span-13-5"><a id="__codelineno-13-5" name="__codelineno-13-5" href="#__codelineno-13-5"></a>s3cmd -c mys3cfg ls
+<div class="language-bash highlight"><pre><span></span><code><span id="__span-14-1"><a id="__codelineno-14-1" name="__codelineno-14-1" href="#__codelineno-14-1"></a><span class="c1"># Create a bucket</span>
+</span><span id="__span-14-2"><a id="__codelineno-14-2" name="__codelineno-14-2" href="#__codelineno-14-2"></a>s3cmd -c mys3cfg mb s3://eoepca
+</span><span id="__span-14-3"><a id="__codelineno-14-3" name="__codelineno-14-3" href="#__codelineno-14-3"></a>
+</span><span id="__span-14-4"><a id="__codelineno-14-4" name="__codelineno-14-4" href="#__codelineno-14-4"></a><span class="c1"># List buckets</span>
+</span><span id="__span-14-5"><a id="__codelineno-14-5" name="__codelineno-14-5" href="#__codelineno-14-5"></a>s3cmd -c mys3cfg ls
 </span></code></pre></div>
 <p>For example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube&hellip;
-<div class="language-bash highlight"><pre><span></span><code><span id="__span-14-1"><a id="__codelineno-14-1" name="__codelineno-14-1" href="#__codelineno-14-1"></a>s3cmd -c deploy/cluster/s3cfg ls
+<div class="language-bash highlight"><pre><span></span><code><span id="__span-15-1"><a id="__codelineno-15-1" name="__codelineno-15-1" href="#__codelineno-15-1"></a>s3cmd -c deploy/cluster/s3cfg ls
 </span></code></pre></div></p>
 <h3 id="references_1">References<a class="headerlink" href="#references_1" title="Permanent link">⚓︎</a></h3>
 <ul>
-<li><a href="https://bitnami.com/stack/minio/helm">MinIO Helm Chart</a></li>
-<li><a href="https://github.com/bitnami/charts/tree/master/bitnami/minio">MinIO Helm Chart on GitHub</a></li>
+<li><a href="https://min.io/">MinIO Website</a></li>
+<li><a href="https://charts.min.io/">MinIO Helm Chart</a></li>
+<li><a href="https://github.com/minio/minio">MinIO on GitHub</a></li>
 </ul>
 
               
diff --git a/current/search/search_index.json b/current/search/search_index.json
index 1db66ed4..8b48c1f7 100644
--- a/current/search/search_index.json
+++ b/current/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Deployment Guide \u2693\ufe0e The Deployment Guide describes how each building-block comprising the EOEPCA Reference Implementation is configured and deployed. A full system deployment is described, in which components are deployed with complementary configurations that facilitate their integration as a coherent system. Nevertheless, each component can be cherry-picked from this system deployment for individual re-use. The deployment is organised into the following sections: Cluster Establish the Kubernetes cluster and other prerequisites for the deployment of the EOEPCA system. EOEPCA Deployment of the EOEPCA components. Examples Deployment examples.","title":"Introduction"},{"location":"#deployment-guide","text":"The Deployment Guide describes how each building-block comprising the EOEPCA Reference Implementation is configured and deployed. A full system deployment is described, in which components are deployed with complementary configurations that facilitate their integration as a coherent system. Nevertheless, each component can be cherry-picked from this system deployment for individual re-use. The deployment is organised into the following sections: Cluster Establish the Kubernetes cluster and other prerequisites for the deployment of the EOEPCA system. EOEPCA Deployment of the EOEPCA components. Examples Deployment examples.","title":"Deployment Guide"},{"location":"cluster/cluster-prerequisites/","text":"Cluster Prerequisites \u2693\ufe0e The following prerequisite components are assumed to be deployed in the cluster. Nginx Ingress Controller \u2693\ufe0e # Add the helm repository helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update # Install the Nginx Ingress Controller helm chart helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx --wait To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx annotation must be applied to the Ingress resource\u2026 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : kubernetes.io/ingress.class : nginx ... Cert Manager \u2693\ufe0e # Add the helm repository helm repo add jetstack https://charts.jetstack.io helm repo update # Install the Cert Manager helm chart helm upgrade -i cert-manager jetstack/cert-manager \\ --namespace cert-manager \\ --create-namespace \\ --set installCRDs = true Letsencrypt Certificates \u2693\ufe0e Once the Certificate Manager is deployed, then we can establish ClusterIssuer operators in the cluster to support use of TLS with service Ingress endpoints. For Letsencrypt we can define two ClusterIssuer - for production and for staging . NOTE that these require the cluster to be publicly accessible, in order for the http01 acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints. Production \u2693\ufe0e apiVersion : cert-manager.io/v1 kind : ClusterIssuer metadata : name : letsencrypt-production spec : acme : # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email : eoepca.systemteam@telespazio.com server : https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef : # Secret resource that will be used to store the account's private key. name : letsencrypt-production-account-key # Add a single challenge solver, HTTP01 using nginx solvers : - http01 : ingress : class : nginx Staging \u2693\ufe0e apiVersion : cert-manager.io/v1 kind : ClusterIssuer metadata : name : letsencrypt-staging spec : acme : # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email : eoepca.systemteam@telespazio.com server : https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef : # Secret resource that will be used to store the account's private key. name : letsencrypt-staging-account-key # Add a single challenge solver, HTTP01 using nginx solvers : - http01 : ingress : class : nginx To exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer annotation must be applied to the Ingress resource. For example\u2026 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : kubernetes.io/ingress.class : nginx cert-manager.io/cluster-issuer : letsencrypt-production ... Sealed Secrets \u2693\ufe0e The EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments. Various Secret are relied upon by the system services. Secrets should not be exposed by commit to GitHub. Instead SealedSecret are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller that runs within the cluster. The sealed-secret-controller decrypts the SealedSecret to a regular Secret (of the same name) that can then be consumed by the cluster components. The sealed-secret-controller is deployed to the cluster using the helm chart\u2026 helm repo add bitnami-sealed-secrets https://bitnami-labs.github.io/sealed-secrets helm repo update helm install --version 2 .1.8 --create-namespace --namespace infra \\ eoepca-sealed-secrets bitnami-sealed-secrets/sealed-secrets Once the controller is deployed within the cluster, then the kubeseal command can be used to create a SealedSecret from a regular Secret , as follows\u2026 Create example Secret\u2026 kubectl -n test create secret generic mysecret \\ --from-literal = password = changeme \\ --dry-run = client -o yaml \\ > mysecret.yaml Create SealedSecret from Secret using kubeseal\u2026 kubeseal -o yaml \\ --controller-name eoepca-sealed-secrets \\ --controller-namespace infra \\ < mysecret.yaml \\ > mysecret-sealed.yaml References \u2693\ufe0e Sealed Secrets on GitHub kubeseal Release MinIO Object Storage \u2693\ufe0e Various building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service. As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by bitnami . # Add the bitnami helm repository helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update # Install the minio helm chart helm upgrade -i minio -f minio-values.yaml bitnami/minio The minio deployment is customised via the values file minio-values.yaml , for example\u2026 auth : rootUser : eoepca rootPassword : changeme ingress : enabled : true ingressClassName : nginx hostname : minio-console.192-168-49-2.nip.io annotations : nginx.ingress.kubernetes.io/proxy-body-size : 0m apiIngress : enabled : true ingressClassName : nginx hostname : minio.192-168-49-2.nip.io annotations : nginx.ingress.kubernetes.io/proxy-body-size : 0m persistence : storageClass : standard Note that the annotation nginx.ingress.kubernetes.io/proxy-body-size was found to be required to allow transfer of large files (such as data products) through the nginx proxy s3cmd Configuration \u2693\ufe0e The s3cmd can be configured for access to the MinIO deployment. The --configure option can be used to prepare a suitable configuration file for s3cmd \u2026 s3cmd -c mys3cfg --configure In response to the prompts, the following configuration selections are applicable to the above settings\u2026 Access Key: eoepca Secret Key: changeme Default Region: us-east-1 S3 Endpoint: minio.192-168-49-2.nip.io DNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Save the configuration file, and check access to the S3 object store with\u2026 # Create a bucket s3cmd -c mys3cfg mb s3://eoepca # List buckets s3cmd -c mys3cfg ls For example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube\u2026 s3cmd -c deploy/cluster/s3cfg ls References \u2693\ufe0e MinIO Helm Chart MinIO Helm Chart on GitHub","title":"Cluster Prerequisites"},{"location":"cluster/cluster-prerequisites/#cluster-prerequisites","text":"The following prerequisite components are assumed to be deployed in the cluster.","title":"Cluster Prerequisites"},{"location":"cluster/cluster-prerequisites/#nginx-ingress-controller","text":"# Add the helm repository helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update # Install the Nginx Ingress Controller helm chart helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx --wait To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx annotation must be applied to the Ingress resource\u2026 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : kubernetes.io/ingress.class : nginx ...","title":"Nginx Ingress Controller"},{"location":"cluster/cluster-prerequisites/#cert-manager","text":"# Add the helm repository helm repo add jetstack https://charts.jetstack.io helm repo update # Install the Cert Manager helm chart helm upgrade -i cert-manager jetstack/cert-manager \\ --namespace cert-manager \\ --create-namespace \\ --set installCRDs = true","title":"Cert Manager"},{"location":"cluster/cluster-prerequisites/#letsencrypt-certificates","text":"Once the Certificate Manager is deployed, then we can establish ClusterIssuer operators in the cluster to support use of TLS with service Ingress endpoints. For Letsencrypt we can define two ClusterIssuer - for production and for staging . NOTE that these require the cluster to be publicly accessible, in order for the http01 acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints.","title":"Letsencrypt Certificates"},{"location":"cluster/cluster-prerequisites/#production","text":"apiVersion : cert-manager.io/v1 kind : ClusterIssuer metadata : name : letsencrypt-production spec : acme : # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email : eoepca.systemteam@telespazio.com server : https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef : # Secret resource that will be used to store the account's private key. name : letsencrypt-production-account-key # Add a single challenge solver, HTTP01 using nginx solvers : - http01 : ingress : class : nginx","title":"Production"},{"location":"cluster/cluster-prerequisites/#staging","text":"apiVersion : cert-manager.io/v1 kind : ClusterIssuer metadata : name : letsencrypt-staging spec : acme : # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email : eoepca.systemteam@telespazio.com server : https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef : # Secret resource that will be used to store the account's private key. name : letsencrypt-staging-account-key # Add a single challenge solver, HTTP01 using nginx solvers : - http01 : ingress : class : nginx To exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer annotation must be applied to the Ingress resource. For example\u2026 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : kubernetes.io/ingress.class : nginx cert-manager.io/cluster-issuer : letsencrypt-production ...","title":"Staging"},{"location":"cluster/cluster-prerequisites/#sealed-secrets","text":"The EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments. Various Secret are relied upon by the system services. Secrets should not be exposed by commit to GitHub. Instead SealedSecret are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller that runs within the cluster. The sealed-secret-controller decrypts the SealedSecret to a regular Secret (of the same name) that can then be consumed by the cluster components. The sealed-secret-controller is deployed to the cluster using the helm chart\u2026 helm repo add bitnami-sealed-secrets https://bitnami-labs.github.io/sealed-secrets helm repo update helm install --version 2 .1.8 --create-namespace --namespace infra \\ eoepca-sealed-secrets bitnami-sealed-secrets/sealed-secrets Once the controller is deployed within the cluster, then the kubeseal command can be used to create a SealedSecret from a regular Secret , as follows\u2026 Create example Secret\u2026 kubectl -n test create secret generic mysecret \\ --from-literal = password = changeme \\ --dry-run = client -o yaml \\ > mysecret.yaml Create SealedSecret from Secret using kubeseal\u2026 kubeseal -o yaml \\ --controller-name eoepca-sealed-secrets \\ --controller-namespace infra \\ < mysecret.yaml \\ > mysecret-sealed.yaml","title":"Sealed Secrets"},{"location":"cluster/cluster-prerequisites/#references","text":"Sealed Secrets on GitHub kubeseal Release","title":"References"},{"location":"cluster/cluster-prerequisites/#minio-object-storage","text":"Various building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service. As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by bitnami . # Add the bitnami helm repository helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update # Install the minio helm chart helm upgrade -i minio -f minio-values.yaml bitnami/minio The minio deployment is customised via the values file minio-values.yaml , for example\u2026 auth : rootUser : eoepca rootPassword : changeme ingress : enabled : true ingressClassName : nginx hostname : minio-console.192-168-49-2.nip.io annotations : nginx.ingress.kubernetes.io/proxy-body-size : 0m apiIngress : enabled : true ingressClassName : nginx hostname : minio.192-168-49-2.nip.io annotations : nginx.ingress.kubernetes.io/proxy-body-size : 0m persistence : storageClass : standard Note that the annotation nginx.ingress.kubernetes.io/proxy-body-size was found to be required to allow transfer of large files (such as data products) through the nginx proxy","title":"MinIO Object Storage"},{"location":"cluster/cluster-prerequisites/#s3cmd-configuration","text":"The s3cmd can be configured for access to the MinIO deployment. The --configure option can be used to prepare a suitable configuration file for s3cmd \u2026 s3cmd -c mys3cfg --configure In response to the prompts, the following configuration selections are applicable to the above settings\u2026 Access Key: eoepca Secret Key: changeme Default Region: us-east-1 S3 Endpoint: minio.192-168-49-2.nip.io DNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Save the configuration file, and check access to the S3 object store with\u2026 # Create a bucket s3cmd -c mys3cfg mb s3://eoepca # List buckets s3cmd -c mys3cfg ls For example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube\u2026 s3cmd -c deploy/cluster/s3cfg ls","title":"s3cmd Configuration"},{"location":"cluster/cluster-prerequisites/#references_1","text":"MinIO Helm Chart MinIO Helm Chart on GitHub","title":"References"},{"location":"cluster/helm-repositories/","text":"Helm Repositories \u2693\ufe0e EOEPCA Helm Charts \u2693\ufe0e The EOEPCA building-blocks are engineered as containers for deployment to a Kubernetes cluster. Each building block defines a Helm Chart to facilitate its deployment. The EOEPCA Helm Chart Repository is configured with helm as follows\u2026 helm repo add eoepca https://eoepca.github.io/helm-charts/ Third-party Helm Charts \u2693\ufe0e In addition to the EOEPCA Helm Chart Repository, the following repositories are also relied upon, and should be configured\u2026 Cert Manager \u2693\ufe0e helm repo add jetstack https://charts.jetstack.io Nginx Ingress Controller \u2693\ufe0e helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx Repo Update \u2693\ufe0e Refresh the local repo cache, after helm repo add \u2026 helm repo update","title":"Helm Repositories"},{"location":"cluster/helm-repositories/#helm-repositories","text":"","title":"Helm Repositories"},{"location":"cluster/helm-repositories/#eoepca-helm-charts","text":"The EOEPCA building-blocks are engineered as containers for deployment to a Kubernetes cluster. Each building block defines a Helm Chart to facilitate its deployment. The EOEPCA Helm Chart Repository is configured with helm as follows\u2026 helm repo add eoepca https://eoepca.github.io/helm-charts/","title":"EOEPCA Helm Charts"},{"location":"cluster/helm-repositories/#third-party-helm-charts","text":"In addition to the EOEPCA Helm Chart Repository, the following repositories are also relied upon, and should be configured\u2026","title":"Third-party Helm Charts"},{"location":"cluster/helm-repositories/#cert-manager","text":"helm repo add jetstack https://charts.jetstack.io","title":"Cert Manager"},{"location":"cluster/helm-repositories/#nginx-ingress-controller","text":"helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx","title":"Nginx Ingress Controller"},{"location":"cluster/helm-repositories/#repo-update","text":"Refresh the local repo cache, after helm repo add \u2026 helm repo update","title":"Repo Update"},{"location":"cluster/kubernetes/","text":"Kubernetes Cluster \u2693\ufe0e The EOEPCA Reference Implementation has been developed with Kubernetes as its deployment target. The system components have been developed, deployed and tested using a cluster at version v1.22.5 . Rancher Kubernetes Engine (RKE) \u2693\ufe0e The development, integration and test clusters have been established using Rancher Kubernetes Engine (RKE) at version v1.22.5 . An example of the creation of the EOEPCA Kubernetes clusters can be found on the GitHub Kubernetes Setup page . CREODIAS has been used for the development hosting infrastructure - which provides OpenStack infrastructure that is backed by Cloudferro . An example of the Terraform configurations used to automate the creation of the cloud infrastructure that underpins the RKE deployment can be found on the GitHub CREODIAS Setup page . Local Kubernetes \u2693\ufe0e To make a full deployment of the EOEPCA Reference Implementation requires a multi-node node cluster with suitable resources. For example, the development cluster comprises: 1 Master node (2 vCPU, 8 GB RAM) 5 Worker nodes (4 vCPU, 16 GB RAM) 1 NFS server (2 vCPU, 8 GB RAM) Limited local deployment can be made using a suitable local single-node kuberbetes deployment using - for example using minikube \u2026 minikube -p eoepca start --cpus max --memory max --kubernetes-version v1.22.5 minikube profile eoepca With such a deployment it is possible to deploy individual building-blocks for local development, or building-blocks in combination - within the constraints of the local host resources.","title":"Kubernetes Cluster"},{"location":"cluster/kubernetes/#kubernetes-cluster","text":"The EOEPCA Reference Implementation has been developed with Kubernetes as its deployment target. The system components have been developed, deployed and tested using a cluster at version v1.22.5 .","title":"Kubernetes Cluster"},{"location":"cluster/kubernetes/#rancher-kubernetes-engine-rke","text":"The development, integration and test clusters have been established using Rancher Kubernetes Engine (RKE) at version v1.22.5 . An example of the creation of the EOEPCA Kubernetes clusters can be found on the GitHub Kubernetes Setup page . CREODIAS has been used for the development hosting infrastructure - which provides OpenStack infrastructure that is backed by Cloudferro . An example of the Terraform configurations used to automate the creation of the cloud infrastructure that underpins the RKE deployment can be found on the GitHub CREODIAS Setup page .","title":"Rancher Kubernetes Engine (RKE)"},{"location":"cluster/kubernetes/#local-kubernetes","text":"To make a full deployment of the EOEPCA Reference Implementation requires a multi-node node cluster with suitable resources. For example, the development cluster comprises: 1 Master node (2 vCPU, 8 GB RAM) 5 Worker nodes (4 vCPU, 16 GB RAM) 1 NFS server (2 vCPU, 8 GB RAM) Limited local deployment can be made using a suitable local single-node kuberbetes deployment using - for example using minikube \u2026 minikube -p eoepca start --cpus max --memory max --kubernetes-version v1.22.5 minikube profile eoepca With such a deployment it is possible to deploy individual building-blocks for local development, or building-blocks in combination - within the constraints of the local host resources.","title":"Local Kubernetes"},{"location":"cluster/prerequisite-tooling/","text":"Prerequisite Tooling \u2693\ufe0e There are some standard tools referenced in this guide. These are detailed in the following subsections. docker \u2693\ufe0e Docker faciliates the creation, management and execution of containers. Whilst not strictly necessary to support deployment to an existing/managed Kubernetes cluster, it can nevertheless be useful to have local access to the docker tooling. For example, if minikube is used to follow this guide using a local k8s cluster, then this is best achieved using minikube\u2019s docker driver. Docker is most easily installed with\u2026 curl -fsSL https://get.docker.com | sh For convenience, add your user to the docker group\u2026 sudo usermod -aG docker ${ USER } Logout/in to refresh your session\u2019s group permissions. kubectl \u2693\ufe0e Kubectl is the main tool for interaction with a Kubernetes cluster. The latest version can be installed with\u2026 mkdir -p $HOME /.local/bin \\ && curl -fsSLo $HOME /.local/bin/kubectl \"https://dl.k8s.io/release/ $( curl -L -s https://dl.k8s.io/release/stable.txt ) /bin/linux/amd64/kubectl\" \\ && chmod +x $HOME /.local/bin/kubectl See the official kubectl installation documentation for more installation options. helm \u2693\ufe0e Helm is the Kubernetes package manager, in which components are deployed to a Kubernetes cluster via helm charts. The helm charts are instantiated for deployment via \u2018values\u2019 that configure the chart templates. The latest helm version can be installed with\u2026 export HELM_INSTALL_DIR = \" $HOME /.local/bin\" \\ && curl -sfL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash See the official helm installation documentation for more installation options. minikube \u2693\ufe0e Minikube is a tool that allows to create a local (single-node) Kubernetes cluster for development/testing. It is not designed for production use. In the absence of access to a \u2018full\u2019 Kubernetes cluster, this guide can be followed using minikube. The latest version of minikube can be installed with\u2026 mkdir -p $HOME /.local/bin \\ && curl -fsSLo $HOME /.local/bin/minikube \"https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\" \\ && chmod +x $HOME /.local/bin/minikube See the official minikube installation documentation for more installation options.","title":"Prerequisite Tooling"},{"location":"cluster/prerequisite-tooling/#prerequisite-tooling","text":"There are some standard tools referenced in this guide. These are detailed in the following subsections.","title":"Prerequisite Tooling"},{"location":"cluster/prerequisite-tooling/#docker","text":"Docker faciliates the creation, management and execution of containers. Whilst not strictly necessary to support deployment to an existing/managed Kubernetes cluster, it can nevertheless be useful to have local access to the docker tooling. For example, if minikube is used to follow this guide using a local k8s cluster, then this is best achieved using minikube\u2019s docker driver. Docker is most easily installed with\u2026 curl -fsSL https://get.docker.com | sh For convenience, add your user to the docker group\u2026 sudo usermod -aG docker ${ USER } Logout/in to refresh your session\u2019s group permissions.","title":"docker"},{"location":"cluster/prerequisite-tooling/#kubectl","text":"Kubectl is the main tool for interaction with a Kubernetes cluster. The latest version can be installed with\u2026 mkdir -p $HOME /.local/bin \\ && curl -fsSLo $HOME /.local/bin/kubectl \"https://dl.k8s.io/release/ $( curl -L -s https://dl.k8s.io/release/stable.txt ) /bin/linux/amd64/kubectl\" \\ && chmod +x $HOME /.local/bin/kubectl See the official kubectl installation documentation for more installation options.","title":"kubectl"},{"location":"cluster/prerequisite-tooling/#helm","text":"Helm is the Kubernetes package manager, in which components are deployed to a Kubernetes cluster via helm charts. The helm charts are instantiated for deployment via \u2018values\u2019 that configure the chart templates. The latest helm version can be installed with\u2026 export HELM_INSTALL_DIR = \" $HOME /.local/bin\" \\ && curl -sfL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash See the official helm installation documentation for more installation options.","title":"helm"},{"location":"cluster/prerequisite-tooling/#minikube","text":"Minikube is a tool that allows to create a local (single-node) Kubernetes cluster for development/testing. It is not designed for production use. In the absence of access to a \u2018full\u2019 Kubernetes cluster, this guide can be followed using minikube. The latest version of minikube can be installed with\u2026 mkdir -p $HOME /.local/bin \\ && curl -fsSLo $HOME /.local/bin/minikube \"https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\" \\ && chmod +x $HOME /.local/bin/minikube See the official minikube installation documentation for more installation options.","title":"minikube"},{"location":"eoepca/ades/","text":"Application Deployment & Execution Service (ADES) \u2693\ufe0e The ADES provides a platform-hosted execution engine through which users can initiate parameterised processing jobs using applications made available within the platform - supporting the efficient execution of the processing \u2018close to the data\u2019. Users can deploy specific \u2018applications\u2019 to the ADES, which may be their own applications, or those published by other platform users. Helm Chart \u2693\ufe0e The ADES is deployed via the ades helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the ades chart . helm install --version 2 .0.4 --values ades-values.yaml ades eoepca/ades Values \u2693\ufe0e At minimum, values for the following attributes should be specified: Details of the S3 Object Store for stage-out of processing results Dynamic provisioning StorageClass of ReadWriteMany storage (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the ADES will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Example ades-values.yaml \u2026 workflowExecutor : inputs : # Stage-in from CREODIAS eodata # (only works within CREODIAS - i.e. on a Cloudferro VM) STAGEIN_AWS_SERVICEURL : http://data.cloudferro.com STAGEIN_AWS_ACCESS_KEY_ID : test STAGEIN_AWS_SECRET_ACCESS_KEY : test STAGEIN_AWS_REGION : RegionOne # Stage-out to minio S3 # (use this if the ADES is not configured to stage-out to the Workspace) STAGEOUT_AWS_SERVICEURL : http://minio.192-168-49-2.nip.io STAGEOUT_AWS_ACCESS_KEY_ID : eoepca STAGEOUT_AWS_SECRET_ACCESS_KEY : changeme STAGEOUT_AWS_REGION : us-east-1 STAGEOUT_OUTPUT : s3://eoepca processingStorageClass : standard processingVolumeTmpSize : \"6Gi\" processingVolumeOutputSize : \"6Gi\" processingMaxRam : \"8Gi\" processingMaxCores : \"4\" wps : pepBaseUrl : \"http://ades-pep:5576\" usePep : \"true\" persistence : storageClass : standard ingress : enabled : true annotations : kubernetes.io/ingress.class : nginx ingress.kubernetes.io/ssl-redirect : \"false\" hosts : - host : ades.192-168-49-2.nip.io paths : - path : / pathType : ImplementationSpecific tls : - hosts : - ades.192-168-49-2.nip.io secretName : ades-tls Stage-in / Stage-out Configuration \u2693\ufe0e The ADES hosts applications that are deployed and invoked in accordance with the OGC Best Practise for Application Package . Thus, the ADES provides a conformant environment within which the application is integrated for execution. A key part of the ADES\u2019s role in this is to faciltate the provision of input data to the application (stage-in), and the handling of the results output at the conclusion of application execution (stage-out). The ADES helm chart configures (by default) the ADES with implementations of the stage-in and stage-out functions that use the Spatio Temporal Asset Router Services utility. The ADES provides hooks for system integrators to override these defaults to implement their own stage-in and stage-out behaviour - for example, to integrate with their platform\u2019s own catalogue and data offering. The stage-in and stage-out are specified as CWL via the helm values\u2026 workflowExecutor : stagein : cwl : | cwlVersion: v1.0 ... stageout : cwl : | cwlVersion: v1.0 ... For a detailed description see ADES stage-in/out configuration in the ADES wiki . Workspace Integration \u2693\ufe0e The ADES has the facility to integrate with the EOEPCA Workspace component for registration of staged-out processing results. This is disabled by default ( useResourceManager: false ). When enabled, the ADES will register the staged-out products with the user\u2019s Workspace, such that they are indexed and available via the user\u2019s Resource Catalogue and Data Access services. Example ades-values.yaml (snippet)\u2026 workflowExecutor : ... useResourceManager : \"true\" resourceManagerWorkspacePrefix : \"guide-user\" resourceManagerEndpoint : \"https://workspace-api.192-168-49-2.nip.io\" platformDomain : \"https://auth.192-168-49-2.nip.io\" ... The value resourceManagerWorkspacePrefix must be consistent with that configured for the Workspace API deployment , (ref. value prefixForName ). Protection \u2693\ufe0e As described in section Resource Protection , the resource-guard component can be inserted into the request path of the ADES service to provide access authorization decisions helm install --version 1 .2.1 --values ades-guard-values.yaml ades-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the ADES for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent \u2026 Example ades-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : ades domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth # customDefaultResources: # - name: \"ADES Service for user 'eric'\" # description: \"Protected Access for eric to his space in the ADES\" # resource_uri: \"/eric\" # scopes: [] # default_owner: \"a9812efe-fc0c-49d3-8115-0f36883a84b9\" # - name: \"ADES Service for user 'bob'\" # description: \"Protected Access for bob to his space in the ADES\" # resource_uri: \"/bob\" # scopes: [] # default_owner: \"4ccae3a1-3fad-4ffe-bfa7-cce851143780\" volumeClaim : name : eoepca-proc-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : ades paths : - path : /(.*) service : name : ades port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"proc-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint Client Secret \u2693\ufe0e The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n proc create secret generic proc-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > proc-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : proc-client namespace : proc data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io. In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml ADES Usage Samples \u2693\ufe0e This section includes some sample requests to test the deployed ADES. NOTES: It assumed that the ADES is subject to access protection (ref. Resource Protection ), in which case a User ID Token must be provided with the request - typically in the HTTP header, such as Authorization: Bearer or X-User-Id . See section User ID Token for more details. The samples assume a user eric The snuggs application is used in the example below. See also Application Package Example . List Processes \u2693\ufe0e List available processes. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/wps3/processes' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Deploy Process \u2693\ufe0e Deploy the sample application snuggs . curl --location --request POST 'https://ades.192-168-49-2.nip.io/eric/wps3/processes' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' \\ --header 'Content-Type: application/json' \\ --data-raw '{ \"inputs\": [ { \"id\": \"applicationPackage\", \"input\": { \"format\": { \"mimeType\": \"application/cwl\" }, \"value\": { \"href\": \"https://raw.githubusercontent.com/EOEPCA/app-snuggs/main/app-package.cwl\" } } } ], \"outputs\": [ { \"format\": { \"mimeType\": \"string\", \"schema\": \"string\", \"encoding\": \"string\" }, \"id\": \"deployResult\", \"transmissionMode\": \"value\" } ], \"mode\": \"auto\", \"response\": \"raw\" }' Get Process Details \u2693\ufe0e Get details for a deployed process. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Execute Process \u2693\ufe0e Execute a process with supplied parameterisation. curl --location --request POST 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0/jobs' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' \\ --header 'Content-Type: application/json' \\ --data-raw '{ \"inputs\": [ { \"id\": \"input_reference\", \"input\": { \"dataType\": { \"name\": \"application/json\" }, \"value\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\" } }, { \"id\": \"s_expression\", \"input\": { \"dataType\": { \"name\": \"string\" }, \"value\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\" } } ], \"outputs\": [ { \"format\": { \"mimeType\": \"string\", \"schema\": \"string\", \"encoding\": \"string\" }, \"id\": \"wf_outputs\", \"transmissionMode\": \"value\" } ], \"mode\": \"auto\", \"response\": \"raw\" }' Job Status \u2693\ufe0e Once a processes execution has been initiated then its progress can monitored via a job-specific URL that is returned in the HTTP response headers of the execute request. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/watchjob/processes/snuggs-0_3_0/jobs/2e0fabf4-4ed6-11ec-b857-626a98159388' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Job Result \u2693\ufe0e Once the job execution has completed, then the results can be obtained. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/watchjob/processes/snuggs-0_3_0/jobs/2e0fabf4-4ed6-11ec-b857-626a98159388/result' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Undeploy Process \u2693\ufe0e A process can be deleted (undeployed). curl --location --request DELETE 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Application Package Example \u2693\ufe0e For a (trivial) example application package see Example Application Package , which provides a description and illustration of the basics of creating an application that integrates with the expectations of the ADES stage-in and stage-out. For further reference see\u2026 Application Packages OGC Best Practise for Application Package Example Application Package Common Workflow Language (CWL) Guide for CWL in Earth Observation CWL Specification CWL User Guide Additional Information \u2693\ufe0e Additional information regarding the ADES can be found at: Helm Chart Wiki GitHub Repository ADES stage-in/out configuration","title":"Application Deployment & Execution"},{"location":"eoepca/ades/#application-deployment-execution-service-ades","text":"The ADES provides a platform-hosted execution engine through which users can initiate parameterised processing jobs using applications made available within the platform - supporting the efficient execution of the processing \u2018close to the data\u2019. Users can deploy specific \u2018applications\u2019 to the ADES, which may be their own applications, or those published by other platform users.","title":"Application Deployment &amp; Execution Service (ADES)"},{"location":"eoepca/ades/#helm-chart","text":"The ADES is deployed via the ades helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the ades chart . helm install --version 2 .0.4 --values ades-values.yaml ades eoepca/ades","title":"Helm Chart"},{"location":"eoepca/ades/#values","text":"At minimum, values for the following attributes should be specified: Details of the S3 Object Store for stage-out of processing results Dynamic provisioning StorageClass of ReadWriteMany storage (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the ADES will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Example ades-values.yaml \u2026 workflowExecutor : inputs : # Stage-in from CREODIAS eodata # (only works within CREODIAS - i.e. on a Cloudferro VM) STAGEIN_AWS_SERVICEURL : http://data.cloudferro.com STAGEIN_AWS_ACCESS_KEY_ID : test STAGEIN_AWS_SECRET_ACCESS_KEY : test STAGEIN_AWS_REGION : RegionOne # Stage-out to minio S3 # (use this if the ADES is not configured to stage-out to the Workspace) STAGEOUT_AWS_SERVICEURL : http://minio.192-168-49-2.nip.io STAGEOUT_AWS_ACCESS_KEY_ID : eoepca STAGEOUT_AWS_SECRET_ACCESS_KEY : changeme STAGEOUT_AWS_REGION : us-east-1 STAGEOUT_OUTPUT : s3://eoepca processingStorageClass : standard processingVolumeTmpSize : \"6Gi\" processingVolumeOutputSize : \"6Gi\" processingMaxRam : \"8Gi\" processingMaxCores : \"4\" wps : pepBaseUrl : \"http://ades-pep:5576\" usePep : \"true\" persistence : storageClass : standard ingress : enabled : true annotations : kubernetes.io/ingress.class : nginx ingress.kubernetes.io/ssl-redirect : \"false\" hosts : - host : ades.192-168-49-2.nip.io paths : - path : / pathType : ImplementationSpecific tls : - hosts : - ades.192-168-49-2.nip.io secretName : ades-tls","title":"Values"},{"location":"eoepca/ades/#stage-in-stage-out-configuration","text":"The ADES hosts applications that are deployed and invoked in accordance with the OGC Best Practise for Application Package . Thus, the ADES provides a conformant environment within which the application is integrated for execution. A key part of the ADES\u2019s role in this is to faciltate the provision of input data to the application (stage-in), and the handling of the results output at the conclusion of application execution (stage-out). The ADES helm chart configures (by default) the ADES with implementations of the stage-in and stage-out functions that use the Spatio Temporal Asset Router Services utility. The ADES provides hooks for system integrators to override these defaults to implement their own stage-in and stage-out behaviour - for example, to integrate with their platform\u2019s own catalogue and data offering. The stage-in and stage-out are specified as CWL via the helm values\u2026 workflowExecutor : stagein : cwl : | cwlVersion: v1.0 ... stageout : cwl : | cwlVersion: v1.0 ... For a detailed description see ADES stage-in/out configuration in the ADES wiki .","title":"Stage-in / Stage-out Configuration"},{"location":"eoepca/ades/#workspace-integration","text":"The ADES has the facility to integrate with the EOEPCA Workspace component for registration of staged-out processing results. This is disabled by default ( useResourceManager: false ). When enabled, the ADES will register the staged-out products with the user\u2019s Workspace, such that they are indexed and available via the user\u2019s Resource Catalogue and Data Access services. Example ades-values.yaml (snippet)\u2026 workflowExecutor : ... useResourceManager : \"true\" resourceManagerWorkspacePrefix : \"guide-user\" resourceManagerEndpoint : \"https://workspace-api.192-168-49-2.nip.io\" platformDomain : \"https://auth.192-168-49-2.nip.io\" ... The value resourceManagerWorkspacePrefix must be consistent with that configured for the Workspace API deployment , (ref. value prefixForName ).","title":"Workspace Integration"},{"location":"eoepca/ades/#protection","text":"As described in section Resource Protection , the resource-guard component can be inserted into the request path of the ADES service to provide access authorization decisions helm install --version 1 .2.1 --values ades-guard-values.yaml ades-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the ADES for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent \u2026 Example ades-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : ades domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth # customDefaultResources: # - name: \"ADES Service for user 'eric'\" # description: \"Protected Access for eric to his space in the ADES\" # resource_uri: \"/eric\" # scopes: [] # default_owner: \"a9812efe-fc0c-49d3-8115-0f36883a84b9\" # - name: \"ADES Service for user 'bob'\" # description: \"Protected Access for bob to his space in the ADES\" # resource_uri: \"/bob\" # scopes: [] # default_owner: \"4ccae3a1-3fad-4ffe-bfa7-cce851143780\" volumeClaim : name : eoepca-proc-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : ades paths : - path : /(.*) service : name : ades port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"proc-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint","title":"Protection"},{"location":"eoepca/ades/#client-secret","text":"The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n proc create secret generic proc-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > proc-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : proc-client namespace : proc data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io. In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml","title":"Client Secret"},{"location":"eoepca/ades/#ades-usage-samples","text":"This section includes some sample requests to test the deployed ADES. NOTES: It assumed that the ADES is subject to access protection (ref. Resource Protection ), in which case a User ID Token must be provided with the request - typically in the HTTP header, such as Authorization: Bearer or X-User-Id . See section User ID Token for more details. The samples assume a user eric The snuggs application is used in the example below. See also Application Package Example .","title":"ADES Usage Samples"},{"location":"eoepca/ades/#list-processes","text":"List available processes. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/wps3/processes' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"List Processes"},{"location":"eoepca/ades/#deploy-process","text":"Deploy the sample application snuggs . curl --location --request POST 'https://ades.192-168-49-2.nip.io/eric/wps3/processes' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' \\ --header 'Content-Type: application/json' \\ --data-raw '{ \"inputs\": [ { \"id\": \"applicationPackage\", \"input\": { \"format\": { \"mimeType\": \"application/cwl\" }, \"value\": { \"href\": \"https://raw.githubusercontent.com/EOEPCA/app-snuggs/main/app-package.cwl\" } } } ], \"outputs\": [ { \"format\": { \"mimeType\": \"string\", \"schema\": \"string\", \"encoding\": \"string\" }, \"id\": \"deployResult\", \"transmissionMode\": \"value\" } ], \"mode\": \"auto\", \"response\": \"raw\" }'","title":"Deploy Process"},{"location":"eoepca/ades/#get-process-details","text":"Get details for a deployed process. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"Get Process Details"},{"location":"eoepca/ades/#execute-process","text":"Execute a process with supplied parameterisation. curl --location --request POST 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0/jobs' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' \\ --header 'Content-Type: application/json' \\ --data-raw '{ \"inputs\": [ { \"id\": \"input_reference\", \"input\": { \"dataType\": { \"name\": \"application/json\" }, \"value\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\" } }, { \"id\": \"s_expression\", \"input\": { \"dataType\": { \"name\": \"string\" }, \"value\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\" } } ], \"outputs\": [ { \"format\": { \"mimeType\": \"string\", \"schema\": \"string\", \"encoding\": \"string\" }, \"id\": \"wf_outputs\", \"transmissionMode\": \"value\" } ], \"mode\": \"auto\", \"response\": \"raw\" }'","title":"Execute Process"},{"location":"eoepca/ades/#job-status","text":"Once a processes execution has been initiated then its progress can monitored via a job-specific URL that is returned in the HTTP response headers of the execute request. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/watchjob/processes/snuggs-0_3_0/jobs/2e0fabf4-4ed6-11ec-b857-626a98159388' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"Job Status"},{"location":"eoepca/ades/#job-result","text":"Once the job execution has completed, then the results can be obtained. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/watchjob/processes/snuggs-0_3_0/jobs/2e0fabf4-4ed6-11ec-b857-626a98159388/result' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"Job Result"},{"location":"eoepca/ades/#undeploy-process","text":"A process can be deleted (undeployed). curl --location --request DELETE 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"Undeploy Process"},{"location":"eoepca/ades/#application-package-example","text":"For a (trivial) example application package see Example Application Package , which provides a description and illustration of the basics of creating an application that integrates with the expectations of the ADES stage-in and stage-out. For further reference see\u2026 Application Packages OGC Best Practise for Application Package Example Application Package Common Workflow Language (CWL) Guide for CWL in Earth Observation CWL Specification CWL User Guide","title":"Application Package Example"},{"location":"eoepca/ades/#additional-information","text":"Additional information regarding the ADES can be found at: Helm Chart Wiki GitHub Repository ADES stage-in/out configuration","title":"Additional Information"},{"location":"eoepca/container-registry/","text":"Container Registry \u2693\ufe0e To support the development (ref. Processor Development Environment ) and deployment/execution (ref. ADES ) of user-defined applications, we deploy a container registry to host container images. This is provied by a deployment of the Harbor artefact repository . Helm Chart \u2693\ufe0e Harbor is deployed via the harbor helm chart from the Harbor Helm Chart Repository . helm install --values harbor-values.yaml harbor harbor --repo https://helm.goharbor.io Values \u2693\ufe0e The chart is configured via values that are fully documented on the Harbor website . Example\u2026 expose : ingress : annotations : kubernetes.io/ingress.class : nginx cert-manager.io/cluster-issuer : letsencrypt-production nginx.ingress.kubernetes.io/proxy-read-timeout : '600' # from chart: ingress.kubernetes.io/ssl-redirect : letsencrypt-production ingress.kubernetes.io/proxy-body-size : \"0\" nginx.ingress.kubernetes.io/ssl-redirect : letsencrypt-production nginx.ingress.kubernetes.io/proxy-body-size : \"0\" hosts : core : harbor.192-168-49-2.nip.io notary : harbor-notary.192-168-49-2.nip.io persistence : persistentVolumeClaim : registry : storageClass : standard chartmuseum : storageClass : standard jobservice : storageClass : standard database : storageClass : standard redis : storageClass : standard trivy : storageClass : standard externalURL : https://harbor.192-168-49-2.nip.io # initial password for logging in with user \"admin\" harborAdminPassword : \"changeme\" chartmuseum : enabled : false trivy : enabled : false notary : enabled : false NOTES: We specify use of \u2018valid\u2019 certificates from Letsencrypt \u2018production\u2019. The Workspace API, which calls the Harbor API, expects valid certificates and will thus fail if presented with TLS certificates that fail validation. The letsencrypt-production Cluster Issuer relies upon the deployment being accessible from the public internet via the expose.ingress.hosts.core DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller. The Workspace API will not like this. Container Registry Usage \u2693\ufe0e After deployemnt Harbor is accessible via its web interface at https://harbor.<domain>/ e.g. https://harbor.192-168-49-2.nip.io/ . Login as the admin user with the password specified in the helm values. Additional Information \u2693\ufe0e Additional information regarding the Container Registry can be found at: Web Site Helm Chart Repository Helm Chart Description Harbor Documentation","title":"Container Registry"},{"location":"eoepca/container-registry/#container-registry","text":"To support the development (ref. Processor Development Environment ) and deployment/execution (ref. ADES ) of user-defined applications, we deploy a container registry to host container images. This is provied by a deployment of the Harbor artefact repository .","title":"Container Registry"},{"location":"eoepca/container-registry/#helm-chart","text":"Harbor is deployed via the harbor helm chart from the Harbor Helm Chart Repository . helm install --values harbor-values.yaml harbor harbor --repo https://helm.goharbor.io","title":"Helm Chart"},{"location":"eoepca/container-registry/#values","text":"The chart is configured via values that are fully documented on the Harbor website . Example\u2026 expose : ingress : annotations : kubernetes.io/ingress.class : nginx cert-manager.io/cluster-issuer : letsencrypt-production nginx.ingress.kubernetes.io/proxy-read-timeout : '600' # from chart: ingress.kubernetes.io/ssl-redirect : letsencrypt-production ingress.kubernetes.io/proxy-body-size : \"0\" nginx.ingress.kubernetes.io/ssl-redirect : letsencrypt-production nginx.ingress.kubernetes.io/proxy-body-size : \"0\" hosts : core : harbor.192-168-49-2.nip.io notary : harbor-notary.192-168-49-2.nip.io persistence : persistentVolumeClaim : registry : storageClass : standard chartmuseum : storageClass : standard jobservice : storageClass : standard database : storageClass : standard redis : storageClass : standard trivy : storageClass : standard externalURL : https://harbor.192-168-49-2.nip.io # initial password for logging in with user \"admin\" harborAdminPassword : \"changeme\" chartmuseum : enabled : false trivy : enabled : false notary : enabled : false NOTES: We specify use of \u2018valid\u2019 certificates from Letsencrypt \u2018production\u2019. The Workspace API, which calls the Harbor API, expects valid certificates and will thus fail if presented with TLS certificates that fail validation. The letsencrypt-production Cluster Issuer relies upon the deployment being accessible from the public internet via the expose.ingress.hosts.core DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller. The Workspace API will not like this.","title":"Values"},{"location":"eoepca/container-registry/#container-registry-usage","text":"After deployemnt Harbor is accessible via its web interface at https://harbor.<domain>/ e.g. https://harbor.192-168-49-2.nip.io/ . Login as the admin user with the password specified in the helm values.","title":"Container Registry Usage"},{"location":"eoepca/container-registry/#additional-information","text":"Additional information regarding the Container Registry can be found at: Web Site Helm Chart Repository Helm Chart Description Harbor Documentation","title":"Additional Information"},{"location":"eoepca/data-access/","text":"Data Access \u2693\ufe0e The Data Access provides standards-based services for access to platform hosted data - including OGC WMS/WMTS for visualisation, and OGC WCS for data retrieval. This component also includes Harvester and Registrar services to discover/watch the existing data holding of the infrastructure data layer and populate/maintain the data access and resource catalogue services accordingly. Helm Chart \u2693\ufe0e The Data Access is deployed via the data-access helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are supplied with the instantiation of the helm release. The EOEPCA data-access chart provides a thin wrapper around the EOX View Server ( vs ) helm chart. The documentation for the View Server can be found here: User Guide: https://vs.pages.eox.at/documentation/user/main/ Operator Guide: https://vs.pages.eox.at/documentation/operator/main/ helm install --version 1 .2.5 --values data-access-values.yaml data-access data-access Values \u2693\ufe0e The Data Access supports many values to configure the service. These are documented in full in the View Server - Operator Guide Configuration page . Core Configuration \u2693\ufe0e Typically, values for the following attributes may be specified to override the chart defaults: The fully-qualified public URL for the service, ref. ( global.ingress.hosts.host[0] ) Metadata describing the service instance Dynamic provisioning StorageClass for persistence Persistent Volume Claims for database and redis components Object storage details for data and cache Container images for renderer and registrar (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Data Access will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . global : env : REGISTRAR_REPLACE : \"true\" CPL_VSIL_CURL_ALLOWED_EXTENSIONS : .TIF,.tif,.xml,.jp2,.jpg,.jpeg AWS_HTTPS : \"FALSE\" startup_scripts : - /registrar_pycsw/registrar_pycsw/initialize-collections.sh ingress : enabled : true annotations : kubernetes.io/ingress.class : nginx kubernetes.io/tls-acme : \"true\" nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" cert-manager.io/cluster-issuer : letsencrypt-production hosts : - host : data-access.192-168-49-2.nip.io tls : - hosts : - data-access.192-168-49-2.nip.io secretName : data-access-tls storage : data : data : type : S3 endpoint_url : http://data.cloudferro.com access_key_id : access secret_access_key : access region_name : RegionOne validate_bucket_name : false cache : type : S3 endpoint_url : \"http://minio.192-168-49-2.nip.io\" host : \"minio.192-168-49-2.nip.io\" access_key_id : xxx secret_access_key : xxx region : us-east-1 bucket : cache-bucket metadata : title : EOEPCA Data Access Service developed by EOX abstract : EOEPCA Data Access Service developed by EOX header : \"EOEPCA Data Access View Server (VS) Client powered by <a href=\\\"//eox.at\\\"><img src=\\\"//eox.at/wp-content/uploads/2017/09/EOX_Logo.svg\\\" alt=\\\"EOX\\\" style=\\\"height:25px;margin-left:10px\\\"/></a>\" url : https://data-access.192-168-49-2.nip.io/ows layers : # see section 'Data-layer Configuration' collections : # see section 'Data-layer Configuration' productTypes : # see section 'Data-layer Configuration' vs : renderer : replicaCount : 4 ingress : enabled : false registrar : replicaCount : 1 harvester : # see section 'Harvester Configuration' replicaCount : 1 client : replicaCount : 1 ingress : enabled : false redis : master : persistence : enabled : true storageClass : standard ingestor : replicaCount : 0 ingress : enabled : false preprocessor : replicaCount : 0 cache : ingress : enabled : false Data-layer Configuration \u2693\ufe0e Configuration of the service data-layer - as described in the View Server Operator Guide . The data-access service data handling is configured by definition of productTypes , collections and layers \u2026 productTypes - Product Types Identify the underlying file assets as WCS coverages and their visual representation collections - Data Collections Provides groupings into which products are organised layers - Layers Specifies the hoe the product visual representations are exposed through the WMS service For more information, see the worked example in section Data Specification for the example CREODIAS deployment . Harvester \u2693\ufe0e The Data Access service includes a Harvester component. The following subsections describe its configuration and usage. Harvester Helm Configuration \u2693\ufe0e The Harvester can be configured through the helm chart values\u2026 vs : harvester : replicaCount : 1 config : redis : host : data-access-redis-master port : 6379 harvesters : - name : Creodias-Opensearch resource : url : https://finder.creodias.eu/resto/api/collections/Sentinel2/describe.xml type : OpenSearch format_config : type : 'application/json' property_mapping : start_datetime : 'startDate' end_datetime : 'completionDate' productIdentifier : 'productIdentifier' query : time : property : sensed begin : 2019-09-10T00:00:00Z end : 2019-09-11T00:00:00Z collection : null bbox : 14.9,47.7,16.4,48.7 filter : {} postprocess : - type : harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor queue : register The harvester.config.harvesters list defines a set of pre-defined harvesters which can be invoked in a later stage. The name property must be unique for each harvester and must be unique among all harvesters in the list. Each harvester is associated with a resource , an optional filter or postprocess function, and a queue . The resource defines where each item is harvested from. This can be a file system, a search service, catalog file or something similar. The example above defines a connection to an OpenSearch service on CREODIAS, with associated default query parameters and a format configuration. The filter allows to filter elements within the harvester, when the resource does not provide a specific filter. This filter can be supplied using CQL2-JSON. The postprocess can adjust the harvested results. In this example the harvested items are not complete, and additional metadata must be retrieved from an object storage. The queue defines where harvested items will be pushed into. Usually this is a registration queue, where the registrar will pick up and start registration according to its configuration. Starting the Harvester \u2693\ufe0e The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue. One-off harvests via the CLI \u2693\ufe0e In order to start a harvest from the CLI, the operator first needs to connect to the kubernetes pod of the harvester. Within that pod, the harvest can be executed like this\u2026 python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch This will invoke the Creodias-Opensearch harvester with default arguments. When some values are to be overridden, the \u2013values switch can be used to pass override values. These values must be a JSON string. The following example adjusts the begin and end times of the query parameters\u2026 python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch --values '{\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}' Harvests via the harvest daemon \u2693\ufe0e The harvester pod runs a service listening on a redis queue. When a message is read from the queue, it will be read as a JSON string, expecting an object with at least a name property. Optionally, it can also have a values property, working in the same way as with CLI --values . To send a harvesting request via the redis queue, it is necessary to connect to the redis pod and execute the redis-cli there. Then the following command can be used to achieve the same result as above with CLI harvesting\u2026 redis-cli LPUSH '{\"name\": \"Creodias-Opensearch\", \"values\": {\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}}' Results of the harvesting \u2693\ufe0e The harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created. Storage \u2693\ufe0e Specification of PVCs and access to object storage. Persistent Volume Claims \u2693\ufe0e The PVCs specified in the helm chart values must be created. PVC for Database \u2693\ufe0e kind : PersistentVolumeClaim apiVersion : v1 metadata : name : data-access-db namespace : rm labels : k8s-app : data-access name : data-access spec : storageClassName : standard accessModes : - ReadWriteMany resources : requests : storage : 100Gi PVC for Redis \u2693\ufe0e kind : PersistentVolumeClaim apiVersion : v1 metadata : name : data-access-redis namespace : rm labels : k8s-app : data-access name : data-access spec : storageClassName : standard accessModes : - ReadWriteMany resources : requests : storage : 1Gi Object Storage \u2693\ufe0e The helm chart values expect specification of object storage details for: data : to access the EO data of the underlying infrastructure cache : a dedicated object storage bucket is used to support the cache function of the data access services Platform EO Data \u2693\ufe0e Specifies the details for the infrastructure object storage that provides direct access to the EO product files. For example, the CREODIAS metadata catalogue provides references to product files in their eodata object storage - the access details for which are configured in the data access services: global : storage : data : data : type : S3 endpoint_url : http://data.cloudferro.com access_key_id : access secret_access_key : access region_name : RegionOne validate_bucket_name : false Data Access Cache \u2693\ufe0e The Data Access services maintain a cache, which relies on the usage of a dedicate object storage bucket for data persistence. This bucket must be created (manual step) and its access details configured in the data access services. Example based upon CREODIAS: global : storage : cache : type : S3 endpoint_url : \"https://cf2.cloudferro.com:8080/cache-bucket\" host : \"cf2.cloudferro.com:8080\" access_key_id : xxx secret_access_key : xxx region : RegionOne bucket : cache-bucket \u2026where xxx must be replaced with the bucket credentials. Protection \u2693\ufe0e As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Data Access service to provide access authorization decisions. helm install --version 1 .2.1 --values data-access-guard-values.yaml data-access-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Data Access for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example data-access-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : data-access domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : data-access paths : - path : /(ows.*) service : name : data-access-renderer port : 80 - path : /(opensearch.*) service : name : data-access-renderer port : 80 - path : /(coverages/metadata.*) service : name : data-access-renderer port : 80 - path : /(admin.*) service : name : data-access-renderer port : 80 - path : /cache/(.*) service : name : data-access-cache port : 80 - path : /(.*) service : name : data-access-client port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint Client Secret \u2693\ufe0e The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io. In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml Data Access Usage \u2693\ufe0e Default Harvesting \u2693\ufe0e At deployment time the harvester helm values include configuration that populates a default harvester configuration, that is prepared in the file /config.yaml in the harvester pod. The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section Data-layer Configuration . The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the rm/harvester service and executing the command\u2026 python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch Ad-hoc Harvesting \u2693\ufe0e Ad-hoc harvesting can be invoked by provision of a suitable config.yaml into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time. The helper script ./deploy/bin/harvest faciltates this\u2026 ./deploy/bin/harvest <path-to-config-file> See directory ./deploy/samples/harvester/ that contains some sample harvesting configuration files. For example\u2026 ./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2019.09.10.yaml Registration of Collections \u2693\ufe0e The helper script ./deploy/bin/register-collection is provided to faciltate the registration of collections that are specfied in STAC Collection format. ./deploy/bin/register-collection <path-to-stac-collection-file> See directory ./deploy/samples/collections/ that contains some same STAC Collection files. For example\u2026 ./deploy/bin/register-collection ./deploy/samples/collections/S2MSI2A.json Additional Information \u2693\ufe0e Additional information regarding the Data Access can be found at: Helm Chart Documentation: User Guide Operator Guide Git Repository","title":"Data Access"},{"location":"eoepca/data-access/#data-access","text":"The Data Access provides standards-based services for access to platform hosted data - including OGC WMS/WMTS for visualisation, and OGC WCS for data retrieval. This component also includes Harvester and Registrar services to discover/watch the existing data holding of the infrastructure data layer and populate/maintain the data access and resource catalogue services accordingly.","title":"Data Access"},{"location":"eoepca/data-access/#helm-chart","text":"The Data Access is deployed via the data-access helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are supplied with the instantiation of the helm release. The EOEPCA data-access chart provides a thin wrapper around the EOX View Server ( vs ) helm chart. The documentation for the View Server can be found here: User Guide: https://vs.pages.eox.at/documentation/user/main/ Operator Guide: https://vs.pages.eox.at/documentation/operator/main/ helm install --version 1 .2.5 --values data-access-values.yaml data-access data-access","title":"Helm Chart"},{"location":"eoepca/data-access/#values","text":"The Data Access supports many values to configure the service. These are documented in full in the View Server - Operator Guide Configuration page .","title":"Values"},{"location":"eoepca/data-access/#core-configuration","text":"Typically, values for the following attributes may be specified to override the chart defaults: The fully-qualified public URL for the service, ref. ( global.ingress.hosts.host[0] ) Metadata describing the service instance Dynamic provisioning StorageClass for persistence Persistent Volume Claims for database and redis components Object storage details for data and cache Container images for renderer and registrar (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Data Access will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . global : env : REGISTRAR_REPLACE : \"true\" CPL_VSIL_CURL_ALLOWED_EXTENSIONS : .TIF,.tif,.xml,.jp2,.jpg,.jpeg AWS_HTTPS : \"FALSE\" startup_scripts : - /registrar_pycsw/registrar_pycsw/initialize-collections.sh ingress : enabled : true annotations : kubernetes.io/ingress.class : nginx kubernetes.io/tls-acme : \"true\" nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" cert-manager.io/cluster-issuer : letsencrypt-production hosts : - host : data-access.192-168-49-2.nip.io tls : - hosts : - data-access.192-168-49-2.nip.io secretName : data-access-tls storage : data : data : type : S3 endpoint_url : http://data.cloudferro.com access_key_id : access secret_access_key : access region_name : RegionOne validate_bucket_name : false cache : type : S3 endpoint_url : \"http://minio.192-168-49-2.nip.io\" host : \"minio.192-168-49-2.nip.io\" access_key_id : xxx secret_access_key : xxx region : us-east-1 bucket : cache-bucket metadata : title : EOEPCA Data Access Service developed by EOX abstract : EOEPCA Data Access Service developed by EOX header : \"EOEPCA Data Access View Server (VS) Client powered by <a href=\\\"//eox.at\\\"><img src=\\\"//eox.at/wp-content/uploads/2017/09/EOX_Logo.svg\\\" alt=\\\"EOX\\\" style=\\\"height:25px;margin-left:10px\\\"/></a>\" url : https://data-access.192-168-49-2.nip.io/ows layers : # see section 'Data-layer Configuration' collections : # see section 'Data-layer Configuration' productTypes : # see section 'Data-layer Configuration' vs : renderer : replicaCount : 4 ingress : enabled : false registrar : replicaCount : 1 harvester : # see section 'Harvester Configuration' replicaCount : 1 client : replicaCount : 1 ingress : enabled : false redis : master : persistence : enabled : true storageClass : standard ingestor : replicaCount : 0 ingress : enabled : false preprocessor : replicaCount : 0 cache : ingress : enabled : false","title":"Core Configuration"},{"location":"eoepca/data-access/#data-layer-configuration","text":"Configuration of the service data-layer - as described in the View Server Operator Guide . The data-access service data handling is configured by definition of productTypes , collections and layers \u2026 productTypes - Product Types Identify the underlying file assets as WCS coverages and their visual representation collections - Data Collections Provides groupings into which products are organised layers - Layers Specifies the hoe the product visual representations are exposed through the WMS service For more information, see the worked example in section Data Specification for the example CREODIAS deployment .","title":"Data-layer Configuration"},{"location":"eoepca/data-access/#harvester","text":"The Data Access service includes a Harvester component. The following subsections describe its configuration and usage.","title":"Harvester"},{"location":"eoepca/data-access/#harvester-helm-configuration","text":"The Harvester can be configured through the helm chart values\u2026 vs : harvester : replicaCount : 1 config : redis : host : data-access-redis-master port : 6379 harvesters : - name : Creodias-Opensearch resource : url : https://finder.creodias.eu/resto/api/collections/Sentinel2/describe.xml type : OpenSearch format_config : type : 'application/json' property_mapping : start_datetime : 'startDate' end_datetime : 'completionDate' productIdentifier : 'productIdentifier' query : time : property : sensed begin : 2019-09-10T00:00:00Z end : 2019-09-11T00:00:00Z collection : null bbox : 14.9,47.7,16.4,48.7 filter : {} postprocess : - type : harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor queue : register The harvester.config.harvesters list defines a set of pre-defined harvesters which can be invoked in a later stage. The name property must be unique for each harvester and must be unique among all harvesters in the list. Each harvester is associated with a resource , an optional filter or postprocess function, and a queue . The resource defines where each item is harvested from. This can be a file system, a search service, catalog file or something similar. The example above defines a connection to an OpenSearch service on CREODIAS, with associated default query parameters and a format configuration. The filter allows to filter elements within the harvester, when the resource does not provide a specific filter. This filter can be supplied using CQL2-JSON. The postprocess can adjust the harvested results. In this example the harvested items are not complete, and additional metadata must be retrieved from an object storage. The queue defines where harvested items will be pushed into. Usually this is a registration queue, where the registrar will pick up and start registration according to its configuration.","title":"Harvester Helm Configuration"},{"location":"eoepca/data-access/#starting-the-harvester","text":"The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue.","title":"Starting the Harvester"},{"location":"eoepca/data-access/#results-of-the-harvesting","text":"The harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created.","title":"Results of the harvesting"},{"location":"eoepca/data-access/#storage","text":"Specification of PVCs and access to object storage.","title":"Storage"},{"location":"eoepca/data-access/#persistent-volume-claims","text":"The PVCs specified in the helm chart values must be created.","title":"Persistent Volume Claims"},{"location":"eoepca/data-access/#pvc-for-database","text":"kind : PersistentVolumeClaim apiVersion : v1 metadata : name : data-access-db namespace : rm labels : k8s-app : data-access name : data-access spec : storageClassName : standard accessModes : - ReadWriteMany resources : requests : storage : 100Gi","title":"PVC for Database"},{"location":"eoepca/data-access/#pvc-for-redis","text":"kind : PersistentVolumeClaim apiVersion : v1 metadata : name : data-access-redis namespace : rm labels : k8s-app : data-access name : data-access spec : storageClassName : standard accessModes : - ReadWriteMany resources : requests : storage : 1Gi","title":"PVC for Redis"},{"location":"eoepca/data-access/#object-storage","text":"The helm chart values expect specification of object storage details for: data : to access the EO data of the underlying infrastructure cache : a dedicated object storage bucket is used to support the cache function of the data access services","title":"Object Storage"},{"location":"eoepca/data-access/#platform-eo-data","text":"Specifies the details for the infrastructure object storage that provides direct access to the EO product files. For example, the CREODIAS metadata catalogue provides references to product files in their eodata object storage - the access details for which are configured in the data access services: global : storage : data : data : type : S3 endpoint_url : http://data.cloudferro.com access_key_id : access secret_access_key : access region_name : RegionOne validate_bucket_name : false","title":"Platform EO Data"},{"location":"eoepca/data-access/#data-access-cache","text":"The Data Access services maintain a cache, which relies on the usage of a dedicate object storage bucket for data persistence. This bucket must be created (manual step) and its access details configured in the data access services. Example based upon CREODIAS: global : storage : cache : type : S3 endpoint_url : \"https://cf2.cloudferro.com:8080/cache-bucket\" host : \"cf2.cloudferro.com:8080\" access_key_id : xxx secret_access_key : xxx region : RegionOne bucket : cache-bucket \u2026where xxx must be replaced with the bucket credentials.","title":"Data Access Cache"},{"location":"eoepca/data-access/#protection","text":"As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Data Access service to provide access authorization decisions. helm install --version 1 .2.1 --values data-access-guard-values.yaml data-access-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Data Access for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example data-access-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : data-access domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : data-access paths : - path : /(ows.*) service : name : data-access-renderer port : 80 - path : /(opensearch.*) service : name : data-access-renderer port : 80 - path : /(coverages/metadata.*) service : name : data-access-renderer port : 80 - path : /(admin.*) service : name : data-access-renderer port : 80 - path : /cache/(.*) service : name : data-access-cache port : 80 - path : /(.*) service : name : data-access-client port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint","title":"Protection"},{"location":"eoepca/data-access/#client-secret","text":"The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io. In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml","title":"Client Secret"},{"location":"eoepca/data-access/#data-access-usage","text":"","title":"Data Access Usage"},{"location":"eoepca/data-access/#default-harvesting","text":"At deployment time the harvester helm values include configuration that populates a default harvester configuration, that is prepared in the file /config.yaml in the harvester pod. The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section Data-layer Configuration . The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the rm/harvester service and executing the command\u2026 python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch","title":"Default Harvesting"},{"location":"eoepca/data-access/#ad-hoc-harvesting","text":"Ad-hoc harvesting can be invoked by provision of a suitable config.yaml into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time. The helper script ./deploy/bin/harvest faciltates this\u2026 ./deploy/bin/harvest <path-to-config-file> See directory ./deploy/samples/harvester/ that contains some sample harvesting configuration files. For example\u2026 ./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2019.09.10.yaml","title":"Ad-hoc Harvesting"},{"location":"eoepca/data-access/#registration-of-collections","text":"The helper script ./deploy/bin/register-collection is provided to faciltate the registration of collections that are specfied in STAC Collection format. ./deploy/bin/register-collection <path-to-stac-collection-file> See directory ./deploy/samples/collections/ that contains some same STAC Collection files. For example\u2026 ./deploy/bin/register-collection ./deploy/samples/collections/S2MSI2A.json","title":"Registration of Collections"},{"location":"eoepca/data-access/#additional-information","text":"Additional information regarding the Data Access can be found at: Helm Chart Documentation: User Guide Operator Guide Git Repository","title":"Additional Information"},{"location":"eoepca/login-service/","text":"Login Service \u2693\ufe0e The Login Service provides the platform Authorization Server for authenticated user identity and request authorization. Helm Chart \u2693\ufe0e The Login Service is deployed via the login-service helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the login-service chart . helm install --version 1 .2.1 --values login-service-values.yaml login-service eoepca/login-service Values \u2693\ufe0e At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Kubernetes namespace for the login-service components Initial password for the admin user Note that the password must meet the complexity: at least 6 characters and include one uppercase letter, one lowercase letter, one digit, and one special character Name of Persistent Volume Claim for login-service persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. TLS Certificate Provider, e.g. letsencrypt-production Example login-service-values.yaml \u2026 global : domain : auth.192-168-49-2.nip.io nginxIp : 192.168.49.2 namespace : um volumeClaim : name : eoepca-userman-pvc create : false config : domain : auth.192-168-49-2.nip.io adminPass : Chang3me! ldapPass : Chang3me! volumeClaim : name : eoepca-userman-pvc opendj : volumeClaim : name : eoepca-userman-pvc oxauth : volumeClaim : name : eoepca-userman-pvc oxtrust : volumeClaim : name : eoepca-userman-pvc nginx : ingress : annotations : cert-manager.io/cluster-issuer : letsencrypt-production hosts : - auth.192-168-49-2.nip.io tls : - hosts : - auth.192-168-49-2.nip.io secretName : login-service-tls Login Service Usage \u2693\ufe0e Once the deployment has been completed successfully, the Login Service is accessed at the endpoint https://auth.<domain>/ , configured by your domain - e.g. https://auth.192-168-49-2.nip.io/ . Login as the admin user with the credentials configured in the helm values - ref. adminPass / ldapPass . Typical first actions to undertake through the Gluu web interface include creation of users and clients. Additional Information \u2693\ufe0e Additional information regarding the Login Service can be found at: Helm Chart Wiki GitHub Repository","title":"Login Service"},{"location":"eoepca/login-service/#login-service","text":"The Login Service provides the platform Authorization Server for authenticated user identity and request authorization.","title":"Login Service"},{"location":"eoepca/login-service/#helm-chart","text":"The Login Service is deployed via the login-service helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the login-service chart . helm install --version 1 .2.1 --values login-service-values.yaml login-service eoepca/login-service","title":"Helm Chart"},{"location":"eoepca/login-service/#values","text":"At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Kubernetes namespace for the login-service components Initial password for the admin user Note that the password must meet the complexity: at least 6 characters and include one uppercase letter, one lowercase letter, one digit, and one special character Name of Persistent Volume Claim for login-service persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. TLS Certificate Provider, e.g. letsencrypt-production Example login-service-values.yaml \u2026 global : domain : auth.192-168-49-2.nip.io nginxIp : 192.168.49.2 namespace : um volumeClaim : name : eoepca-userman-pvc create : false config : domain : auth.192-168-49-2.nip.io adminPass : Chang3me! ldapPass : Chang3me! volumeClaim : name : eoepca-userman-pvc opendj : volumeClaim : name : eoepca-userman-pvc oxauth : volumeClaim : name : eoepca-userman-pvc oxtrust : volumeClaim : name : eoepca-userman-pvc nginx : ingress : annotations : cert-manager.io/cluster-issuer : letsencrypt-production hosts : - auth.192-168-49-2.nip.io tls : - hosts : - auth.192-168-49-2.nip.io secretName : login-service-tls","title":"Values"},{"location":"eoepca/login-service/#login-service-usage","text":"Once the deployment has been completed successfully, the Login Service is accessed at the endpoint https://auth.<domain>/ , configured by your domain - e.g. https://auth.192-168-49-2.nip.io/ . Login as the admin user with the credentials configured in the helm values - ref. adminPass / ldapPass . Typical first actions to undertake through the Gluu web interface include creation of users and clients.","title":"Login Service Usage"},{"location":"eoepca/login-service/#additional-information","text":"Additional information regarding the Login Service can be found at: Helm Chart Wiki GitHub Repository","title":"Additional Information"},{"location":"eoepca/pde/","text":"Processor Development Environment (PDE) \u2693\ufe0e The Processor Development Environment (PDE) provides a web-based application that allows the user to perform platform-hosted interactive analysis and application development. Helm Chart \u2693\ufe0e The Processor Development Environment is deployed via the eoepca/jupyterhub helm chart from the EOEPCA Helm Chart Repository . The chart is derived from the public chart jupyterhub/k8s-hub that is available on DockerHub - https://hub.docker.com/r/jupyterhub/k8s-hub . helm install --version 1 .1.12 --values pde-values.yaml pde eoepca/jupyterhub Values \u2693\ufe0e The jupyterhub/k8s-hub chart supports many values. Typically, values for the following attributes may be specified: The persistence storage-class to be used URLs for OAuth integration with the EOEPCA Login Service (OIDC Provider) Ingress configuration Certificate Issuer for TLS Example pde-values.yaml \u2026 hub : db : pvc : storageClassName : standard extraEnv : OAUTH_CALLBACK_URL : \"https://pde.192-168-49-2.nip.io/hub/oauth_callback\" OAUTH2_USERDATA_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/userinfo\" OAUTH2_TOKEN_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/token\" OAUTH2_AUTHORIZE_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/authorize\" OAUTH_LOGOUT_REDIRECT_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/end_session?post_logout_redirect_uri=https://pde.192-168-49-2.nip.io\" STORAGE_CLASS : \"standard\" ingress : enabled : true annotations : cert-manager.io/cluster-issuer : letsencrypt-production hosts : - host : pde.192-168-49-2.nip.io paths : - path : / tls : - hosts : - pde.192-168-49-2.nip.io secretName : pde-tls JupyterHub Secret \u2693\ufe0e The PDE relies upon a Kubernetes secret jupyterhub-secrets that provides confidential values, including the client credentials for the Login Service. The credentials of the client registered for resource protection (ref. Client Registration ) can be re-used from the local file client.yaml as follows\u2026 kubectl -n pde create secret generic jupyterhub-secrets \\ --from-literal=JUPYTERHUB_CRYPT_KEY=\"$(openssl rand -hex 32)\" \\ --from-literal=OAUTH_CLIENT_ID=\"$(cat client.yaml | grep client-id | cut -d\\ -f2)\" \\ --from-literal=OAUTH_CLIENT_SECRET=\"$(cat client.yaml | grep client-secret | cut -d\\ -f2)\" PDE Usage \u2693\ufe0e The PDE is accessed at the endpoint https://pde.<domain>/ , configured by your domain - e.g. https://pde.192-168-49-2.nip.io/ . Additional Information \u2693\ufe0e Additional information regarding the Processor Development Environment can be found at: Helm Chart GitHub Repository","title":"Processor Development Environment"},{"location":"eoepca/pde/#processor-development-environment-pde","text":"The Processor Development Environment (PDE) provides a web-based application that allows the user to perform platform-hosted interactive analysis and application development.","title":"Processor Development Environment (PDE)"},{"location":"eoepca/pde/#helm-chart","text":"The Processor Development Environment is deployed via the eoepca/jupyterhub helm chart from the EOEPCA Helm Chart Repository . The chart is derived from the public chart jupyterhub/k8s-hub that is available on DockerHub - https://hub.docker.com/r/jupyterhub/k8s-hub . helm install --version 1 .1.12 --values pde-values.yaml pde eoepca/jupyterhub","title":"Helm Chart"},{"location":"eoepca/pde/#values","text":"The jupyterhub/k8s-hub chart supports many values. Typically, values for the following attributes may be specified: The persistence storage-class to be used URLs for OAuth integration with the EOEPCA Login Service (OIDC Provider) Ingress configuration Certificate Issuer for TLS Example pde-values.yaml \u2026 hub : db : pvc : storageClassName : standard extraEnv : OAUTH_CALLBACK_URL : \"https://pde.192-168-49-2.nip.io/hub/oauth_callback\" OAUTH2_USERDATA_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/userinfo\" OAUTH2_TOKEN_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/token\" OAUTH2_AUTHORIZE_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/authorize\" OAUTH_LOGOUT_REDIRECT_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/end_session?post_logout_redirect_uri=https://pde.192-168-49-2.nip.io\" STORAGE_CLASS : \"standard\" ingress : enabled : true annotations : cert-manager.io/cluster-issuer : letsencrypt-production hosts : - host : pde.192-168-49-2.nip.io paths : - path : / tls : - hosts : - pde.192-168-49-2.nip.io secretName : pde-tls","title":"Values"},{"location":"eoepca/pde/#jupyterhub-secret","text":"The PDE relies upon a Kubernetes secret jupyterhub-secrets that provides confidential values, including the client credentials for the Login Service. The credentials of the client registered for resource protection (ref. Client Registration ) can be re-used from the local file client.yaml as follows\u2026 kubectl -n pde create secret generic jupyterhub-secrets \\ --from-literal=JUPYTERHUB_CRYPT_KEY=\"$(openssl rand -hex 32)\" \\ --from-literal=OAUTH_CLIENT_ID=\"$(cat client.yaml | grep client-id | cut -d\\ -f2)\" \\ --from-literal=OAUTH_CLIENT_SECRET=\"$(cat client.yaml | grep client-secret | cut -d\\ -f2)\"","title":"JupyterHub Secret"},{"location":"eoepca/pde/#pde-usage","text":"The PDE is accessed at the endpoint https://pde.<domain>/ , configured by your domain - e.g. https://pde.192-168-49-2.nip.io/ .","title":"PDE Usage"},{"location":"eoepca/pde/#additional-information","text":"Additional information regarding the Processor Development Environment can be found at: Helm Chart GitHub Repository","title":"Additional Information"},{"location":"eoepca/pdp/","text":"Policy Decision Point \u2693\ufe0e The Policy Decision Point (PDP) provides the platform policy database and associated service for access policy decision requests. Helm Chart \u2693\ufe0e The PDP is deployed via the pdp-engine helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the pdp-engine chart . helm install --version 1 .1.6 --values pdp-values.yaml pdp eoepca/pdp-engine Values \u2693\ufe0e At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for pdp-engine persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. Example pdp-values.yaml \u2026 global : nginxIp : 192.168.49.2 domain : auth.192-168-49-2.nip.io volumeClaim : name : eoepca-userman-pvc create : false Additional Information \u2693\ufe0e Additional information regarding the PDP can be found at: Helm Chart Wiki GitHub Repository","title":"Policy Decision Point"},{"location":"eoepca/pdp/#policy-decision-point","text":"The Policy Decision Point (PDP) provides the platform policy database and associated service for access policy decision requests.","title":"Policy Decision Point"},{"location":"eoepca/pdp/#helm-chart","text":"The PDP is deployed via the pdp-engine helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the pdp-engine chart . helm install --version 1 .1.6 --values pdp-values.yaml pdp eoepca/pdp-engine","title":"Helm Chart"},{"location":"eoepca/pdp/#values","text":"At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for pdp-engine persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. Example pdp-values.yaml \u2026 global : nginxIp : 192.168.49.2 domain : auth.192-168-49-2.nip.io volumeClaim : name : eoepca-userman-pvc create : false","title":"Values"},{"location":"eoepca/pdp/#additional-information","text":"Additional information regarding the PDP can be found at: Helm Chart Wiki GitHub Repository","title":"Additional Information"},{"location":"eoepca/persistence/","text":"Persistence \u2693\ufe0e The EOEPCA building-blocks rely upon Kubernetes Persistent Volumes for their component persistence. Components integrate with the storage provided in the cluster by means of configurable Persistent Volume Claims and/or dynamic Storage Class that are specfied as values at time of deployment. Some components require storage of type ReadWriteMany - which, for a multi-node cluster, implies a network-based storage solution. ReadWriteMany Storage \u2693\ufe0e For the EOEPCA development deployment, an NFS server has been established to provide the persistence layer for ReadWriteMany storage. Pre-defined Persistent Volume Claims \u2693\ufe0e The EOEPCA development deployment establishes the following pre-defined Persistent Volume Claims, to provide a simple storage architecture that is organised around the \u2018domain areas\u2019 into which the Reference Implementation is split. Resource Managment ( resman ) - persistentvolumeclaim/eoepca-resman-pvc Processing & Chaining ( proc ) - persistentvolumeclaim/eoepca-proc-pvc User Management ( userman ) - persistentvolumeclaim/eoepca-userman-pvc NOTE that this is offered only as an example thay suits the approach of the development team. Each building-block has configuration through which its persistence (PV/PVC) can be configured according the needs of the deployment. The following Kubernetes yaml provides an example of provisioning such domain-specific PersistentVolumeClaims within the cluster - in this case using the minikube built-in storage-class standard for dynamic provisioning\u2026 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-proc-pvc namespace: proc spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-resman-pvc namespace: rm spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-userman-pvc namespace: um spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi Once established, these PersistentVolumeClaims are then referenced within the deployment configurations of the building-blocks. Dynamic ReadWriteMany Storage Provisioning \u2693\ufe0e In addition to the pre-defined PV/PVCs, the EOEPCA Reference Implementation also defines NFS-based storage classes for dynamic storage provisioning: managed-nfs-storage With a Reclaim Policy of Delete . managed-nfs-storage-retain With a Reclaim Policy of Retain . The building-blocks simply reference the required Storage Class in their volume specifications, to receive a Persistent Volume Claim that is dynamically provisioned at deployment time. This is acheived through the nfs-provisioner helm chart , with the following typical configurations\u2026 Reclaim Policy Delete \u2026 provisionerName : nfs-storage storageClass : name : managed-nfs-storage create : true reclaimPolicy : Delete archiveOnDelete : false allowVolumeExpansion : true nfs : server : \"<your-nfs-ip-address-here>\" path : /data/dynamic # your NFS server path here Reclaim Policy Retain \u2026 provisionerName : nfs-storage-retain storageClass : name : managed-nfs-storage-retain create : true reclaimPolicy : Retain allowVolumeExpansion : true nfs : server : \"<your-nfs-ip-address-here>\" path : /data/dynamic # your NFS server path here Clustered Storage Solutions \u2693\ufe0e Clustered storage approaches offer an alternative to NFS. Clustered Storage provides a network-attached storage through a set of commodity hosts whose storage is aggregated to form a distributed file-system. Capacity is scaled by adding additional nodes or adding additional storage to the existing nodes. In the context of a multi-node Kubernetes cluster, then it is typical that the same commodity nodes provide both the cluster members and storage resources, i.e. the clustered storage is spread across the Kubernetes worker nodes. Candidate clustered storage solutions include: GlusterFS GlusterFS is deployed as an operating system service across each node participating in the storage solution. Thus, with GlusterFS, the distributed storage nodes do not need to be one-and-the-same with the compute (cluster) nodes \u2013 although this may preferably be the case. Longhorn Longhorn offers a solution that is similar to that of GlusterFS, except that Longhorn is \u2018cloud-native\u2019 in that its service layer deploys within the Kubernetes cluster itself. Thus, the storage nodes are also the cluster compute nodes by design. All things being equal, Longhorn is recommended as the best approach for Kubernetes clusters. Local Cluster Storage \u2693\ufe0e For the purposes of the EOEPCA deployment, the default Storage Class included with the local Kubernetes distribution can be used for all storage concerns - e.g. standard for minikube which provides the ReadWriteMany persistence that is required by the ADES.","title":"Persistence"},{"location":"eoepca/persistence/#persistence","text":"The EOEPCA building-blocks rely upon Kubernetes Persistent Volumes for their component persistence. Components integrate with the storage provided in the cluster by means of configurable Persistent Volume Claims and/or dynamic Storage Class that are specfied as values at time of deployment. Some components require storage of type ReadWriteMany - which, for a multi-node cluster, implies a network-based storage solution.","title":"Persistence"},{"location":"eoepca/persistence/#readwritemany-storage","text":"For the EOEPCA development deployment, an NFS server has been established to provide the persistence layer for ReadWriteMany storage.","title":"ReadWriteMany Storage"},{"location":"eoepca/persistence/#pre-defined-persistent-volume-claims","text":"The EOEPCA development deployment establishes the following pre-defined Persistent Volume Claims, to provide a simple storage architecture that is organised around the \u2018domain areas\u2019 into which the Reference Implementation is split. Resource Managment ( resman ) - persistentvolumeclaim/eoepca-resman-pvc Processing & Chaining ( proc ) - persistentvolumeclaim/eoepca-proc-pvc User Management ( userman ) - persistentvolumeclaim/eoepca-userman-pvc NOTE that this is offered only as an example thay suits the approach of the development team. Each building-block has configuration through which its persistence (PV/PVC) can be configured according the needs of the deployment. The following Kubernetes yaml provides an example of provisioning such domain-specific PersistentVolumeClaims within the cluster - in this case using the minikube built-in storage-class standard for dynamic provisioning\u2026 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-proc-pvc namespace: proc spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-resman-pvc namespace: rm spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-userman-pvc namespace: um spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi Once established, these PersistentVolumeClaims are then referenced within the deployment configurations of the building-blocks.","title":"Pre-defined Persistent Volume Claims"},{"location":"eoepca/persistence/#dynamic-readwritemany-storage-provisioning","text":"In addition to the pre-defined PV/PVCs, the EOEPCA Reference Implementation also defines NFS-based storage classes for dynamic storage provisioning: managed-nfs-storage With a Reclaim Policy of Delete . managed-nfs-storage-retain With a Reclaim Policy of Retain . The building-blocks simply reference the required Storage Class in their volume specifications, to receive a Persistent Volume Claim that is dynamically provisioned at deployment time. This is acheived through the nfs-provisioner helm chart , with the following typical configurations\u2026 Reclaim Policy Delete \u2026 provisionerName : nfs-storage storageClass : name : managed-nfs-storage create : true reclaimPolicy : Delete archiveOnDelete : false allowVolumeExpansion : true nfs : server : \"<your-nfs-ip-address-here>\" path : /data/dynamic # your NFS server path here Reclaim Policy Retain \u2026 provisionerName : nfs-storage-retain storageClass : name : managed-nfs-storage-retain create : true reclaimPolicy : Retain allowVolumeExpansion : true nfs : server : \"<your-nfs-ip-address-here>\" path : /data/dynamic # your NFS server path here","title":"Dynamic ReadWriteMany Storage Provisioning"},{"location":"eoepca/persistence/#clustered-storage-solutions","text":"Clustered storage approaches offer an alternative to NFS. Clustered Storage provides a network-attached storage through a set of commodity hosts whose storage is aggregated to form a distributed file-system. Capacity is scaled by adding additional nodes or adding additional storage to the existing nodes. In the context of a multi-node Kubernetes cluster, then it is typical that the same commodity nodes provide both the cluster members and storage resources, i.e. the clustered storage is spread across the Kubernetes worker nodes. Candidate clustered storage solutions include: GlusterFS GlusterFS is deployed as an operating system service across each node participating in the storage solution. Thus, with GlusterFS, the distributed storage nodes do not need to be one-and-the-same with the compute (cluster) nodes \u2013 although this may preferably be the case. Longhorn Longhorn offers a solution that is similar to that of GlusterFS, except that Longhorn is \u2018cloud-native\u2019 in that its service layer deploys within the Kubernetes cluster itself. Thus, the storage nodes are also the cluster compute nodes by design. All things being equal, Longhorn is recommended as the best approach for Kubernetes clusters.","title":"Clustered Storage Solutions"},{"location":"eoepca/persistence/#local-cluster-storage","text":"For the purposes of the EOEPCA deployment, the default Storage Class included with the local Kubernetes distribution can be used for all storage concerns - e.g. standard for minikube which provides the ReadWriteMany persistence that is required by the ADES.","title":"Local Cluster Storage"},{"location":"eoepca/resource-catalogue/","text":"Resource Catalogue \u2693\ufe0e The Resource Catalogue provides a standards-based EO metadata catalogue that includes support for OGC CSW / API Records, STAC and OpenSearch. Helm Chart \u2693\ufe0e The Resource Catalogue is deployed via the rm-resource-catalogue helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the rm-resource-catalogue chart . helm install --version 1 .2.0 --values resource-catalogue-values.yaml resource-catalogue eoepca/rm-resource-catalogue Values \u2693\ufe0e The Resource Catalogue supports many values to configure the service - as described in the Values section of the chart README . Typically, values for the following attributes may be specified: The fully-qualified public URL for the service Dynamic provisioning StorageClass for database persistence (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Resource Catalogue will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Metadata describing the Catalogue instance Tuning configuration for PostgreSQL - see values db.config.XXX . Example resource-catalogue-values.yaml \u2026 global : namespace : rm # For protected access disable this ingress, and rely upon the resource-guard # for ingress with protection. ingress : # Enabled for unprotected 'open' access to the resource-catalogue. enabled : true name : resource-catalogue host : resource-catalogue.192-168-49-2.nip.io tls_host : resource-catalogue.192-168-49-2.nip.io tls_secret_name : resource-catalogue-tls annotations : cert-manager.io/cluster-issuer : letsencrypt-production db : volume_storage_type : standard # config: # enabled: true # shared_buffers: 2GB # effective_cache_size: 6GB # maintenance_work_mem: 512MB # checkpoint_completion_target: 0.9 # wal_buffers: 16MB # default_statistics_target: 100 # random_page_cost: 4 # work_mem: 4MB # cpu_tuple_cost: 0.4 pycsw : config : server : url : https://resource-catalogue.192-168-49-2.nip.io/ Protection \u2693\ufe0e As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Resource Catalogue service to provide access authorization decisions helm install --version 1 .2.1 --values resource-catalogue-guard-values.yaml resource-catalogue-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Resource Catalogue for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example resource-catalogue-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : resource-catalogue domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : resource-catalogue paths : - path : /(.*) service : name : resource-catalogue-service port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint Client Secret \u2693\ufe0e The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io . In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml Resource Catalogue Usage \u2693\ufe0e The Resource Catalogue is initially populated during the initialisation of the Data Access service. See section Data-layer Configuration . The Resource Catalogue is accessed at the endpoint https://resource-catalogue.<domain>/ , configured by your domain - e.g. https://resource-catalogue.192-168-49-2.nip.io/ . Loading Records \u2693\ufe0e As described in the pycsw documentation , ISO XML records can be loaded into the resource-catalogue using the pycsw-admin.py admin utility\u2026 pycsw-admin.py load_records -c /path/to/cfg -p /path/to/records The /path/to/records can either be a single metadata file, or a directory containing multiple metadata files. This is most easily achieved via connection to the pycsw pod, which includes the pycsw-admin.py utility and the pycsw configuration file at /etc/pycsw/pycsw.cfg \u2026 kubectl -n rm cp \"<metadata-file-or-directory>\" \"<pycsw-pod-name>\" :/tmp/metadata kubectl -n rm exec -i \"<pycsw-pod-name>\" -- pycsw-admin.py load-records -c /etc/pycsw/pycsw.cfg -p /tmp/metadata The name of the pycsw pod can be obtained using kubectl \u2026 kubectl -n rm get pod --selector = 'io.kompose.service=pycsw' --output = jsonpath ={ .items [ 0 ] .metadata.name } To facilitate the loading of records via the pycsw pod, a helper script load-records has been provided in the git repository that hosts this document \u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide ./deploy/bin/load-records \"<metadata-file-or-directory>\" The helper script identifies the pycsw pod, copies the metadata files to the pod, and runs pycsw-admin.py load-records within the pod to load the records. Additional Information \u2693\ufe0e Additional information regarding the Resource Catalogue can be found at: Helm Chart pycsw Documentation GitHub Repository","title":"Resource Catalogue"},{"location":"eoepca/resource-catalogue/#resource-catalogue","text":"The Resource Catalogue provides a standards-based EO metadata catalogue that includes support for OGC CSW / API Records, STAC and OpenSearch.","title":"Resource Catalogue"},{"location":"eoepca/resource-catalogue/#helm-chart","text":"The Resource Catalogue is deployed via the rm-resource-catalogue helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the rm-resource-catalogue chart . helm install --version 1 .2.0 --values resource-catalogue-values.yaml resource-catalogue eoepca/rm-resource-catalogue","title":"Helm Chart"},{"location":"eoepca/resource-catalogue/#values","text":"The Resource Catalogue supports many values to configure the service - as described in the Values section of the chart README . Typically, values for the following attributes may be specified: The fully-qualified public URL for the service Dynamic provisioning StorageClass for database persistence (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Resource Catalogue will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Metadata describing the Catalogue instance Tuning configuration for PostgreSQL - see values db.config.XXX . Example resource-catalogue-values.yaml \u2026 global : namespace : rm # For protected access disable this ingress, and rely upon the resource-guard # for ingress with protection. ingress : # Enabled for unprotected 'open' access to the resource-catalogue. enabled : true name : resource-catalogue host : resource-catalogue.192-168-49-2.nip.io tls_host : resource-catalogue.192-168-49-2.nip.io tls_secret_name : resource-catalogue-tls annotations : cert-manager.io/cluster-issuer : letsencrypt-production db : volume_storage_type : standard # config: # enabled: true # shared_buffers: 2GB # effective_cache_size: 6GB # maintenance_work_mem: 512MB # checkpoint_completion_target: 0.9 # wal_buffers: 16MB # default_statistics_target: 100 # random_page_cost: 4 # work_mem: 4MB # cpu_tuple_cost: 0.4 pycsw : config : server : url : https://resource-catalogue.192-168-49-2.nip.io/","title":"Values"},{"location":"eoepca/resource-catalogue/#protection","text":"As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Resource Catalogue service to provide access authorization decisions helm install --version 1 .2.1 --values resource-catalogue-guard-values.yaml resource-catalogue-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Resource Catalogue for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example resource-catalogue-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : resource-catalogue domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : resource-catalogue paths : - path : /(.*) service : name : resource-catalogue-service port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint","title":"Protection"},{"location":"eoepca/resource-catalogue/#client-secret","text":"The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io . In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml","title":"Client Secret"},{"location":"eoepca/resource-catalogue/#resource-catalogue-usage","text":"The Resource Catalogue is initially populated during the initialisation of the Data Access service. See section Data-layer Configuration . The Resource Catalogue is accessed at the endpoint https://resource-catalogue.<domain>/ , configured by your domain - e.g. https://resource-catalogue.192-168-49-2.nip.io/ .","title":"Resource Catalogue Usage"},{"location":"eoepca/resource-catalogue/#loading-records","text":"As described in the pycsw documentation , ISO XML records can be loaded into the resource-catalogue using the pycsw-admin.py admin utility\u2026 pycsw-admin.py load_records -c /path/to/cfg -p /path/to/records The /path/to/records can either be a single metadata file, or a directory containing multiple metadata files. This is most easily achieved via connection to the pycsw pod, which includes the pycsw-admin.py utility and the pycsw configuration file at /etc/pycsw/pycsw.cfg \u2026 kubectl -n rm cp \"<metadata-file-or-directory>\" \"<pycsw-pod-name>\" :/tmp/metadata kubectl -n rm exec -i \"<pycsw-pod-name>\" -- pycsw-admin.py load-records -c /etc/pycsw/pycsw.cfg -p /tmp/metadata The name of the pycsw pod can be obtained using kubectl \u2026 kubectl -n rm get pod --selector = 'io.kompose.service=pycsw' --output = jsonpath ={ .items [ 0 ] .metadata.name } To facilitate the loading of records via the pycsw pod, a helper script load-records has been provided in the git repository that hosts this document \u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide ./deploy/bin/load-records \"<metadata-file-or-directory>\" The helper script identifies the pycsw pod, copies the metadata files to the pod, and runs pycsw-admin.py load-records within the pod to load the records.","title":"Loading Records"},{"location":"eoepca/resource-catalogue/#additional-information","text":"Additional information regarding the Resource Catalogue can be found at: Helm Chart pycsw Documentation GitHub Repository","title":"Additional Information"},{"location":"eoepca/resource-protection/","text":"Resource Protection \u2693\ufe0e EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes: Login Service (Authorization Server) Policy Decision Point (PDP) Policy Enforcement Point (PEP) Building Blocks that act as a Resource Server are individually protected by a Policy Enforcement Point (PEP). The PEP enforces the authorization decision in collaboration with the Login Service and Policy Decision Point (PDP). The PEP expects to interface to a client (user agent, e.g. browser) using User Managed Access (UMA) flows. It is not typical for a client to support UMA flows , and so the PEP can be deployed with a companion UMA User Agent component that interfaces between the client and the PEP, and performs the UMA Flow on behalf of the client. The Resource Guard is a \u2018convenience\u2019 component that deploys the PEP & UMA User Agent as a cooperating pair. The Resource Guard \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, the Resource Guard deploys with an Ingress specification that: Configures the auth_request module to defer access authorization to the uma-user-agent service Configures the ingress rules (host/path) for the target Resource Server Helm Chart \u2693\ufe0e The Resource Guard is deployed via the resource-guard helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the resource-guard chart . It is expected to deploy multiple instances of the Resource Guard chart, one for each Resource Server to be protected. helm install --version 1 .0.7 --values myservice-guard-values.yaml myservice-guard eoepca/resource-guard Values \u2693\ufe0e The helm chart is deployed with values that are passed through to the subcharts for the pep-engine and uma-user-agent . Typical values to be specified include: Host/domain details for the Login Service and PDP, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for pep-engine persistence, e.g. myservice-pep-pvc TLS Certificate Provider, e.g. letsencrypt-production Optional specification of default resources with which to initialise the policy database for the component Ingress rules definition for reverse-proxy to the target Resource Server Name of Secret that contains the client credentials used by the uma-user-agent to interface with the Login Service. See section Client Secret below Example myservice-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : myservice domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : workingMode : PARTIAL asHostname : auth pdpHostname : auth customDefaultResources : - name : \"Eric's space\" description : \"Protected Access for eric to his space in myservice\" resource_uri : \"/ericspace\" scopes : [] default_owner : \"d3688daa-385d-45b0-8e04-2062e3e2cd86\" nginxIntegration : enabled : false volumeClaim : name : myservice-pep-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : myservice paths : - path : /(.*) service : name : myservice port : 80 - path : /(doc.*) service : name : myservice-docs port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"myservice-agent\" logging : level : \"debug\" unauthorizedResponse : 'Bearer realm=\"https://auth.192-168-49-2.nip.io/oxauth/auth/passport/passportlogin.htm\"' #--------------------------------------------------------------------------- # END values #--------------------------------------------------------------------------- Client Credentials \u2693\ufe0e The uma-user-agent requires Client Credentials for its interactions with the login-service . The uma-user-agent expects to read these credentials from the file client.yaml , in the form\u2026 client-id: <my-client-id> client-secret: <my-secret> Client Registration \u2693\ufe0e To obtain the Client Credentials required by the uma-user-agent it is necessary to register a client with the login-service , or use the credentials for an existing client. A helper script is provided to register a basic client and obtain the required credentials. The script is available in the deployment-guide repository , and can be obtained as follows\u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide The register-client helper script requires some command-line arguments\u2026 Usage: register_client <authorization-server-hostname> <client-name> [<redirect-uri> [<logout-uri>]] For example\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient INFO: Preparing docker image... [ done ] Client successfully registered. Make a note of the credentials: client-id: a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret: 73914cfc-c7dd-4b54-8807-ce17c3645558 Or to register OIDC redirect URLs\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient https://portal.192-168-49-2.nip.io/oidc/callback/ https://portal.192-168-49-2.nip.io/logout The script writes the \u2018client credentials\u2019 to stdout - in the expected YAML configuration file format - which can be redirected to file\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient | tee client.yaml \u2026writes the client credentials to the file client.yaml . NOTE that the register-client helper relies upon docker to build and run the script. Client Secret \u2693\ufe0e The client.yaml configuration file is made available via a Kubernetes Secret\u2026 kubectl -n myservice-ns create secret generic myservice-agent \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > myservice-agent-secret.yaml apiVersion : v1 kind : Secret metadata : name : myservice-agent namespace : myservice-ns data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The resource-guard deployment is configured with the name of the Secret through the helm chart value client.credentialsSecretName . User ID Token \u2693\ufe0e As described in the README for the Resource Guard , it is necessary for a request to a protected resource to provide the User ID Token in the request header. Obtaining the User ID Token \u2693\ufe0e In the simple case of a user with username/password held within the Login Service, the User ID Token can be obtained as follows: curl --location --request POST 'https://auth.192-168-49-2.nip.io/oxauth/restv1/token' \\ --header 'Cache-Control: no-cache' \\ --header 'Content-Type: application/x-www-form-urlencoded' \\ --data-urlencode 'scope=openid user_name is_operator' \\ --data-urlencode 'grant_type=password' \\ --data-urlencode 'username=<username>' \\ --data-urlencode 'password=<password>' \\ --data-urlencode 'client_id=<client-id>' \\ --data-urlencode 'client_secret=<client-password>' The User ID Token is included in the id_token field of the json response. Alternatively, OAuth/OIDC flows can be followed to authenticate via external identity providers. User ID Token in HTTP requests \u2693\ufe0e The Resource Guard protection supports presentation of the User ID Token via the following HTTP request headers (in order of priority)\u2026 Authorization header as a bearer token - in the form: Authorization: Bearer <token> X-User-Id header Cookie: auth_user_id=<token> Note that the name of the cookie is configurable Additional Information \u2693\ufe0e Additional information regarding the Resource Guard can be found at: Helm Chart README GitHub Repository: pep-engine uma-user-agent","title":"Resource Protection"},{"location":"eoepca/resource-protection/#resource-protection","text":"EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes: Login Service (Authorization Server) Policy Decision Point (PDP) Policy Enforcement Point (PEP) Building Blocks that act as a Resource Server are individually protected by a Policy Enforcement Point (PEP). The PEP enforces the authorization decision in collaboration with the Login Service and Policy Decision Point (PDP). The PEP expects to interface to a client (user agent, e.g. browser) using User Managed Access (UMA) flows. It is not typical for a client to support UMA flows , and so the PEP can be deployed with a companion UMA User Agent component that interfaces between the client and the PEP, and performs the UMA Flow on behalf of the client. The Resource Guard is a \u2018convenience\u2019 component that deploys the PEP & UMA User Agent as a cooperating pair. The Resource Guard \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, the Resource Guard deploys with an Ingress specification that: Configures the auth_request module to defer access authorization to the uma-user-agent service Configures the ingress rules (host/path) for the target Resource Server","title":"Resource Protection"},{"location":"eoepca/resource-protection/#helm-chart","text":"The Resource Guard is deployed via the resource-guard helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the resource-guard chart . It is expected to deploy multiple instances of the Resource Guard chart, one for each Resource Server to be protected. helm install --version 1 .0.7 --values myservice-guard-values.yaml myservice-guard eoepca/resource-guard","title":"Helm Chart"},{"location":"eoepca/resource-protection/#values","text":"The helm chart is deployed with values that are passed through to the subcharts for the pep-engine and uma-user-agent . Typical values to be specified include: Host/domain details for the Login Service and PDP, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for pep-engine persistence, e.g. myservice-pep-pvc TLS Certificate Provider, e.g. letsencrypt-production Optional specification of default resources with which to initialise the policy database for the component Ingress rules definition for reverse-proxy to the target Resource Server Name of Secret that contains the client credentials used by the uma-user-agent to interface with the Login Service. See section Client Secret below Example myservice-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : myservice domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : workingMode : PARTIAL asHostname : auth pdpHostname : auth customDefaultResources : - name : \"Eric's space\" description : \"Protected Access for eric to his space in myservice\" resource_uri : \"/ericspace\" scopes : [] default_owner : \"d3688daa-385d-45b0-8e04-2062e3e2cd86\" nginxIntegration : enabled : false volumeClaim : name : myservice-pep-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : myservice paths : - path : /(.*) service : name : myservice port : 80 - path : /(doc.*) service : name : myservice-docs port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"myservice-agent\" logging : level : \"debug\" unauthorizedResponse : 'Bearer realm=\"https://auth.192-168-49-2.nip.io/oxauth/auth/passport/passportlogin.htm\"' #--------------------------------------------------------------------------- # END values #---------------------------------------------------------------------------","title":"Values"},{"location":"eoepca/resource-protection/#client-credentials","text":"The uma-user-agent requires Client Credentials for its interactions with the login-service . The uma-user-agent expects to read these credentials from the file client.yaml , in the form\u2026 client-id: <my-client-id> client-secret: <my-secret>","title":"Client Credentials"},{"location":"eoepca/resource-protection/#client-registration","text":"To obtain the Client Credentials required by the uma-user-agent it is necessary to register a client with the login-service , or use the credentials for an existing client. A helper script is provided to register a basic client and obtain the required credentials. The script is available in the deployment-guide repository , and can be obtained as follows\u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide The register-client helper script requires some command-line arguments\u2026 Usage: register_client <authorization-server-hostname> <client-name> [<redirect-uri> [<logout-uri>]] For example\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient INFO: Preparing docker image... [ done ] Client successfully registered. Make a note of the credentials: client-id: a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret: 73914cfc-c7dd-4b54-8807-ce17c3645558 Or to register OIDC redirect URLs\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient https://portal.192-168-49-2.nip.io/oidc/callback/ https://portal.192-168-49-2.nip.io/logout The script writes the \u2018client credentials\u2019 to stdout - in the expected YAML configuration file format - which can be redirected to file\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient | tee client.yaml \u2026writes the client credentials to the file client.yaml . NOTE that the register-client helper relies upon docker to build and run the script.","title":"Client Registration"},{"location":"eoepca/resource-protection/#client-secret","text":"The client.yaml configuration file is made available via a Kubernetes Secret\u2026 kubectl -n myservice-ns create secret generic myservice-agent \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > myservice-agent-secret.yaml apiVersion : v1 kind : Secret metadata : name : myservice-agent namespace : myservice-ns data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The resource-guard deployment is configured with the name of the Secret through the helm chart value client.credentialsSecretName .","title":"Client Secret"},{"location":"eoepca/resource-protection/#user-id-token","text":"As described in the README for the Resource Guard , it is necessary for a request to a protected resource to provide the User ID Token in the request header.","title":"User ID Token"},{"location":"eoepca/resource-protection/#obtaining-the-user-id-token","text":"In the simple case of a user with username/password held within the Login Service, the User ID Token can be obtained as follows: curl --location --request POST 'https://auth.192-168-49-2.nip.io/oxauth/restv1/token' \\ --header 'Cache-Control: no-cache' \\ --header 'Content-Type: application/x-www-form-urlencoded' \\ --data-urlencode 'scope=openid user_name is_operator' \\ --data-urlencode 'grant_type=password' \\ --data-urlencode 'username=<username>' \\ --data-urlencode 'password=<password>' \\ --data-urlencode 'client_id=<client-id>' \\ --data-urlencode 'client_secret=<client-password>' The User ID Token is included in the id_token field of the json response. Alternatively, OAuth/OIDC flows can be followed to authenticate via external identity providers.","title":"Obtaining the User ID Token"},{"location":"eoepca/resource-protection/#user-id-token-in-http-requests","text":"The Resource Guard protection supports presentation of the User ID Token via the following HTTP request headers (in order of priority)\u2026 Authorization header as a bearer token - in the form: Authorization: Bearer <token> X-User-Id header Cookie: auth_user_id=<token> Note that the name of the cookie is configurable","title":"User ID Token in HTTP requests"},{"location":"eoepca/resource-protection/#additional-information","text":"Additional information regarding the Resource Guard can be found at: Helm Chart README GitHub Repository: pep-engine uma-user-agent","title":"Additional Information"},{"location":"eoepca/user-profile/","text":"User Profile \u2693\ufe0e The User Profile represents the user\u2019s \u2018account\u2019 within the platform. Helm Chart \u2693\ufe0e The User Profile is deployed via the user-profile helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the user-profile chart . helm install --version 1 .1.6 --values user-profile-values.yaml user-profile eoepca/user-profile Values \u2693\ufe0e At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for user-profile persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. Example user-profile-values.yaml \u2026 global : domain : auth.192-168-49-2.nip.io nginxIp : 192.168.49.2 volumeClaim : name : eoepca-userman-pvc create : false User Profile Usage \u2693\ufe0e The User Profile is accessed through the /web_ui path of the Login Service, e.g. http://auth.kube.guide.eoepca.org/web_ui. Additional Information \u2693\ufe0e Additional information regarding the User Profile can be found at: Helm Chart Wiki GitHub Repository","title":"User Profile"},{"location":"eoepca/user-profile/#user-profile","text":"The User Profile represents the user\u2019s \u2018account\u2019 within the platform.","title":"User Profile"},{"location":"eoepca/user-profile/#helm-chart","text":"The User Profile is deployed via the user-profile helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the user-profile chart . helm install --version 1 .1.6 --values user-profile-values.yaml user-profile eoepca/user-profile","title":"Helm Chart"},{"location":"eoepca/user-profile/#values","text":"At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for user-profile persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. Example user-profile-values.yaml \u2026 global : domain : auth.192-168-49-2.nip.io nginxIp : 192.168.49.2 volumeClaim : name : eoepca-userman-pvc create : false","title":"Values"},{"location":"eoepca/user-profile/#user-profile-usage","text":"The User Profile is accessed through the /web_ui path of the Login Service, e.g. http://auth.kube.guide.eoepca.org/web_ui.","title":"User Profile Usage"},{"location":"eoepca/user-profile/#additional-information","text":"Additional information regarding the User Profile can be found at: Helm Chart Wiki GitHub Repository","title":"Additional Information"},{"location":"eoepca/workspace/","text":"Workspace \u2693\ufe0e The Workspace provides protected user resource management that includes dedicated storage and services for resource discovery and access. Workspace API \u2693\ufe0e The Workspace API provides a REST service through which user workspaces can be created, interrogated, managed and deleted. Helm Chart \u2693\ufe0e The Workspace API is deployed via the rm-workspace-api helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the um-workspace-api chart . helm install --version 1 .2.0 --values workspace-api-values.yaml workspace-api eoepca/rm-workspace-api Values \u2693\ufe0e At minimum, values for the following attributes should be specified: The fully-qualified public URL for the service (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Workspace API will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Prefix for user projects in OpenStack Details for underlying S3 object storage service Identification of secret that provides the client credentials for resource protection Whether flux components should be installed - otherwise they must already be present - Flux Dependency Name of the ConfigMap for user workspace templates - See User Workspace Templates Example workspace-api-values.yaml \u2026 fullnameOverride : workspace-api ingress : enabled : true hosts : - host : workspace-api-open.192-168-49-2.nip.io paths : [ \"/\" ] tls : - hosts : - workspace-api-open.192-168-49-2.nip.io secretName : workspace-api-open-tls fluxHelmOperator : enabled : true prefixForName : \"guide-user\" workspaceSecretName : \"bucket\" namespaceForBucketResource : \"rm\" s3Endpoint : \"https://cf2.cloudferro.com:8080\" s3Region : \"RegionOne\" harborUrl : \"https://harbor.192-168-49-2.nip.io\" harborUsername : \"admin\" harborPassword : \"changeme\" umaClientSecretName : \"resman-client\" umaClientSecretNamespace : \"rm\" workspaceChartsConfigMap : \"workspace-charts\" NOTES: The Workspace API assumes a deployment of the Harbor Container Regsitry, as configured by the harborXXX values above. See section Container Registry . Flux Dependency \u2693\ufe0e Workspaces are created by instantiating the rm-user-workspace helm chart for each user/group. The Workspace API uses Flux CD as a helper to manage these subordinate helm charts - via flux resources of type HelmRelease . Thus, it is necessary to deploy within the cluster the aspects of flux that support this helm chart management - namely the flux helm-controller , source-controller and the Kubernetes Custom Resource Definitions (CRD) for HelmRelease and HelmRepository . In case you are not already using flux within your clsuter, then the Workspace API helm chart can be configured to deploy the required flux components\u2026 fluxHelmOperator: enabled: true # true = install flux for me, false = I already have flux User Workspace Templates \u2693\ufe0e The Workspace API instantiates for each user a set of services, including a Resource Catalogue and Data Access services. These user services are instantiated via helm using templates. The templates are provided to the Workspace API in a ConfigMap that is, by default, named workspace-charts . Each file in the config-map is expected to be of kind HelmRelease . During creation of a new workspace, the Worksapce API applies each file to the cluster in the namespace of the newly created namespace. The default ConfigMap that is included with this guide contains the following templates: Data Access : template-hr-data-access.yaml Resource Catalogue : template-hr-resource-catalogue.yaml Protection : template-hr-resource-guard.yaml Each of these templates is expressed as a flux HelmRelease object that describes the helm chart and values required to deploy the service. Templates ConfigMap \u2693\ufe0e The templates are provided to the Workspace API as a ConfigMap in the namespace of the Workspace API deployment\u2026 (for full examples see https://github.com/ EOEPCA /deployment-guide/tree/main/deploy/eoepca/workspace-templates ) apiVersion : v1 kind : ConfigMap metadata : name : workspace-charts data : template-hr-resource-catalogue.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: rm-resource-catalogue spec: interval: 5m chart: spec: chart: rm-resource-catalogue version: 1.2.0 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... template-hr-data-access.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: vs spec: interval: 5m chart: spec: chart: data-access version: 1.2.5 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... template-hr-resource-guard.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: resource-guard spec: interval: 5m chart: spec: chart: resource-guard version: 1.2.1 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... HelmRepositories for Templates \u2693\ufe0e As can be seen above, the HelmRelease templates rely upon objects of type HelmRepository that define the hosting helm chart repository. Thus, in support of the workspace templates, appropriate HelmRepository objects must be provisioned within the cluster. For example, in support of the above examples that rely upon the EOEPCA Helm Chart Repository \u2026 apiVersion : source.toolkit.fluxcd.io/v1beta1 kind : HelmRepository metadata : name : eoepca namespace : rm spec : interval : 2m url : https://eoepca.github.io/helm-charts/ Helm Template Parameters \u2693\ufe0e The Workspace API uses the jinja2 templating engine when applying the HelmReleases for a user workspace. The current parameters are currently supported: workspace_name The name of the workspace - {{ workspace_name }} used to ensure unique naming of cluster resources, such as service ingress default_owner The uuid of the owner of the workspace - {{ default_owner }} used to initialise the workspace protection Protection \u2693\ufe0e As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Workspace API service to provide access authorization decisions helm install --version 1 .2.1 --values workspace-api-guard-values.yaml workspace-api-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Workspace API for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example workspace-api-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : workspace-api domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth # customDefaultResources: # - name: \"Eric's workspace\" # description: \"Protected Access for eric to his user workspace\" # resource_uri: \"/workspaces/guide-user-eric\" # scopes: [] # default_owner: \"d3688daa-385d-45b0-8e04-2062e3e2cd86\" # - name: \"Bob's workspace\" # description: \"Protected Access for bob to his user workspace\" # resource_uri: \"/workspaces/guide-user-bob\" # scopes: [] # default_owner: \"f12c2592-0332-49f4-a4fb-7063b3c2a889\" volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : workspace-api paths : - path : /(.*) service : name : workspace-api port : http annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint Client Secret \u2693\ufe0e The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io . In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml Workspace API Usage \u2693\ufe0e The Workspace API provides a REST interface that is accessed at the endpoint https://workspace-api.192-168-49-2.nip.io/. See the Swagger Docs . Additional Information \u2693\ufe0e Additional information regarding the Workspace API can be found at: Helm Chart Wiki GitHub Repository Bucket Operator \u2693\ufe0e The Workspace API creates workspaces for individual users. In doing so, dedicated object storage buckets are created associated to each user workspace - for self-contained storage of user owned resources (data, processing applications, etc.). The bucket creation relies upon the object storage services of the underlying cloud infrastructure. We have created a Bucket abstraction as a Kubernetes Custom Resource Definition . This is served by a Bucket Operator service that deploys into the Kubernetes cluster to satisfy requests for resources of type Bucket . We provide a Bucket Operator implementation that currently supports the creation of buckets in OpenStack object storage - currently tested only on the CREODIAS (Cloudferro). The Bucket Operator is deployed via the rm-bucket-operator helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the um-bucket-operator chart . helm install --version 0 .9.9 --values bucket-operator-values.yaml bucket-operator eoepca/rm-bucket-operator Values \u2693\ufe0e At minimum, values for the following attributes should be specified: The fully-qualified public URL for the service OpenStack access details Cluster Issuer for TLS Example bucket-operator-values.yaml \u2026 domain : 192-168-49-2.nip.io data : OS_MEMBERROLEID : \"9ee2ff9ee4384b1894a90878d3e92bab\" OS_SERVICEPROJECTID : \"d21467d0a0414252a79e29d38f03ff98\" USER_EMAIL_PATTERN : \"eoepca+<name>@192-168-49-2.nip.io\" ingress : annotations : cert-manager.io/cluster-issuer : letsencrypt-production kubernetes.io/ingress.class : nginx nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" OpenStack Secret \u2693\ufe0e The Bucket Operator requires privileged access to the OpenStack API, for which credentials are required. These are provided via a Kubernetes secret named openstack created in the namespace of the Bucket Operator. For example\u2026 kubectl -n rm create secret generic openstack \\ --from-literal=username=\"${OS_USERNAME}\" \\ --from-literal=password=\"${OS_PASSWORD}\" \\ --from-literal=domainname=\"${OS_DOMAINNAME}\" See the README for the Bucket Operator , which describes the configuration required for integration with your OpenStack account. For a worked example see our Scripted Example Deployment - in particular: Openstack Configuration Deployment Script","title":"Workspace"},{"location":"eoepca/workspace/#workspace","text":"The Workspace provides protected user resource management that includes dedicated storage and services for resource discovery and access.","title":"Workspace"},{"location":"eoepca/workspace/#workspace-api","text":"The Workspace API provides a REST service through which user workspaces can be created, interrogated, managed and deleted.","title":"Workspace API"},{"location":"eoepca/workspace/#helm-chart","text":"The Workspace API is deployed via the rm-workspace-api helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the um-workspace-api chart . helm install --version 1 .2.0 --values workspace-api-values.yaml workspace-api eoepca/rm-workspace-api","title":"Helm Chart"},{"location":"eoepca/workspace/#values","text":"At minimum, values for the following attributes should be specified: The fully-qualified public URL for the service (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Workspace API will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Prefix for user projects in OpenStack Details for underlying S3 object storage service Identification of secret that provides the client credentials for resource protection Whether flux components should be installed - otherwise they must already be present - Flux Dependency Name of the ConfigMap for user workspace templates - See User Workspace Templates Example workspace-api-values.yaml \u2026 fullnameOverride : workspace-api ingress : enabled : true hosts : - host : workspace-api-open.192-168-49-2.nip.io paths : [ \"/\" ] tls : - hosts : - workspace-api-open.192-168-49-2.nip.io secretName : workspace-api-open-tls fluxHelmOperator : enabled : true prefixForName : \"guide-user\" workspaceSecretName : \"bucket\" namespaceForBucketResource : \"rm\" s3Endpoint : \"https://cf2.cloudferro.com:8080\" s3Region : \"RegionOne\" harborUrl : \"https://harbor.192-168-49-2.nip.io\" harborUsername : \"admin\" harborPassword : \"changeme\" umaClientSecretName : \"resman-client\" umaClientSecretNamespace : \"rm\" workspaceChartsConfigMap : \"workspace-charts\" NOTES: The Workspace API assumes a deployment of the Harbor Container Regsitry, as configured by the harborXXX values above. See section Container Registry .","title":"Values"},{"location":"eoepca/workspace/#flux-dependency","text":"Workspaces are created by instantiating the rm-user-workspace helm chart for each user/group. The Workspace API uses Flux CD as a helper to manage these subordinate helm charts - via flux resources of type HelmRelease . Thus, it is necessary to deploy within the cluster the aspects of flux that support this helm chart management - namely the flux helm-controller , source-controller and the Kubernetes Custom Resource Definitions (CRD) for HelmRelease and HelmRepository . In case you are not already using flux within your clsuter, then the Workspace API helm chart can be configured to deploy the required flux components\u2026 fluxHelmOperator: enabled: true # true = install flux for me, false = I already have flux","title":"Flux Dependency"},{"location":"eoepca/workspace/#user-workspace-templates","text":"The Workspace API instantiates for each user a set of services, including a Resource Catalogue and Data Access services. These user services are instantiated via helm using templates. The templates are provided to the Workspace API in a ConfigMap that is, by default, named workspace-charts . Each file in the config-map is expected to be of kind HelmRelease . During creation of a new workspace, the Worksapce API applies each file to the cluster in the namespace of the newly created namespace. The default ConfigMap that is included with this guide contains the following templates: Data Access : template-hr-data-access.yaml Resource Catalogue : template-hr-resource-catalogue.yaml Protection : template-hr-resource-guard.yaml Each of these templates is expressed as a flux HelmRelease object that describes the helm chart and values required to deploy the service.","title":"User Workspace Templates"},{"location":"eoepca/workspace/#templates-configmap","text":"The templates are provided to the Workspace API as a ConfigMap in the namespace of the Workspace API deployment\u2026 (for full examples see https://github.com/ EOEPCA /deployment-guide/tree/main/deploy/eoepca/workspace-templates ) apiVersion : v1 kind : ConfigMap metadata : name : workspace-charts data : template-hr-resource-catalogue.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: rm-resource-catalogue spec: interval: 5m chart: spec: chart: rm-resource-catalogue version: 1.2.0 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... template-hr-data-access.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: vs spec: interval: 5m chart: spec: chart: data-access version: 1.2.5 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... template-hr-resource-guard.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: resource-guard spec: interval: 5m chart: spec: chart: resource-guard version: 1.2.1 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ...","title":"Templates ConfigMap"},{"location":"eoepca/workspace/#helmrepositories-for-templates","text":"As can be seen above, the HelmRelease templates rely upon objects of type HelmRepository that define the hosting helm chart repository. Thus, in support of the workspace templates, appropriate HelmRepository objects must be provisioned within the cluster. For example, in support of the above examples that rely upon the EOEPCA Helm Chart Repository \u2026 apiVersion : source.toolkit.fluxcd.io/v1beta1 kind : HelmRepository metadata : name : eoepca namespace : rm spec : interval : 2m url : https://eoepca.github.io/helm-charts/","title":"HelmRepositories for Templates"},{"location":"eoepca/workspace/#helm-template-parameters","text":"The Workspace API uses the jinja2 templating engine when applying the HelmReleases for a user workspace. The current parameters are currently supported: workspace_name The name of the workspace - {{ workspace_name }} used to ensure unique naming of cluster resources, such as service ingress default_owner The uuid of the owner of the workspace - {{ default_owner }} used to initialise the workspace protection","title":"Helm Template Parameters"},{"location":"eoepca/workspace/#protection","text":"As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Workspace API service to provide access authorization decisions helm install --version 1 .2.1 --values workspace-api-guard-values.yaml workspace-api-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Workspace API for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example workspace-api-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : workspace-api domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth # customDefaultResources: # - name: \"Eric's workspace\" # description: \"Protected Access for eric to his user workspace\" # resource_uri: \"/workspaces/guide-user-eric\" # scopes: [] # default_owner: \"d3688daa-385d-45b0-8e04-2062e3e2cd86\" # - name: \"Bob's workspace\" # description: \"Protected Access for bob to his user workspace\" # resource_uri: \"/workspaces/guide-user-bob\" # scopes: [] # default_owner: \"f12c2592-0332-49f4-a4fb-7063b3c2a889\" volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : workspace-api paths : - path : /(.*) service : name : workspace-api port : http annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint","title":"Protection"},{"location":"eoepca/workspace/#client-secret","text":"The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io . In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml","title":"Client Secret"},{"location":"eoepca/workspace/#workspace-api-usage","text":"The Workspace API provides a REST interface that is accessed at the endpoint https://workspace-api.192-168-49-2.nip.io/. See the Swagger Docs .","title":"Workspace API Usage"},{"location":"eoepca/workspace/#additional-information","text":"Additional information regarding the Workspace API can be found at: Helm Chart Wiki GitHub Repository","title":"Additional Information"},{"location":"eoepca/workspace/#bucket-operator","text":"The Workspace API creates workspaces for individual users. In doing so, dedicated object storage buckets are created associated to each user workspace - for self-contained storage of user owned resources (data, processing applications, etc.). The bucket creation relies upon the object storage services of the underlying cloud infrastructure. We have created a Bucket abstraction as a Kubernetes Custom Resource Definition . This is served by a Bucket Operator service that deploys into the Kubernetes cluster to satisfy requests for resources of type Bucket . We provide a Bucket Operator implementation that currently supports the creation of buckets in OpenStack object storage - currently tested only on the CREODIAS (Cloudferro). The Bucket Operator is deployed via the rm-bucket-operator helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the um-bucket-operator chart . helm install --version 0 .9.9 --values bucket-operator-values.yaml bucket-operator eoepca/rm-bucket-operator","title":"Bucket Operator"},{"location":"eoepca/workspace/#values_1","text":"At minimum, values for the following attributes should be specified: The fully-qualified public URL for the service OpenStack access details Cluster Issuer for TLS Example bucket-operator-values.yaml \u2026 domain : 192-168-49-2.nip.io data : OS_MEMBERROLEID : \"9ee2ff9ee4384b1894a90878d3e92bab\" OS_SERVICEPROJECTID : \"d21467d0a0414252a79e29d38f03ff98\" USER_EMAIL_PATTERN : \"eoepca+<name>@192-168-49-2.nip.io\" ingress : annotations : cert-manager.io/cluster-issuer : letsencrypt-production kubernetes.io/ingress.class : nginx nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\"","title":"Values"},{"location":"eoepca/workspace/#openstack-secret","text":"The Bucket Operator requires privileged access to the OpenStack API, for which credentials are required. These are provided via a Kubernetes secret named openstack created in the namespace of the Bucket Operator. For example\u2026 kubectl -n rm create secret generic openstack \\ --from-literal=username=\"${OS_USERNAME}\" \\ --from-literal=password=\"${OS_PASSWORD}\" \\ --from-literal=domainname=\"${OS_DOMAINNAME}\" See the README for the Bucket Operator , which describes the configuration required for integration with your OpenStack account. For a worked example see our Scripted Example Deployment - in particular: Openstack Configuration Deployment Script","title":"OpenStack Secret"},{"location":"quickstart/creodias-deployment/","text":"CREODIAS Deployment \u2693\ufe0e Based upon our development experiences on CREODIAS, there is a wrapper script creodias with particular customisations suited to the CREODIAS infrastructure and data offering. The customisations are expressed through environment variables that are captured in the file creodias-options . These scripts are examples that can be seen as a starting point, from which they can be adapted to your needs. The CREODIAS deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Protected service endpoints requiring IAM authorization See Endpoint Protection below for further information With reference to the file creodias-options , particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment\u2026 Passwords: LOGIN_SERVICE_ADMIN_PASSWORD , MINIO_ROOT_PASSWORD , HARBOR_ADMIN_PASSWORD OpenStack details: see section Openstack Configuration If configuring an external deployment - ref. Public Deployment \u2026 public_ip - The public IP address through which the deployment is exposed via the ingress-controller domain - The DNS domain name through which the deployment is accessed - forming the stem for all service hostnames in the ingress rules Once the file creodias-options has been well populated for your environment, then the deployment is initiated with\u2026 ./deploy/creodias/creodias \u2026noting that this step is a customised version of that described in section Deployment . Endpoint Protection \u2693\ufe0e Similarly the script creodias-protection is a customised version of that described in section Apply Protection . Once the main deployment has completed, then the test users can be created , their IDs ( Inum ) set in script creodias-protection , and the resource protection can then be applied\u2026 ./deploy/creodias/creodias-protection Harvest CREODIAS Data \u2693\ufe0e The harvester can be deployed with a default configuration file at /config.yaml . As described in the Data Access section , harvesting according to this configuration can be triggered with\u2026 kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch See the Harvester section below for an explanation of this harvester configuration. Data Specification Walkthrough \u2693\ufe0e The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables . This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example. Harvester \u2693\ufe0e The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment. The harvester is configured as follows\u2026 harvester: image: repository: eoepca/rm-harvester tag: 1.1.0 config: redis: host: data-access-redis-master port: 6379 harvesters: - name: Creodias-Opensearch resource: url: https://finder.creodias.eu/resto/api/collections/Sentinel2/describe.xml type: OpenSearch format_config: type: 'application/json' property_mapping: start_datetime: 'startDate' end_datetime: 'completionDate' productIdentifier: 'productIdentifier' query: time: property: sensed begin: 2019-09-10T00:00:00Z end: 2019-09-11T00:00:00Z collection: null bbox: 14.9,47.7,16.4,48.7 filter: {} postprocess: - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor queue: register_queue Based upon this harvester configuration we expect that the following query is made to discover data - i.e. an OpenSearch query, with json response representation, for a defined spatial and temporal extent\u2026 https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2019-09-10T00:00:00Z&completionDate=2019-09-11T00:00:00Z&box=14.9,47.7,16.4,48.7 From the result returned, the path to each product ( feature ) is obtained from the productIdentifier property, e.g. { \"type\": \"FeatureCollection\", \"features\": [ { \"type\": \"Feature\", \"properties\": { \"productIdentifier\": \"/eodata/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE\" ... } ... } ... ] } The harvester is configured with a Sentinel-2/CREODIAS specific post-processor harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor which transforms the product path from /eodata/... to s3://EODATA/... . The harvester post-processor follows this path to the Sentinel-2 scene and uses stactools (with built-in support for Sentinel-2) to establish a STAC item representing the product. This includes enumeration of assets for inspire-metadata and product-metadata - which are used by the registrar pycsw backend to embelesh the product record metadata. The harvester outputs the STAC item for each product, which is pushed to the registrar via the register_queue redis queue. Registration \u2693\ufe0e The registrar is configured at deployment to have the access details for the CREODIAS data in S3\u2026 global: storage: data: data: type: S3 endpoint_url: http://data.cloudferro.com access_key_id: access secret_access_key: access region_name: RegionOne validate_bucket_name: false Using this S3 configuration, the registrar pycsw backend uses the product metadata linked in the STAC item (ref. assets inspire-metadata and product-metadata ) to embelesh the metadata. For example, product-metadata in the file\u2026 s3://EODATA/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE/MTD_MSIL1C.xml The registrar uses this information to create the ISO XML metadata that is loaded into the resource-catalogue. Product Type \u2693\ufe0e The registrar recognises the product as Sentinel-2 and so reads its metadata XML files to obtain additional information. From the metadata XML file (e.g. MTD_MSIL1C.xml ) the registrar obtains the Product Type for each product from the field <PRODUCT_TYPE> \u2026 <n1:Level-1C_User_Product> <n1:General_Info> <Product_Info> <PRODUCT_TYPE>S2MSI1C</PRODUCT_TYPE> ... </Product_Info> ... </n1:General_Info> ... <n1:Level-1C_User_Product> Resource Catalogue Collections \u2693\ufe0e The registrar ( eoepca/rm-data-access-core ) container image is pre-loaded with two collections at the path /registrar_pycsw/registrar_pycsw/resources , (in the built container the files are at the path /usr/local/lib/python3.8/dist-packages/registrar_pycsw/resources/ ): S2MSI1C.yml - identifier: S2MSI1C S2MSI2A.yml - identifier: S2MSI2A The registrar applies these collections into the resource-catalogue during start-up - to create pre-defined out-of-the-box collections in pycsw. During registration, the PycswBackend of the registrar uses the Product Type to map the product into the collection of the same name - using metadata field parentidentifier . Data Specification \u2693\ufe0e The data-access service data handling is configured by definition of productTypes , collections and layers \u2026 productTypes identify the underlying file assets as WCS coverages and their visual representation collections provide groupings into which products are organised layers specifies the hoe the product visual representations are exposed through the WMS service productType \u2693\ufe0e During registration, products are mapped into a productType via a filter that is applied against the STAC item metadata. The registrar uses the product_type of each product to determine the collection into which the product should be registered - noting that the name of the product type does not take part in the matching logic (and hence can be any text name)\u2026 productTypes: - name: S2MSI1C filter: s2:product_type: S2MSI1C In the above example, the field s2:product_type is populated by the stactools that prepares the STAC item from the product metadata. productType - coverages \u2693\ufe0e coverages defines the coverages for the WCS service. Each coverage links to the assets that are defined within the product STAC item. productType - browses \u2693\ufe0e browses defines the images that are visualised in the View Server Client. Expressions are used to map the product assets into their visual representation. collections \u2693\ufe0e Collections are defined by reference to the defined productTypes and coverages . layers \u2693\ufe0e layers defines the layers that are presented through the WMS service - each layer being linked to the underlying browse that provides the image source. Layers are defined via their id that relies upon the naming convection <collection>__<browse> to identify the browse and so define the layer. Example Configuration \u2693\ufe0e Example configuration for Sentinel-2 L1C and L2A data. global : layers : - id : S2L1C title : Sentinel-2 Level 1C True Color abstract : Sentinel-2 Level 2A True Color displayColor : '#eb3700' grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__TRUE_COLOR title : Sentinel-2 Level 1C True Color abstract : Sentinel-2 Level 2A True Color grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__masked_clouds title : Sentinel-2 Level 1C True Color with cloud masks abstract : Sentinel-2 Level 1C True Color with cloud masks grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__FALSE_COLOR title : Sentinel-2 Level 1C False Color abstract : Sentinel-2 Level 1C False Color grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__NDVI title : Sentinel-2 Level 21CNDVI abstract : Sentinel-2 Level 1C NDVI grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L2A title : Sentinel-2 Level 2A True Color abstract : Sentinel-2 Level 2A True Color displayColor : '#eb3700' grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__TRUE_COLOR title : Sentinel-2 Level 2A True Color abstract : Sentinel-2 Level 2A True Color grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__masked_clouds title : Sentinel-2 Level 2A True Color with cloud masks abstract : Sentinel-2 Level 2A True Color with cloud masks grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__FALSE_COLOR title : Sentinel-2 Level 2A False Color abstract : Sentinel-2 Level 2A False Color grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__NDVI title : Sentinel-2 Level 2A NDVI abstract : Sentinel-2 Level 2A NDVI grids : - name : WGS84 zoom : 13 parentLayer : S2L2A collections : S2L1C : product_types : - S2MSI1C coverage_types : - S2L1C_B01 - S2L1C_B02 - S2L1C_B03 - S2L1C_B04 - S2L1C_B05 - S2L1C_B06 - S2L1C_B07 - S2L1C_B08 - S2L1C_B8A - S2L1C_B09 - S2L1C_B10 - S2L1C_B11 - S2L1C_B12 S2L2A : product_types : - S2MSI2A product_levels : - Level-2A coverage_types : - S2L2A_B01 - S2L2A_B02 - S2L2A_B03 - S2L2A_B04 - S2L2A_B05 - S2L2A_B06 - S2L2A_B07 - S2L2A_B08 - S2L2A_B8A - S2L2A_B09 - S2L2A_B11 - S2L2A_B12 productTypes : - name : S2MSI1C filter : s2:product_type : S2MSI1C metadata_assets : [] coverages : S2L1C_B01 : assets : - B01 S2L1C_B02 : assets : - B02 S2L1C_B03 : assets : - B03 S2L1C_B04 : assets : - B04 S2L1C_B05 : assets : - B05 S2L1C_B06 : assets : - B06 S2L1C_B07 : assets : - B07 S2L1C_B08 : assets : - B08 S2L1C_B8A : assets : - B8A S2L1C_B09 : assets : - B09 S2L1C_B10 : assets : - B10 S2L1C_B11 : assets : - B11 S2L1C_B12 : assets : - B12 defaultBrowse : TRUE_COLOR browses : TRUE_COLOR : asset : visual red : expression : B04 range : [ 0 , 4000 ] nodata : 0 green : expression : B03 range : [ 0 , 4000 ] nodata : 0 blue : expression : B02 range : [ 0 , 4000 ] nodata : 0 FALSE_COLOR : red : expression : B08 range : [ 0 , 4000 ] nodata : 0 green : expression : B04 range : [ 0 , 4000 ] nodata : 0 blue : expression : B03 range : [ 0 , 4000 ] nodata : 0 NDVI : grey : expression : (B08-B04)/(B08+B04) range : [ -1 , 1 ] masks : clouds : validity : false - name : S2MSI2A filter : s2:product_type : S2MSI2A metadata_assets : [] coverages : S2L2A_B01 : assets : - B01 S2L2A_B02 : assets : - B02 S2L2A_B03 : assets : - B03 S2L2A_B04 : assets : - B04 S2L2A_B05 : assets : - B05 S2L2A_B06 : assets : - B06 S2L2A_B07 : assets : - B07 S2L2A_B08 : assets : - B08 S2L2A_B8A : assets : - B8A S2L2A_B09 : assets : - B09 S2L2A_B11 : assets : - B11 S2L2A_B12 : assets : - B12 default_browse_locator : TCI_10m browses : TRUE_COLOR : asset : visual-10m red : expression : B04 range : [ 0 , 4000 ] nodata : 0 green : expression : B03 range : [ 0 , 4000 ] nodata : 0 blue : expression : B02 range : [ 0 , 4000 ] nodata : 0 FALSE_COLOR : red : expression : B08 range : [ 0 , 4000 ] nodata : 0 green : expression : B04 range : [ 0 , 4000 ] nodata : 0 blue : expression : B03 range : [ 0 , 4000 ] nodata : 0 NDVI : grey : expression : (B08-B04)/(B08+B04) range : [ -1 , 1 ] masks : clouds : validity : false","title":"CREODIAS Deployment"},{"location":"quickstart/creodias-deployment/#creodias-deployment","text":"Based upon our development experiences on CREODIAS, there is a wrapper script creodias with particular customisations suited to the CREODIAS infrastructure and data offering. The customisations are expressed through environment variables that are captured in the file creodias-options . These scripts are examples that can be seen as a starting point, from which they can be adapted to your needs. The CREODIAS deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Protected service endpoints requiring IAM authorization See Endpoint Protection below for further information With reference to the file creodias-options , particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment\u2026 Passwords: LOGIN_SERVICE_ADMIN_PASSWORD , MINIO_ROOT_PASSWORD , HARBOR_ADMIN_PASSWORD OpenStack details: see section Openstack Configuration If configuring an external deployment - ref. Public Deployment \u2026 public_ip - The public IP address through which the deployment is exposed via the ingress-controller domain - The DNS domain name through which the deployment is accessed - forming the stem for all service hostnames in the ingress rules Once the file creodias-options has been well populated for your environment, then the deployment is initiated with\u2026 ./deploy/creodias/creodias \u2026noting that this step is a customised version of that described in section Deployment .","title":"CREODIAS Deployment"},{"location":"quickstart/creodias-deployment/#endpoint-protection","text":"Similarly the script creodias-protection is a customised version of that described in section Apply Protection . Once the main deployment has completed, then the test users can be created , their IDs ( Inum ) set in script creodias-protection , and the resource protection can then be applied\u2026 ./deploy/creodias/creodias-protection","title":"Endpoint Protection"},{"location":"quickstart/creodias-deployment/#harvest-creodias-data","text":"The harvester can be deployed with a default configuration file at /config.yaml . As described in the Data Access section , harvesting according to this configuration can be triggered with\u2026 kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch See the Harvester section below for an explanation of this harvester configuration.","title":"Harvest CREODIAS Data"},{"location":"quickstart/creodias-deployment/#data-specification-walkthrough","text":"The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables . This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example.","title":"Data Specification Walkthrough"},{"location":"quickstart/creodias-deployment/#harvester","text":"The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment. The harvester is configured as follows\u2026 harvester: image: repository: eoepca/rm-harvester tag: 1.1.0 config: redis: host: data-access-redis-master port: 6379 harvesters: - name: Creodias-Opensearch resource: url: https://finder.creodias.eu/resto/api/collections/Sentinel2/describe.xml type: OpenSearch format_config: type: 'application/json' property_mapping: start_datetime: 'startDate' end_datetime: 'completionDate' productIdentifier: 'productIdentifier' query: time: property: sensed begin: 2019-09-10T00:00:00Z end: 2019-09-11T00:00:00Z collection: null bbox: 14.9,47.7,16.4,48.7 filter: {} postprocess: - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor queue: register_queue Based upon this harvester configuration we expect that the following query is made to discover data - i.e. an OpenSearch query, with json response representation, for a defined spatial and temporal extent\u2026 https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2019-09-10T00:00:00Z&completionDate=2019-09-11T00:00:00Z&box=14.9,47.7,16.4,48.7 From the result returned, the path to each product ( feature ) is obtained from the productIdentifier property, e.g. { \"type\": \"FeatureCollection\", \"features\": [ { \"type\": \"Feature\", \"properties\": { \"productIdentifier\": \"/eodata/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE\" ... } ... } ... ] } The harvester is configured with a Sentinel-2/CREODIAS specific post-processor harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor which transforms the product path from /eodata/... to s3://EODATA/... . The harvester post-processor follows this path to the Sentinel-2 scene and uses stactools (with built-in support for Sentinel-2) to establish a STAC item representing the product. This includes enumeration of assets for inspire-metadata and product-metadata - which are used by the registrar pycsw backend to embelesh the product record metadata. The harvester outputs the STAC item for each product, which is pushed to the registrar via the register_queue redis queue.","title":"Harvester"},{"location":"quickstart/creodias-deployment/#registration","text":"The registrar is configured at deployment to have the access details for the CREODIAS data in S3\u2026 global: storage: data: data: type: S3 endpoint_url: http://data.cloudferro.com access_key_id: access secret_access_key: access region_name: RegionOne validate_bucket_name: false Using this S3 configuration, the registrar pycsw backend uses the product metadata linked in the STAC item (ref. assets inspire-metadata and product-metadata ) to embelesh the metadata. For example, product-metadata in the file\u2026 s3://EODATA/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE/MTD_MSIL1C.xml The registrar uses this information to create the ISO XML metadata that is loaded into the resource-catalogue.","title":"Registration"},{"location":"quickstart/creodias-deployment/#product-type","text":"The registrar recognises the product as Sentinel-2 and so reads its metadata XML files to obtain additional information. From the metadata XML file (e.g. MTD_MSIL1C.xml ) the registrar obtains the Product Type for each product from the field <PRODUCT_TYPE> \u2026 <n1:Level-1C_User_Product> <n1:General_Info> <Product_Info> <PRODUCT_TYPE>S2MSI1C</PRODUCT_TYPE> ... </Product_Info> ... </n1:General_Info> ... <n1:Level-1C_User_Product>","title":"Product Type"},{"location":"quickstart/creodias-deployment/#resource-catalogue-collections","text":"The registrar ( eoepca/rm-data-access-core ) container image is pre-loaded with two collections at the path /registrar_pycsw/registrar_pycsw/resources , (in the built container the files are at the path /usr/local/lib/python3.8/dist-packages/registrar_pycsw/resources/ ): S2MSI1C.yml - identifier: S2MSI1C S2MSI2A.yml - identifier: S2MSI2A The registrar applies these collections into the resource-catalogue during start-up - to create pre-defined out-of-the-box collections in pycsw. During registration, the PycswBackend of the registrar uses the Product Type to map the product into the collection of the same name - using metadata field parentidentifier .","title":"Resource Catalogue Collections"},{"location":"quickstart/creodias-deployment/#data-specification","text":"The data-access service data handling is configured by definition of productTypes , collections and layers \u2026 productTypes identify the underlying file assets as WCS coverages and their visual representation collections provide groupings into which products are organised layers specifies the hoe the product visual representations are exposed through the WMS service","title":"Data Specification"},{"location":"quickstart/creodias-deployment/#producttype","text":"During registration, products are mapped into a productType via a filter that is applied against the STAC item metadata. The registrar uses the product_type of each product to determine the collection into which the product should be registered - noting that the name of the product type does not take part in the matching logic (and hence can be any text name)\u2026 productTypes: - name: S2MSI1C filter: s2:product_type: S2MSI1C In the above example, the field s2:product_type is populated by the stactools that prepares the STAC item from the product metadata.","title":"productType"},{"location":"quickstart/creodias-deployment/#collections","text":"Collections are defined by reference to the defined productTypes and coverages .","title":"collections"},{"location":"quickstart/creodias-deployment/#layers","text":"layers defines the layers that are presented through the WMS service - each layer being linked to the underlying browse that provides the image source. Layers are defined via their id that relies upon the naming convection <collection>__<browse> to identify the browse and so define the layer.","title":"layers"},{"location":"quickstart/creodias-deployment/#example-configuration","text":"Example configuration for Sentinel-2 L1C and L2A data. global : layers : - id : S2L1C title : Sentinel-2 Level 1C True Color abstract : Sentinel-2 Level 2A True Color displayColor : '#eb3700' grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__TRUE_COLOR title : Sentinel-2 Level 1C True Color abstract : Sentinel-2 Level 2A True Color grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__masked_clouds title : Sentinel-2 Level 1C True Color with cloud masks abstract : Sentinel-2 Level 1C True Color with cloud masks grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__FALSE_COLOR title : Sentinel-2 Level 1C False Color abstract : Sentinel-2 Level 1C False Color grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__NDVI title : Sentinel-2 Level 21CNDVI abstract : Sentinel-2 Level 1C NDVI grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L2A title : Sentinel-2 Level 2A True Color abstract : Sentinel-2 Level 2A True Color displayColor : '#eb3700' grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__TRUE_COLOR title : Sentinel-2 Level 2A True Color abstract : Sentinel-2 Level 2A True Color grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__masked_clouds title : Sentinel-2 Level 2A True Color with cloud masks abstract : Sentinel-2 Level 2A True Color with cloud masks grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__FALSE_COLOR title : Sentinel-2 Level 2A False Color abstract : Sentinel-2 Level 2A False Color grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__NDVI title : Sentinel-2 Level 2A NDVI abstract : Sentinel-2 Level 2A NDVI grids : - name : WGS84 zoom : 13 parentLayer : S2L2A collections : S2L1C : product_types : - S2MSI1C coverage_types : - S2L1C_B01 - S2L1C_B02 - S2L1C_B03 - S2L1C_B04 - S2L1C_B05 - S2L1C_B06 - S2L1C_B07 - S2L1C_B08 - S2L1C_B8A - S2L1C_B09 - S2L1C_B10 - S2L1C_B11 - S2L1C_B12 S2L2A : product_types : - S2MSI2A product_levels : - Level-2A coverage_types : - S2L2A_B01 - S2L2A_B02 - S2L2A_B03 - S2L2A_B04 - S2L2A_B05 - S2L2A_B06 - S2L2A_B07 - S2L2A_B08 - S2L2A_B8A - S2L2A_B09 - S2L2A_B11 - S2L2A_B12 productTypes : - name : S2MSI1C filter : s2:product_type : S2MSI1C metadata_assets : [] coverages : S2L1C_B01 : assets : - B01 S2L1C_B02 : assets : - B02 S2L1C_B03 : assets : - B03 S2L1C_B04 : assets : - B04 S2L1C_B05 : assets : - B05 S2L1C_B06 : assets : - B06 S2L1C_B07 : assets : - B07 S2L1C_B08 : assets : - B08 S2L1C_B8A : assets : - B8A S2L1C_B09 : assets : - B09 S2L1C_B10 : assets : - B10 S2L1C_B11 : assets : - B11 S2L1C_B12 : assets : - B12 defaultBrowse : TRUE_COLOR browses : TRUE_COLOR : asset : visual red : expression : B04 range : [ 0 , 4000 ] nodata : 0 green : expression : B03 range : [ 0 , 4000 ] nodata : 0 blue : expression : B02 range : [ 0 , 4000 ] nodata : 0 FALSE_COLOR : red : expression : B08 range : [ 0 , 4000 ] nodata : 0 green : expression : B04 range : [ 0 , 4000 ] nodata : 0 blue : expression : B03 range : [ 0 , 4000 ] nodata : 0 NDVI : grey : expression : (B08-B04)/(B08+B04) range : [ -1 , 1 ] masks : clouds : validity : false - name : S2MSI2A filter : s2:product_type : S2MSI2A metadata_assets : [] coverages : S2L2A_B01 : assets : - B01 S2L2A_B02 : assets : - B02 S2L2A_B03 : assets : - B03 S2L2A_B04 : assets : - B04 S2L2A_B05 : assets : - B05 S2L2A_B06 : assets : - B06 S2L2A_B07 : assets : - B07 S2L2A_B08 : assets : - B08 S2L2A_B8A : assets : - B8A S2L2A_B09 : assets : - B09 S2L2A_B11 : assets : - B11 S2L2A_B12 : assets : - B12 default_browse_locator : TCI_10m browses : TRUE_COLOR : asset : visual-10m red : expression : B04 range : [ 0 , 4000 ] nodata : 0 green : expression : B03 range : [ 0 , 4000 ] nodata : 0 blue : expression : B02 range : [ 0 , 4000 ] nodata : 0 FALSE_COLOR : red : expression : B08 range : [ 0 , 4000 ] nodata : 0 green : expression : B04 range : [ 0 , 4000 ] nodata : 0 blue : expression : B03 range : [ 0 , 4000 ] nodata : 0 NDVI : grey : expression : (B08-B04)/(B08+B04) range : [ -1 , 1 ] masks : clouds : validity : false","title":"Example Configuration"},{"location":"quickstart/data-access-deployment/","text":"Data Access Deployment \u2693\ufe0e A deployment wrapper script has been prepared for a \u2018data access\u2019 deployment - that is focused on the Resource Catalogue and Data Access services. The script deploy/data-access/data-access achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/data-access/data-access-options . The data-access deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: Resource Catalogue for data discovery Data Access for data visualisation and download Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Open ingress are enabled for unauthenticated access to resource-catalogue and data-access services Other eoepca services not deployed Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/data-access/data-access The Resource Catalogue is accessed at the endpoint resource-catalogue-open.<domain> - e.g. resource-catalogue-open.192-168-49-2.nip.io . The Data Access View Server is accessed at the endpoint data-access-open.<domain> - e.g. data-access-open.192-168-49-2.nip.io . Data Harvesting \u2693\ufe0e See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.","title":"Data Access"},{"location":"quickstart/data-access-deployment/#data-access-deployment","text":"A deployment wrapper script has been prepared for a \u2018data access\u2019 deployment - that is focused on the Resource Catalogue and Data Access services. The script deploy/data-access/data-access achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/data-access/data-access-options . The data-access deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: Resource Catalogue for data discovery Data Access for data visualisation and download Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Open ingress are enabled for unauthenticated access to resource-catalogue and data-access services Other eoepca services not deployed","title":"Data Access Deployment"},{"location":"quickstart/data-access-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/data-access/data-access The Resource Catalogue is accessed at the endpoint resource-catalogue-open.<domain> - e.g. resource-catalogue-open.192-168-49-2.nip.io . The Data Access View Server is accessed at the endpoint data-access-open.<domain> - e.g. data-access-open.192-168-49-2.nip.io .","title":"Initiate Deployment"},{"location":"quickstart/data-access-deployment/#data-harvesting","text":"See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.","title":"Data Harvesting"},{"location":"quickstart/exploitation-deployment/","text":"Exploitation Deployment \u2693\ufe0e A deployment wrapper script has been prepared for an \u2018exploitation\u2019 deployment - that provides deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services. The script deploy/exploitation/exploitation achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/exploitation/exploitation-options . The exploitation deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: ADES for processing Resource Catalogue for data discovery Data Access for data visualisation and download Minio for S3 object storage ADES stage-out to Minio Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Open ingress are enabled for unauthenticated access to ADES, resource-catalogue and data-access services Other eoepca services not deployed Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/exploitation/exploitation The ADES service is accessed at the endpoint ades-open.<domain> - e.g. ades-open.192-168-49-2.nip.io . The Resource Catalogue is accessed at the endpoint resource-catalogue-open.<domain> - e.g. resource-catalogue-open.192-168-49-2.nip.io . The Data Access View Server is accessed at the endpoint data-access-open.<domain> - e.g. data-access-open.192-168-49-2.nip.io . Example Requests - s-expression on CREODIAS \u2693\ufe0e NOTE that this example processing request requires harvesting data from CREODIAS, which can only be performed if the deployment is made to a CREODIAS VM with access to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION . Section Processing provides an example of a simple self-contained processing deployment and execution , and access to the processing results . In addition to the snuggs example, the file deploy/samples/requests/processing/s-expression.http has been prepared to exploit data that has been registered within the Resource Catalogue and Data Access services. First the input data for processing must be harvested into the resource management services. Sentinel-2 data on 2nd Sept 2020\u2026 ./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2020.09.02.yaml Then the s-expression.http file provides sample requests for OGC API Processes operations: List Processes Deploy Process Get Process Details Execute Process Get Job Status Get Job Results NOTE that the first requests in the file provide optional calls to obtain a user ID token ( openidConfiguration / authenticate ) - to be used in the case that protected (not \u2018open\u2019) endpoints are deployed. The file snuggs.http describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client . Install in vscode with ext install humao.rest-client . Various variables, such as to specify the @domain for your deployment, can be configured at the top of the file. At the completion of successful processing execution, the procesing results are obtained as described in section Processing Results . Data Harvesting \u2693\ufe0e See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.","title":"Exploitation"},{"location":"quickstart/exploitation-deployment/#exploitation-deployment","text":"A deployment wrapper script has been prepared for an \u2018exploitation\u2019 deployment - that provides deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services. The script deploy/exploitation/exploitation achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/exploitation/exploitation-options . The exploitation deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: ADES for processing Resource Catalogue for data discovery Data Access for data visualisation and download Minio for S3 object storage ADES stage-out to Minio Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Open ingress are enabled for unauthenticated access to ADES, resource-catalogue and data-access services Other eoepca services not deployed","title":"Exploitation Deployment"},{"location":"quickstart/exploitation-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/exploitation/exploitation The ADES service is accessed at the endpoint ades-open.<domain> - e.g. ades-open.192-168-49-2.nip.io . The Resource Catalogue is accessed at the endpoint resource-catalogue-open.<domain> - e.g. resource-catalogue-open.192-168-49-2.nip.io . The Data Access View Server is accessed at the endpoint data-access-open.<domain> - e.g. data-access-open.192-168-49-2.nip.io .","title":"Initiate Deployment"},{"location":"quickstart/exploitation-deployment/#example-requests-s-expression-on-creodias","text":"NOTE that this example processing request requires harvesting data from CREODIAS, which can only be performed if the deployment is made to a CREODIAS VM with access to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION . Section Processing provides an example of a simple self-contained processing deployment and execution , and access to the processing results . In addition to the snuggs example, the file deploy/samples/requests/processing/s-expression.http has been prepared to exploit data that has been registered within the Resource Catalogue and Data Access services. First the input data for processing must be harvested into the resource management services. Sentinel-2 data on 2nd Sept 2020\u2026 ./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2020.09.02.yaml Then the s-expression.http file provides sample requests for OGC API Processes operations: List Processes Deploy Process Get Process Details Execute Process Get Job Status Get Job Results NOTE that the first requests in the file provide optional calls to obtain a user ID token ( openidConfiguration / authenticate ) - to be used in the case that protected (not \u2018open\u2019) endpoints are deployed. The file snuggs.http describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client . Install in vscode with ext install humao.rest-client . Various variables, such as to specify the @domain for your deployment, can be configured at the top of the file. At the completion of successful processing execution, the procesing results are obtained as described in section Processing Results .","title":"Example Requests - s-expression on CREODIAS"},{"location":"quickstart/exploitation-deployment/#data-harvesting","text":"See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.","title":"Data Harvesting"},{"location":"quickstart/processing-deployment/","text":"Processing Deployment \u2693\ufe0e A deployment wrapper script has been prepared for a \u2018processing\u2019 deployment - that is focused on the ADES and the deployment/execution of processing jobs. The script deploy/processing/processing achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/processing/processing-options . The processing deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: ADES for processing Minio for S3 object storage ADES stage-out to Minio Open ingress are enabled for unauthenticated access to ADES service Other eoepca services not deployed Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/processing/processing The ADES service is accessed at the endpoint ades-open.<domain> - e.g. ades-open.192-168-49-2.nip.io . Example Requests - snuggs application \u2693\ufe0e The file deploy/samples/requests/processing/snuggs.http has been prepared with sample requests for OGC API Processes operations: List Processes Deploy Process Get Process Details Execute Process Get Job Status Get Job Results Note The first requests in the file provide optional calls to obtain a user ID token ( openidConfiguration / authenticate ). These are to be used in the case that protected (not \u2018open\u2019) endpoints are deployed. The file describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client . Install in vscode with ext install humao.rest-client . The variables @hostname and @domain can be configured at the top of the file. Requests using CuRL Alternatively, the following curl commands can be used instead\u2026 List Processes curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes \\ --header 'accept: application/json' Deploy Process curl -k \\ --request POST \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes \\ --header 'accept: application/json' \\ --header 'content-type: application/json' \\ --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/app-snuggs/main/app-package.cwl\",\"type\": \"application/cwl\"}}' Get Process Details curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0 \\ --header 'accept: application/json' Execute Process curl -k -v \\ --request POST \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0/execution \\ --header 'accept: application/json' \\ --header 'content-type: application/json' \\ --header 'prefer: respond-async' \\ --data '{\"inputs\": {\"input_reference\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\",\"s_expression\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\"},\"response\":\"raw\"}' Get Job Status This request requires the Location header from the response to the execute request. This will be of the form /{user}/wps3/jobs/{job-id} - e.g. /eric/wps3/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e . curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io { location-header } \\ --header 'accept: application/json' Get Job Results This request uses the same URL as Get Job Status , with the additional URL path /result - i.e. /{user}/wps3/jobs/{job-id}/result - e.g. /eric/wps3/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/result curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io { location-header } /result \\ --header 'accept: application/json' The response indicates the location of the results, which should be in the minio object storage. See Processing Results List Jobs curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/jobs \\ --header 'accept: application/json' Processing Results \u2693\ufe0e The outputs are published as a static STAC catalogue to a path that includes the unique job ID. In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint console.minio.<domain> e.g. https://console.minio.192-168-49-2.nip.io/, or using an S3 client such as\u2026 s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca For the default credentials to connect to Minio see Minio Object Storage Default Credentials .","title":"Processing Deployment"},{"location":"quickstart/processing-deployment/#processing-deployment","text":"A deployment wrapper script has been prepared for a \u2018processing\u2019 deployment - that is focused on the ADES and the deployment/execution of processing jobs. The script deploy/processing/processing achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/processing/processing-options . The processing deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: ADES for processing Minio for S3 object storage ADES stage-out to Minio Open ingress are enabled for unauthenticated access to ADES service Other eoepca services not deployed","title":"Processing Deployment"},{"location":"quickstart/processing-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/processing/processing The ADES service is accessed at the endpoint ades-open.<domain> - e.g. ades-open.192-168-49-2.nip.io .","title":"Initiate Deployment"},{"location":"quickstart/processing-deployment/#example-requests-snuggs-application","text":"The file deploy/samples/requests/processing/snuggs.http has been prepared with sample requests for OGC API Processes operations: List Processes Deploy Process Get Process Details Execute Process Get Job Status Get Job Results Note The first requests in the file provide optional calls to obtain a user ID token ( openidConfiguration / authenticate ). These are to be used in the case that protected (not \u2018open\u2019) endpoints are deployed. The file describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client . Install in vscode with ext install humao.rest-client . The variables @hostname and @domain can be configured at the top of the file. Requests using CuRL Alternatively, the following curl commands can be used instead\u2026 List Processes curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes \\ --header 'accept: application/json' Deploy Process curl -k \\ --request POST \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes \\ --header 'accept: application/json' \\ --header 'content-type: application/json' \\ --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/app-snuggs/main/app-package.cwl\",\"type\": \"application/cwl\"}}' Get Process Details curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0 \\ --header 'accept: application/json' Execute Process curl -k -v \\ --request POST \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0/execution \\ --header 'accept: application/json' \\ --header 'content-type: application/json' \\ --header 'prefer: respond-async' \\ --data '{\"inputs\": {\"input_reference\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\",\"s_expression\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\"},\"response\":\"raw\"}' Get Job Status This request requires the Location header from the response to the execute request. This will be of the form /{user}/wps3/jobs/{job-id} - e.g. /eric/wps3/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e . curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io { location-header } \\ --header 'accept: application/json' Get Job Results This request uses the same URL as Get Job Status , with the additional URL path /result - i.e. /{user}/wps3/jobs/{job-id}/result - e.g. /eric/wps3/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/result curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io { location-header } /result \\ --header 'accept: application/json' The response indicates the location of the results, which should be in the minio object storage. See Processing Results List Jobs curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/jobs \\ --header 'accept: application/json'","title":"Example Requests - snuggs application"},{"location":"quickstart/processing-deployment/#processing-results","text":"The outputs are published as a static STAC catalogue to a path that includes the unique job ID. In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint console.minio.<domain> e.g. https://console.minio.192-168-49-2.nip.io/, or using an S3 client such as\u2026 s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca For the default credentials to connect to Minio see Minio Object Storage Default Credentials .","title":"Processing Results"},{"location":"quickstart/quickstart/","text":"Quickstart \u2693\ufe0e The deployment of the EOEPCA components and the supporting Kubernetes cluster is described in the sections Cluster and EOEPCA . These sections should be consulted for more detailed information. Scripted Deployment \u2693\ufe0e As a companion to these descriptions, we have developed a set of scripts to provide a demonstration of an example deployment. This is described in the following section Scripted Deployment . Note The scripted deployment assumes that installation of the Prerequisite Tooling has been performed Customised Deployments \u2693\ufe0e The Scripted Deployment can be quickly exploited through the following customisations for particular use cases: Simple Basic local deployment Processing Deployment focused on processing Data Access Deployment focused on the Resource Catalogue and Data Access services Exploitation Deployment providing deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services User Management Deployment focused on the User Management services CREODIAS Deployment with access to CREODIAS EO data Each customisation is introduced in their respective sections. Quick Example \u2693\ufe0e Follow these steps to create a simple local deployment in minikube\u2026 Prerequisite Tooling Follow the steps in section Prerequisite Tooling to install the required tooling. Clone the repository git clone https://github.com/EOEPCA/deployment-guide Initiate the deployment cd deployment-guide ./deploy/simple/simple Wait for deployment ready List pod status watch kubectl get pod -A Wait until all pods report either Running or Completed This may take 10-20 mins depending on the capabilities of your platform. Test the deployment Make the sample requests to the ADES processing service.","title":"Quick Start"},{"location":"quickstart/quickstart/#quickstart","text":"The deployment of the EOEPCA components and the supporting Kubernetes cluster is described in the sections Cluster and EOEPCA . These sections should be consulted for more detailed information.","title":"Quickstart"},{"location":"quickstart/quickstart/#scripted-deployment","text":"As a companion to these descriptions, we have developed a set of scripts to provide a demonstration of an example deployment. This is described in the following section Scripted Deployment . Note The scripted deployment assumes that installation of the Prerequisite Tooling has been performed","title":"Scripted Deployment"},{"location":"quickstart/quickstart/#customised-deployments","text":"The Scripted Deployment can be quickly exploited through the following customisations for particular use cases: Simple Basic local deployment Processing Deployment focused on processing Data Access Deployment focused on the Resource Catalogue and Data Access services Exploitation Deployment providing deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services User Management Deployment focused on the User Management services CREODIAS Deployment with access to CREODIAS EO data Each customisation is introduced in their respective sections.","title":"Customised Deployments"},{"location":"quickstart/quickstart/#quick-example","text":"Follow these steps to create a simple local deployment in minikube\u2026 Prerequisite Tooling Follow the steps in section Prerequisite Tooling to install the required tooling. Clone the repository git clone https://github.com/EOEPCA/deployment-guide Initiate the deployment cd deployment-guide ./deploy/simple/simple Wait for deployment ready List pod status watch kubectl get pod -A Wait until all pods report either Running or Completed This may take 10-20 mins depending on the capabilities of your platform. Test the deployment Make the sample requests to the ADES processing service.","title":"Quick Example"},{"location":"quickstart/scripted-deployment/","text":"Scripted Deployment \u2693\ufe0e The Scripted Deployment provides a demonstration of an example deployment, and can found in the subdirectory deployment-guide/deploy of the source repository for this guide\u2026 git clone https://github.com/EOEPCA/deployment-guide \\ && cd deployment-guide \\ && ls deploy The script deploy/eoepca/eoepca.sh acts as an entry-point to the full system deployment. In order to tailor the deployment for your target environment, the script is configured through environment variables and command-line arguments. By default the script assumes deployment to a local minikube. Note The scripted deployment assumes that installation of the Prerequisite Tooling has been performed. The following subsections lead through the steps for a full local deployment. Whilst minikube is assumed, minimal adaptions are required to make the deployment to your existing Kubernetes cluster. The deployment follows these broad steps: Configuration Tailoring of deployment options. Deployment Creation of cluster and deployment of eoepca services. Protection Application of protection for authorized access to services. The Protection step is split from Deployment as there are some manual steps to be performed before the Protection can be applied. Configuration \u2693\ufe0e The script deploy/eoepca/eoepca.sh is configured by some environment variables and command-line arguments. Environment Variables \u2693\ufe0e Environment Variables Variable Description Default REQUIRE_<cluster-component> A set of variables that can be used to control which CLUSTER components are deployed by the script, as follows (with defaults): REQUIRE_MINIKUBE=true REQUIRE_INGRESS_NGINX=true REQUIRE_CERT_MANAGER=true REQUIRE_LETSENCRYPT=true REQUIRE_SEALED_SECRETS=false REQUIRE_MINIO=false see description REQUIRE_<eoepca-component> A set of variables that can be used to control which EOEPCA components are deployed by the script, as follows (with defaults): REQUIRE_STORAGE=true REQUIRE_DUMMY_SERVICE=false REQUIRE_LOGIN_SERVICE=true REQUIRE_PDP=true REQUIRE_USER_PROFILE=true REQUIRE_ADES=true REQUIRE_RESOURCE_CATALOGUE=true REQUIRE_DATA_ACCESS=true REQUIRE_WORKSPACE_API=true REQUIRE_BUCKET_OPERATOR=true REQUIRE_HARBOR=true REQUIRE_PORTAL=true REQUIRE_PDE=true see description REQUIRE_<protection-component> A set of variables that can be used to control which PROTECTION components are deployed by the script, as follows (with defaults): REQUIRE_DUMMY_SERVICE_PROTECTION=false REQUIRE_ADES_PROTECTION=true REQUIRE_RESOURCE_CATALOGUE_PROTECTION=true REQUIRE_DATA_ACCESS_PROTECTION=true REQUIRE_WORKSPACE_API_PROTECTION=true see description MINIKUBE_KUBERNETES_VERSION The Kubernetes version to be used by minikube Note that the EOEPCA development has been conducted primarily using version 1.22.5. v1.22.5 MINIKUBE_MEMORY_AMOUNT Amount of memory to allocate to the docker containers used by minikube to implement the cluster. 12g USE_METALLB Enable use of minikube\u2019s built-in load-balancer. The load-balancer can be used to facilitate exposing services publicly. However, the same can be achieved using minikube\u2019s built-in ingress-controller. Therefore, this option is suppressed by default. false USE_INGRESS_NGINX_HELM Install the ingress-nginx controller using the published helm chart, rather than relying upon the version that is built-in to minikube. By default we prefer the version that is built in to minikube. false USE_INGRESS_NGINX_LOADBALANCER Patch the built-in minikube nginx-ingress-controller to offer a service of type LoadBalancer , rather than the default NodePort . It was initially thought that this would be necessary to achieve public access to the ingress services - but was subsequently found that the default NodePort configuration of the ingress-controller was sufficient. This option is left in case it proves useful. Only applicable for USE_INGRESS_NGINX_HELM=false (i.e. when using the minikube built-in ) false OPEN_INGRESS Create \u2018open\u2019 ingress endpoints that are not subject to authorization protection. For a secure system the open endpoints should be disabled ( false ) and access to resource should be protected via ingress that apply protection false USE_TLS Indicates whether TLS will be configured for service Ingress rules. If not (i.e. USE_TLS=false ), then the ingress-controller is configured to disable ssl-redirect , and TLS_CLUSTER_ISSUER=notls is set. true TLS_CLUSTER_ISSUER The name of the ClusterIssuer to satisfy ingress tls certificates. Out-of-the-box ClusterIssuer instances are configured in the file deploy/cluster/letsencrypt.sh . letsencrypt-staging LOGIN_SERVICE_ADMIN_PASSWORD Initial password for the admin user in the login-service. changeme MINIO_ROOT_USER Name of the \u2018root\u2019 user for the Minio object storage service. eoepca MINIO_ROOT_PASSWORD Password for the \u2018root\u2019 user for the Minio object storage service. changeme HARBOR_ADMIN_PASSWORD Password for the \u2018admin\u2019 user for the Harbor artefact registry service. changeme PROCESSING_MAX_RAM Max RAM allocated to an individual processing job 8Gi PROCESSING_MAX_CORES Max number of CPU cores allocated to an individual processing job 4 STAGEOUT_TARGET Configures the ADES with the destination to which it should push processing results: workspace - via the Workspace API minio - to minio S3 object storage workspace INSTALL_FLUX The Workspace API relies upon Flux CI/CD , and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value false to suppress the Workspace API flux install true CREODIAS_DATA_SPECIFICATION Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services. Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the eodata network. false Openstack Configuration \u2693\ufe0e There are some additional environment variables that configure the BucketOperator with details of the infrastructure Openstack layer. Attention This is only applicable for an Openstack deployment and has only been tested on the CREODIAS. Environment Variables Variable Description Default OS_DOMAINNAME Openstack domain of the admin account in the cloud provider. cloud_XXXXX OS_USERNAME Openstack username of the admin account in the cloud provider. user@cloud.com OS_PASSWORD Openstack password of the admin account in the cloud provider. none OS_MEMBERROLEID ID of a specific role (e.g. the \u2018 member \u2018 role) for operations users (to allow administration), e.g. 7fe2ff9ee5384b1894a90838d3e92bab . none OS_SERVICEPROJECTID ID of a project containing the user identity requiring write access to the created user buckets, e.g. 573916ef342a4bf1aea807d0c6058c1e . none USER_EMAIL_PATTERN Email associated to the created user within the created user project. Note: <name> is templated and will be replaced. eoepca-<name>@platform.com Command-line Arguments \u2693\ufe0e The eoepca.sh script is further configured via command-line arguments\u2026 eoepca.sh <action> <cluster-name> <public-ip> <domain> Arguments Argument Description Default action Action to perform: apply | delete | template . apply makes the deployment delete removes the deployment template outputs generated kubernetes yaml to stdout apply cluster-name The name of the minikube \u2018profile\u2019 for the created minikube cluster eoepca public-ip The public IP address through which the deployment is exposed via the ingress-controller. By default, the value is deduced from the assigned cluster minikube IP address - ref. command minikube ip . <minikube-ip> domain The DNS domain name through which the deployment is accessed. Forms the stem for all service hostnames in the ingress rules - i.e. <service-name>.<domain> . By default, the value is deduced from the assigned cluster minikube IP address, using nip.io to establish a DNS lookup - i.e. <minikube ip>.nip.io . <minikube ip>.nip.io Public Deployment \u2693\ufe0e For simplicity, the out-of-the-box scripts assume a \u2018private\u2019 deployment - with no public IP / DNS and hence no use of TLS for service ingress endpoints. In the case that an external-facing public deployment is desired, then the following configuration selections should be made: public_ip - set to the external-facing public IP of your deployment, e.g. the floating IP of your load-balancer in a cloud deployment domain - set to the domain (as per DNS records) for your deployment Note that the EOEPCA components typically configure their ingress with hostname prefixes applied to this domain . Thus, it is necessary that the DNS record for the domain is established as a wildcard record - i.e. *.<domain> USE_TLS=true - to enable configuration of TLS endpoints in each component service ingress TLS_CLUSTER_ISSUER=<issuer> - should be configured ~ e.g. using the letsencrypt-production or letsencrypt-staging (testing only) Cluster Issuer that are configured by the scripted deployment Deployment \u2693\ufe0e The deployment is initiated by setting the appropriate environment variables and invoking the eoepca.sh script with suitable command-line arguments . You may find it convenient to do so using a wrapper script that customises the environment varaibles according to your cluster, and then invokes the eoepca.sh script. Customised examples are provided for Simple, CREODIAS and Processing deployments. NOTE that if a prior deployment has been attempted then, before redeploying, a clean-up should be performed as described in the Clean-up section below. This is particularly important in the case that the minikube none driver is used, as the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. Initiate the deployment\u2026 ./deploy/eoepca/eoepca.sh apply \"<cluster-name>\" \"<public-ip>\" \"<domain>\" The deployment takes 10+ minutes - depending on the resources of your host/cluster. The progress can be monitored\u2026 kubectl get pods -A The deployment is ready once all pods are either Running or Completed . This can be further confirmed by accessing the login-service web interface at https://auth.<domain>/ and logging in as user admin using the credentials configured via LOGIN_SERVICE_ADMIN_PASSWORD . Default Credentials \u2693\ufe0e Login Service \u2693\ufe0e By default, the Login Service is accessed at the URL https://auth.<domain>/ with the credentials\u2026 username: admin password: Chang3me! \u2026unless the password is overridden via the variable LOGIN_SERVICE_ADMIN_PASSWORD . Minio Object Storage \u2693\ufe0e By default, Minio is accessed at the URL http://console.minio.<domain>/ with the credentials\u2026 username: eoepca password: changeme \u2026unless the username/password are overridden via the variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD . Harbor Container Registry \u2693\ufe0e By default, Harbor is accessed at the URL https://harbor.<domain>/ with the credentials\u2026 username: admin password: changeme \u2026unless the password is overridden via the variable HARBOR_ADMIN_PASSWORD . Protection \u2693\ufe0e The protection of resource server endpoints is applied with the script deploy/eoepca/eoepca-protection.sh . This script should be executed with environment variables and command-line options that are consistent with those of the main deployment (ref. script eoepca.sh ). The script eoepca-protection.sh introduces two users eric and bob to demonstrate the application of authorized access to various service endpoints: ADES, Workspace API and dummy-service (simple endpoint used for debugging). Thus, the users must first be created in the login-service and their unique IDs passed to the protection script. Usage: eoepca-protection.sh <action> <eric-id> <bob-id> <public-ip> <domain> Create Test Users \u2693\ufe0e Access the login-service web interface ( https://auth.<domain>/ ) as user admin using the credentials configured via LOGIN_SERVICE_ADMIN_PASSWORD . Select Users -> Add person to add users eric and bob (dummy details can be used). Note the Inum (unique user ID) for each user for use with the eoepca-protection.sh script. Apply Protection \u2693\ufe0e Apply the protection\u2026 Ensure that the script is executed with the environment variables and command-line options that are consistent with those of the main deployment . ./deploy/eoepca/eoepca-protection.sh apply \"<eric-id>\" \"<bob-id>\" \"<public-ip>\" \"<domain>\" Post-deploy Manual Steps \u2693\ufe0e The scripted deployment has been designed, as far as possible, to automate the configuration of the deployed components. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026 Fix the setting UMA Resource Lifetime - change 2592000 -> 2147483647 secs Fix the setting Authentication method for the Token Endpoint for the ApplicationHub - client_secret_post -> client_secret_basic Add groups group-1 , group-2 , group-3 to ApplicationHub, and add users eric , bob to these groups Create User Workspaces \u2693\ufe0e The protection steps created the test users eric and bob . For completeness we use the Workspace API to create their user workspaces, which hold their personal resources (data, processing results, etc.) within the platform - see Workspace . Using Workspace Swagger UI \u2693\ufe0e The Workspace API provides a Swagger UI that facilitates interaction with the API - at the URL https://workspace-api.<domain>/docs# . Access to The Workspace API is protected, such that the necessary access tokens must be supplied in requests, which is most easily achieved by logging in via the \u2018portal\u2019. The portal is accessed at https://portal.<domain>/ . It is a rudimentary web service that facilitates establishing the appropriate tokens in the user\u2019s browser context. Login to the portal as the admin user, using the configured credentials. Access the Workspace Swagger UI at https://workspace-api.<domain>/docs . Workspaces are created using POST /workspaces (Create Workspace) . Expand the node and select Try it out . Complete the request body, such as\u2026 { \"preferred_name\" : \"eric\" , \"default_owner\" : \"d95b0c2b-ea74-4b3f-9c6a-85198dec974d\" } \u2026where the default_owner is the user ID ( Inum ) for the user - thus protecting the created workspace for the identified user. Using curl \u2693\ufe0e The same can be achieved with a straight http request, for example using curl \u2026 curl -X 'POST' \\ 'https://workspace-api.192-168-49-2.nip.io/workspaces' \\ -H 'accept: application/json' \\ -H 'Content-Type: application/json' \\ -H 'X-User-Id: <admin-id-token>' \\ -d '{ \"preferred_name\": \"<workspace-name>\", \"default_owner\": \"<user-inum>\" }' Values must be provided for: admin-id-token - User ID token for the admin user workspace-name - name of the workspace, typically the username user-inum - the ID of the user for which the created workspace will be protected The ID token for the admin user can be obtained with a call to the token endpoint of the Login Service - supplying the credentials for the admin user and the pre-registered client\u2026 curl -L -X POST 'https://auth.<domain>/oxauth/restv1/token' \\ -H 'Cache-Control: no-cache' \\ -H 'Content-Type: application/x-www-form-urlencoded' \\ --data-urlencode 'scope=openid user_name is_operator' \\ --data-urlencode 'grant_type=password' \\ --data-urlencode 'username=admin' \\ --data-urlencode 'password=<admin-password>' \\ --data-urlencode 'client_id=<client-id>' \\ --data-urlencode 'client_secret=<client-secret>' A json response is returned, in which the field id_token provides the user ID token for the admin user. Using create-workspace helper script \u2693\ufe0e As an aide there is a helper script create-workspace . The script is available in the deployment-guide repository , and can be obtained as follows\u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide The create-workspace helper script requires some command-line arguments\u2026 Usage: create-workspace <domain> <user> <user-inum> [<client-id> <client-secret>] For example\u2026 ./deploy/bin/create-workspace 192 -168-49-2.nip.io eric d95b0c2b-ea74-4b3f-9c6a-85198dec974d The script prompts for the password of the admin user. By default <client-id> and <client-secret> are read from the client.yaml file that is created by the deployment script, which auto-registers a Login Service client. Thus, these args can be ommited to use the default client credentials. Clean-up \u2693\ufe0e Before initiating a fresh deployment, if a prior deployment has been attempted, then it is necessary to remove any persistent artefacts of the prior deployment. This includes\u2026 Minikube cluster Delete the minikube cluster\u2026 minikube delete If necessary specify the cluster (profile)\u2026 minikube -p <profile> delete Persistent Data In the case that the minikube none driver is used, the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. In this case, the minikube standard StorageClass is fulfilled by the hostpath provisioner, whose persistence is removed as follows\u2026 sudo rm -rf /tmp/hostpath-provisioner Client Credentials During the deployment a client of the Authorisation Server is registered, and its credentials stored for reuse in the file client.yaml . Once the cluster has been destroyed, then these client credentials become stale and so should be removed to avoid polluting subsequent deployments\u2026 rm -rf ./deploy/eoepca/client.yaml There is a helper script clean that can be used for steps 2 and 3 above, (the script does not delete the cluster). ./deploy/cluster/clean","title":"Scripted Deployment"},{"location":"quickstart/scripted-deployment/#scripted-deployment","text":"The Scripted Deployment provides a demonstration of an example deployment, and can found in the subdirectory deployment-guide/deploy of the source repository for this guide\u2026 git clone https://github.com/EOEPCA/deployment-guide \\ && cd deployment-guide \\ && ls deploy The script deploy/eoepca/eoepca.sh acts as an entry-point to the full system deployment. In order to tailor the deployment for your target environment, the script is configured through environment variables and command-line arguments. By default the script assumes deployment to a local minikube. Note The scripted deployment assumes that installation of the Prerequisite Tooling has been performed. The following subsections lead through the steps for a full local deployment. Whilst minikube is assumed, minimal adaptions are required to make the deployment to your existing Kubernetes cluster. The deployment follows these broad steps: Configuration Tailoring of deployment options. Deployment Creation of cluster and deployment of eoepca services. Protection Application of protection for authorized access to services. The Protection step is split from Deployment as there are some manual steps to be performed before the Protection can be applied.","title":"Scripted Deployment"},{"location":"quickstart/scripted-deployment/#configuration","text":"The script deploy/eoepca/eoepca.sh is configured by some environment variables and command-line arguments.","title":"Configuration"},{"location":"quickstart/scripted-deployment/#environment-variables","text":"Environment Variables Variable Description Default REQUIRE_<cluster-component> A set of variables that can be used to control which CLUSTER components are deployed by the script, as follows (with defaults): REQUIRE_MINIKUBE=true REQUIRE_INGRESS_NGINX=true REQUIRE_CERT_MANAGER=true REQUIRE_LETSENCRYPT=true REQUIRE_SEALED_SECRETS=false REQUIRE_MINIO=false see description REQUIRE_<eoepca-component> A set of variables that can be used to control which EOEPCA components are deployed by the script, as follows (with defaults): REQUIRE_STORAGE=true REQUIRE_DUMMY_SERVICE=false REQUIRE_LOGIN_SERVICE=true REQUIRE_PDP=true REQUIRE_USER_PROFILE=true REQUIRE_ADES=true REQUIRE_RESOURCE_CATALOGUE=true REQUIRE_DATA_ACCESS=true REQUIRE_WORKSPACE_API=true REQUIRE_BUCKET_OPERATOR=true REQUIRE_HARBOR=true REQUIRE_PORTAL=true REQUIRE_PDE=true see description REQUIRE_<protection-component> A set of variables that can be used to control which PROTECTION components are deployed by the script, as follows (with defaults): REQUIRE_DUMMY_SERVICE_PROTECTION=false REQUIRE_ADES_PROTECTION=true REQUIRE_RESOURCE_CATALOGUE_PROTECTION=true REQUIRE_DATA_ACCESS_PROTECTION=true REQUIRE_WORKSPACE_API_PROTECTION=true see description MINIKUBE_KUBERNETES_VERSION The Kubernetes version to be used by minikube Note that the EOEPCA development has been conducted primarily using version 1.22.5. v1.22.5 MINIKUBE_MEMORY_AMOUNT Amount of memory to allocate to the docker containers used by minikube to implement the cluster. 12g USE_METALLB Enable use of minikube\u2019s built-in load-balancer. The load-balancer can be used to facilitate exposing services publicly. However, the same can be achieved using minikube\u2019s built-in ingress-controller. Therefore, this option is suppressed by default. false USE_INGRESS_NGINX_HELM Install the ingress-nginx controller using the published helm chart, rather than relying upon the version that is built-in to minikube. By default we prefer the version that is built in to minikube. false USE_INGRESS_NGINX_LOADBALANCER Patch the built-in minikube nginx-ingress-controller to offer a service of type LoadBalancer , rather than the default NodePort . It was initially thought that this would be necessary to achieve public access to the ingress services - but was subsequently found that the default NodePort configuration of the ingress-controller was sufficient. This option is left in case it proves useful. Only applicable for USE_INGRESS_NGINX_HELM=false (i.e. when using the minikube built-in ) false OPEN_INGRESS Create \u2018open\u2019 ingress endpoints that are not subject to authorization protection. For a secure system the open endpoints should be disabled ( false ) and access to resource should be protected via ingress that apply protection false USE_TLS Indicates whether TLS will be configured for service Ingress rules. If not (i.e. USE_TLS=false ), then the ingress-controller is configured to disable ssl-redirect , and TLS_CLUSTER_ISSUER=notls is set. true TLS_CLUSTER_ISSUER The name of the ClusterIssuer to satisfy ingress tls certificates. Out-of-the-box ClusterIssuer instances are configured in the file deploy/cluster/letsencrypt.sh . letsencrypt-staging LOGIN_SERVICE_ADMIN_PASSWORD Initial password for the admin user in the login-service. changeme MINIO_ROOT_USER Name of the \u2018root\u2019 user for the Minio object storage service. eoepca MINIO_ROOT_PASSWORD Password for the \u2018root\u2019 user for the Minio object storage service. changeme HARBOR_ADMIN_PASSWORD Password for the \u2018admin\u2019 user for the Harbor artefact registry service. changeme PROCESSING_MAX_RAM Max RAM allocated to an individual processing job 8Gi PROCESSING_MAX_CORES Max number of CPU cores allocated to an individual processing job 4 STAGEOUT_TARGET Configures the ADES with the destination to which it should push processing results: workspace - via the Workspace API minio - to minio S3 object storage workspace INSTALL_FLUX The Workspace API relies upon Flux CI/CD , and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value false to suppress the Workspace API flux install true CREODIAS_DATA_SPECIFICATION Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services. Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the eodata network. false","title":"Environment Variables"},{"location":"quickstart/scripted-deployment/#openstack-configuration","text":"There are some additional environment variables that configure the BucketOperator with details of the infrastructure Openstack layer. Attention This is only applicable for an Openstack deployment and has only been tested on the CREODIAS. Environment Variables Variable Description Default OS_DOMAINNAME Openstack domain of the admin account in the cloud provider. cloud_XXXXX OS_USERNAME Openstack username of the admin account in the cloud provider. user@cloud.com OS_PASSWORD Openstack password of the admin account in the cloud provider. none OS_MEMBERROLEID ID of a specific role (e.g. the \u2018 member \u2018 role) for operations users (to allow administration), e.g. 7fe2ff9ee5384b1894a90838d3e92bab . none OS_SERVICEPROJECTID ID of a project containing the user identity requiring write access to the created user buckets, e.g. 573916ef342a4bf1aea807d0c6058c1e . none USER_EMAIL_PATTERN Email associated to the created user within the created user project. Note: <name> is templated and will be replaced. eoepca-<name>@platform.com","title":"Openstack Configuration"},{"location":"quickstart/scripted-deployment/#command-line-arguments","text":"The eoepca.sh script is further configured via command-line arguments\u2026 eoepca.sh <action> <cluster-name> <public-ip> <domain> Arguments Argument Description Default action Action to perform: apply | delete | template . apply makes the deployment delete removes the deployment template outputs generated kubernetes yaml to stdout apply cluster-name The name of the minikube \u2018profile\u2019 for the created minikube cluster eoepca public-ip The public IP address through which the deployment is exposed via the ingress-controller. By default, the value is deduced from the assigned cluster minikube IP address - ref. command minikube ip . <minikube-ip> domain The DNS domain name through which the deployment is accessed. Forms the stem for all service hostnames in the ingress rules - i.e. <service-name>.<domain> . By default, the value is deduced from the assigned cluster minikube IP address, using nip.io to establish a DNS lookup - i.e. <minikube ip>.nip.io . <minikube ip>.nip.io","title":"Command-line Arguments"},{"location":"quickstart/scripted-deployment/#public-deployment","text":"For simplicity, the out-of-the-box scripts assume a \u2018private\u2019 deployment - with no public IP / DNS and hence no use of TLS for service ingress endpoints. In the case that an external-facing public deployment is desired, then the following configuration selections should be made: public_ip - set to the external-facing public IP of your deployment, e.g. the floating IP of your load-balancer in a cloud deployment domain - set to the domain (as per DNS records) for your deployment Note that the EOEPCA components typically configure their ingress with hostname prefixes applied to this domain . Thus, it is necessary that the DNS record for the domain is established as a wildcard record - i.e. *.<domain> USE_TLS=true - to enable configuration of TLS endpoints in each component service ingress TLS_CLUSTER_ISSUER=<issuer> - should be configured ~ e.g. using the letsencrypt-production or letsencrypt-staging (testing only) Cluster Issuer that are configured by the scripted deployment","title":"Public Deployment"},{"location":"quickstart/scripted-deployment/#deployment","text":"The deployment is initiated by setting the appropriate environment variables and invoking the eoepca.sh script with suitable command-line arguments . You may find it convenient to do so using a wrapper script that customises the environment varaibles according to your cluster, and then invokes the eoepca.sh script. Customised examples are provided for Simple, CREODIAS and Processing deployments. NOTE that if a prior deployment has been attempted then, before redeploying, a clean-up should be performed as described in the Clean-up section below. This is particularly important in the case that the minikube none driver is used, as the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. Initiate the deployment\u2026 ./deploy/eoepca/eoepca.sh apply \"<cluster-name>\" \"<public-ip>\" \"<domain>\" The deployment takes 10+ minutes - depending on the resources of your host/cluster. The progress can be monitored\u2026 kubectl get pods -A The deployment is ready once all pods are either Running or Completed . This can be further confirmed by accessing the login-service web interface at https://auth.<domain>/ and logging in as user admin using the credentials configured via LOGIN_SERVICE_ADMIN_PASSWORD .","title":"Deployment"},{"location":"quickstart/scripted-deployment/#default-credentials","text":"","title":"Default Credentials"},{"location":"quickstart/scripted-deployment/#login-service","text":"By default, the Login Service is accessed at the URL https://auth.<domain>/ with the credentials\u2026 username: admin password: Chang3me! \u2026unless the password is overridden via the variable LOGIN_SERVICE_ADMIN_PASSWORD .","title":"Login Service"},{"location":"quickstart/scripted-deployment/#minio-object-storage","text":"By default, Minio is accessed at the URL http://console.minio.<domain>/ with the credentials\u2026 username: eoepca password: changeme \u2026unless the username/password are overridden via the variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD .","title":"Minio Object Storage"},{"location":"quickstart/scripted-deployment/#harbor-container-registry","text":"By default, Harbor is accessed at the URL https://harbor.<domain>/ with the credentials\u2026 username: admin password: changeme \u2026unless the password is overridden via the variable HARBOR_ADMIN_PASSWORD .","title":"Harbor Container Registry"},{"location":"quickstart/scripted-deployment/#protection","text":"The protection of resource server endpoints is applied with the script deploy/eoepca/eoepca-protection.sh . This script should be executed with environment variables and command-line options that are consistent with those of the main deployment (ref. script eoepca.sh ). The script eoepca-protection.sh introduces two users eric and bob to demonstrate the application of authorized access to various service endpoints: ADES, Workspace API and dummy-service (simple endpoint used for debugging). Thus, the users must first be created in the login-service and their unique IDs passed to the protection script. Usage: eoepca-protection.sh <action> <eric-id> <bob-id> <public-ip> <domain>","title":"Protection"},{"location":"quickstart/scripted-deployment/#create-test-users","text":"Access the login-service web interface ( https://auth.<domain>/ ) as user admin using the credentials configured via LOGIN_SERVICE_ADMIN_PASSWORD . Select Users -> Add person to add users eric and bob (dummy details can be used). Note the Inum (unique user ID) for each user for use with the eoepca-protection.sh script.","title":"Create Test Users"},{"location":"quickstart/scripted-deployment/#apply-protection","text":"Apply the protection\u2026 Ensure that the script is executed with the environment variables and command-line options that are consistent with those of the main deployment . ./deploy/eoepca/eoepca-protection.sh apply \"<eric-id>\" \"<bob-id>\" \"<public-ip>\" \"<domain>\"","title":"Apply Protection"},{"location":"quickstart/scripted-deployment/#post-deploy-manual-steps","text":"The scripted deployment has been designed, as far as possible, to automate the configuration of the deployed components. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026 Fix the setting UMA Resource Lifetime - change 2592000 -> 2147483647 secs Fix the setting Authentication method for the Token Endpoint for the ApplicationHub - client_secret_post -> client_secret_basic Add groups group-1 , group-2 , group-3 to ApplicationHub, and add users eric , bob to these groups","title":"Post-deploy Manual Steps"},{"location":"quickstart/scripted-deployment/#create-user-workspaces","text":"The protection steps created the test users eric and bob . For completeness we use the Workspace API to create their user workspaces, which hold their personal resources (data, processing results, etc.) within the platform - see Workspace .","title":"Create User Workspaces"},{"location":"quickstart/scripted-deployment/#using-workspace-swagger-ui","text":"The Workspace API provides a Swagger UI that facilitates interaction with the API - at the URL https://workspace-api.<domain>/docs# . Access to The Workspace API is protected, such that the necessary access tokens must be supplied in requests, which is most easily achieved by logging in via the \u2018portal\u2019. The portal is accessed at https://portal.<domain>/ . It is a rudimentary web service that facilitates establishing the appropriate tokens in the user\u2019s browser context. Login to the portal as the admin user, using the configured credentials. Access the Workspace Swagger UI at https://workspace-api.<domain>/docs . Workspaces are created using POST /workspaces (Create Workspace) . Expand the node and select Try it out . Complete the request body, such as\u2026 { \"preferred_name\" : \"eric\" , \"default_owner\" : \"d95b0c2b-ea74-4b3f-9c6a-85198dec974d\" } \u2026where the default_owner is the user ID ( Inum ) for the user - thus protecting the created workspace for the identified user.","title":"Using Workspace Swagger UI"},{"location":"quickstart/scripted-deployment/#using-curl","text":"The same can be achieved with a straight http request, for example using curl \u2026 curl -X 'POST' \\ 'https://workspace-api.192-168-49-2.nip.io/workspaces' \\ -H 'accept: application/json' \\ -H 'Content-Type: application/json' \\ -H 'X-User-Id: <admin-id-token>' \\ -d '{ \"preferred_name\": \"<workspace-name>\", \"default_owner\": \"<user-inum>\" }' Values must be provided for: admin-id-token - User ID token for the admin user workspace-name - name of the workspace, typically the username user-inum - the ID of the user for which the created workspace will be protected The ID token for the admin user can be obtained with a call to the token endpoint of the Login Service - supplying the credentials for the admin user and the pre-registered client\u2026 curl -L -X POST 'https://auth.<domain>/oxauth/restv1/token' \\ -H 'Cache-Control: no-cache' \\ -H 'Content-Type: application/x-www-form-urlencoded' \\ --data-urlencode 'scope=openid user_name is_operator' \\ --data-urlencode 'grant_type=password' \\ --data-urlencode 'username=admin' \\ --data-urlencode 'password=<admin-password>' \\ --data-urlencode 'client_id=<client-id>' \\ --data-urlencode 'client_secret=<client-secret>' A json response is returned, in which the field id_token provides the user ID token for the admin user.","title":"Using curl"},{"location":"quickstart/scripted-deployment/#using-create-workspace-helper-script","text":"As an aide there is a helper script create-workspace . The script is available in the deployment-guide repository , and can be obtained as follows\u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide The create-workspace helper script requires some command-line arguments\u2026 Usage: create-workspace <domain> <user> <user-inum> [<client-id> <client-secret>] For example\u2026 ./deploy/bin/create-workspace 192 -168-49-2.nip.io eric d95b0c2b-ea74-4b3f-9c6a-85198dec974d The script prompts for the password of the admin user. By default <client-id> and <client-secret> are read from the client.yaml file that is created by the deployment script, which auto-registers a Login Service client. Thus, these args can be ommited to use the default client credentials.","title":"Using create-workspace helper script"},{"location":"quickstart/scripted-deployment/#clean-up","text":"Before initiating a fresh deployment, if a prior deployment has been attempted, then it is necessary to remove any persistent artefacts of the prior deployment. This includes\u2026 Minikube cluster Delete the minikube cluster\u2026 minikube delete If necessary specify the cluster (profile)\u2026 minikube -p <profile> delete Persistent Data In the case that the minikube none driver is used, the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. In this case, the minikube standard StorageClass is fulfilled by the hostpath provisioner, whose persistence is removed as follows\u2026 sudo rm -rf /tmp/hostpath-provisioner Client Credentials During the deployment a client of the Authorisation Server is registered, and its credentials stored for reuse in the file client.yaml . Once the cluster has been destroyed, then these client credentials become stale and so should be removed to avoid polluting subsequent deployments\u2026 rm -rf ./deploy/eoepca/client.yaml There is a helper script clean that can be used for steps 2 and 3 above, (the script does not delete the cluster). ./deploy/cluster/clean","title":"Clean-up"},{"location":"quickstart/simple-deployment/","text":"Simple Deployment \u2693\ufe0e A deployment wrapper script has been prepared for a \u2018simple\u2019 deployment - designed to get a core local deployment of the primary servies. The script deploy/simple/simple achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/simple/simple-options . The simple deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Configuration of \u2018open\u2019 interfaces - i.e. service/API endpoints that are not protected and can accessed without authentication. This facilitates experimentation with the services Configuration of ADES stage-out to a local instance of minio , on the assumption that access to CREODIAS buckets for stage-out (via Workspace) is not an option Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/simple/simple See section Deployment for more details regarding the outcome of the scripted deployment. Protection \u2693\ufe0e See section Protection for more details regarding the protection of the deployed services - which, for the simple deployment, is performed via the script deploy/simple/simple-protection \u2026 ./deploy/simple/simple-protection","title":"Simple Deployment"},{"location":"quickstart/simple-deployment/#simple-deployment","text":"A deployment wrapper script has been prepared for a \u2018simple\u2019 deployment - designed to get a core local deployment of the primary servies. The script deploy/simple/simple achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/simple/simple-options . The simple deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Configuration of \u2018open\u2019 interfaces - i.e. service/API endpoints that are not protected and can accessed without authentication. This facilitates experimentation with the services Configuration of ADES stage-out to a local instance of minio , on the assumption that access to CREODIAS buckets for stage-out (via Workspace) is not an option","title":"Simple Deployment"},{"location":"quickstart/simple-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/simple/simple See section Deployment for more details regarding the outcome of the scripted deployment.","title":"Initiate Deployment"},{"location":"quickstart/simple-deployment/#protection","text":"See section Protection for more details regarding the protection of the deployed services - which, for the simple deployment, is performed via the script deploy/simple/simple-protection \u2026 ./deploy/simple/simple-protection","title":"Protection"},{"location":"quickstart/userman-deployment/","text":"User Management Deployment \u2693\ufe0e A deployment wrapper script has been prepared for a \u2018user management\u2019 deployment - that is focused on the Login Service , PDP and User Profile . The script deploy/userman/userman achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/userman/userman-options . The user-management deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: Login Service Policy Decision Point (PDP) User Profile Other eoepca services not deployed Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/userman/userman The Login Service is accessed at the endpoint auth.<domain> - e.g. auth.192-168-49-2.nip.io .","title":"User Management"},{"location":"quickstart/userman-deployment/#user-management-deployment","text":"A deployment wrapper script has been prepared for a \u2018user management\u2019 deployment - that is focused on the Login Service , PDP and User Profile . The script deploy/userman/userman achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/userman/userman-options . The user-management deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: Login Service Policy Decision Point (PDP) User Profile Other eoepca services not deployed","title":"User Management Deployment"},{"location":"quickstart/userman-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/userman/userman The Login Service is accessed at the endpoint auth.<domain> - e.g. auth.192-168-49-2.nip.io .","title":"Initiate Deployment"}]}
\ No newline at end of file
+{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Deployment Guide \u2693\ufe0e The Deployment Guide describes how each building-block comprising the EOEPCA Reference Implementation is configured and deployed. A full system deployment is described, in which components are deployed with complementary configurations that facilitate their integration as a coherent system. Nevertheless, each component can be cherry-picked from this system deployment for individual re-use. The deployment is organised into the following sections: Cluster Establish the Kubernetes cluster and other prerequisites for the deployment of the EOEPCA system. EOEPCA Deployment of the EOEPCA components. Examples Deployment examples.","title":"Introduction"},{"location":"#deployment-guide","text":"The Deployment Guide describes how each building-block comprising the EOEPCA Reference Implementation is configured and deployed. A full system deployment is described, in which components are deployed with complementary configurations that facilitate their integration as a coherent system. Nevertheless, each component can be cherry-picked from this system deployment for individual re-use. The deployment is organised into the following sections: Cluster Establish the Kubernetes cluster and other prerequisites for the deployment of the EOEPCA system. EOEPCA Deployment of the EOEPCA components. Examples Deployment examples.","title":"Deployment Guide"},{"location":"cluster/cluster-prerequisites/","text":"Cluster Prerequisites \u2693\ufe0e The following prerequisite components are assumed to be deployed in the cluster. Nginx Ingress Controller \u2693\ufe0e # Add the helm repository helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update # Install the Nginx Ingress Controller helm chart helm upgrade -i --version = '<4.5.0' ingress-nginx ingress-nginx/ingress-nginx --wait Note For Kubernetes version 1.22 and earlier the version of the Nginx Ingress Controller must be before v4.5.0. To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx annotation must be applied to the Ingress resource\u2026 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : kubernetes.io/ingress.class : nginx ... Cert Manager \u2693\ufe0e # Add the helm repository helm repo add jetstack https://charts.jetstack.io helm repo update # Install the Cert Manager helm chart helm upgrade -i cert-manager jetstack/cert-manager \\ --namespace cert-manager \\ --create-namespace \\ --set installCRDs = true Letsencrypt Certificates \u2693\ufe0e Once the Certificate Manager is deployed, then we can establish ClusterIssuer operators in the cluster to support use of TLS with service Ingress endpoints. For Letsencrypt we can define two ClusterIssuer - for production and for staging . NOTE that these require the cluster to be publicly accessible, in order for the http01 acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints. Production \u2693\ufe0e apiVersion : cert-manager.io/v1 kind : ClusterIssuer metadata : name : letsencrypt-production spec : acme : # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email : eoepca.systemteam@telespazio.com server : https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef : # Secret resource that will be used to store the account's private key. name : letsencrypt-production-account-key # Add a single challenge solver, HTTP01 using nginx solvers : - http01 : ingress : class : nginx Staging \u2693\ufe0e apiVersion : cert-manager.io/v1 kind : ClusterIssuer metadata : name : letsencrypt-staging spec : acme : # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email : eoepca.systemteam@telespazio.com server : https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef : # Secret resource that will be used to store the account's private key. name : letsencrypt-staging-account-key # Add a single challenge solver, HTTP01 using nginx solvers : - http01 : ingress : class : nginx To exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer annotation must be applied to the Ingress resource. For example\u2026 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : kubernetes.io/ingress.class : nginx cert-manager.io/cluster-issuer : letsencrypt-production ... Sealed Secrets \u2693\ufe0e The EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments. Various Secret are relied upon by the system services. Secrets should not be exposed by commit to GitHub. Instead SealedSecret are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller that runs within the cluster. The sealed-secret-controller decrypts the SealedSecret to a regular Secret (of the same name) that can then be consumed by the cluster components. The sealed-secret-controller is deployed to the cluster using the helm chart\u2026 helm repo add bitnami-sealed-secrets https://bitnami-labs.github.io/sealed-secrets helm repo update helm install --version 2 .1.8 --create-namespace --namespace infra \\ eoepca-sealed-secrets bitnami-sealed-secrets/sealed-secrets Once the controller is deployed within the cluster, then the kubeseal command can be used to create a SealedSecret from a regular Secret , as follows\u2026 Create example Secret\u2026 kubectl -n test create secret generic mysecret \\ --from-literal = password = changeme \\ --dry-run = client -o yaml \\ > mysecret.yaml Create SealedSecret from Secret using kubeseal\u2026 kubeseal -o yaml \\ --controller-name eoepca-sealed-secrets \\ --controller-namespace infra \\ < mysecret.yaml \\ > mysecret-sealed.yaml References \u2693\ufe0e Sealed Secrets on GitHub kubeseal Release MinIO Object Storage \u2693\ufe0e Various building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service. As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by the MinIO Project . # Install the minio helm chart helm upgrade -i minio -f minio-values.yaml bitnami/minio \\ --repo https://charts.min.io/ \\ --namespace rm --create-namespace \\ --wait Note The Kubernetes namespace rm is used above as an example, and can be changed according to your deployment preference. The minio deployment is customised via the values file minio-values.yaml , for example\u2026 existingSecret : minio-auth replicas : 2 ingress : enabled : true ingressClassName : nginx annotations : cert-manager.io/cluster-issuer : \"letsencrypt\" nginx.ingress.kubernetes.io/ssl-redirect : \"true\" nginx.ingress.kubernetes.io/proxy-body-size : \"0\" nginx.ingress.kubernetes.io/proxy-read-timeout : '600' path : / hosts : - minio.192-168-49-2.nip.io tls : - secretName : minio-tls hosts : - minio.192-168-49-2.nip.io consoleIngress : enabled : true ingressClassName : nginx annotations : cert-manager.io/cluster-issuer : \"letsencrypt\" nginx.ingress.kubernetes.io/ssl-redirect : \"true\" nginx.ingress.kubernetes.io/proxy-body-size : \"0\" nginx.ingress.kubernetes.io/proxy-read-timeout : '600' path : / hosts : - console.minio.192-168-49-2.nip.io tls : - secretName : minio-console-tls hosts : - console.minio.192-168-49-2.nip.io resources : requests : memory : 1Gi persistence : storageClass : standard buckets : - name : eoepca - name : cache-bucket Note The example values assuming a TLS configuration using letsencrypt certificate provider The admin credentials are provided by the Kubernetes secret named minio-auth - see below The annotation nginx.ingress.kubernetes.io/proxy-body-size was found to be required to allow transfer of large files (such as data products) through the nginx proxy Minio Credentials Secret \u2693\ufe0e The Minio admin credentials are provided via a Kubernetes secret that is referenced from the Minio helm chart deployment values. For example\u2026 kubectl -n rm create secret generic minio-auth \\ --from-literal=rootUser=\"eoepca\" \\ --from-literal=rootPassword=\"changeme\" Note The secret must be created in the same Kubernetes namespace as the Minio service deployment - e.g. rm namespce in the example above. s3cmd Configuration \u2693\ufe0e The s3cmd can be configured for access to the MinIO deployment. The --configure option can be used to prepare a suitable configuration file for s3cmd \u2026 s3cmd -c mys3cfg --configure In response to the prompts, the following configuration selections are applicable to the above settings\u2026 Access Key: eoepca Secret Key: changeme Default Region: us-east-1 S3 Endpoint: minio.192-168-49-2.nip.io DNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Save the configuration file, and check access to the S3 object store with\u2026 # Create a bucket s3cmd -c mys3cfg mb s3://eoepca # List buckets s3cmd -c mys3cfg ls For example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube\u2026 s3cmd -c deploy/cluster/s3cfg ls References \u2693\ufe0e MinIO Website MinIO Helm Chart MinIO on GitHub","title":"Cluster Prerequisites"},{"location":"cluster/cluster-prerequisites/#cluster-prerequisites","text":"The following prerequisite components are assumed to be deployed in the cluster.","title":"Cluster Prerequisites"},{"location":"cluster/cluster-prerequisites/#nginx-ingress-controller","text":"# Add the helm repository helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update # Install the Nginx Ingress Controller helm chart helm upgrade -i --version = '<4.5.0' ingress-nginx ingress-nginx/ingress-nginx --wait Note For Kubernetes version 1.22 and earlier the version of the Nginx Ingress Controller must be before v4.5.0. To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx annotation must be applied to the Ingress resource\u2026 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : kubernetes.io/ingress.class : nginx ...","title":"Nginx Ingress Controller"},{"location":"cluster/cluster-prerequisites/#cert-manager","text":"# Add the helm repository helm repo add jetstack https://charts.jetstack.io helm repo update # Install the Cert Manager helm chart helm upgrade -i cert-manager jetstack/cert-manager \\ --namespace cert-manager \\ --create-namespace \\ --set installCRDs = true","title":"Cert Manager"},{"location":"cluster/cluster-prerequisites/#letsencrypt-certificates","text":"Once the Certificate Manager is deployed, then we can establish ClusterIssuer operators in the cluster to support use of TLS with service Ingress endpoints. For Letsencrypt we can define two ClusterIssuer - for production and for staging . NOTE that these require the cluster to be publicly accessible, in order for the http01 acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints.","title":"Letsencrypt Certificates"},{"location":"cluster/cluster-prerequisites/#production","text":"apiVersion : cert-manager.io/v1 kind : ClusterIssuer metadata : name : letsencrypt-production spec : acme : # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email : eoepca.systemteam@telespazio.com server : https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef : # Secret resource that will be used to store the account's private key. name : letsencrypt-production-account-key # Add a single challenge solver, HTTP01 using nginx solvers : - http01 : ingress : class : nginx","title":"Production"},{"location":"cluster/cluster-prerequisites/#staging","text":"apiVersion : cert-manager.io/v1 kind : ClusterIssuer metadata : name : letsencrypt-staging spec : acme : # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email : eoepca.systemteam@telespazio.com server : https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef : # Secret resource that will be used to store the account's private key. name : letsencrypt-staging-account-key # Add a single challenge solver, HTTP01 using nginx solvers : - http01 : ingress : class : nginx To exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer annotation must be applied to the Ingress resource. For example\u2026 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : kubernetes.io/ingress.class : nginx cert-manager.io/cluster-issuer : letsencrypt-production ...","title":"Staging"},{"location":"cluster/cluster-prerequisites/#sealed-secrets","text":"The EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments. Various Secret are relied upon by the system services. Secrets should not be exposed by commit to GitHub. Instead SealedSecret are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller that runs within the cluster. The sealed-secret-controller decrypts the SealedSecret to a regular Secret (of the same name) that can then be consumed by the cluster components. The sealed-secret-controller is deployed to the cluster using the helm chart\u2026 helm repo add bitnami-sealed-secrets https://bitnami-labs.github.io/sealed-secrets helm repo update helm install --version 2 .1.8 --create-namespace --namespace infra \\ eoepca-sealed-secrets bitnami-sealed-secrets/sealed-secrets Once the controller is deployed within the cluster, then the kubeseal command can be used to create a SealedSecret from a regular Secret , as follows\u2026 Create example Secret\u2026 kubectl -n test create secret generic mysecret \\ --from-literal = password = changeme \\ --dry-run = client -o yaml \\ > mysecret.yaml Create SealedSecret from Secret using kubeseal\u2026 kubeseal -o yaml \\ --controller-name eoepca-sealed-secrets \\ --controller-namespace infra \\ < mysecret.yaml \\ > mysecret-sealed.yaml","title":"Sealed Secrets"},{"location":"cluster/cluster-prerequisites/#references","text":"Sealed Secrets on GitHub kubeseal Release","title":"References"},{"location":"cluster/cluster-prerequisites/#minio-object-storage","text":"Various building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service. As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by the MinIO Project . # Install the minio helm chart helm upgrade -i minio -f minio-values.yaml bitnami/minio \\ --repo https://charts.min.io/ \\ --namespace rm --create-namespace \\ --wait Note The Kubernetes namespace rm is used above as an example, and can be changed according to your deployment preference. The minio deployment is customised via the values file minio-values.yaml , for example\u2026 existingSecret : minio-auth replicas : 2 ingress : enabled : true ingressClassName : nginx annotations : cert-manager.io/cluster-issuer : \"letsencrypt\" nginx.ingress.kubernetes.io/ssl-redirect : \"true\" nginx.ingress.kubernetes.io/proxy-body-size : \"0\" nginx.ingress.kubernetes.io/proxy-read-timeout : '600' path : / hosts : - minio.192-168-49-2.nip.io tls : - secretName : minio-tls hosts : - minio.192-168-49-2.nip.io consoleIngress : enabled : true ingressClassName : nginx annotations : cert-manager.io/cluster-issuer : \"letsencrypt\" nginx.ingress.kubernetes.io/ssl-redirect : \"true\" nginx.ingress.kubernetes.io/proxy-body-size : \"0\" nginx.ingress.kubernetes.io/proxy-read-timeout : '600' path : / hosts : - console.minio.192-168-49-2.nip.io tls : - secretName : minio-console-tls hosts : - console.minio.192-168-49-2.nip.io resources : requests : memory : 1Gi persistence : storageClass : standard buckets : - name : eoepca - name : cache-bucket Note The example values assuming a TLS configuration using letsencrypt certificate provider The admin credentials are provided by the Kubernetes secret named minio-auth - see below The annotation nginx.ingress.kubernetes.io/proxy-body-size was found to be required to allow transfer of large files (such as data products) through the nginx proxy","title":"MinIO Object Storage"},{"location":"cluster/cluster-prerequisites/#minio-credentials-secret","text":"The Minio admin credentials are provided via a Kubernetes secret that is referenced from the Minio helm chart deployment values. For example\u2026 kubectl -n rm create secret generic minio-auth \\ --from-literal=rootUser=\"eoepca\" \\ --from-literal=rootPassword=\"changeme\" Note The secret must be created in the same Kubernetes namespace as the Minio service deployment - e.g. rm namespce in the example above.","title":"Minio Credentials Secret"},{"location":"cluster/cluster-prerequisites/#s3cmd-configuration","text":"The s3cmd can be configured for access to the MinIO deployment. The --configure option can be used to prepare a suitable configuration file for s3cmd \u2026 s3cmd -c mys3cfg --configure In response to the prompts, the following configuration selections are applicable to the above settings\u2026 Access Key: eoepca Secret Key: changeme Default Region: us-east-1 S3 Endpoint: minio.192-168-49-2.nip.io DNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Save the configuration file, and check access to the S3 object store with\u2026 # Create a bucket s3cmd -c mys3cfg mb s3://eoepca # List buckets s3cmd -c mys3cfg ls For example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube\u2026 s3cmd -c deploy/cluster/s3cfg ls","title":"s3cmd Configuration"},{"location":"cluster/cluster-prerequisites/#references_1","text":"MinIO Website MinIO Helm Chart MinIO on GitHub","title":"References"},{"location":"cluster/helm-repositories/","text":"Helm Repositories \u2693\ufe0e EOEPCA Helm Charts \u2693\ufe0e The EOEPCA building-blocks are engineered as containers for deployment to a Kubernetes cluster. Each building block defines a Helm Chart to facilitate its deployment. The EOEPCA Helm Chart Repository is configured with helm as follows\u2026 helm repo add eoepca https://eoepca.github.io/helm-charts/ Third-party Helm Charts \u2693\ufe0e In addition to the EOEPCA Helm Chart Repository, the following repositories are also relied upon, and should be configured\u2026 Cert Manager \u2693\ufe0e helm repo add jetstack https://charts.jetstack.io Nginx Ingress Controller \u2693\ufe0e helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx Repo Update \u2693\ufe0e Refresh the local repo cache, after helm repo add \u2026 helm repo update","title":"Helm Repositories"},{"location":"cluster/helm-repositories/#helm-repositories","text":"","title":"Helm Repositories"},{"location":"cluster/helm-repositories/#eoepca-helm-charts","text":"The EOEPCA building-blocks are engineered as containers for deployment to a Kubernetes cluster. Each building block defines a Helm Chart to facilitate its deployment. The EOEPCA Helm Chart Repository is configured with helm as follows\u2026 helm repo add eoepca https://eoepca.github.io/helm-charts/","title":"EOEPCA Helm Charts"},{"location":"cluster/helm-repositories/#third-party-helm-charts","text":"In addition to the EOEPCA Helm Chart Repository, the following repositories are also relied upon, and should be configured\u2026","title":"Third-party Helm Charts"},{"location":"cluster/helm-repositories/#cert-manager","text":"helm repo add jetstack https://charts.jetstack.io","title":"Cert Manager"},{"location":"cluster/helm-repositories/#nginx-ingress-controller","text":"helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx","title":"Nginx Ingress Controller"},{"location":"cluster/helm-repositories/#repo-update","text":"Refresh the local repo cache, after helm repo add \u2026 helm repo update","title":"Repo Update"},{"location":"cluster/kubernetes/","text":"Kubernetes Cluster \u2693\ufe0e The EOEPCA Reference Implementation has been developed with Kubernetes as its deployment target. The system components have been developed, deployed and tested using a cluster at version v1.22.5 . Rancher Kubernetes Engine (RKE) \u2693\ufe0e The development, integration and test clusters have been established using Rancher Kubernetes Engine (RKE) at version v1.22.5 . An example of the creation of the EOEPCA Kubernetes clusters can be found on the GitHub Kubernetes Setup page . CREODIAS has been used for the development hosting infrastructure - which provides OpenStack infrastructure that is backed by Cloudferro . An example of the Terraform configurations used to automate the creation of the cloud infrastructure that underpins the RKE deployment can be found on the GitHub CREODIAS Setup page . Local Kubernetes \u2693\ufe0e To make a full deployment of the EOEPCA Reference Implementation requires a multi-node node cluster with suitable resources. For example, the development cluster comprises: 1 Master node (2 vCPU, 8 GB RAM) 5 Worker nodes (4 vCPU, 16 GB RAM) 1 NFS server (2 vCPU, 8 GB RAM) Limited local deployment can be made using a suitable local single-node kuberbetes deployment using - for example using minikube \u2026 minikube -p eoepca start --cpus max --memory max --kubernetes-version v1.22.5 minikube profile eoepca With such a deployment it is possible to deploy individual building-blocks for local development, or building-blocks in combination - within the constraints of the local host resources.","title":"Kubernetes Cluster"},{"location":"cluster/kubernetes/#kubernetes-cluster","text":"The EOEPCA Reference Implementation has been developed with Kubernetes as its deployment target. The system components have been developed, deployed and tested using a cluster at version v1.22.5 .","title":"Kubernetes Cluster"},{"location":"cluster/kubernetes/#rancher-kubernetes-engine-rke","text":"The development, integration and test clusters have been established using Rancher Kubernetes Engine (RKE) at version v1.22.5 . An example of the creation of the EOEPCA Kubernetes clusters can be found on the GitHub Kubernetes Setup page . CREODIAS has been used for the development hosting infrastructure - which provides OpenStack infrastructure that is backed by Cloudferro . An example of the Terraform configurations used to automate the creation of the cloud infrastructure that underpins the RKE deployment can be found on the GitHub CREODIAS Setup page .","title":"Rancher Kubernetes Engine (RKE)"},{"location":"cluster/kubernetes/#local-kubernetes","text":"To make a full deployment of the EOEPCA Reference Implementation requires a multi-node node cluster with suitable resources. For example, the development cluster comprises: 1 Master node (2 vCPU, 8 GB RAM) 5 Worker nodes (4 vCPU, 16 GB RAM) 1 NFS server (2 vCPU, 8 GB RAM) Limited local deployment can be made using a suitable local single-node kuberbetes deployment using - for example using minikube \u2026 minikube -p eoepca start --cpus max --memory max --kubernetes-version v1.22.5 minikube profile eoepca With such a deployment it is possible to deploy individual building-blocks for local development, or building-blocks in combination - within the constraints of the local host resources.","title":"Local Kubernetes"},{"location":"cluster/prerequisite-tooling/","text":"Prerequisite Tooling \u2693\ufe0e There are some standard tools referenced in this guide. These are detailed in the following subsections. docker \u2693\ufe0e Docker faciliates the creation, management and execution of containers. Whilst not strictly necessary to support deployment to an existing/managed Kubernetes cluster, it can nevertheless be useful to have local access to the docker tooling. For example, if minikube is used to follow this guide using a local k8s cluster, then this is best achieved using minikube\u2019s docker driver. Docker is most easily installed with\u2026 curl -fsSL https://get.docker.com | sh For convenience, add your user to the docker group\u2026 sudo usermod -aG docker ${ USER } Logout/in to refresh your session\u2019s group permissions. kubectl \u2693\ufe0e Kubectl is the main tool for interaction with a Kubernetes cluster. The latest version can be installed with\u2026 mkdir -p $HOME /.local/bin \\ && curl -fsSLo $HOME /.local/bin/kubectl \"https://dl.k8s.io/release/ $( curl -L -s https://dl.k8s.io/release/stable.txt ) /bin/linux/amd64/kubectl\" \\ && chmod +x $HOME /.local/bin/kubectl See the official kubectl installation documentation for more installation options. helm \u2693\ufe0e Helm is the Kubernetes package manager, in which components are deployed to a Kubernetes cluster via helm charts. The helm charts are instantiated for deployment via \u2018values\u2019 that configure the chart templates. The latest helm version can be installed with\u2026 export HELM_INSTALL_DIR = \" $HOME /.local/bin\" \\ && curl -sfL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash See the official helm installation documentation for more installation options. minikube \u2693\ufe0e Minikube is a tool that allows to create a local (single-node) Kubernetes cluster for development/testing. It is not designed for production use. In the absence of access to a \u2018full\u2019 Kubernetes cluster, this guide can be followed using minikube. The latest version of minikube can be installed with\u2026 mkdir -p $HOME /.local/bin \\ && curl -fsSLo $HOME /.local/bin/minikube \"https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\" \\ && chmod +x $HOME /.local/bin/minikube See the official minikube installation documentation for more installation options.","title":"Prerequisite Tooling"},{"location":"cluster/prerequisite-tooling/#prerequisite-tooling","text":"There are some standard tools referenced in this guide. These are detailed in the following subsections.","title":"Prerequisite Tooling"},{"location":"cluster/prerequisite-tooling/#docker","text":"Docker faciliates the creation, management and execution of containers. Whilst not strictly necessary to support deployment to an existing/managed Kubernetes cluster, it can nevertheless be useful to have local access to the docker tooling. For example, if minikube is used to follow this guide using a local k8s cluster, then this is best achieved using minikube\u2019s docker driver. Docker is most easily installed with\u2026 curl -fsSL https://get.docker.com | sh For convenience, add your user to the docker group\u2026 sudo usermod -aG docker ${ USER } Logout/in to refresh your session\u2019s group permissions.","title":"docker"},{"location":"cluster/prerequisite-tooling/#kubectl","text":"Kubectl is the main tool for interaction with a Kubernetes cluster. The latest version can be installed with\u2026 mkdir -p $HOME /.local/bin \\ && curl -fsSLo $HOME /.local/bin/kubectl \"https://dl.k8s.io/release/ $( curl -L -s https://dl.k8s.io/release/stable.txt ) /bin/linux/amd64/kubectl\" \\ && chmod +x $HOME /.local/bin/kubectl See the official kubectl installation documentation for more installation options.","title":"kubectl"},{"location":"cluster/prerequisite-tooling/#helm","text":"Helm is the Kubernetes package manager, in which components are deployed to a Kubernetes cluster via helm charts. The helm charts are instantiated for deployment via \u2018values\u2019 that configure the chart templates. The latest helm version can be installed with\u2026 export HELM_INSTALL_DIR = \" $HOME /.local/bin\" \\ && curl -sfL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash See the official helm installation documentation for more installation options.","title":"helm"},{"location":"cluster/prerequisite-tooling/#minikube","text":"Minikube is a tool that allows to create a local (single-node) Kubernetes cluster for development/testing. It is not designed for production use. In the absence of access to a \u2018full\u2019 Kubernetes cluster, this guide can be followed using minikube. The latest version of minikube can be installed with\u2026 mkdir -p $HOME /.local/bin \\ && curl -fsSLo $HOME /.local/bin/minikube \"https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\" \\ && chmod +x $HOME /.local/bin/minikube See the official minikube installation documentation for more installation options.","title":"minikube"},{"location":"eoepca/ades/","text":"Application Deployment & Execution Service (ADES) \u2693\ufe0e The ADES provides a platform-hosted execution engine through which users can initiate parameterised processing jobs using applications made available within the platform - supporting the efficient execution of the processing \u2018close to the data\u2019. Users can deploy specific \u2018applications\u2019 to the ADES, which may be their own applications, or those published by other platform users. Helm Chart \u2693\ufe0e The ADES is deployed via the ades helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the ades chart . helm install --version 2 .0.4 --values ades-values.yaml ades eoepca/ades Values \u2693\ufe0e At minimum, values for the following attributes should be specified: Details of the S3 Object Store for stage-out of processing results Dynamic provisioning StorageClass of ReadWriteMany storage (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the ADES will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Example ades-values.yaml \u2026 workflowExecutor : inputs : # Stage-in from CREODIAS eodata # (only works within CREODIAS - i.e. on a Cloudferro VM) STAGEIN_AWS_SERVICEURL : http://data.cloudferro.com STAGEIN_AWS_ACCESS_KEY_ID : test STAGEIN_AWS_SECRET_ACCESS_KEY : test STAGEIN_AWS_REGION : RegionOne # Stage-out to minio S3 # (use this if the ADES is not configured to stage-out to the Workspace) STAGEOUT_AWS_SERVICEURL : http://minio.192-168-49-2.nip.io STAGEOUT_AWS_ACCESS_KEY_ID : eoepca STAGEOUT_AWS_SECRET_ACCESS_KEY : changeme STAGEOUT_AWS_REGION : us-east-1 STAGEOUT_OUTPUT : s3://eoepca processingStorageClass : standard processingVolumeTmpSize : \"6Gi\" processingVolumeOutputSize : \"6Gi\" processingMaxRam : \"8Gi\" processingMaxCores : \"4\" wps : pepBaseUrl : \"http://ades-pep:5576\" usePep : \"true\" persistence : storageClass : standard ingress : enabled : true annotations : kubernetes.io/ingress.class : nginx ingress.kubernetes.io/ssl-redirect : \"false\" hosts : - host : ades.192-168-49-2.nip.io paths : - path : / pathType : ImplementationSpecific tls : - hosts : - ades.192-168-49-2.nip.io secretName : ades-tls Stage-in / Stage-out Configuration \u2693\ufe0e The ADES hosts applications that are deployed and invoked in accordance with the OGC Best Practise for Application Package . Thus, the ADES provides a conformant environment within which the application is integrated for execution. A key part of the ADES\u2019s role in this is to faciltate the provision of input data to the application (stage-in), and the handling of the results output at the conclusion of application execution (stage-out). The ADES helm chart configures (by default) the ADES with implementations of the stage-in and stage-out functions that use the Spatio Temporal Asset Router Services utility. The ADES provides hooks for system integrators to override these defaults to implement their own stage-in and stage-out behaviour - for example, to integrate with their platform\u2019s own catalogue and data offering. The stage-in and stage-out are specified as CWL via the helm values\u2026 workflowExecutor : stagein : cwl : | cwlVersion: v1.0 ... stageout : cwl : | cwlVersion: v1.0 ... For a detailed description see ADES stage-in/out configuration in the ADES wiki . Workspace Integration \u2693\ufe0e The ADES has the facility to integrate with the EOEPCA Workspace component for registration of staged-out processing results. This is disabled by default ( useResourceManager: false ). When enabled, the ADES will register the staged-out products with the user\u2019s Workspace, such that they are indexed and available via the user\u2019s Resource Catalogue and Data Access services. Example ades-values.yaml (snippet)\u2026 workflowExecutor : ... useResourceManager : \"true\" resourceManagerWorkspacePrefix : \"guide-user\" resourceManagerEndpoint : \"https://workspace-api.192-168-49-2.nip.io\" platformDomain : \"https://auth.192-168-49-2.nip.io\" ... The value resourceManagerWorkspacePrefix must be consistent with that configured for the Workspace API deployment , (ref. value prefixForName ). Protection \u2693\ufe0e As described in section Resource Protection , the resource-guard component can be inserted into the request path of the ADES service to provide access authorization decisions helm install --version 1 .2.1 --values ades-guard-values.yaml ades-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the ADES for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent \u2026 Example ades-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : ades domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth # customDefaultResources: # - name: \"ADES Service for user 'eric'\" # description: \"Protected Access for eric to his space in the ADES\" # resource_uri: \"/eric\" # scopes: [] # default_owner: \"a9812efe-fc0c-49d3-8115-0f36883a84b9\" # - name: \"ADES Service for user 'bob'\" # description: \"Protected Access for bob to his space in the ADES\" # resource_uri: \"/bob\" # scopes: [] # default_owner: \"4ccae3a1-3fad-4ffe-bfa7-cce851143780\" volumeClaim : name : eoepca-proc-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : ades paths : - path : /(.*) service : name : ades port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"proc-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint Client Secret \u2693\ufe0e The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n proc create secret generic proc-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > proc-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : proc-client namespace : proc data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io. In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml ADES Usage Samples \u2693\ufe0e This section includes some sample requests to test the deployed ADES. NOTES: It assumed that the ADES is subject to access protection (ref. Resource Protection ), in which case a User ID Token must be provided with the request - typically in the HTTP header, such as Authorization: Bearer or X-User-Id . See section User ID Token for more details. The samples assume a user eric The snuggs application is used in the example below. See also Application Package Example . List Processes \u2693\ufe0e List available processes. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/wps3/processes' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Deploy Process \u2693\ufe0e Deploy the sample application snuggs . curl --location --request POST 'https://ades.192-168-49-2.nip.io/eric/wps3/processes' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' \\ --header 'Content-Type: application/json' \\ --data-raw '{ \"inputs\": [ { \"id\": \"applicationPackage\", \"input\": { \"format\": { \"mimeType\": \"application/cwl\" }, \"value\": { \"href\": \"https://raw.githubusercontent.com/EOEPCA/app-snuggs/main/app-package.cwl\" } } } ], \"outputs\": [ { \"format\": { \"mimeType\": \"string\", \"schema\": \"string\", \"encoding\": \"string\" }, \"id\": \"deployResult\", \"transmissionMode\": \"value\" } ], \"mode\": \"auto\", \"response\": \"raw\" }' Get Process Details \u2693\ufe0e Get details for a deployed process. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Execute Process \u2693\ufe0e Execute a process with supplied parameterisation. curl --location --request POST 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0/jobs' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' \\ --header 'Content-Type: application/json' \\ --data-raw '{ \"inputs\": [ { \"id\": \"input_reference\", \"input\": { \"dataType\": { \"name\": \"application/json\" }, \"value\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\" } }, { \"id\": \"s_expression\", \"input\": { \"dataType\": { \"name\": \"string\" }, \"value\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\" } } ], \"outputs\": [ { \"format\": { \"mimeType\": \"string\", \"schema\": \"string\", \"encoding\": \"string\" }, \"id\": \"wf_outputs\", \"transmissionMode\": \"value\" } ], \"mode\": \"auto\", \"response\": \"raw\" }' Job Status \u2693\ufe0e Once a processes execution has been initiated then its progress can monitored via a job-specific URL that is returned in the HTTP response headers of the execute request. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/watchjob/processes/snuggs-0_3_0/jobs/2e0fabf4-4ed6-11ec-b857-626a98159388' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Job Result \u2693\ufe0e Once the job execution has completed, then the results can be obtained. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/watchjob/processes/snuggs-0_3_0/jobs/2e0fabf4-4ed6-11ec-b857-626a98159388/result' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Undeploy Process \u2693\ufe0e A process can be deleted (undeployed). curl --location --request DELETE 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' Application Package Example \u2693\ufe0e For a (trivial) example application package see Example Application Package , which provides a description and illustration of the basics of creating an application that integrates with the expectations of the ADES stage-in and stage-out. For further reference see\u2026 Application Packages OGC Best Practise for Application Package Example Application Package Common Workflow Language (CWL) Guide for CWL in Earth Observation CWL Specification CWL User Guide Additional Information \u2693\ufe0e Additional information regarding the ADES can be found at: Helm Chart Wiki GitHub Repository ADES stage-in/out configuration","title":"Application Deployment & Execution"},{"location":"eoepca/ades/#application-deployment-execution-service-ades","text":"The ADES provides a platform-hosted execution engine through which users can initiate parameterised processing jobs using applications made available within the platform - supporting the efficient execution of the processing \u2018close to the data\u2019. Users can deploy specific \u2018applications\u2019 to the ADES, which may be their own applications, or those published by other platform users.","title":"Application Deployment &amp; Execution Service (ADES)"},{"location":"eoepca/ades/#helm-chart","text":"The ADES is deployed via the ades helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the ades chart . helm install --version 2 .0.4 --values ades-values.yaml ades eoepca/ades","title":"Helm Chart"},{"location":"eoepca/ades/#values","text":"At minimum, values for the following attributes should be specified: Details of the S3 Object Store for stage-out of processing results Dynamic provisioning StorageClass of ReadWriteMany storage (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the ADES will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Example ades-values.yaml \u2026 workflowExecutor : inputs : # Stage-in from CREODIAS eodata # (only works within CREODIAS - i.e. on a Cloudferro VM) STAGEIN_AWS_SERVICEURL : http://data.cloudferro.com STAGEIN_AWS_ACCESS_KEY_ID : test STAGEIN_AWS_SECRET_ACCESS_KEY : test STAGEIN_AWS_REGION : RegionOne # Stage-out to minio S3 # (use this if the ADES is not configured to stage-out to the Workspace) STAGEOUT_AWS_SERVICEURL : http://minio.192-168-49-2.nip.io STAGEOUT_AWS_ACCESS_KEY_ID : eoepca STAGEOUT_AWS_SECRET_ACCESS_KEY : changeme STAGEOUT_AWS_REGION : us-east-1 STAGEOUT_OUTPUT : s3://eoepca processingStorageClass : standard processingVolumeTmpSize : \"6Gi\" processingVolumeOutputSize : \"6Gi\" processingMaxRam : \"8Gi\" processingMaxCores : \"4\" wps : pepBaseUrl : \"http://ades-pep:5576\" usePep : \"true\" persistence : storageClass : standard ingress : enabled : true annotations : kubernetes.io/ingress.class : nginx ingress.kubernetes.io/ssl-redirect : \"false\" hosts : - host : ades.192-168-49-2.nip.io paths : - path : / pathType : ImplementationSpecific tls : - hosts : - ades.192-168-49-2.nip.io secretName : ades-tls","title":"Values"},{"location":"eoepca/ades/#stage-in-stage-out-configuration","text":"The ADES hosts applications that are deployed and invoked in accordance with the OGC Best Practise for Application Package . Thus, the ADES provides a conformant environment within which the application is integrated for execution. A key part of the ADES\u2019s role in this is to faciltate the provision of input data to the application (stage-in), and the handling of the results output at the conclusion of application execution (stage-out). The ADES helm chart configures (by default) the ADES with implementations of the stage-in and stage-out functions that use the Spatio Temporal Asset Router Services utility. The ADES provides hooks for system integrators to override these defaults to implement their own stage-in and stage-out behaviour - for example, to integrate with their platform\u2019s own catalogue and data offering. The stage-in and stage-out are specified as CWL via the helm values\u2026 workflowExecutor : stagein : cwl : | cwlVersion: v1.0 ... stageout : cwl : | cwlVersion: v1.0 ... For a detailed description see ADES stage-in/out configuration in the ADES wiki .","title":"Stage-in / Stage-out Configuration"},{"location":"eoepca/ades/#workspace-integration","text":"The ADES has the facility to integrate with the EOEPCA Workspace component for registration of staged-out processing results. This is disabled by default ( useResourceManager: false ). When enabled, the ADES will register the staged-out products with the user\u2019s Workspace, such that they are indexed and available via the user\u2019s Resource Catalogue and Data Access services. Example ades-values.yaml (snippet)\u2026 workflowExecutor : ... useResourceManager : \"true\" resourceManagerWorkspacePrefix : \"guide-user\" resourceManagerEndpoint : \"https://workspace-api.192-168-49-2.nip.io\" platformDomain : \"https://auth.192-168-49-2.nip.io\" ... The value resourceManagerWorkspacePrefix must be consistent with that configured for the Workspace API deployment , (ref. value prefixForName ).","title":"Workspace Integration"},{"location":"eoepca/ades/#protection","text":"As described in section Resource Protection , the resource-guard component can be inserted into the request path of the ADES service to provide access authorization decisions helm install --version 1 .2.1 --values ades-guard-values.yaml ades-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the ADES for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent \u2026 Example ades-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : ades domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth # customDefaultResources: # - name: \"ADES Service for user 'eric'\" # description: \"Protected Access for eric to his space in the ADES\" # resource_uri: \"/eric\" # scopes: [] # default_owner: \"a9812efe-fc0c-49d3-8115-0f36883a84b9\" # - name: \"ADES Service for user 'bob'\" # description: \"Protected Access for bob to his space in the ADES\" # resource_uri: \"/bob\" # scopes: [] # default_owner: \"4ccae3a1-3fad-4ffe-bfa7-cce851143780\" volumeClaim : name : eoepca-proc-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : ades paths : - path : /(.*) service : name : ades port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"proc-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint","title":"Protection"},{"location":"eoepca/ades/#client-secret","text":"The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n proc create secret generic proc-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > proc-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : proc-client namespace : proc data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io. In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml","title":"Client Secret"},{"location":"eoepca/ades/#ades-usage-samples","text":"This section includes some sample requests to test the deployed ADES. NOTES: It assumed that the ADES is subject to access protection (ref. Resource Protection ), in which case a User ID Token must be provided with the request - typically in the HTTP header, such as Authorization: Bearer or X-User-Id . See section User ID Token for more details. The samples assume a user eric The snuggs application is used in the example below. See also Application Package Example .","title":"ADES Usage Samples"},{"location":"eoepca/ades/#list-processes","text":"List available processes. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/wps3/processes' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"List Processes"},{"location":"eoepca/ades/#deploy-process","text":"Deploy the sample application snuggs . curl --location --request POST 'https://ades.192-168-49-2.nip.io/eric/wps3/processes' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' \\ --header 'Content-Type: application/json' \\ --data-raw '{ \"inputs\": [ { \"id\": \"applicationPackage\", \"input\": { \"format\": { \"mimeType\": \"application/cwl\" }, \"value\": { \"href\": \"https://raw.githubusercontent.com/EOEPCA/app-snuggs/main/app-package.cwl\" } } } ], \"outputs\": [ { \"format\": { \"mimeType\": \"string\", \"schema\": \"string\", \"encoding\": \"string\" }, \"id\": \"deployResult\", \"transmissionMode\": \"value\" } ], \"mode\": \"auto\", \"response\": \"raw\" }'","title":"Deploy Process"},{"location":"eoepca/ades/#get-process-details","text":"Get details for a deployed process. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"Get Process Details"},{"location":"eoepca/ades/#execute-process","text":"Execute a process with supplied parameterisation. curl --location --request POST 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0/jobs' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json' \\ --header 'Content-Type: application/json' \\ --data-raw '{ \"inputs\": [ { \"id\": \"input_reference\", \"input\": { \"dataType\": { \"name\": \"application/json\" }, \"value\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\" } }, { \"id\": \"s_expression\", \"input\": { \"dataType\": { \"name\": \"string\" }, \"value\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\" } } ], \"outputs\": [ { \"format\": { \"mimeType\": \"string\", \"schema\": \"string\", \"encoding\": \"string\" }, \"id\": \"wf_outputs\", \"transmissionMode\": \"value\" } ], \"mode\": \"auto\", \"response\": \"raw\" }'","title":"Execute Process"},{"location":"eoepca/ades/#job-status","text":"Once a processes execution has been initiated then its progress can monitored via a job-specific URL that is returned in the HTTP response headers of the execute request. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/watchjob/processes/snuggs-0_3_0/jobs/2e0fabf4-4ed6-11ec-b857-626a98159388' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"Job Status"},{"location":"eoepca/ades/#job-result","text":"Once the job execution has completed, then the results can be obtained. curl --location --request GET 'https://ades.192-168-49-2.nip.io/eric/watchjob/processes/snuggs-0_3_0/jobs/2e0fabf4-4ed6-11ec-b857-626a98159388/result' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"Job Result"},{"location":"eoepca/ades/#undeploy-process","text":"A process can be deleted (undeployed). curl --location --request DELETE 'https://ades.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0' \\ --header 'X-User-Id: <user-id-token>' \\ --header 'Accept: application/json'","title":"Undeploy Process"},{"location":"eoepca/ades/#application-package-example","text":"For a (trivial) example application package see Example Application Package , which provides a description and illustration of the basics of creating an application that integrates with the expectations of the ADES stage-in and stage-out. For further reference see\u2026 Application Packages OGC Best Practise for Application Package Example Application Package Common Workflow Language (CWL) Guide for CWL in Earth Observation CWL Specification CWL User Guide","title":"Application Package Example"},{"location":"eoepca/ades/#additional-information","text":"Additional information regarding the ADES can be found at: Helm Chart Wiki GitHub Repository ADES stage-in/out configuration","title":"Additional Information"},{"location":"eoepca/container-registry/","text":"Container Registry \u2693\ufe0e To support the development (ref. Processor Development Environment ) and deployment/execution (ref. ADES ) of user-defined applications, we deploy a container registry to host container images. This is provied by a deployment of the Harbor artefact repository . Helm Chart \u2693\ufe0e Harbor is deployed via the harbor helm chart from the Harbor Helm Chart Repository . helm install --values harbor-values.yaml harbor harbor --repo https://helm.goharbor.io Values \u2693\ufe0e The chart is configured via values that are fully documented on the Harbor website . Example\u2026 expose : ingress : annotations : kubernetes.io/ingress.class : nginx cert-manager.io/cluster-issuer : letsencrypt-production nginx.ingress.kubernetes.io/proxy-read-timeout : '600' # from chart: ingress.kubernetes.io/ssl-redirect : letsencrypt-production ingress.kubernetes.io/proxy-body-size : \"0\" nginx.ingress.kubernetes.io/ssl-redirect : letsencrypt-production nginx.ingress.kubernetes.io/proxy-body-size : \"0\" hosts : core : harbor.192-168-49-2.nip.io notary : harbor-notary.192-168-49-2.nip.io persistence : persistentVolumeClaim : registry : storageClass : standard chartmuseum : storageClass : standard jobservice : storageClass : standard database : storageClass : standard redis : storageClass : standard trivy : storageClass : standard externalURL : https://harbor.192-168-49-2.nip.io # initial password for logging in with user \"admin\" harborAdminPassword : \"changeme\" chartmuseum : enabled : false trivy : enabled : false notary : enabled : false NOTES: We specify use of \u2018valid\u2019 certificates from Letsencrypt \u2018production\u2019. The Workspace API, which calls the Harbor API, expects valid certificates and will thus fail if presented with TLS certificates that fail validation. The letsencrypt-production Cluster Issuer relies upon the deployment being accessible from the public internet via the expose.ingress.hosts.core DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller. The Workspace API will not like this. Container Registry Usage \u2693\ufe0e After deployemnt Harbor is accessible via its web interface at https://harbor.<domain>/ e.g. https://harbor.192-168-49-2.nip.io/ . Login as the admin user with the password specified in the helm values. Additional Information \u2693\ufe0e Additional information regarding the Container Registry can be found at: Web Site Helm Chart Repository Helm Chart Description Harbor Documentation","title":"Container Registry"},{"location":"eoepca/container-registry/#container-registry","text":"To support the development (ref. Processor Development Environment ) and deployment/execution (ref. ADES ) of user-defined applications, we deploy a container registry to host container images. This is provied by a deployment of the Harbor artefact repository .","title":"Container Registry"},{"location":"eoepca/container-registry/#helm-chart","text":"Harbor is deployed via the harbor helm chart from the Harbor Helm Chart Repository . helm install --values harbor-values.yaml harbor harbor --repo https://helm.goharbor.io","title":"Helm Chart"},{"location":"eoepca/container-registry/#values","text":"The chart is configured via values that are fully documented on the Harbor website . Example\u2026 expose : ingress : annotations : kubernetes.io/ingress.class : nginx cert-manager.io/cluster-issuer : letsencrypt-production nginx.ingress.kubernetes.io/proxy-read-timeout : '600' # from chart: ingress.kubernetes.io/ssl-redirect : letsencrypt-production ingress.kubernetes.io/proxy-body-size : \"0\" nginx.ingress.kubernetes.io/ssl-redirect : letsencrypt-production nginx.ingress.kubernetes.io/proxy-body-size : \"0\" hosts : core : harbor.192-168-49-2.nip.io notary : harbor-notary.192-168-49-2.nip.io persistence : persistentVolumeClaim : registry : storageClass : standard chartmuseum : storageClass : standard jobservice : storageClass : standard database : storageClass : standard redis : storageClass : standard trivy : storageClass : standard externalURL : https://harbor.192-168-49-2.nip.io # initial password for logging in with user \"admin\" harborAdminPassword : \"changeme\" chartmuseum : enabled : false trivy : enabled : false notary : enabled : false NOTES: We specify use of \u2018valid\u2019 certificates from Letsencrypt \u2018production\u2019. The Workspace API, which calls the Harbor API, expects valid certificates and will thus fail if presented with TLS certificates that fail validation. The letsencrypt-production Cluster Issuer relies upon the deployment being accessible from the public internet via the expose.ingress.hosts.core DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller. The Workspace API will not like this.","title":"Values"},{"location":"eoepca/container-registry/#container-registry-usage","text":"After deployemnt Harbor is accessible via its web interface at https://harbor.<domain>/ e.g. https://harbor.192-168-49-2.nip.io/ . Login as the admin user with the password specified in the helm values.","title":"Container Registry Usage"},{"location":"eoepca/container-registry/#additional-information","text":"Additional information regarding the Container Registry can be found at: Web Site Helm Chart Repository Helm Chart Description Harbor Documentation","title":"Additional Information"},{"location":"eoepca/data-access/","text":"Data Access \u2693\ufe0e The Data Access provides standards-based services for access to platform hosted data - including OGC WMS/WMTS for visualisation, and OGC WCS for data retrieval. This component also includes Harvester and Registrar services to discover/watch the existing data holding of the infrastructure data layer and populate/maintain the data access and resource catalogue services accordingly. Helm Chart \u2693\ufe0e The Data Access is deployed via the data-access helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are supplied with the instantiation of the helm release. The EOEPCA data-access chart provides a thin wrapper around the EOX View Server ( vs ) helm chart. The documentation for the View Server can be found here: User Guide: https://vs.pages.eox.at/documentation/user/main/ Operator Guide: https://vs.pages.eox.at/documentation/operator/main/ helm install --version 1 .2.5 --values data-access-values.yaml data-access data-access Values \u2693\ufe0e The Data Access supports many values to configure the service. These are documented in full in the View Server - Operator Guide Configuration page . Core Configuration \u2693\ufe0e Typically, values for the following attributes may be specified to override the chart defaults: The fully-qualified public URL for the service, ref. ( global.ingress.hosts.host[0] ) Metadata describing the service instance Dynamic provisioning StorageClass for persistence Persistent Volume Claims for database and redis components Object storage details for data and cache Container images for renderer and registrar (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Data Access will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . global : env : REGISTRAR_REPLACE : \"true\" CPL_VSIL_CURL_ALLOWED_EXTENSIONS : .TIF,.tif,.xml,.jp2,.jpg,.jpeg AWS_HTTPS : \"FALSE\" startup_scripts : - /registrar_pycsw/registrar_pycsw/initialize-collections.sh ingress : enabled : true annotations : kubernetes.io/ingress.class : nginx kubernetes.io/tls-acme : \"true\" nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" cert-manager.io/cluster-issuer : letsencrypt-production hosts : - host : data-access.192-168-49-2.nip.io tls : - hosts : - data-access.192-168-49-2.nip.io secretName : data-access-tls storage : data : data : type : S3 endpoint_url : http://data.cloudferro.com access_key_id : access secret_access_key : access region_name : RegionOne validate_bucket_name : false cache : type : S3 endpoint_url : \"http://minio.192-168-49-2.nip.io\" host : \"minio.192-168-49-2.nip.io\" access_key_id : xxx secret_access_key : xxx region : us-east-1 bucket : cache-bucket metadata : title : EOEPCA Data Access Service developed by EOX abstract : EOEPCA Data Access Service developed by EOX header : \"EOEPCA Data Access View Server (VS) Client powered by <a href=\\\"//eox.at\\\"><img src=\\\"//eox.at/wp-content/uploads/2017/09/EOX_Logo.svg\\\" alt=\\\"EOX\\\" style=\\\"height:25px;margin-left:10px\\\"/></a>\" url : https://data-access.192-168-49-2.nip.io/ows layers : # see section 'Data-layer Configuration' collections : # see section 'Data-layer Configuration' productTypes : # see section 'Data-layer Configuration' vs : renderer : replicaCount : 4 ingress : enabled : false registrar : replicaCount : 1 harvester : # see section 'Harvester Configuration' replicaCount : 1 client : replicaCount : 1 ingress : enabled : false redis : master : persistence : enabled : true storageClass : standard ingestor : replicaCount : 0 ingress : enabled : false preprocessor : replicaCount : 0 cache : ingress : enabled : false Data-layer Configuration \u2693\ufe0e Configuration of the service data-layer - as described in the View Server Operator Guide . The data-access service data handling is configured by definition of productTypes , collections and layers \u2026 productTypes - Product Types Identify the underlying file assets as WCS coverages and their visual representation collections - Data Collections Provides groupings into which products are organised layers - Layers Specifies the hoe the product visual representations are exposed through the WMS service For more information, see the worked example in section Data Specification for the example CREODIAS deployment . Harvester \u2693\ufe0e The Data Access service includes a Harvester component. The following subsections describe its configuration and usage. Harvester Helm Configuration \u2693\ufe0e The Harvester can be configured through the helm chart values\u2026 vs : harvester : replicaCount : 1 config : redis : host : data-access-redis-master port : 6379 harvesters : - name : Creodias-Opensearch resource : url : https://finder.creodias.eu/resto/api/collections/Sentinel2/describe.xml type : OpenSearch format_config : type : 'application/json' property_mapping : start_datetime : 'startDate' end_datetime : 'completionDate' productIdentifier : 'productIdentifier' query : time : property : sensed begin : 2019-09-10T00:00:00Z end : 2019-09-11T00:00:00Z collection : null bbox : 14.9,47.7,16.4,48.7 filter : {} postprocess : - type : harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor queue : register The harvester.config.harvesters list defines a set of pre-defined harvesters which can be invoked in a later stage. The name property must be unique for each harvester and must be unique among all harvesters in the list. Each harvester is associated with a resource , an optional filter or postprocess function, and a queue . The resource defines where each item is harvested from. This can be a file system, a search service, catalog file or something similar. The example above defines a connection to an OpenSearch service on CREODIAS, with associated default query parameters and a format configuration. The filter allows to filter elements within the harvester, when the resource does not provide a specific filter. This filter can be supplied using CQL2-JSON. The postprocess can adjust the harvested results. In this example the harvested items are not complete, and additional metadata must be retrieved from an object storage. The queue defines where harvested items will be pushed into. Usually this is a registration queue, where the registrar will pick up and start registration according to its configuration. Starting the Harvester \u2693\ufe0e The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue. One-off harvests via the CLI \u2693\ufe0e In order to start a harvest from the CLI, the operator first needs to connect to the kubernetes pod of the harvester. Within that pod, the harvest can be executed like this\u2026 python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch This will invoke the Creodias-Opensearch harvester with default arguments. When some values are to be overridden, the \u2013values switch can be used to pass override values. These values must be a JSON string. The following example adjusts the begin and end times of the query parameters\u2026 python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch --values '{\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}' Harvests via the harvest daemon \u2693\ufe0e The harvester pod runs a service listening on a redis queue. When a message is read from the queue, it will be read as a JSON string, expecting an object with at least a name property. Optionally, it can also have a values property, working in the same way as with CLI --values . To send a harvesting request via the redis queue, it is necessary to connect to the redis pod and execute the redis-cli there. Then the following command can be used to achieve the same result as above with CLI harvesting\u2026 redis-cli LPUSH '{\"name\": \"Creodias-Opensearch\", \"values\": {\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}}' Results of the harvesting \u2693\ufe0e The harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created. Storage \u2693\ufe0e Specification of PVCs and access to object storage. Persistent Volume Claims \u2693\ufe0e The PVCs specified in the helm chart values must be created. PVC for Database \u2693\ufe0e kind : PersistentVolumeClaim apiVersion : v1 metadata : name : data-access-db namespace : rm labels : k8s-app : data-access name : data-access spec : storageClassName : standard accessModes : - ReadWriteMany resources : requests : storage : 100Gi PVC for Redis \u2693\ufe0e kind : PersistentVolumeClaim apiVersion : v1 metadata : name : data-access-redis namespace : rm labels : k8s-app : data-access name : data-access spec : storageClassName : standard accessModes : - ReadWriteMany resources : requests : storage : 1Gi Object Storage \u2693\ufe0e The helm chart values expect specification of object storage details for: data : to access the EO data of the underlying infrastructure cache : a dedicated object storage bucket is used to support the cache function of the data access services Platform EO Data \u2693\ufe0e Specifies the details for the infrastructure object storage that provides direct access to the EO product files. For example, the CREODIAS metadata catalogue provides references to product files in their eodata object storage - the access details for which are configured in the data access services: global : storage : data : data : type : S3 endpoint_url : http://data.cloudferro.com access_key_id : access secret_access_key : access region_name : RegionOne validate_bucket_name : false Data Access Cache \u2693\ufe0e The Data Access services maintain a cache, which relies on the usage of a dedicate object storage bucket for data persistence. This bucket must be created (manual step) and its access details configured in the data access services. Example based upon CREODIAS: global : storage : cache : type : S3 endpoint_url : \"https://cf2.cloudferro.com:8080/cache-bucket\" host : \"cf2.cloudferro.com:8080\" access_key_id : xxx secret_access_key : xxx region : RegionOne bucket : cache-bucket \u2026where xxx must be replaced with the bucket credentials. Protection \u2693\ufe0e As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Data Access service to provide access authorization decisions. helm install --version 1 .2.1 --values data-access-guard-values.yaml data-access-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Data Access for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example data-access-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : data-access domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : data-access paths : - path : /(ows.*) service : name : data-access-renderer port : 80 - path : /(opensearch.*) service : name : data-access-renderer port : 80 - path : /(coverages/metadata.*) service : name : data-access-renderer port : 80 - path : /(admin.*) service : name : data-access-renderer port : 80 - path : /cache/(.*) service : name : data-access-cache port : 80 - path : /(.*) service : name : data-access-client port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint Client Secret \u2693\ufe0e The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io. In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml Data Access Usage \u2693\ufe0e Default Harvesting \u2693\ufe0e At deployment time the harvester helm values include configuration that populates a default harvester configuration, that is prepared in the file /config.yaml in the harvester pod. The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section Data-layer Configuration . The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the rm/harvester service and executing the command\u2026 python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch Ad-hoc Harvesting \u2693\ufe0e Ad-hoc harvesting can be invoked by provision of a suitable config.yaml into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time. The helper script ./deploy/bin/harvest faciltates this\u2026 ./deploy/bin/harvest <path-to-config-file> See directory ./deploy/samples/harvester/ that contains some sample harvesting configuration files. For example\u2026 ./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2019.09.10.yaml Registration of Collections \u2693\ufe0e The helper script ./deploy/bin/register-collection is provided to faciltate the registration of collections that are specfied in STAC Collection format. ./deploy/bin/register-collection <path-to-stac-collection-file> See directory ./deploy/samples/collections/ that contains some same STAC Collection files. For example\u2026 ./deploy/bin/register-collection ./deploy/samples/collections/S2MSI2A.json Additional Information \u2693\ufe0e Additional information regarding the Data Access can be found at: Helm Chart Documentation: User Guide Operator Guide Git Repository","title":"Data Access"},{"location":"eoepca/data-access/#data-access","text":"The Data Access provides standards-based services for access to platform hosted data - including OGC WMS/WMTS for visualisation, and OGC WCS for data retrieval. This component also includes Harvester and Registrar services to discover/watch the existing data holding of the infrastructure data layer and populate/maintain the data access and resource catalogue services accordingly.","title":"Data Access"},{"location":"eoepca/data-access/#helm-chart","text":"The Data Access is deployed via the data-access helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are supplied with the instantiation of the helm release. The EOEPCA data-access chart provides a thin wrapper around the EOX View Server ( vs ) helm chart. The documentation for the View Server can be found here: User Guide: https://vs.pages.eox.at/documentation/user/main/ Operator Guide: https://vs.pages.eox.at/documentation/operator/main/ helm install --version 1 .2.5 --values data-access-values.yaml data-access data-access","title":"Helm Chart"},{"location":"eoepca/data-access/#values","text":"The Data Access supports many values to configure the service. These are documented in full in the View Server - Operator Guide Configuration page .","title":"Values"},{"location":"eoepca/data-access/#core-configuration","text":"Typically, values for the following attributes may be specified to override the chart defaults: The fully-qualified public URL for the service, ref. ( global.ingress.hosts.host[0] ) Metadata describing the service instance Dynamic provisioning StorageClass for persistence Persistent Volume Claims for database and redis components Object storage details for data and cache Container images for renderer and registrar (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Data Access will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . global : env : REGISTRAR_REPLACE : \"true\" CPL_VSIL_CURL_ALLOWED_EXTENSIONS : .TIF,.tif,.xml,.jp2,.jpg,.jpeg AWS_HTTPS : \"FALSE\" startup_scripts : - /registrar_pycsw/registrar_pycsw/initialize-collections.sh ingress : enabled : true annotations : kubernetes.io/ingress.class : nginx kubernetes.io/tls-acme : \"true\" nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" cert-manager.io/cluster-issuer : letsencrypt-production hosts : - host : data-access.192-168-49-2.nip.io tls : - hosts : - data-access.192-168-49-2.nip.io secretName : data-access-tls storage : data : data : type : S3 endpoint_url : http://data.cloudferro.com access_key_id : access secret_access_key : access region_name : RegionOne validate_bucket_name : false cache : type : S3 endpoint_url : \"http://minio.192-168-49-2.nip.io\" host : \"minio.192-168-49-2.nip.io\" access_key_id : xxx secret_access_key : xxx region : us-east-1 bucket : cache-bucket metadata : title : EOEPCA Data Access Service developed by EOX abstract : EOEPCA Data Access Service developed by EOX header : \"EOEPCA Data Access View Server (VS) Client powered by <a href=\\\"//eox.at\\\"><img src=\\\"//eox.at/wp-content/uploads/2017/09/EOX_Logo.svg\\\" alt=\\\"EOX\\\" style=\\\"height:25px;margin-left:10px\\\"/></a>\" url : https://data-access.192-168-49-2.nip.io/ows layers : # see section 'Data-layer Configuration' collections : # see section 'Data-layer Configuration' productTypes : # see section 'Data-layer Configuration' vs : renderer : replicaCount : 4 ingress : enabled : false registrar : replicaCount : 1 harvester : # see section 'Harvester Configuration' replicaCount : 1 client : replicaCount : 1 ingress : enabled : false redis : master : persistence : enabled : true storageClass : standard ingestor : replicaCount : 0 ingress : enabled : false preprocessor : replicaCount : 0 cache : ingress : enabled : false","title":"Core Configuration"},{"location":"eoepca/data-access/#data-layer-configuration","text":"Configuration of the service data-layer - as described in the View Server Operator Guide . The data-access service data handling is configured by definition of productTypes , collections and layers \u2026 productTypes - Product Types Identify the underlying file assets as WCS coverages and their visual representation collections - Data Collections Provides groupings into which products are organised layers - Layers Specifies the hoe the product visual representations are exposed through the WMS service For more information, see the worked example in section Data Specification for the example CREODIAS deployment .","title":"Data-layer Configuration"},{"location":"eoepca/data-access/#harvester","text":"The Data Access service includes a Harvester component. The following subsections describe its configuration and usage.","title":"Harvester"},{"location":"eoepca/data-access/#harvester-helm-configuration","text":"The Harvester can be configured through the helm chart values\u2026 vs : harvester : replicaCount : 1 config : redis : host : data-access-redis-master port : 6379 harvesters : - name : Creodias-Opensearch resource : url : https://finder.creodias.eu/resto/api/collections/Sentinel2/describe.xml type : OpenSearch format_config : type : 'application/json' property_mapping : start_datetime : 'startDate' end_datetime : 'completionDate' productIdentifier : 'productIdentifier' query : time : property : sensed begin : 2019-09-10T00:00:00Z end : 2019-09-11T00:00:00Z collection : null bbox : 14.9,47.7,16.4,48.7 filter : {} postprocess : - type : harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor queue : register The harvester.config.harvesters list defines a set of pre-defined harvesters which can be invoked in a later stage. The name property must be unique for each harvester and must be unique among all harvesters in the list. Each harvester is associated with a resource , an optional filter or postprocess function, and a queue . The resource defines where each item is harvested from. This can be a file system, a search service, catalog file or something similar. The example above defines a connection to an OpenSearch service on CREODIAS, with associated default query parameters and a format configuration. The filter allows to filter elements within the harvester, when the resource does not provide a specific filter. This filter can be supplied using CQL2-JSON. The postprocess can adjust the harvested results. In this example the harvested items are not complete, and additional metadata must be retrieved from an object storage. The queue defines where harvested items will be pushed into. Usually this is a registration queue, where the registrar will pick up and start registration according to its configuration.","title":"Harvester Helm Configuration"},{"location":"eoepca/data-access/#starting-the-harvester","text":"The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue.","title":"Starting the Harvester"},{"location":"eoepca/data-access/#results-of-the-harvesting","text":"The harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created.","title":"Results of the harvesting"},{"location":"eoepca/data-access/#storage","text":"Specification of PVCs and access to object storage.","title":"Storage"},{"location":"eoepca/data-access/#persistent-volume-claims","text":"The PVCs specified in the helm chart values must be created.","title":"Persistent Volume Claims"},{"location":"eoepca/data-access/#pvc-for-database","text":"kind : PersistentVolumeClaim apiVersion : v1 metadata : name : data-access-db namespace : rm labels : k8s-app : data-access name : data-access spec : storageClassName : standard accessModes : - ReadWriteMany resources : requests : storage : 100Gi","title":"PVC for Database"},{"location":"eoepca/data-access/#pvc-for-redis","text":"kind : PersistentVolumeClaim apiVersion : v1 metadata : name : data-access-redis namespace : rm labels : k8s-app : data-access name : data-access spec : storageClassName : standard accessModes : - ReadWriteMany resources : requests : storage : 1Gi","title":"PVC for Redis"},{"location":"eoepca/data-access/#object-storage","text":"The helm chart values expect specification of object storage details for: data : to access the EO data of the underlying infrastructure cache : a dedicated object storage bucket is used to support the cache function of the data access services","title":"Object Storage"},{"location":"eoepca/data-access/#platform-eo-data","text":"Specifies the details for the infrastructure object storage that provides direct access to the EO product files. For example, the CREODIAS metadata catalogue provides references to product files in their eodata object storage - the access details for which are configured in the data access services: global : storage : data : data : type : S3 endpoint_url : http://data.cloudferro.com access_key_id : access secret_access_key : access region_name : RegionOne validate_bucket_name : false","title":"Platform EO Data"},{"location":"eoepca/data-access/#data-access-cache","text":"The Data Access services maintain a cache, which relies on the usage of a dedicate object storage bucket for data persistence. This bucket must be created (manual step) and its access details configured in the data access services. Example based upon CREODIAS: global : storage : cache : type : S3 endpoint_url : \"https://cf2.cloudferro.com:8080/cache-bucket\" host : \"cf2.cloudferro.com:8080\" access_key_id : xxx secret_access_key : xxx region : RegionOne bucket : cache-bucket \u2026where xxx must be replaced with the bucket credentials.","title":"Data Access Cache"},{"location":"eoepca/data-access/#protection","text":"As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Data Access service to provide access authorization decisions. helm install --version 1 .2.1 --values data-access-guard-values.yaml data-access-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Data Access for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example data-access-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : data-access domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : data-access paths : - path : /(ows.*) service : name : data-access-renderer port : 80 - path : /(opensearch.*) service : name : data-access-renderer port : 80 - path : /(coverages/metadata.*) service : name : data-access-renderer port : 80 - path : /(admin.*) service : name : data-access-renderer port : 80 - path : /cache/(.*) service : name : data-access-cache port : 80 - path : /(.*) service : name : data-access-client port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint","title":"Protection"},{"location":"eoepca/data-access/#client-secret","text":"The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io. In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml","title":"Client Secret"},{"location":"eoepca/data-access/#data-access-usage","text":"","title":"Data Access Usage"},{"location":"eoepca/data-access/#default-harvesting","text":"At deployment time the harvester helm values include configuration that populates a default harvester configuration, that is prepared in the file /config.yaml in the harvester pod. The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section Data-layer Configuration . The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the rm/harvester service and executing the command\u2026 python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch","title":"Default Harvesting"},{"location":"eoepca/data-access/#ad-hoc-harvesting","text":"Ad-hoc harvesting can be invoked by provision of a suitable config.yaml into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time. The helper script ./deploy/bin/harvest faciltates this\u2026 ./deploy/bin/harvest <path-to-config-file> See directory ./deploy/samples/harvester/ that contains some sample harvesting configuration files. For example\u2026 ./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2019.09.10.yaml","title":"Ad-hoc Harvesting"},{"location":"eoepca/data-access/#registration-of-collections","text":"The helper script ./deploy/bin/register-collection is provided to faciltate the registration of collections that are specfied in STAC Collection format. ./deploy/bin/register-collection <path-to-stac-collection-file> See directory ./deploy/samples/collections/ that contains some same STAC Collection files. For example\u2026 ./deploy/bin/register-collection ./deploy/samples/collections/S2MSI2A.json","title":"Registration of Collections"},{"location":"eoepca/data-access/#additional-information","text":"Additional information regarding the Data Access can be found at: Helm Chart Documentation: User Guide Operator Guide Git Repository","title":"Additional Information"},{"location":"eoepca/login-service/","text":"Login Service \u2693\ufe0e The Login Service provides the platform Authorization Server for authenticated user identity and request authorization. Helm Chart \u2693\ufe0e The Login Service is deployed via the login-service helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the login-service chart . helm install --version 1 .2.1 --values login-service-values.yaml login-service eoepca/login-service Values \u2693\ufe0e At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Kubernetes namespace for the login-service components Initial password for the admin user Note that the password must meet the complexity: at least 6 characters and include one uppercase letter, one lowercase letter, one digit, and one special character Name of Persistent Volume Claim for login-service persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. TLS Certificate Provider, e.g. letsencrypt-production Example login-service-values.yaml \u2026 global : domain : auth.192-168-49-2.nip.io nginxIp : 192.168.49.2 namespace : um volumeClaim : name : eoepca-userman-pvc create : false config : domain : auth.192-168-49-2.nip.io adminPass : Chang3me! ldapPass : Chang3me! volumeClaim : name : eoepca-userman-pvc opendj : volumeClaim : name : eoepca-userman-pvc oxauth : volumeClaim : name : eoepca-userman-pvc oxtrust : volumeClaim : name : eoepca-userman-pvc nginx : ingress : annotations : cert-manager.io/cluster-issuer : letsencrypt-production hosts : - auth.192-168-49-2.nip.io tls : - hosts : - auth.192-168-49-2.nip.io secretName : login-service-tls Login Service Usage \u2693\ufe0e Once the deployment has been completed successfully, the Login Service is accessed at the endpoint https://auth.<domain>/ , configured by your domain - e.g. https://auth.192-168-49-2.nip.io/ . Login as the admin user with the credentials configured in the helm values - ref. adminPass / ldapPass . Typical first actions to undertake through the Gluu web interface include creation of users and clients. Additional Information \u2693\ufe0e Additional information regarding the Login Service can be found at: Helm Chart Wiki GitHub Repository","title":"Login Service"},{"location":"eoepca/login-service/#login-service","text":"The Login Service provides the platform Authorization Server for authenticated user identity and request authorization.","title":"Login Service"},{"location":"eoepca/login-service/#helm-chart","text":"The Login Service is deployed via the login-service helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the login-service chart . helm install --version 1 .2.1 --values login-service-values.yaml login-service eoepca/login-service","title":"Helm Chart"},{"location":"eoepca/login-service/#values","text":"At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Kubernetes namespace for the login-service components Initial password for the admin user Note that the password must meet the complexity: at least 6 characters and include one uppercase letter, one lowercase letter, one digit, and one special character Name of Persistent Volume Claim for login-service persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. TLS Certificate Provider, e.g. letsencrypt-production Example login-service-values.yaml \u2026 global : domain : auth.192-168-49-2.nip.io nginxIp : 192.168.49.2 namespace : um volumeClaim : name : eoepca-userman-pvc create : false config : domain : auth.192-168-49-2.nip.io adminPass : Chang3me! ldapPass : Chang3me! volumeClaim : name : eoepca-userman-pvc opendj : volumeClaim : name : eoepca-userman-pvc oxauth : volumeClaim : name : eoepca-userman-pvc oxtrust : volumeClaim : name : eoepca-userman-pvc nginx : ingress : annotations : cert-manager.io/cluster-issuer : letsencrypt-production hosts : - auth.192-168-49-2.nip.io tls : - hosts : - auth.192-168-49-2.nip.io secretName : login-service-tls","title":"Values"},{"location":"eoepca/login-service/#login-service-usage","text":"Once the deployment has been completed successfully, the Login Service is accessed at the endpoint https://auth.<domain>/ , configured by your domain - e.g. https://auth.192-168-49-2.nip.io/ . Login as the admin user with the credentials configured in the helm values - ref. adminPass / ldapPass . Typical first actions to undertake through the Gluu web interface include creation of users and clients.","title":"Login Service Usage"},{"location":"eoepca/login-service/#additional-information","text":"Additional information regarding the Login Service can be found at: Helm Chart Wiki GitHub Repository","title":"Additional Information"},{"location":"eoepca/pde/","text":"Processor Development Environment (PDE) \u2693\ufe0e The Processor Development Environment (PDE) provides a web-based application that allows the user to perform platform-hosted interactive analysis and application development. Helm Chart \u2693\ufe0e The Processor Development Environment is deployed via the eoepca/jupyterhub helm chart from the EOEPCA Helm Chart Repository . The chart is derived from the public chart jupyterhub/k8s-hub that is available on DockerHub - https://hub.docker.com/r/jupyterhub/k8s-hub . helm install --version 1 .1.12 --values pde-values.yaml pde eoepca/jupyterhub Values \u2693\ufe0e The jupyterhub/k8s-hub chart supports many values. Typically, values for the following attributes may be specified: The persistence storage-class to be used URLs for OAuth integration with the EOEPCA Login Service (OIDC Provider) Ingress configuration Certificate Issuer for TLS Example pde-values.yaml \u2026 hub : db : pvc : storageClassName : standard extraEnv : OAUTH_CALLBACK_URL : \"https://pde.192-168-49-2.nip.io/hub/oauth_callback\" OAUTH2_USERDATA_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/userinfo\" OAUTH2_TOKEN_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/token\" OAUTH2_AUTHORIZE_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/authorize\" OAUTH_LOGOUT_REDIRECT_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/end_session?post_logout_redirect_uri=https://pde.192-168-49-2.nip.io\" STORAGE_CLASS : \"standard\" ingress : enabled : true annotations : cert-manager.io/cluster-issuer : letsencrypt-production hosts : - host : pde.192-168-49-2.nip.io paths : - path : / tls : - hosts : - pde.192-168-49-2.nip.io secretName : pde-tls JupyterHub Secret \u2693\ufe0e The PDE relies upon a Kubernetes secret jupyterhub-secrets that provides confidential values, including the client credentials for the Login Service. The credentials of the client registered for resource protection (ref. Client Registration ) can be re-used from the local file client.yaml as follows\u2026 kubectl -n pde create secret generic jupyterhub-secrets \\ --from-literal=JUPYTERHUB_CRYPT_KEY=\"$(openssl rand -hex 32)\" \\ --from-literal=OAUTH_CLIENT_ID=\"$(cat client.yaml | grep client-id | cut -d\\ -f2)\" \\ --from-literal=OAUTH_CLIENT_SECRET=\"$(cat client.yaml | grep client-secret | cut -d\\ -f2)\" PDE Usage \u2693\ufe0e The PDE is accessed at the endpoint https://pde.<domain>/ , configured by your domain - e.g. https://pde.192-168-49-2.nip.io/ . Additional Information \u2693\ufe0e Additional information regarding the Processor Development Environment can be found at: Helm Chart GitHub Repository","title":"Processor Development Environment"},{"location":"eoepca/pde/#processor-development-environment-pde","text":"The Processor Development Environment (PDE) provides a web-based application that allows the user to perform platform-hosted interactive analysis and application development.","title":"Processor Development Environment (PDE)"},{"location":"eoepca/pde/#helm-chart","text":"The Processor Development Environment is deployed via the eoepca/jupyterhub helm chart from the EOEPCA Helm Chart Repository . The chart is derived from the public chart jupyterhub/k8s-hub that is available on DockerHub - https://hub.docker.com/r/jupyterhub/k8s-hub . helm install --version 1 .1.12 --values pde-values.yaml pde eoepca/jupyterhub","title":"Helm Chart"},{"location":"eoepca/pde/#values","text":"The jupyterhub/k8s-hub chart supports many values. Typically, values for the following attributes may be specified: The persistence storage-class to be used URLs for OAuth integration with the EOEPCA Login Service (OIDC Provider) Ingress configuration Certificate Issuer for TLS Example pde-values.yaml \u2026 hub : db : pvc : storageClassName : standard extraEnv : OAUTH_CALLBACK_URL : \"https://pde.192-168-49-2.nip.io/hub/oauth_callback\" OAUTH2_USERDATA_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/userinfo\" OAUTH2_TOKEN_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/token\" OAUTH2_AUTHORIZE_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/authorize\" OAUTH_LOGOUT_REDIRECT_URL : \"https://auth.192-168-49-2.nip.io/oxauth/restv1/end_session?post_logout_redirect_uri=https://pde.192-168-49-2.nip.io\" STORAGE_CLASS : \"standard\" ingress : enabled : true annotations : cert-manager.io/cluster-issuer : letsencrypt-production hosts : - host : pde.192-168-49-2.nip.io paths : - path : / tls : - hosts : - pde.192-168-49-2.nip.io secretName : pde-tls","title":"Values"},{"location":"eoepca/pde/#jupyterhub-secret","text":"The PDE relies upon a Kubernetes secret jupyterhub-secrets that provides confidential values, including the client credentials for the Login Service. The credentials of the client registered for resource protection (ref. Client Registration ) can be re-used from the local file client.yaml as follows\u2026 kubectl -n pde create secret generic jupyterhub-secrets \\ --from-literal=JUPYTERHUB_CRYPT_KEY=\"$(openssl rand -hex 32)\" \\ --from-literal=OAUTH_CLIENT_ID=\"$(cat client.yaml | grep client-id | cut -d\\ -f2)\" \\ --from-literal=OAUTH_CLIENT_SECRET=\"$(cat client.yaml | grep client-secret | cut -d\\ -f2)\"","title":"JupyterHub Secret"},{"location":"eoepca/pde/#pde-usage","text":"The PDE is accessed at the endpoint https://pde.<domain>/ , configured by your domain - e.g. https://pde.192-168-49-2.nip.io/ .","title":"PDE Usage"},{"location":"eoepca/pde/#additional-information","text":"Additional information regarding the Processor Development Environment can be found at: Helm Chart GitHub Repository","title":"Additional Information"},{"location":"eoepca/pdp/","text":"Policy Decision Point \u2693\ufe0e The Policy Decision Point (PDP) provides the platform policy database and associated service for access policy decision requests. Helm Chart \u2693\ufe0e The PDP is deployed via the pdp-engine helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the pdp-engine chart . helm install --version 1 .1.6 --values pdp-values.yaml pdp eoepca/pdp-engine Values \u2693\ufe0e At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for pdp-engine persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. Example pdp-values.yaml \u2026 global : nginxIp : 192.168.49.2 domain : auth.192-168-49-2.nip.io volumeClaim : name : eoepca-userman-pvc create : false Additional Information \u2693\ufe0e Additional information regarding the PDP can be found at: Helm Chart Wiki GitHub Repository","title":"Policy Decision Point"},{"location":"eoepca/pdp/#policy-decision-point","text":"The Policy Decision Point (PDP) provides the platform policy database and associated service for access policy decision requests.","title":"Policy Decision Point"},{"location":"eoepca/pdp/#helm-chart","text":"The PDP is deployed via the pdp-engine helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the pdp-engine chart . helm install --version 1 .1.6 --values pdp-values.yaml pdp eoepca/pdp-engine","title":"Helm Chart"},{"location":"eoepca/pdp/#values","text":"At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for pdp-engine persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. Example pdp-values.yaml \u2026 global : nginxIp : 192.168.49.2 domain : auth.192-168-49-2.nip.io volumeClaim : name : eoepca-userman-pvc create : false","title":"Values"},{"location":"eoepca/pdp/#additional-information","text":"Additional information regarding the PDP can be found at: Helm Chart Wiki GitHub Repository","title":"Additional Information"},{"location":"eoepca/persistence/","text":"Persistence \u2693\ufe0e The EOEPCA building-blocks rely upon Kubernetes Persistent Volumes for their component persistence. Components integrate with the storage provided in the cluster by means of configurable Persistent Volume Claims and/or dynamic Storage Class that are specfied as values at time of deployment. Some components require storage of type ReadWriteMany - which, for a multi-node cluster, implies a network-based storage solution. ReadWriteMany Storage \u2693\ufe0e For the EOEPCA development deployment, an NFS server has been established to provide the persistence layer for ReadWriteMany storage. Pre-defined Persistent Volume Claims \u2693\ufe0e The EOEPCA development deployment establishes the following pre-defined Persistent Volume Claims, to provide a simple storage architecture that is organised around the \u2018domain areas\u2019 into which the Reference Implementation is split. Resource Managment ( resman ) - persistentvolumeclaim/eoepca-resman-pvc Processing & Chaining ( proc ) - persistentvolumeclaim/eoepca-proc-pvc User Management ( userman ) - persistentvolumeclaim/eoepca-userman-pvc NOTE that this is offered only as an example thay suits the approach of the development team. Each building-block has configuration through which its persistence (PV/PVC) can be configured according the needs of the deployment. The following Kubernetes yaml provides an example of provisioning such domain-specific PersistentVolumeClaims within the cluster - in this case using the minikube built-in storage-class standard for dynamic provisioning\u2026 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-proc-pvc namespace: proc spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-resman-pvc namespace: rm spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-userman-pvc namespace: um spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi Once established, these PersistentVolumeClaims are then referenced within the deployment configurations of the building-blocks. Dynamic ReadWriteMany Storage Provisioning \u2693\ufe0e In addition to the pre-defined PV/PVCs, the EOEPCA Reference Implementation also defines NFS-based storage classes for dynamic storage provisioning: managed-nfs-storage With a Reclaim Policy of Delete . managed-nfs-storage-retain With a Reclaim Policy of Retain . The building-blocks simply reference the required Storage Class in their volume specifications, to receive a Persistent Volume Claim that is dynamically provisioned at deployment time. This is acheived through the nfs-provisioner helm chart , with the following typical configurations\u2026 Reclaim Policy Delete \u2026 provisionerName : nfs-storage storageClass : name : managed-nfs-storage create : true reclaimPolicy : Delete archiveOnDelete : false allowVolumeExpansion : true nfs : server : \"<your-nfs-ip-address-here>\" path : /data/dynamic # your NFS server path here Reclaim Policy Retain \u2026 provisionerName : nfs-storage-retain storageClass : name : managed-nfs-storage-retain create : true reclaimPolicy : Retain allowVolumeExpansion : true nfs : server : \"<your-nfs-ip-address-here>\" path : /data/dynamic # your NFS server path here Clustered Storage Solutions \u2693\ufe0e Clustered storage approaches offer an alternative to NFS. Clustered Storage provides a network-attached storage through a set of commodity hosts whose storage is aggregated to form a distributed file-system. Capacity is scaled by adding additional nodes or adding additional storage to the existing nodes. In the context of a multi-node Kubernetes cluster, then it is typical that the same commodity nodes provide both the cluster members and storage resources, i.e. the clustered storage is spread across the Kubernetes worker nodes. Candidate clustered storage solutions include: GlusterFS GlusterFS is deployed as an operating system service across each node participating in the storage solution. Thus, with GlusterFS, the distributed storage nodes do not need to be one-and-the-same with the compute (cluster) nodes \u2013 although this may preferably be the case. Longhorn Longhorn offers a solution that is similar to that of GlusterFS, except that Longhorn is \u2018cloud-native\u2019 in that its service layer deploys within the Kubernetes cluster itself. Thus, the storage nodes are also the cluster compute nodes by design. All things being equal, Longhorn is recommended as the best approach for Kubernetes clusters. Local Cluster Storage \u2693\ufe0e For the purposes of the EOEPCA deployment, the default Storage Class included with the local Kubernetes distribution can be used for all storage concerns - e.g. standard for minikube which provides the ReadWriteMany persistence that is required by the ADES.","title":"Persistence"},{"location":"eoepca/persistence/#persistence","text":"The EOEPCA building-blocks rely upon Kubernetes Persistent Volumes for their component persistence. Components integrate with the storage provided in the cluster by means of configurable Persistent Volume Claims and/or dynamic Storage Class that are specfied as values at time of deployment. Some components require storage of type ReadWriteMany - which, for a multi-node cluster, implies a network-based storage solution.","title":"Persistence"},{"location":"eoepca/persistence/#readwritemany-storage","text":"For the EOEPCA development deployment, an NFS server has been established to provide the persistence layer for ReadWriteMany storage.","title":"ReadWriteMany Storage"},{"location":"eoepca/persistence/#pre-defined-persistent-volume-claims","text":"The EOEPCA development deployment establishes the following pre-defined Persistent Volume Claims, to provide a simple storage architecture that is organised around the \u2018domain areas\u2019 into which the Reference Implementation is split. Resource Managment ( resman ) - persistentvolumeclaim/eoepca-resman-pvc Processing & Chaining ( proc ) - persistentvolumeclaim/eoepca-proc-pvc User Management ( userman ) - persistentvolumeclaim/eoepca-userman-pvc NOTE that this is offered only as an example thay suits the approach of the development team. Each building-block has configuration through which its persistence (PV/PVC) can be configured according the needs of the deployment. The following Kubernetes yaml provides an example of provisioning such domain-specific PersistentVolumeClaims within the cluster - in this case using the minikube built-in storage-class standard for dynamic provisioning\u2026 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-proc-pvc namespace: proc spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-resman-pvc namespace: rm spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: eoepca-userman-pvc namespace: um spec: accessModes: - ReadWriteMany storageClassName: standard resources: requests: storage: 5Gi Once established, these PersistentVolumeClaims are then referenced within the deployment configurations of the building-blocks.","title":"Pre-defined Persistent Volume Claims"},{"location":"eoepca/persistence/#dynamic-readwritemany-storage-provisioning","text":"In addition to the pre-defined PV/PVCs, the EOEPCA Reference Implementation also defines NFS-based storage classes for dynamic storage provisioning: managed-nfs-storage With a Reclaim Policy of Delete . managed-nfs-storage-retain With a Reclaim Policy of Retain . The building-blocks simply reference the required Storage Class in their volume specifications, to receive a Persistent Volume Claim that is dynamically provisioned at deployment time. This is acheived through the nfs-provisioner helm chart , with the following typical configurations\u2026 Reclaim Policy Delete \u2026 provisionerName : nfs-storage storageClass : name : managed-nfs-storage create : true reclaimPolicy : Delete archiveOnDelete : false allowVolumeExpansion : true nfs : server : \"<your-nfs-ip-address-here>\" path : /data/dynamic # your NFS server path here Reclaim Policy Retain \u2026 provisionerName : nfs-storage-retain storageClass : name : managed-nfs-storage-retain create : true reclaimPolicy : Retain allowVolumeExpansion : true nfs : server : \"<your-nfs-ip-address-here>\" path : /data/dynamic # your NFS server path here","title":"Dynamic ReadWriteMany Storage Provisioning"},{"location":"eoepca/persistence/#clustered-storage-solutions","text":"Clustered storage approaches offer an alternative to NFS. Clustered Storage provides a network-attached storage through a set of commodity hosts whose storage is aggregated to form a distributed file-system. Capacity is scaled by adding additional nodes or adding additional storage to the existing nodes. In the context of a multi-node Kubernetes cluster, then it is typical that the same commodity nodes provide both the cluster members and storage resources, i.e. the clustered storage is spread across the Kubernetes worker nodes. Candidate clustered storage solutions include: GlusterFS GlusterFS is deployed as an operating system service across each node participating in the storage solution. Thus, with GlusterFS, the distributed storage nodes do not need to be one-and-the-same with the compute (cluster) nodes \u2013 although this may preferably be the case. Longhorn Longhorn offers a solution that is similar to that of GlusterFS, except that Longhorn is \u2018cloud-native\u2019 in that its service layer deploys within the Kubernetes cluster itself. Thus, the storage nodes are also the cluster compute nodes by design. All things being equal, Longhorn is recommended as the best approach for Kubernetes clusters.","title":"Clustered Storage Solutions"},{"location":"eoepca/persistence/#local-cluster-storage","text":"For the purposes of the EOEPCA deployment, the default Storage Class included with the local Kubernetes distribution can be used for all storage concerns - e.g. standard for minikube which provides the ReadWriteMany persistence that is required by the ADES.","title":"Local Cluster Storage"},{"location":"eoepca/resource-catalogue/","text":"Resource Catalogue \u2693\ufe0e The Resource Catalogue provides a standards-based EO metadata catalogue that includes support for OGC CSW / API Records, STAC and OpenSearch. Helm Chart \u2693\ufe0e The Resource Catalogue is deployed via the rm-resource-catalogue helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the rm-resource-catalogue chart . helm install --version 1 .2.0 --values resource-catalogue-values.yaml resource-catalogue eoepca/rm-resource-catalogue Values \u2693\ufe0e The Resource Catalogue supports many values to configure the service - as described in the Values section of the chart README . Typically, values for the following attributes may be specified: The fully-qualified public URL for the service Dynamic provisioning StorageClass for database persistence (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Resource Catalogue will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Metadata describing the Catalogue instance Tuning configuration for PostgreSQL - see values db.config.XXX . Example resource-catalogue-values.yaml \u2026 global : namespace : rm # For protected access disable this ingress, and rely upon the resource-guard # for ingress with protection. ingress : # Enabled for unprotected 'open' access to the resource-catalogue. enabled : true name : resource-catalogue host : resource-catalogue.192-168-49-2.nip.io tls_host : resource-catalogue.192-168-49-2.nip.io tls_secret_name : resource-catalogue-tls annotations : cert-manager.io/cluster-issuer : letsencrypt-production db : volume_storage_type : standard # config: # enabled: true # shared_buffers: 2GB # effective_cache_size: 6GB # maintenance_work_mem: 512MB # checkpoint_completion_target: 0.9 # wal_buffers: 16MB # default_statistics_target: 100 # random_page_cost: 4 # work_mem: 4MB # cpu_tuple_cost: 0.4 pycsw : config : server : url : https://resource-catalogue.192-168-49-2.nip.io/ Protection \u2693\ufe0e As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Resource Catalogue service to provide access authorization decisions helm install --version 1 .2.1 --values resource-catalogue-guard-values.yaml resource-catalogue-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Resource Catalogue for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example resource-catalogue-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : resource-catalogue domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : resource-catalogue paths : - path : /(.*) service : name : resource-catalogue-service port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint Client Secret \u2693\ufe0e The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io . In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml Resource Catalogue Usage \u2693\ufe0e The Resource Catalogue is initially populated during the initialisation of the Data Access service. See section Data-layer Configuration . The Resource Catalogue is accessed at the endpoint https://resource-catalogue.<domain>/ , configured by your domain - e.g. https://resource-catalogue.192-168-49-2.nip.io/ . Loading Records \u2693\ufe0e As described in the pycsw documentation , ISO XML records can be loaded into the resource-catalogue using the pycsw-admin.py admin utility\u2026 pycsw-admin.py load_records -c /path/to/cfg -p /path/to/records The /path/to/records can either be a single metadata file, or a directory containing multiple metadata files. This is most easily achieved via connection to the pycsw pod, which includes the pycsw-admin.py utility and the pycsw configuration file at /etc/pycsw/pycsw.cfg \u2026 kubectl -n rm cp \"<metadata-file-or-directory>\" \"<pycsw-pod-name>\" :/tmp/metadata kubectl -n rm exec -i \"<pycsw-pod-name>\" -- pycsw-admin.py load-records -c /etc/pycsw/pycsw.cfg -p /tmp/metadata The name of the pycsw pod can be obtained using kubectl \u2026 kubectl -n rm get pod --selector = 'io.kompose.service=pycsw' --output = jsonpath ={ .items [ 0 ] .metadata.name } To facilitate the loading of records via the pycsw pod, a helper script load-records has been provided in the git repository that hosts this document \u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide ./deploy/bin/load-records \"<metadata-file-or-directory>\" The helper script identifies the pycsw pod, copies the metadata files to the pod, and runs pycsw-admin.py load-records within the pod to load the records. Additional Information \u2693\ufe0e Additional information regarding the Resource Catalogue can be found at: Helm Chart pycsw Documentation GitHub Repository","title":"Resource Catalogue"},{"location":"eoepca/resource-catalogue/#resource-catalogue","text":"The Resource Catalogue provides a standards-based EO metadata catalogue that includes support for OGC CSW / API Records, STAC and OpenSearch.","title":"Resource Catalogue"},{"location":"eoepca/resource-catalogue/#helm-chart","text":"The Resource Catalogue is deployed via the rm-resource-catalogue helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the rm-resource-catalogue chart . helm install --version 1 .2.0 --values resource-catalogue-values.yaml resource-catalogue eoepca/rm-resource-catalogue","title":"Helm Chart"},{"location":"eoepca/resource-catalogue/#values","text":"The Resource Catalogue supports many values to configure the service - as described in the Values section of the chart README . Typically, values for the following attributes may be specified: The fully-qualified public URL for the service Dynamic provisioning StorageClass for database persistence (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Resource Catalogue will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Metadata describing the Catalogue instance Tuning configuration for PostgreSQL - see values db.config.XXX . Example resource-catalogue-values.yaml \u2026 global : namespace : rm # For protected access disable this ingress, and rely upon the resource-guard # for ingress with protection. ingress : # Enabled for unprotected 'open' access to the resource-catalogue. enabled : true name : resource-catalogue host : resource-catalogue.192-168-49-2.nip.io tls_host : resource-catalogue.192-168-49-2.nip.io tls_secret_name : resource-catalogue-tls annotations : cert-manager.io/cluster-issuer : letsencrypt-production db : volume_storage_type : standard # config: # enabled: true # shared_buffers: 2GB # effective_cache_size: 6GB # maintenance_work_mem: 512MB # checkpoint_completion_target: 0.9 # wal_buffers: 16MB # default_statistics_target: 100 # random_page_cost: 4 # work_mem: 4MB # cpu_tuple_cost: 0.4 pycsw : config : server : url : https://resource-catalogue.192-168-49-2.nip.io/","title":"Values"},{"location":"eoepca/resource-catalogue/#protection","text":"As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Resource Catalogue service to provide access authorization decisions helm install --version 1 .2.1 --values resource-catalogue-guard-values.yaml resource-catalogue-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Resource Catalogue for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example resource-catalogue-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : resource-catalogue domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : resource-catalogue paths : - path : /(.*) service : name : resource-catalogue-service port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint","title":"Protection"},{"location":"eoepca/resource-catalogue/#client-secret","text":"The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io . In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml","title":"Client Secret"},{"location":"eoepca/resource-catalogue/#resource-catalogue-usage","text":"The Resource Catalogue is initially populated during the initialisation of the Data Access service. See section Data-layer Configuration . The Resource Catalogue is accessed at the endpoint https://resource-catalogue.<domain>/ , configured by your domain - e.g. https://resource-catalogue.192-168-49-2.nip.io/ .","title":"Resource Catalogue Usage"},{"location":"eoepca/resource-catalogue/#loading-records","text":"As described in the pycsw documentation , ISO XML records can be loaded into the resource-catalogue using the pycsw-admin.py admin utility\u2026 pycsw-admin.py load_records -c /path/to/cfg -p /path/to/records The /path/to/records can either be a single metadata file, or a directory containing multiple metadata files. This is most easily achieved via connection to the pycsw pod, which includes the pycsw-admin.py utility and the pycsw configuration file at /etc/pycsw/pycsw.cfg \u2026 kubectl -n rm cp \"<metadata-file-or-directory>\" \"<pycsw-pod-name>\" :/tmp/metadata kubectl -n rm exec -i \"<pycsw-pod-name>\" -- pycsw-admin.py load-records -c /etc/pycsw/pycsw.cfg -p /tmp/metadata The name of the pycsw pod can be obtained using kubectl \u2026 kubectl -n rm get pod --selector = 'io.kompose.service=pycsw' --output = jsonpath ={ .items [ 0 ] .metadata.name } To facilitate the loading of records via the pycsw pod, a helper script load-records has been provided in the git repository that hosts this document \u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide ./deploy/bin/load-records \"<metadata-file-or-directory>\" The helper script identifies the pycsw pod, copies the metadata files to the pod, and runs pycsw-admin.py load-records within the pod to load the records.","title":"Loading Records"},{"location":"eoepca/resource-catalogue/#additional-information","text":"Additional information regarding the Resource Catalogue can be found at: Helm Chart pycsw Documentation GitHub Repository","title":"Additional Information"},{"location":"eoepca/resource-protection/","text":"Resource Protection \u2693\ufe0e EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes: Login Service (Authorization Server) Policy Decision Point (PDP) Policy Enforcement Point (PEP) Building Blocks that act as a Resource Server are individually protected by a Policy Enforcement Point (PEP). The PEP enforces the authorization decision in collaboration with the Login Service and Policy Decision Point (PDP). The PEP expects to interface to a client (user agent, e.g. browser) using User Managed Access (UMA) flows. It is not typical for a client to support UMA flows , and so the PEP can be deployed with a companion UMA User Agent component that interfaces between the client and the PEP, and performs the UMA Flow on behalf of the client. The Resource Guard is a \u2018convenience\u2019 component that deploys the PEP & UMA User Agent as a cooperating pair. The Resource Guard \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, the Resource Guard deploys with an Ingress specification that: Configures the auth_request module to defer access authorization to the uma-user-agent service Configures the ingress rules (host/path) for the target Resource Server Helm Chart \u2693\ufe0e The Resource Guard is deployed via the resource-guard helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the resource-guard chart . It is expected to deploy multiple instances of the Resource Guard chart, one for each Resource Server to be protected. helm install --version 1 .0.7 --values myservice-guard-values.yaml myservice-guard eoepca/resource-guard Values \u2693\ufe0e The helm chart is deployed with values that are passed through to the subcharts for the pep-engine and uma-user-agent . Typical values to be specified include: Host/domain details for the Login Service and PDP, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for pep-engine persistence, e.g. myservice-pep-pvc TLS Certificate Provider, e.g. letsencrypt-production Optional specification of default resources with which to initialise the policy database for the component Ingress rules definition for reverse-proxy to the target Resource Server Name of Secret that contains the client credentials used by the uma-user-agent to interface with the Login Service. See section Client Secret below Example myservice-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : myservice domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : workingMode : PARTIAL asHostname : auth pdpHostname : auth customDefaultResources : - name : \"Eric's space\" description : \"Protected Access for eric to his space in myservice\" resource_uri : \"/ericspace\" scopes : [] default_owner : \"d3688daa-385d-45b0-8e04-2062e3e2cd86\" nginxIntegration : enabled : false volumeClaim : name : myservice-pep-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : myservice paths : - path : /(.*) service : name : myservice port : 80 - path : /(doc.*) service : name : myservice-docs port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"myservice-agent\" logging : level : \"debug\" unauthorizedResponse : 'Bearer realm=\"https://auth.192-168-49-2.nip.io/oxauth/auth/passport/passportlogin.htm\"' #--------------------------------------------------------------------------- # END values #--------------------------------------------------------------------------- Client Credentials \u2693\ufe0e The uma-user-agent requires Client Credentials for its interactions with the login-service . The uma-user-agent expects to read these credentials from the file client.yaml , in the form\u2026 client-id: <my-client-id> client-secret: <my-secret> Client Registration \u2693\ufe0e To obtain the Client Credentials required by the uma-user-agent it is necessary to register a client with the login-service , or use the credentials for an existing client. A helper script is provided to register a basic client and obtain the required credentials. The script is available in the deployment-guide repository , and can be obtained as follows\u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide The register-client helper script requires some command-line arguments\u2026 Usage: register_client <authorization-server-hostname> <client-name> [<redirect-uri> [<logout-uri>]] For example\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient INFO: Preparing docker image... [ done ] Client successfully registered. Make a note of the credentials: client-id: a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret: 73914cfc-c7dd-4b54-8807-ce17c3645558 Or to register OIDC redirect URLs\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient https://portal.192-168-49-2.nip.io/oidc/callback/ https://portal.192-168-49-2.nip.io/logout The script writes the \u2018client credentials\u2019 to stdout - in the expected YAML configuration file format - which can be redirected to file\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient | tee client.yaml \u2026writes the client credentials to the file client.yaml . NOTE that the register-client helper relies upon docker to build and run the script. Client Secret \u2693\ufe0e The client.yaml configuration file is made available via a Kubernetes Secret\u2026 kubectl -n myservice-ns create secret generic myservice-agent \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > myservice-agent-secret.yaml apiVersion : v1 kind : Secret metadata : name : myservice-agent namespace : myservice-ns data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The resource-guard deployment is configured with the name of the Secret through the helm chart value client.credentialsSecretName . User ID Token \u2693\ufe0e As described in the README for the Resource Guard , it is necessary for a request to a protected resource to provide the User ID Token in the request header. Obtaining the User ID Token \u2693\ufe0e In the simple case of a user with username/password held within the Login Service, the User ID Token can be obtained as follows: curl --location --request POST 'https://auth.192-168-49-2.nip.io/oxauth/restv1/token' \\ --header 'Cache-Control: no-cache' \\ --header 'Content-Type: application/x-www-form-urlencoded' \\ --data-urlencode 'scope=openid user_name is_operator' \\ --data-urlencode 'grant_type=password' \\ --data-urlencode 'username=<username>' \\ --data-urlencode 'password=<password>' \\ --data-urlencode 'client_id=<client-id>' \\ --data-urlencode 'client_secret=<client-password>' The User ID Token is included in the id_token field of the json response. Alternatively, OAuth/OIDC flows can be followed to authenticate via external identity providers. User ID Token in HTTP requests \u2693\ufe0e The Resource Guard protection supports presentation of the User ID Token via the following HTTP request headers (in order of priority)\u2026 Authorization header as a bearer token - in the form: Authorization: Bearer <token> X-User-Id header Cookie: auth_user_id=<token> Note that the name of the cookie is configurable Additional Information \u2693\ufe0e Additional information regarding the Resource Guard can be found at: Helm Chart README GitHub Repository: pep-engine uma-user-agent","title":"Resource Protection"},{"location":"eoepca/resource-protection/#resource-protection","text":"EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes: Login Service (Authorization Server) Policy Decision Point (PDP) Policy Enforcement Point (PEP) Building Blocks that act as a Resource Server are individually protected by a Policy Enforcement Point (PEP). The PEP enforces the authorization decision in collaboration with the Login Service and Policy Decision Point (PDP). The PEP expects to interface to a client (user agent, e.g. browser) using User Managed Access (UMA) flows. It is not typical for a client to support UMA flows , and so the PEP can be deployed with a companion UMA User Agent component that interfaces between the client and the PEP, and performs the UMA Flow on behalf of the client. The Resource Guard is a \u2018convenience\u2019 component that deploys the PEP & UMA User Agent as a cooperating pair. The Resource Guard \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, the Resource Guard deploys with an Ingress specification that: Configures the auth_request module to defer access authorization to the uma-user-agent service Configures the ingress rules (host/path) for the target Resource Server","title":"Resource Protection"},{"location":"eoepca/resource-protection/#helm-chart","text":"The Resource Guard is deployed via the resource-guard helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the resource-guard chart . It is expected to deploy multiple instances of the Resource Guard chart, one for each Resource Server to be protected. helm install --version 1 .0.7 --values myservice-guard-values.yaml myservice-guard eoepca/resource-guard","title":"Helm Chart"},{"location":"eoepca/resource-protection/#values","text":"The helm chart is deployed with values that are passed through to the subcharts for the pep-engine and uma-user-agent . Typical values to be specified include: Host/domain details for the Login Service and PDP, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for pep-engine persistence, e.g. myservice-pep-pvc TLS Certificate Provider, e.g. letsencrypt-production Optional specification of default resources with which to initialise the policy database for the component Ingress rules definition for reverse-proxy to the target Resource Server Name of Secret that contains the client credentials used by the uma-user-agent to interface with the Login Service. See section Client Secret below Example myservice-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : myservice domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : workingMode : PARTIAL asHostname : auth pdpHostname : auth customDefaultResources : - name : \"Eric's space\" description : \"Protected Access for eric to his space in myservice\" resource_uri : \"/ericspace\" scopes : [] default_owner : \"d3688daa-385d-45b0-8e04-2062e3e2cd86\" nginxIntegration : enabled : false volumeClaim : name : myservice-pep-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : myservice paths : - path : /(.*) service : name : myservice port : 80 - path : /(doc.*) service : name : myservice-docs port : 80 annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"myservice-agent\" logging : level : \"debug\" unauthorizedResponse : 'Bearer realm=\"https://auth.192-168-49-2.nip.io/oxauth/auth/passport/passportlogin.htm\"' #--------------------------------------------------------------------------- # END values #---------------------------------------------------------------------------","title":"Values"},{"location":"eoepca/resource-protection/#client-credentials","text":"The uma-user-agent requires Client Credentials for its interactions with the login-service . The uma-user-agent expects to read these credentials from the file client.yaml , in the form\u2026 client-id: <my-client-id> client-secret: <my-secret>","title":"Client Credentials"},{"location":"eoepca/resource-protection/#client-registration","text":"To obtain the Client Credentials required by the uma-user-agent it is necessary to register a client with the login-service , or use the credentials for an existing client. A helper script is provided to register a basic client and obtain the required credentials. The script is available in the deployment-guide repository , and can be obtained as follows\u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide The register-client helper script requires some command-line arguments\u2026 Usage: register_client <authorization-server-hostname> <client-name> [<redirect-uri> [<logout-uri>]] For example\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient INFO: Preparing docker image... [ done ] Client successfully registered. Make a note of the credentials: client-id: a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret: 73914cfc-c7dd-4b54-8807-ce17c3645558 Or to register OIDC redirect URLs\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient https://portal.192-168-49-2.nip.io/oidc/callback/ https://portal.192-168-49-2.nip.io/logout The script writes the \u2018client credentials\u2019 to stdout - in the expected YAML configuration file format - which can be redirected to file\u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io myclient | tee client.yaml \u2026writes the client credentials to the file client.yaml . NOTE that the register-client helper relies upon docker to build and run the script.","title":"Client Registration"},{"location":"eoepca/resource-protection/#client-secret","text":"The client.yaml configuration file is made available via a Kubernetes Secret\u2026 kubectl -n myservice-ns create secret generic myservice-agent \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > myservice-agent-secret.yaml apiVersion : v1 kind : Secret metadata : name : myservice-agent namespace : myservice-ns data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The resource-guard deployment is configured with the name of the Secret through the helm chart value client.credentialsSecretName .","title":"Client Secret"},{"location":"eoepca/resource-protection/#user-id-token","text":"As described in the README for the Resource Guard , it is necessary for a request to a protected resource to provide the User ID Token in the request header.","title":"User ID Token"},{"location":"eoepca/resource-protection/#obtaining-the-user-id-token","text":"In the simple case of a user with username/password held within the Login Service, the User ID Token can be obtained as follows: curl --location --request POST 'https://auth.192-168-49-2.nip.io/oxauth/restv1/token' \\ --header 'Cache-Control: no-cache' \\ --header 'Content-Type: application/x-www-form-urlencoded' \\ --data-urlencode 'scope=openid user_name is_operator' \\ --data-urlencode 'grant_type=password' \\ --data-urlencode 'username=<username>' \\ --data-urlencode 'password=<password>' \\ --data-urlencode 'client_id=<client-id>' \\ --data-urlencode 'client_secret=<client-password>' The User ID Token is included in the id_token field of the json response. Alternatively, OAuth/OIDC flows can be followed to authenticate via external identity providers.","title":"Obtaining the User ID Token"},{"location":"eoepca/resource-protection/#user-id-token-in-http-requests","text":"The Resource Guard protection supports presentation of the User ID Token via the following HTTP request headers (in order of priority)\u2026 Authorization header as a bearer token - in the form: Authorization: Bearer <token> X-User-Id header Cookie: auth_user_id=<token> Note that the name of the cookie is configurable","title":"User ID Token in HTTP requests"},{"location":"eoepca/resource-protection/#additional-information","text":"Additional information regarding the Resource Guard can be found at: Helm Chart README GitHub Repository: pep-engine uma-user-agent","title":"Additional Information"},{"location":"eoepca/user-profile/","text":"User Profile \u2693\ufe0e The User Profile represents the user\u2019s \u2018account\u2019 within the platform. Helm Chart \u2693\ufe0e The User Profile is deployed via the user-profile helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the user-profile chart . helm install --version 1 .1.6 --values user-profile-values.yaml user-profile eoepca/user-profile Values \u2693\ufe0e At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for user-profile persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. Example user-profile-values.yaml \u2026 global : domain : auth.192-168-49-2.nip.io nginxIp : 192.168.49.2 volumeClaim : name : eoepca-userman-pvc create : false User Profile Usage \u2693\ufe0e The User Profile is accessed through the /web_ui path of the Login Service, e.g. http://auth.kube.guide.eoepca.org/web_ui. Additional Information \u2693\ufe0e Additional information regarding the User Profile can be found at: Helm Chart Wiki GitHub Repository","title":"User Profile"},{"location":"eoepca/user-profile/#user-profile","text":"The User Profile represents the user\u2019s \u2018account\u2019 within the platform.","title":"User Profile"},{"location":"eoepca/user-profile/#helm-chart","text":"The User Profile is deployed via the user-profile helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the user-profile chart . helm install --version 1 .1.6 --values user-profile-values.yaml user-profile eoepca/user-profile","title":"Helm Chart"},{"location":"eoepca/user-profile/#values","text":"At minimum, values for the following attributes should be specified: Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2 Name of Persistent Volume Claim for user-profile persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage. Example user-profile-values.yaml \u2026 global : domain : auth.192-168-49-2.nip.io nginxIp : 192.168.49.2 volumeClaim : name : eoepca-userman-pvc create : false","title":"Values"},{"location":"eoepca/user-profile/#user-profile-usage","text":"The User Profile is accessed through the /web_ui path of the Login Service, e.g. http://auth.kube.guide.eoepca.org/web_ui.","title":"User Profile Usage"},{"location":"eoepca/user-profile/#additional-information","text":"Additional information regarding the User Profile can be found at: Helm Chart Wiki GitHub Repository","title":"Additional Information"},{"location":"eoepca/workspace/","text":"Workspace \u2693\ufe0e The Workspace provides protected user resource management that includes dedicated storage and services for resource discovery and access. Workspace API \u2693\ufe0e The Workspace API provides a REST service through which user workspaces can be created, interrogated, managed and deleted. Helm Chart \u2693\ufe0e The Workspace API is deployed via the rm-workspace-api helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the um-workspace-api chart . helm install --version 1 .2.0 --values workspace-api-values.yaml workspace-api eoepca/rm-workspace-api Values \u2693\ufe0e At minimum, values for the following attributes should be specified: The fully-qualified public URL for the service (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Workspace API will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Prefix for user projects in OpenStack Details for underlying S3 object storage service Identification of secret that provides the client credentials for resource protection Whether flux components should be installed - otherwise they must already be present - Flux Dependency Name of the ConfigMap for user workspace templates - See User Workspace Templates Example workspace-api-values.yaml \u2026 fullnameOverride : workspace-api ingress : enabled : true hosts : - host : workspace-api-open.192-168-49-2.nip.io paths : [ \"/\" ] tls : - hosts : - workspace-api-open.192-168-49-2.nip.io secretName : workspace-api-open-tls fluxHelmOperator : enabled : true prefixForName : \"guide-user\" workspaceSecretName : \"bucket\" namespaceForBucketResource : \"rm\" s3Endpoint : \"https://cf2.cloudferro.com:8080\" s3Region : \"RegionOne\" harborUrl : \"https://harbor.192-168-49-2.nip.io\" harborUsername : \"admin\" harborPassword : \"changeme\" umaClientSecretName : \"resman-client\" umaClientSecretNamespace : \"rm\" workspaceChartsConfigMap : \"workspace-charts\" NOTES: The Workspace API assumes a deployment of the Harbor Container Regsitry, as configured by the harborXXX values above. See section Container Registry . Flux Dependency \u2693\ufe0e Workspaces are created by instantiating the rm-user-workspace helm chart for each user/group. The Workspace API uses Flux CD as a helper to manage these subordinate helm charts - via flux resources of type HelmRelease . Thus, it is necessary to deploy within the cluster the aspects of flux that support this helm chart management - namely the flux helm-controller , source-controller and the Kubernetes Custom Resource Definitions (CRD) for HelmRelease and HelmRepository . In case you are not already using flux within your clsuter, then the Workspace API helm chart can be configured to deploy the required flux components\u2026 fluxHelmOperator: enabled: true # true = install flux for me, false = I already have flux User Workspace Templates \u2693\ufe0e The Workspace API instantiates for each user a set of services, including a Resource Catalogue and Data Access services. These user services are instantiated via helm using templates. The templates are provided to the Workspace API in a ConfigMap that is, by default, named workspace-charts . Each file in the config-map is expected to be of kind HelmRelease . During creation of a new workspace, the Worksapce API applies each file to the cluster in the namespace of the newly created namespace. The default ConfigMap that is included with this guide contains the following templates: Data Access : template-hr-data-access.yaml Resource Catalogue : template-hr-resource-catalogue.yaml Protection : template-hr-resource-guard.yaml Each of these templates is expressed as a flux HelmRelease object that describes the helm chart and values required to deploy the service. Templates ConfigMap \u2693\ufe0e The templates are provided to the Workspace API as a ConfigMap in the namespace of the Workspace API deployment\u2026 (for full examples see https://github.com/ EOEPCA /deployment-guide/tree/main/deploy/eoepca/workspace-templates ) apiVersion : v1 kind : ConfigMap metadata : name : workspace-charts data : template-hr-resource-catalogue.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: rm-resource-catalogue spec: interval: 5m chart: spec: chart: rm-resource-catalogue version: 1.2.0 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... template-hr-data-access.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: vs spec: interval: 5m chart: spec: chart: data-access version: 1.2.5 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... template-hr-resource-guard.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: resource-guard spec: interval: 5m chart: spec: chart: resource-guard version: 1.2.1 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... HelmRepositories for Templates \u2693\ufe0e As can be seen above, the HelmRelease templates rely upon objects of type HelmRepository that define the hosting helm chart repository. Thus, in support of the workspace templates, appropriate HelmRepository objects must be provisioned within the cluster. For example, in support of the above examples that rely upon the EOEPCA Helm Chart Repository \u2026 apiVersion : source.toolkit.fluxcd.io/v1beta1 kind : HelmRepository metadata : name : eoepca namespace : rm spec : interval : 2m url : https://eoepca.github.io/helm-charts/ Helm Template Parameters \u2693\ufe0e The Workspace API uses the jinja2 templating engine when applying the HelmReleases for a user workspace. The current parameters are currently supported: workspace_name The name of the workspace - {{ workspace_name }} used to ensure unique naming of cluster resources, such as service ingress default_owner The uuid of the owner of the workspace - {{ default_owner }} used to initialise the workspace protection Protection \u2693\ufe0e As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Workspace API service to provide access authorization decisions helm install --version 1 .2.1 --values workspace-api-guard-values.yaml workspace-api-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Workspace API for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example workspace-api-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : workspace-api domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth # customDefaultResources: # - name: \"Eric's workspace\" # description: \"Protected Access for eric to his user workspace\" # resource_uri: \"/workspaces/guide-user-eric\" # scopes: [] # default_owner: \"d3688daa-385d-45b0-8e04-2062e3e2cd86\" # - name: \"Bob's workspace\" # description: \"Protected Access for bob to his user workspace\" # resource_uri: \"/workspaces/guide-user-bob\" # scopes: [] # default_owner: \"f12c2592-0332-49f4-a4fb-7063b3c2a889\" volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : workspace-api paths : - path : /(.*) service : name : workspace-api port : http annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint Client Secret \u2693\ufe0e The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io . In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml Workspace API Usage \u2693\ufe0e The Workspace API provides a REST interface that is accessed at the endpoint https://workspace-api.192-168-49-2.nip.io/. See the Swagger Docs . Additional Information \u2693\ufe0e Additional information regarding the Workspace API can be found at: Helm Chart Wiki GitHub Repository Bucket Operator \u2693\ufe0e The Workspace API creates workspaces for individual users. In doing so, dedicated object storage buckets are created associated to each user workspace - for self-contained storage of user owned resources (data, processing applications, etc.). The bucket creation relies upon the object storage services of the underlying cloud infrastructure. We have created a Bucket abstraction as a Kubernetes Custom Resource Definition . This is served by a Bucket Operator service that deploys into the Kubernetes cluster to satisfy requests for resources of type Bucket . We provide a Bucket Operator implementation that currently supports the creation of buckets in OpenStack object storage - currently tested only on the CREODIAS (Cloudferro). The Bucket Operator is deployed via the rm-bucket-operator helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the um-bucket-operator chart . helm install --version 0 .9.9 --values bucket-operator-values.yaml bucket-operator eoepca/rm-bucket-operator Values \u2693\ufe0e At minimum, values for the following attributes should be specified: The fully-qualified public URL for the service OpenStack access details Cluster Issuer for TLS Example bucket-operator-values.yaml \u2026 domain : 192-168-49-2.nip.io data : OS_MEMBERROLEID : \"9ee2ff9ee4384b1894a90878d3e92bab\" OS_SERVICEPROJECTID : \"d21467d0a0414252a79e29d38f03ff98\" USER_EMAIL_PATTERN : \"eoepca+<name>@192-168-49-2.nip.io\" ingress : annotations : cert-manager.io/cluster-issuer : letsencrypt-production kubernetes.io/ingress.class : nginx nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" OpenStack Secret \u2693\ufe0e The Bucket Operator requires privileged access to the OpenStack API, for which credentials are required. These are provided via a Kubernetes secret named openstack created in the namespace of the Bucket Operator. For example\u2026 kubectl -n rm create secret generic openstack \\ --from-literal=username=\"${OS_USERNAME}\" \\ --from-literal=password=\"${OS_PASSWORD}\" \\ --from-literal=domainname=\"${OS_DOMAINNAME}\" See the README for the Bucket Operator , which describes the configuration required for integration with your OpenStack account. For a worked example see our Scripted Example Deployment - in particular: Openstack Configuration Deployment Script","title":"Workspace"},{"location":"eoepca/workspace/#workspace","text":"The Workspace provides protected user resource management that includes dedicated storage and services for resource discovery and access.","title":"Workspace"},{"location":"eoepca/workspace/#workspace-api","text":"The Workspace API provides a REST service through which user workspaces can be created, interrogated, managed and deleted.","title":"Workspace API"},{"location":"eoepca/workspace/#helm-chart","text":"The Workspace API is deployed via the rm-workspace-api helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the um-workspace-api chart . helm install --version 1 .2.0 --values workspace-api-values.yaml workspace-api eoepca/rm-workspace-api","title":"Helm Chart"},{"location":"eoepca/workspace/#values","text":"At minimum, values for the following attributes should be specified: The fully-qualified public URL for the service (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Workspace API will not be protected by the resource-guard component - ref. Resource Protection . Otherwise the ingress will be handled by the resource-guard - use ingress.enabled: false . Prefix for user projects in OpenStack Details for underlying S3 object storage service Identification of secret that provides the client credentials for resource protection Whether flux components should be installed - otherwise they must already be present - Flux Dependency Name of the ConfigMap for user workspace templates - See User Workspace Templates Example workspace-api-values.yaml \u2026 fullnameOverride : workspace-api ingress : enabled : true hosts : - host : workspace-api-open.192-168-49-2.nip.io paths : [ \"/\" ] tls : - hosts : - workspace-api-open.192-168-49-2.nip.io secretName : workspace-api-open-tls fluxHelmOperator : enabled : true prefixForName : \"guide-user\" workspaceSecretName : \"bucket\" namespaceForBucketResource : \"rm\" s3Endpoint : \"https://cf2.cloudferro.com:8080\" s3Region : \"RegionOne\" harborUrl : \"https://harbor.192-168-49-2.nip.io\" harborUsername : \"admin\" harborPassword : \"changeme\" umaClientSecretName : \"resman-client\" umaClientSecretNamespace : \"rm\" workspaceChartsConfigMap : \"workspace-charts\" NOTES: The Workspace API assumes a deployment of the Harbor Container Regsitry, as configured by the harborXXX values above. See section Container Registry .","title":"Values"},{"location":"eoepca/workspace/#flux-dependency","text":"Workspaces are created by instantiating the rm-user-workspace helm chart for each user/group. The Workspace API uses Flux CD as a helper to manage these subordinate helm charts - via flux resources of type HelmRelease . Thus, it is necessary to deploy within the cluster the aspects of flux that support this helm chart management - namely the flux helm-controller , source-controller and the Kubernetes Custom Resource Definitions (CRD) for HelmRelease and HelmRepository . In case you are not already using flux within your clsuter, then the Workspace API helm chart can be configured to deploy the required flux components\u2026 fluxHelmOperator: enabled: true # true = install flux for me, false = I already have flux","title":"Flux Dependency"},{"location":"eoepca/workspace/#user-workspace-templates","text":"The Workspace API instantiates for each user a set of services, including a Resource Catalogue and Data Access services. These user services are instantiated via helm using templates. The templates are provided to the Workspace API in a ConfigMap that is, by default, named workspace-charts . Each file in the config-map is expected to be of kind HelmRelease . During creation of a new workspace, the Worksapce API applies each file to the cluster in the namespace of the newly created namespace. The default ConfigMap that is included with this guide contains the following templates: Data Access : template-hr-data-access.yaml Resource Catalogue : template-hr-resource-catalogue.yaml Protection : template-hr-resource-guard.yaml Each of these templates is expressed as a flux HelmRelease object that describes the helm chart and values required to deploy the service.","title":"User Workspace Templates"},{"location":"eoepca/workspace/#templates-configmap","text":"The templates are provided to the Workspace API as a ConfigMap in the namespace of the Workspace API deployment\u2026 (for full examples see https://github.com/ EOEPCA /deployment-guide/tree/main/deploy/eoepca/workspace-templates ) apiVersion : v1 kind : ConfigMap metadata : name : workspace-charts data : template-hr-resource-catalogue.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: rm-resource-catalogue spec: interval: 5m chart: spec: chart: rm-resource-catalogue version: 1.2.0 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... template-hr-data-access.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: vs spec: interval: 5m chart: spec: chart: data-access version: 1.2.5 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ... template-hr-resource-guard.yaml : | apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: resource-guard spec: interval: 5m chart: spec: chart: resource-guard version: 1.2.1 sourceRef: kind: HelmRepository name: eoepca namespace: rm values: ...","title":"Templates ConfigMap"},{"location":"eoepca/workspace/#helmrepositories-for-templates","text":"As can be seen above, the HelmRelease templates rely upon objects of type HelmRepository that define the hosting helm chart repository. Thus, in support of the workspace templates, appropriate HelmRepository objects must be provisioned within the cluster. For example, in support of the above examples that rely upon the EOEPCA Helm Chart Repository \u2026 apiVersion : source.toolkit.fluxcd.io/v1beta1 kind : HelmRepository metadata : name : eoepca namespace : rm spec : interval : 2m url : https://eoepca.github.io/helm-charts/","title":"HelmRepositories for Templates"},{"location":"eoepca/workspace/#helm-template-parameters","text":"The Workspace API uses the jinja2 templating engine when applying the HelmReleases for a user workspace. The current parameters are currently supported: workspace_name The name of the workspace - {{ workspace_name }} used to ensure unique naming of cluster resources, such as service ingress default_owner The uuid of the owner of the workspace - {{ default_owner }} used to initialise the workspace protection","title":"Helm Template Parameters"},{"location":"eoepca/workspace/#protection","text":"As described in section Resource Protection , the resource-guard component can be inserted into the request path of the Workspace API service to provide access authorization decisions helm install --version 1 .2.1 --values workspace-api-guard-values.yaml workspace-api-guard eoepca/resource-guard The resource-guard must be configured with the values applicable to the Workspace API for the Policy Enforcement Point ( pep-engine ) and the UMA User Agent ( uma-user-agent )\u2026 Example workspace-api-guard-values.yaml \u2026 #--------------------------------------------------------------------------- # Global values #--------------------------------------------------------------------------- global : context : workspace-api domain : 192-168-49-2.nip.io nginxIp : 192.168.49.2 certManager : clusterIssuer : letsencrypt-production #--------------------------------------------------------------------------- # PEP values #--------------------------------------------------------------------------- pep-engine : configMap : asHostname : auth pdpHostname : auth # customDefaultResources: # - name: \"Eric's workspace\" # description: \"Protected Access for eric to his user workspace\" # resource_uri: \"/workspaces/guide-user-eric\" # scopes: [] # default_owner: \"d3688daa-385d-45b0-8e04-2062e3e2cd86\" # - name: \"Bob's workspace\" # description: \"Protected Access for bob to his user workspace\" # resource_uri: \"/workspaces/guide-user-bob\" # scopes: [] # default_owner: \"f12c2592-0332-49f4-a4fb-7063b3c2a889\" volumeClaim : name : eoepca-resman-pvc create : false #--------------------------------------------------------------------------- # UMA User Agent values #--------------------------------------------------------------------------- uma-user-agent : nginxIntegration : enabled : true hosts : - host : workspace-api paths : - path : /(.*) service : name : workspace-api port : http annotations : nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\" nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/rewrite-target : /$1 client : credentialsSecretName : \"resman-client\" logging : level : \"info\" unauthorizedResponse : 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"' openAccess : false insecureTlsSkipVerify : true NOTES: TLS is enabled by the specification of certManager.clusterIssuer The letsencrypt Cluster Issuer relies upon the deployment being accessible from the public internet via the global.domain DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller insecureTlsSkipVerify may be required in the case that good TLS certificates cannot be established, e.g. if letsencrypt cannot be used for a local deployment. Otherwise the certificates offered by login-service Authorization Server will fail validation in the Resource Guard . customDefaultResources can be specified to apply initial protection to the endpoint","title":"Protection"},{"location":"eoepca/workspace/#client-secret","text":"The Resource Guard requires confidential client credentials to be configured through the file client.yaml , delivered via a kubernetes secret.. Example client.yaml \u2026 client-id : a98ba66e-e876-46e1-8619-5e130a38d1a4 client-secret : 73914cfc-c7dd-4b54-8807-ce17c3645558 Example Secret \u2026 kubectl -n rm create secret generic resman-client \\ --from-file = client.yaml \\ --dry-run = client -o yaml \\ > resman-client-secret.yaml apiVersion : v1 kind : Secret metadata : name : resman-client namespace : rm data : client.yaml : Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4 The client credentials are obtained by registration of a client at the login service web interface - e.g. https://auth.192-168-49-2.nip.io . In addition there is a helper script that can be used to create a basic client and obtain the credentials, as described in section Resource Protection \u2026 ./deploy/bin/register-client auth.192-168-49-2.nip.io \"Resource Guard\" | tee client.yaml","title":"Client Secret"},{"location":"eoepca/workspace/#workspace-api-usage","text":"The Workspace API provides a REST interface that is accessed at the endpoint https://workspace-api.192-168-49-2.nip.io/. See the Swagger Docs .","title":"Workspace API Usage"},{"location":"eoepca/workspace/#additional-information","text":"Additional information regarding the Workspace API can be found at: Helm Chart Wiki GitHub Repository","title":"Additional Information"},{"location":"eoepca/workspace/#bucket-operator","text":"The Workspace API creates workspaces for individual users. In doing so, dedicated object storage buckets are created associated to each user workspace - for self-contained storage of user owned resources (data, processing applications, etc.). The bucket creation relies upon the object storage services of the underlying cloud infrastructure. We have created a Bucket abstraction as a Kubernetes Custom Resource Definition . This is served by a Bucket Operator service that deploys into the Kubernetes cluster to satisfy requests for resources of type Bucket . We provide a Bucket Operator implementation that currently supports the creation of buckets in OpenStack object storage - currently tested only on the CREODIAS (Cloudferro). The Bucket Operator is deployed via the rm-bucket-operator helm chart from the EOEPCA Helm Chart Repository . The chart is configured via values that are fully documented in the README for the um-bucket-operator chart . helm install --version 0 .9.9 --values bucket-operator-values.yaml bucket-operator eoepca/rm-bucket-operator","title":"Bucket Operator"},{"location":"eoepca/workspace/#values_1","text":"At minimum, values for the following attributes should be specified: The fully-qualified public URL for the service OpenStack access details Cluster Issuer for TLS Example bucket-operator-values.yaml \u2026 domain : 192-168-49-2.nip.io data : OS_MEMBERROLEID : \"9ee2ff9ee4384b1894a90878d3e92bab\" OS_SERVICEPROJECTID : \"d21467d0a0414252a79e29d38f03ff98\" USER_EMAIL_PATTERN : \"eoepca+<name>@192-168-49-2.nip.io\" ingress : annotations : cert-manager.io/cluster-issuer : letsencrypt-production kubernetes.io/ingress.class : nginx nginx.ingress.kubernetes.io/enable-cors : \"true\" nginx.ingress.kubernetes.io/proxy-read-timeout : \"600\"","title":"Values"},{"location":"eoepca/workspace/#openstack-secret","text":"The Bucket Operator requires privileged access to the OpenStack API, for which credentials are required. These are provided via a Kubernetes secret named openstack created in the namespace of the Bucket Operator. For example\u2026 kubectl -n rm create secret generic openstack \\ --from-literal=username=\"${OS_USERNAME}\" \\ --from-literal=password=\"${OS_PASSWORD}\" \\ --from-literal=domainname=\"${OS_DOMAINNAME}\" See the README for the Bucket Operator , which describes the configuration required for integration with your OpenStack account. For a worked example see our Scripted Example Deployment - in particular: Openstack Configuration Deployment Script","title":"OpenStack Secret"},{"location":"quickstart/creodias-deployment/","text":"CREODIAS Deployment \u2693\ufe0e Based upon our development experiences on CREODIAS, there is a wrapper script creodias with particular customisations suited to the CREODIAS infrastructure and data offering. The customisations are expressed through environment variables that are captured in the file creodias-options . These scripts are examples that can be seen as a starting point, from which they can be adapted to your needs. The CREODIAS deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Protected service endpoints requiring IAM authorization See Endpoint Protection below for further information With reference to the file creodias-options , particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment\u2026 Passwords: LOGIN_SERVICE_ADMIN_PASSWORD , MINIO_ROOT_PASSWORD , HARBOR_ADMIN_PASSWORD OpenStack details: see section Openstack Configuration If configuring an external deployment - ref. Public Deployment \u2026 public_ip - The public IP address through which the deployment is exposed via the ingress-controller domain - The DNS domain name through which the deployment is accessed - forming the stem for all service hostnames in the ingress rules Once the file creodias-options has been well populated for your environment, then the deployment is initiated with\u2026 ./deploy/creodias/creodias \u2026noting that this step is a customised version of that described in section Deployment . Endpoint Protection \u2693\ufe0e Similarly the script creodias-protection is a customised version of that described in section Apply Protection . Once the main deployment has completed, then the test users can be created , their IDs ( Inum ) set in script creodias-protection , and the resource protection can then be applied\u2026 ./deploy/creodias/creodias-protection Harvest CREODIAS Data \u2693\ufe0e The harvester can be deployed with a default configuration file at /config.yaml . As described in the Data Access section , harvesting according to this configuration can be triggered with\u2026 kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch See the Harvester section below for an explanation of this harvester configuration. Data Specification Walkthrough \u2693\ufe0e The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables . This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example. Harvester \u2693\ufe0e The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment. The harvester is configured as follows\u2026 harvester: image: repository: eoepca/rm-harvester tag: 1.1.0 config: redis: host: data-access-redis-master port: 6379 harvesters: - name: Creodias-Opensearch resource: url: https://finder.creodias.eu/resto/api/collections/Sentinel2/describe.xml type: OpenSearch format_config: type: 'application/json' property_mapping: start_datetime: 'startDate' end_datetime: 'completionDate' productIdentifier: 'productIdentifier' query: time: property: sensed begin: 2019-09-10T00:00:00Z end: 2019-09-11T00:00:00Z collection: null bbox: 14.9,47.7,16.4,48.7 filter: {} postprocess: - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor queue: register_queue Based upon this harvester configuration we expect that the following query is made to discover data - i.e. an OpenSearch query, with json response representation, for a defined spatial and temporal extent\u2026 https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2019-09-10T00:00:00Z&completionDate=2019-09-11T00:00:00Z&box=14.9,47.7,16.4,48.7 From the result returned, the path to each product ( feature ) is obtained from the productIdentifier property, e.g. { \"type\": \"FeatureCollection\", \"features\": [ { \"type\": \"Feature\", \"properties\": { \"productIdentifier\": \"/eodata/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE\" ... } ... } ... ] } The harvester is configured with a Sentinel-2/CREODIAS specific post-processor harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor which transforms the product path from /eodata/... to s3://EODATA/... . The harvester post-processor follows this path to the Sentinel-2 scene and uses stactools (with built-in support for Sentinel-2) to establish a STAC item representing the product. This includes enumeration of assets for inspire-metadata and product-metadata - which are used by the registrar pycsw backend to embelesh the product record metadata. The harvester outputs the STAC item for each product, which is pushed to the registrar via the register_queue redis queue. Registration \u2693\ufe0e The registrar is configured at deployment to have the access details for the CREODIAS data in S3\u2026 global: storage: data: data: type: S3 endpoint_url: http://data.cloudferro.com access_key_id: access secret_access_key: access region_name: RegionOne validate_bucket_name: false Using this S3 configuration, the registrar pycsw backend uses the product metadata linked in the STAC item (ref. assets inspire-metadata and product-metadata ) to embelesh the metadata. For example, product-metadata in the file\u2026 s3://EODATA/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE/MTD_MSIL1C.xml The registrar uses this information to create the ISO XML metadata that is loaded into the resource-catalogue. Product Type \u2693\ufe0e The registrar recognises the product as Sentinel-2 and so reads its metadata XML files to obtain additional information. From the metadata XML file (e.g. MTD_MSIL1C.xml ) the registrar obtains the Product Type for each product from the field <PRODUCT_TYPE> \u2026 <n1:Level-1C_User_Product> <n1:General_Info> <Product_Info> <PRODUCT_TYPE>S2MSI1C</PRODUCT_TYPE> ... </Product_Info> ... </n1:General_Info> ... <n1:Level-1C_User_Product> Resource Catalogue Collections \u2693\ufe0e The registrar ( eoepca/rm-data-access-core ) container image is pre-loaded with two collections at the path /registrar_pycsw/registrar_pycsw/resources , (in the built container the files are at the path /usr/local/lib/python3.8/dist-packages/registrar_pycsw/resources/ ): S2MSI1C.yml - identifier: S2MSI1C S2MSI2A.yml - identifier: S2MSI2A The registrar applies these collections into the resource-catalogue during start-up - to create pre-defined out-of-the-box collections in pycsw. During registration, the PycswBackend of the registrar uses the Product Type to map the product into the collection of the same name - using metadata field parentidentifier . Data Specification \u2693\ufe0e The data-access service data handling is configured by definition of productTypes , collections and layers \u2026 productTypes identify the underlying file assets as WCS coverages and their visual representation collections provide groupings into which products are organised layers specifies the hoe the product visual representations are exposed through the WMS service productType \u2693\ufe0e During registration, products are mapped into a productType via a filter that is applied against the STAC item metadata. The registrar uses the product_type of each product to determine the collection into which the product should be registered - noting that the name of the product type does not take part in the matching logic (and hence can be any text name)\u2026 productTypes: - name: S2MSI1C filter: s2:product_type: S2MSI1C In the above example, the field s2:product_type is populated by the stactools that prepares the STAC item from the product metadata. productType - coverages \u2693\ufe0e coverages defines the coverages for the WCS service. Each coverage links to the assets that are defined within the product STAC item. productType - browses \u2693\ufe0e browses defines the images that are visualised in the View Server Client. Expressions are used to map the product assets into their visual representation. collections \u2693\ufe0e Collections are defined by reference to the defined productTypes and coverages . layers \u2693\ufe0e layers defines the layers that are presented through the WMS service - each layer being linked to the underlying browse that provides the image source. Layers are defined via their id that relies upon the naming convection <collection>__<browse> to identify the browse and so define the layer. Example Configuration \u2693\ufe0e Example configuration for Sentinel-2 L1C and L2A data. global : layers : - id : S2L1C title : Sentinel-2 Level 1C True Color abstract : Sentinel-2 Level 2A True Color displayColor : '#eb3700' grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__TRUE_COLOR title : Sentinel-2 Level 1C True Color abstract : Sentinel-2 Level 2A True Color grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__masked_clouds title : Sentinel-2 Level 1C True Color with cloud masks abstract : Sentinel-2 Level 1C True Color with cloud masks grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__FALSE_COLOR title : Sentinel-2 Level 1C False Color abstract : Sentinel-2 Level 1C False Color grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__NDVI title : Sentinel-2 Level 21CNDVI abstract : Sentinel-2 Level 1C NDVI grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L2A title : Sentinel-2 Level 2A True Color abstract : Sentinel-2 Level 2A True Color displayColor : '#eb3700' grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__TRUE_COLOR title : Sentinel-2 Level 2A True Color abstract : Sentinel-2 Level 2A True Color grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__masked_clouds title : Sentinel-2 Level 2A True Color with cloud masks abstract : Sentinel-2 Level 2A True Color with cloud masks grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__FALSE_COLOR title : Sentinel-2 Level 2A False Color abstract : Sentinel-2 Level 2A False Color grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__NDVI title : Sentinel-2 Level 2A NDVI abstract : Sentinel-2 Level 2A NDVI grids : - name : WGS84 zoom : 13 parentLayer : S2L2A collections : S2L1C : product_types : - S2MSI1C coverage_types : - S2L1C_B01 - S2L1C_B02 - S2L1C_B03 - S2L1C_B04 - S2L1C_B05 - S2L1C_B06 - S2L1C_B07 - S2L1C_B08 - S2L1C_B8A - S2L1C_B09 - S2L1C_B10 - S2L1C_B11 - S2L1C_B12 S2L2A : product_types : - S2MSI2A product_levels : - Level-2A coverage_types : - S2L2A_B01 - S2L2A_B02 - S2L2A_B03 - S2L2A_B04 - S2L2A_B05 - S2L2A_B06 - S2L2A_B07 - S2L2A_B08 - S2L2A_B8A - S2L2A_B09 - S2L2A_B11 - S2L2A_B12 productTypes : - name : S2MSI1C filter : s2:product_type : S2MSI1C metadata_assets : [] coverages : S2L1C_B01 : assets : - B01 S2L1C_B02 : assets : - B02 S2L1C_B03 : assets : - B03 S2L1C_B04 : assets : - B04 S2L1C_B05 : assets : - B05 S2L1C_B06 : assets : - B06 S2L1C_B07 : assets : - B07 S2L1C_B08 : assets : - B08 S2L1C_B8A : assets : - B8A S2L1C_B09 : assets : - B09 S2L1C_B10 : assets : - B10 S2L1C_B11 : assets : - B11 S2L1C_B12 : assets : - B12 defaultBrowse : TRUE_COLOR browses : TRUE_COLOR : asset : visual red : expression : B04 range : [ 0 , 4000 ] nodata : 0 green : expression : B03 range : [ 0 , 4000 ] nodata : 0 blue : expression : B02 range : [ 0 , 4000 ] nodata : 0 FALSE_COLOR : red : expression : B08 range : [ 0 , 4000 ] nodata : 0 green : expression : B04 range : [ 0 , 4000 ] nodata : 0 blue : expression : B03 range : [ 0 , 4000 ] nodata : 0 NDVI : grey : expression : (B08-B04)/(B08+B04) range : [ -1 , 1 ] masks : clouds : validity : false - name : S2MSI2A filter : s2:product_type : S2MSI2A metadata_assets : [] coverages : S2L2A_B01 : assets : - B01 S2L2A_B02 : assets : - B02 S2L2A_B03 : assets : - B03 S2L2A_B04 : assets : - B04 S2L2A_B05 : assets : - B05 S2L2A_B06 : assets : - B06 S2L2A_B07 : assets : - B07 S2L2A_B08 : assets : - B08 S2L2A_B8A : assets : - B8A S2L2A_B09 : assets : - B09 S2L2A_B11 : assets : - B11 S2L2A_B12 : assets : - B12 default_browse_locator : TCI_10m browses : TRUE_COLOR : asset : visual-10m red : expression : B04 range : [ 0 , 4000 ] nodata : 0 green : expression : B03 range : [ 0 , 4000 ] nodata : 0 blue : expression : B02 range : [ 0 , 4000 ] nodata : 0 FALSE_COLOR : red : expression : B08 range : [ 0 , 4000 ] nodata : 0 green : expression : B04 range : [ 0 , 4000 ] nodata : 0 blue : expression : B03 range : [ 0 , 4000 ] nodata : 0 NDVI : grey : expression : (B08-B04)/(B08+B04) range : [ -1 , 1 ] masks : clouds : validity : false","title":"CREODIAS Deployment"},{"location":"quickstart/creodias-deployment/#creodias-deployment","text":"Based upon our development experiences on CREODIAS, there is a wrapper script creodias with particular customisations suited to the CREODIAS infrastructure and data offering. The customisations are expressed through environment variables that are captured in the file creodias-options . These scripts are examples that can be seen as a starting point, from which they can be adapted to your needs. The CREODIAS deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Protected service endpoints requiring IAM authorization See Endpoint Protection below for further information With reference to the file creodias-options , particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment\u2026 Passwords: LOGIN_SERVICE_ADMIN_PASSWORD , MINIO_ROOT_PASSWORD , HARBOR_ADMIN_PASSWORD OpenStack details: see section Openstack Configuration If configuring an external deployment - ref. Public Deployment \u2026 public_ip - The public IP address through which the deployment is exposed via the ingress-controller domain - The DNS domain name through which the deployment is accessed - forming the stem for all service hostnames in the ingress rules Once the file creodias-options has been well populated for your environment, then the deployment is initiated with\u2026 ./deploy/creodias/creodias \u2026noting that this step is a customised version of that described in section Deployment .","title":"CREODIAS Deployment"},{"location":"quickstart/creodias-deployment/#endpoint-protection","text":"Similarly the script creodias-protection is a customised version of that described in section Apply Protection . Once the main deployment has completed, then the test users can be created , their IDs ( Inum ) set in script creodias-protection , and the resource protection can then be applied\u2026 ./deploy/creodias/creodias-protection","title":"Endpoint Protection"},{"location":"quickstart/creodias-deployment/#harvest-creodias-data","text":"The harvester can be deployed with a default configuration file at /config.yaml . As described in the Data Access section , harvesting according to this configuration can be triggered with\u2026 kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch See the Harvester section below for an explanation of this harvester configuration.","title":"Harvest CREODIAS Data"},{"location":"quickstart/creodias-deployment/#data-specification-walkthrough","text":"The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables . This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example.","title":"Data Specification Walkthrough"},{"location":"quickstart/creodias-deployment/#harvester","text":"The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment. The harvester is configured as follows\u2026 harvester: image: repository: eoepca/rm-harvester tag: 1.1.0 config: redis: host: data-access-redis-master port: 6379 harvesters: - name: Creodias-Opensearch resource: url: https://finder.creodias.eu/resto/api/collections/Sentinel2/describe.xml type: OpenSearch format_config: type: 'application/json' property_mapping: start_datetime: 'startDate' end_datetime: 'completionDate' productIdentifier: 'productIdentifier' query: time: property: sensed begin: 2019-09-10T00:00:00Z end: 2019-09-11T00:00:00Z collection: null bbox: 14.9,47.7,16.4,48.7 filter: {} postprocess: - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor queue: register_queue Based upon this harvester configuration we expect that the following query is made to discover data - i.e. an OpenSearch query, with json response representation, for a defined spatial and temporal extent\u2026 https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2019-09-10T00:00:00Z&completionDate=2019-09-11T00:00:00Z&box=14.9,47.7,16.4,48.7 From the result returned, the path to each product ( feature ) is obtained from the productIdentifier property, e.g. { \"type\": \"FeatureCollection\", \"features\": [ { \"type\": \"Feature\", \"properties\": { \"productIdentifier\": \"/eodata/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE\" ... } ... } ... ] } The harvester is configured with a Sentinel-2/CREODIAS specific post-processor harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor which transforms the product path from /eodata/... to s3://EODATA/... . The harvester post-processor follows this path to the Sentinel-2 scene and uses stactools (with built-in support for Sentinel-2) to establish a STAC item representing the product. This includes enumeration of assets for inspire-metadata and product-metadata - which are used by the registrar pycsw backend to embelesh the product record metadata. The harvester outputs the STAC item for each product, which is pushed to the registrar via the register_queue redis queue.","title":"Harvester"},{"location":"quickstart/creodias-deployment/#registration","text":"The registrar is configured at deployment to have the access details for the CREODIAS data in S3\u2026 global: storage: data: data: type: S3 endpoint_url: http://data.cloudferro.com access_key_id: access secret_access_key: access region_name: RegionOne validate_bucket_name: false Using this S3 configuration, the registrar pycsw backend uses the product metadata linked in the STAC item (ref. assets inspire-metadata and product-metadata ) to embelesh the metadata. For example, product-metadata in the file\u2026 s3://EODATA/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE/MTD_MSIL1C.xml The registrar uses this information to create the ISO XML metadata that is loaded into the resource-catalogue.","title":"Registration"},{"location":"quickstart/creodias-deployment/#product-type","text":"The registrar recognises the product as Sentinel-2 and so reads its metadata XML files to obtain additional information. From the metadata XML file (e.g. MTD_MSIL1C.xml ) the registrar obtains the Product Type for each product from the field <PRODUCT_TYPE> \u2026 <n1:Level-1C_User_Product> <n1:General_Info> <Product_Info> <PRODUCT_TYPE>S2MSI1C</PRODUCT_TYPE> ... </Product_Info> ... </n1:General_Info> ... <n1:Level-1C_User_Product>","title":"Product Type"},{"location":"quickstart/creodias-deployment/#resource-catalogue-collections","text":"The registrar ( eoepca/rm-data-access-core ) container image is pre-loaded with two collections at the path /registrar_pycsw/registrar_pycsw/resources , (in the built container the files are at the path /usr/local/lib/python3.8/dist-packages/registrar_pycsw/resources/ ): S2MSI1C.yml - identifier: S2MSI1C S2MSI2A.yml - identifier: S2MSI2A The registrar applies these collections into the resource-catalogue during start-up - to create pre-defined out-of-the-box collections in pycsw. During registration, the PycswBackend of the registrar uses the Product Type to map the product into the collection of the same name - using metadata field parentidentifier .","title":"Resource Catalogue Collections"},{"location":"quickstart/creodias-deployment/#data-specification","text":"The data-access service data handling is configured by definition of productTypes , collections and layers \u2026 productTypes identify the underlying file assets as WCS coverages and their visual representation collections provide groupings into which products are organised layers specifies the hoe the product visual representations are exposed through the WMS service","title":"Data Specification"},{"location":"quickstart/creodias-deployment/#producttype","text":"During registration, products are mapped into a productType via a filter that is applied against the STAC item metadata. The registrar uses the product_type of each product to determine the collection into which the product should be registered - noting that the name of the product type does not take part in the matching logic (and hence can be any text name)\u2026 productTypes: - name: S2MSI1C filter: s2:product_type: S2MSI1C In the above example, the field s2:product_type is populated by the stactools that prepares the STAC item from the product metadata.","title":"productType"},{"location":"quickstart/creodias-deployment/#collections","text":"Collections are defined by reference to the defined productTypes and coverages .","title":"collections"},{"location":"quickstart/creodias-deployment/#layers","text":"layers defines the layers that are presented through the WMS service - each layer being linked to the underlying browse that provides the image source. Layers are defined via their id that relies upon the naming convection <collection>__<browse> to identify the browse and so define the layer.","title":"layers"},{"location":"quickstart/creodias-deployment/#example-configuration","text":"Example configuration for Sentinel-2 L1C and L2A data. global : layers : - id : S2L1C title : Sentinel-2 Level 1C True Color abstract : Sentinel-2 Level 2A True Color displayColor : '#eb3700' grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__TRUE_COLOR title : Sentinel-2 Level 1C True Color abstract : Sentinel-2 Level 2A True Color grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__masked_clouds title : Sentinel-2 Level 1C True Color with cloud masks abstract : Sentinel-2 Level 1C True Color with cloud masks grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__FALSE_COLOR title : Sentinel-2 Level 1C False Color abstract : Sentinel-2 Level 1C False Color grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L1C__NDVI title : Sentinel-2 Level 21CNDVI abstract : Sentinel-2 Level 1C NDVI grids : - name : WGS84 zoom : 13 parentLayer : S2L1C - id : S2L2A title : Sentinel-2 Level 2A True Color abstract : Sentinel-2 Level 2A True Color displayColor : '#eb3700' grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__TRUE_COLOR title : Sentinel-2 Level 2A True Color abstract : Sentinel-2 Level 2A True Color grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__masked_clouds title : Sentinel-2 Level 2A True Color with cloud masks abstract : Sentinel-2 Level 2A True Color with cloud masks grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__FALSE_COLOR title : Sentinel-2 Level 2A False Color abstract : Sentinel-2 Level 2A False Color grids : - name : WGS84 zoom : 13 parentLayer : S2L2A - id : S2L2A__NDVI title : Sentinel-2 Level 2A NDVI abstract : Sentinel-2 Level 2A NDVI grids : - name : WGS84 zoom : 13 parentLayer : S2L2A collections : S2L1C : product_types : - S2MSI1C coverage_types : - S2L1C_B01 - S2L1C_B02 - S2L1C_B03 - S2L1C_B04 - S2L1C_B05 - S2L1C_B06 - S2L1C_B07 - S2L1C_B08 - S2L1C_B8A - S2L1C_B09 - S2L1C_B10 - S2L1C_B11 - S2L1C_B12 S2L2A : product_types : - S2MSI2A product_levels : - Level-2A coverage_types : - S2L2A_B01 - S2L2A_B02 - S2L2A_B03 - S2L2A_B04 - S2L2A_B05 - S2L2A_B06 - S2L2A_B07 - S2L2A_B08 - S2L2A_B8A - S2L2A_B09 - S2L2A_B11 - S2L2A_B12 productTypes : - name : S2MSI1C filter : s2:product_type : S2MSI1C metadata_assets : [] coverages : S2L1C_B01 : assets : - B01 S2L1C_B02 : assets : - B02 S2L1C_B03 : assets : - B03 S2L1C_B04 : assets : - B04 S2L1C_B05 : assets : - B05 S2L1C_B06 : assets : - B06 S2L1C_B07 : assets : - B07 S2L1C_B08 : assets : - B08 S2L1C_B8A : assets : - B8A S2L1C_B09 : assets : - B09 S2L1C_B10 : assets : - B10 S2L1C_B11 : assets : - B11 S2L1C_B12 : assets : - B12 defaultBrowse : TRUE_COLOR browses : TRUE_COLOR : asset : visual red : expression : B04 range : [ 0 , 4000 ] nodata : 0 green : expression : B03 range : [ 0 , 4000 ] nodata : 0 blue : expression : B02 range : [ 0 , 4000 ] nodata : 0 FALSE_COLOR : red : expression : B08 range : [ 0 , 4000 ] nodata : 0 green : expression : B04 range : [ 0 , 4000 ] nodata : 0 blue : expression : B03 range : [ 0 , 4000 ] nodata : 0 NDVI : grey : expression : (B08-B04)/(B08+B04) range : [ -1 , 1 ] masks : clouds : validity : false - name : S2MSI2A filter : s2:product_type : S2MSI2A metadata_assets : [] coverages : S2L2A_B01 : assets : - B01 S2L2A_B02 : assets : - B02 S2L2A_B03 : assets : - B03 S2L2A_B04 : assets : - B04 S2L2A_B05 : assets : - B05 S2L2A_B06 : assets : - B06 S2L2A_B07 : assets : - B07 S2L2A_B08 : assets : - B08 S2L2A_B8A : assets : - B8A S2L2A_B09 : assets : - B09 S2L2A_B11 : assets : - B11 S2L2A_B12 : assets : - B12 default_browse_locator : TCI_10m browses : TRUE_COLOR : asset : visual-10m red : expression : B04 range : [ 0 , 4000 ] nodata : 0 green : expression : B03 range : [ 0 , 4000 ] nodata : 0 blue : expression : B02 range : [ 0 , 4000 ] nodata : 0 FALSE_COLOR : red : expression : B08 range : [ 0 , 4000 ] nodata : 0 green : expression : B04 range : [ 0 , 4000 ] nodata : 0 blue : expression : B03 range : [ 0 , 4000 ] nodata : 0 NDVI : grey : expression : (B08-B04)/(B08+B04) range : [ -1 , 1 ] masks : clouds : validity : false","title":"Example Configuration"},{"location":"quickstart/data-access-deployment/","text":"Data Access Deployment \u2693\ufe0e A deployment wrapper script has been prepared for a \u2018data access\u2019 deployment - that is focused on the Resource Catalogue and Data Access services. The script deploy/data-access/data-access achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/data-access/data-access-options . The data-access deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: Resource Catalogue for data discovery Data Access for data visualisation and download Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Open ingress are enabled for unauthenticated access to resource-catalogue and data-access services Other eoepca services not deployed Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/data-access/data-access The Resource Catalogue is accessed at the endpoint resource-catalogue-open.<domain> - e.g. resource-catalogue-open.192-168-49-2.nip.io . The Data Access View Server is accessed at the endpoint data-access-open.<domain> - e.g. data-access-open.192-168-49-2.nip.io . Data Harvesting \u2693\ufe0e See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.","title":"Data Access"},{"location":"quickstart/data-access-deployment/#data-access-deployment","text":"A deployment wrapper script has been prepared for a \u2018data access\u2019 deployment - that is focused on the Resource Catalogue and Data Access services. The script deploy/data-access/data-access achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/data-access/data-access-options . The data-access deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: Resource Catalogue for data discovery Data Access for data visualisation and download Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Open ingress are enabled for unauthenticated access to resource-catalogue and data-access services Other eoepca services not deployed","title":"Data Access Deployment"},{"location":"quickstart/data-access-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/data-access/data-access The Resource Catalogue is accessed at the endpoint resource-catalogue-open.<domain> - e.g. resource-catalogue-open.192-168-49-2.nip.io . The Data Access View Server is accessed at the endpoint data-access-open.<domain> - e.g. data-access-open.192-168-49-2.nip.io .","title":"Initiate Deployment"},{"location":"quickstart/data-access-deployment/#data-harvesting","text":"See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.","title":"Data Harvesting"},{"location":"quickstart/exploitation-deployment/","text":"Exploitation Deployment \u2693\ufe0e A deployment wrapper script has been prepared for an \u2018exploitation\u2019 deployment - that provides deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services. The script deploy/exploitation/exploitation achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/exploitation/exploitation-options . The exploitation deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: ADES for processing Resource Catalogue for data discovery Data Access for data visualisation and download Minio for S3 object storage ADES stage-out to Minio Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Open ingress are enabled for unauthenticated access to ADES, resource-catalogue and data-access services Other eoepca services not deployed Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/exploitation/exploitation The ADES service is accessed at the endpoint ades-open.<domain> - e.g. ades-open.192-168-49-2.nip.io . The Resource Catalogue is accessed at the endpoint resource-catalogue-open.<domain> - e.g. resource-catalogue-open.192-168-49-2.nip.io . The Data Access View Server is accessed at the endpoint data-access-open.<domain> - e.g. data-access-open.192-168-49-2.nip.io . Example Requests - s-expression on CREODIAS \u2693\ufe0e NOTE that this example processing request requires harvesting data from CREODIAS, which can only be performed if the deployment is made to a CREODIAS VM with access to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION . Section Processing provides an example of a simple self-contained processing deployment and execution , and access to the processing results . In addition to the snuggs example, the file deploy/samples/requests/processing/s-expression.http has been prepared to exploit data that has been registered within the Resource Catalogue and Data Access services. First the input data for processing must be harvested into the resource management services. Sentinel-2 data on 2nd Sept 2020\u2026 ./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2020.09.02.yaml Then the s-expression.http file provides sample requests for OGC API Processes operations: List Processes Deploy Process Get Process Details Execute Process Get Job Status Get Job Results NOTE that the first requests in the file provide optional calls to obtain a user ID token ( openidConfiguration / authenticate ) - to be used in the case that protected (not \u2018open\u2019) endpoints are deployed. The file snuggs.http describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client . Install in vscode with ext install humao.rest-client . Various variables, such as to specify the @domain for your deployment, can be configured at the top of the file. At the completion of successful processing execution, the procesing results are obtained as described in section Processing Results . Data Harvesting \u2693\ufe0e See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.","title":"Exploitation"},{"location":"quickstart/exploitation-deployment/#exploitation-deployment","text":"A deployment wrapper script has been prepared for an \u2018exploitation\u2019 deployment - that provides deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services. The script deploy/exploitation/exploitation achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/exploitation/exploitation-options . The exploitation deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: ADES for processing Resource Catalogue for data discovery Data Access for data visualisation and download Minio for S3 object storage ADES stage-out to Minio Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Open ingress are enabled for unauthenticated access to ADES, resource-catalogue and data-access services Other eoepca services not deployed","title":"Exploitation Deployment"},{"location":"quickstart/exploitation-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/exploitation/exploitation The ADES service is accessed at the endpoint ades-open.<domain> - e.g. ades-open.192-168-49-2.nip.io . The Resource Catalogue is accessed at the endpoint resource-catalogue-open.<domain> - e.g. resource-catalogue-open.192-168-49-2.nip.io . The Data Access View Server is accessed at the endpoint data-access-open.<domain> - e.g. data-access-open.192-168-49-2.nip.io .","title":"Initiate Deployment"},{"location":"quickstart/exploitation-deployment/#example-requests-s-expression-on-creodias","text":"NOTE that this example processing request requires harvesting data from CREODIAS, which can only be performed if the deployment is made to a CREODIAS VM with access to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION . Section Processing provides an example of a simple self-contained processing deployment and execution , and access to the processing results . In addition to the snuggs example, the file deploy/samples/requests/processing/s-expression.http has been prepared to exploit data that has been registered within the Resource Catalogue and Data Access services. First the input data for processing must be harvested into the resource management services. Sentinel-2 data on 2nd Sept 2020\u2026 ./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2020.09.02.yaml Then the s-expression.http file provides sample requests for OGC API Processes operations: List Processes Deploy Process Get Process Details Execute Process Get Job Status Get Job Results NOTE that the first requests in the file provide optional calls to obtain a user ID token ( openidConfiguration / authenticate ) - to be used in the case that protected (not \u2018open\u2019) endpoints are deployed. The file snuggs.http describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client . Install in vscode with ext install humao.rest-client . Various variables, such as to specify the @domain for your deployment, can be configured at the top of the file. At the completion of successful processing execution, the procesing results are obtained as described in section Processing Results .","title":"Example Requests - s-expression on CREODIAS"},{"location":"quickstart/exploitation-deployment/#data-harvesting","text":"See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.","title":"Data Harvesting"},{"location":"quickstart/processing-deployment/","text":"Processing Deployment \u2693\ufe0e A deployment wrapper script has been prepared for a \u2018processing\u2019 deployment - that is focused on the ADES and the deployment/execution of processing jobs. The script deploy/processing/processing achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/processing/processing-options . The processing deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: ADES for processing Minio for S3 object storage ADES stage-out to Minio Open ingress are enabled for unauthenticated access to ADES service Other eoepca services not deployed Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/processing/processing The ADES service is accessed at the endpoint ades-open.<domain> - e.g. ades-open.192-168-49-2.nip.io . Example Requests - snuggs application \u2693\ufe0e The file deploy/samples/requests/processing/snuggs.http has been prepared with sample requests for OGC API Processes operations: List Processes Deploy Process Get Process Details Execute Process Get Job Status Get Job Results Note The first requests in the file provide optional calls to obtain a user ID token ( openidConfiguration / authenticate ). These are to be used in the case that protected (not \u2018open\u2019) endpoints are deployed. The file describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client . Install in vscode with ext install humao.rest-client . The variables @hostname and @domain can be configured at the top of the file. Requests using CuRL Alternatively, the following curl commands can be used instead\u2026 List Processes curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes \\ --header 'accept: application/json' Deploy Process curl -k \\ --request POST \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes \\ --header 'accept: application/json' \\ --header 'content-type: application/json' \\ --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/app-snuggs/main/app-package.cwl\",\"type\": \"application/cwl\"}}' Get Process Details curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0 \\ --header 'accept: application/json' Execute Process curl -k -v \\ --request POST \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0/execution \\ --header 'accept: application/json' \\ --header 'content-type: application/json' \\ --header 'prefer: respond-async' \\ --data '{\"inputs\": {\"input_reference\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\",\"s_expression\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\"},\"response\":\"raw\"}' Get Job Status This request requires the Location header from the response to the execute request. This will be of the form /{user}/wps3/jobs/{job-id} - e.g. /eric/wps3/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e . curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io { location-header } \\ --header 'accept: application/json' Get Job Results This request uses the same URL as Get Job Status , with the additional URL path /result - i.e. /{user}/wps3/jobs/{job-id}/result - e.g. /eric/wps3/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/result curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io { location-header } /result \\ --header 'accept: application/json' The response indicates the location of the results, which should be in the minio object storage. See Processing Results List Jobs curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/jobs \\ --header 'accept: application/json' Processing Results \u2693\ufe0e The outputs are published as a static STAC catalogue to a path that includes the unique job ID. In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint console.minio.<domain> e.g. https://console.minio.192-168-49-2.nip.io/, or using an S3 client such as\u2026 s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca For the default credentials to connect to Minio see Minio Object Storage Default Credentials .","title":"Processing Deployment"},{"location":"quickstart/processing-deployment/#processing-deployment","text":"A deployment wrapper script has been prepared for a \u2018processing\u2019 deployment - that is focused on the ADES and the deployment/execution of processing jobs. The script deploy/processing/processing achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/processing/processing-options . The processing deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: ADES for processing Minio for S3 object storage ADES stage-out to Minio Open ingress are enabled for unauthenticated access to ADES service Other eoepca services not deployed","title":"Processing Deployment"},{"location":"quickstart/processing-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/processing/processing The ADES service is accessed at the endpoint ades-open.<domain> - e.g. ades-open.192-168-49-2.nip.io .","title":"Initiate Deployment"},{"location":"quickstart/processing-deployment/#example-requests-snuggs-application","text":"The file deploy/samples/requests/processing/snuggs.http has been prepared with sample requests for OGC API Processes operations: List Processes Deploy Process Get Process Details Execute Process Get Job Status Get Job Results Note The first requests in the file provide optional calls to obtain a user ID token ( openidConfiguration / authenticate ). These are to be used in the case that protected (not \u2018open\u2019) endpoints are deployed. The file describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client . Install in vscode with ext install humao.rest-client . The variables @hostname and @domain can be configured at the top of the file. Requests using CuRL Alternatively, the following curl commands can be used instead\u2026 List Processes curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes \\ --header 'accept: application/json' Deploy Process curl -k \\ --request POST \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes \\ --header 'accept: application/json' \\ --header 'content-type: application/json' \\ --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/app-snuggs/main/app-package.cwl\",\"type\": \"application/cwl\"}}' Get Process Details curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0 \\ --header 'accept: application/json' Execute Process curl -k -v \\ --request POST \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/processes/snuggs-0_3_0/execution \\ --header 'accept: application/json' \\ --header 'content-type: application/json' \\ --header 'prefer: respond-async' \\ --data '{\"inputs\": {\"input_reference\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\",\"s_expression\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\"},\"response\":\"raw\"}' Get Job Status This request requires the Location header from the response to the execute request. This will be of the form /{user}/wps3/jobs/{job-id} - e.g. /eric/wps3/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e . curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io { location-header } \\ --header 'accept: application/json' Get Job Results This request uses the same URL as Get Job Status , with the additional URL path /result - i.e. /{user}/wps3/jobs/{job-id}/result - e.g. /eric/wps3/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/result curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io { location-header } /result \\ --header 'accept: application/json' The response indicates the location of the results, which should be in the minio object storage. See Processing Results List Jobs curl -k \\ --request GET \\ --url https://ades-open.192-168-49-2.nip.io/eric/wps3/jobs \\ --header 'accept: application/json'","title":"Example Requests - snuggs application"},{"location":"quickstart/processing-deployment/#processing-results","text":"The outputs are published as a static STAC catalogue to a path that includes the unique job ID. In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint console.minio.<domain> e.g. https://console.minio.192-168-49-2.nip.io/, or using an S3 client such as\u2026 s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca For the default credentials to connect to Minio see Minio Object Storage Default Credentials .","title":"Processing Results"},{"location":"quickstart/quickstart/","text":"Quickstart \u2693\ufe0e The deployment of the EOEPCA components and the supporting Kubernetes cluster is described in the sections Cluster and EOEPCA . These sections should be consulted for more detailed information. Scripted Deployment \u2693\ufe0e As a companion to these descriptions, we have developed a set of scripts to provide a demonstration of an example deployment. This is described in the following section Scripted Deployment . Note The scripted deployment assumes that installation of the Prerequisite Tooling has been performed Customised Deployments \u2693\ufe0e The Scripted Deployment can be quickly exploited through the following customisations for particular use cases: Simple Basic local deployment Processing Deployment focused on processing Data Access Deployment focused on the Resource Catalogue and Data Access services Exploitation Deployment providing deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services User Management Deployment focused on the User Management services CREODIAS Deployment with access to CREODIAS EO data Each customisation is introduced in their respective sections. Quick Example \u2693\ufe0e Follow these steps to create a simple local deployment in minikube\u2026 Prerequisite Tooling Follow the steps in section Prerequisite Tooling to install the required tooling. Clone the repository git clone https://github.com/EOEPCA/deployment-guide Initiate the deployment cd deployment-guide ./deploy/simple/simple Wait for deployment ready List pod status watch kubectl get pod -A Wait until all pods report either Running or Completed This may take 10-20 mins depending on the capabilities of your platform. Test the deployment Make the sample requests to the ADES processing service.","title":"Quick Start"},{"location":"quickstart/quickstart/#quickstart","text":"The deployment of the EOEPCA components and the supporting Kubernetes cluster is described in the sections Cluster and EOEPCA . These sections should be consulted for more detailed information.","title":"Quickstart"},{"location":"quickstart/quickstart/#scripted-deployment","text":"As a companion to these descriptions, we have developed a set of scripts to provide a demonstration of an example deployment. This is described in the following section Scripted Deployment . Note The scripted deployment assumes that installation of the Prerequisite Tooling has been performed","title":"Scripted Deployment"},{"location":"quickstart/quickstart/#customised-deployments","text":"The Scripted Deployment can be quickly exploited through the following customisations for particular use cases: Simple Basic local deployment Processing Deployment focused on processing Data Access Deployment focused on the Resource Catalogue and Data Access services Exploitation Deployment providing deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services User Management Deployment focused on the User Management services CREODIAS Deployment with access to CREODIAS EO data Each customisation is introduced in their respective sections.","title":"Customised Deployments"},{"location":"quickstart/quickstart/#quick-example","text":"Follow these steps to create a simple local deployment in minikube\u2026 Prerequisite Tooling Follow the steps in section Prerequisite Tooling to install the required tooling. Clone the repository git clone https://github.com/EOEPCA/deployment-guide Initiate the deployment cd deployment-guide ./deploy/simple/simple Wait for deployment ready List pod status watch kubectl get pod -A Wait until all pods report either Running or Completed This may take 10-20 mins depending on the capabilities of your platform. Test the deployment Make the sample requests to the ADES processing service.","title":"Quick Example"},{"location":"quickstart/scripted-deployment/","text":"Scripted Deployment \u2693\ufe0e The Scripted Deployment provides a demonstration of an example deployment, and can found in the subdirectory deployment-guide/deploy of the source repository for this guide\u2026 git clone https://github.com/EOEPCA/deployment-guide \\ && cd deployment-guide \\ && ls deploy The script deploy/eoepca/eoepca.sh acts as an entry-point to the full system deployment. In order to tailor the deployment for your target environment, the script is configured through environment variables and command-line arguments. By default the script assumes deployment to a local minikube. Note The scripted deployment assumes that installation of the Prerequisite Tooling has been performed. The following subsections lead through the steps for a full local deployment. Whilst minikube is assumed, minimal adaptions are required to make the deployment to your existing Kubernetes cluster. The deployment follows these broad steps: Configuration Tailoring of deployment options. Deployment Creation of cluster and deployment of eoepca services. Protection Application of protection for authorized access to services. The Protection step is split from Deployment as there are some manual steps to be performed before the Protection can be applied. Configuration \u2693\ufe0e The script deploy/eoepca/eoepca.sh is configured by some environment variables and command-line arguments. Environment Variables \u2693\ufe0e Environment Variables Variable Description Default REQUIRE_<cluster-component> A set of variables that can be used to control which CLUSTER components are deployed by the script, as follows (with defaults): REQUIRE_MINIKUBE=true REQUIRE_INGRESS_NGINX=true REQUIRE_CERT_MANAGER=true REQUIRE_LETSENCRYPT=true REQUIRE_SEALED_SECRETS=false REQUIRE_MINIO=false see description REQUIRE_<eoepca-component> A set of variables that can be used to control which EOEPCA components are deployed by the script, as follows (with defaults): REQUIRE_STORAGE=true REQUIRE_DUMMY_SERVICE=false REQUIRE_LOGIN_SERVICE=true REQUIRE_PDP=true REQUIRE_USER_PROFILE=true REQUIRE_ADES=true REQUIRE_RESOURCE_CATALOGUE=true REQUIRE_DATA_ACCESS=true REQUIRE_WORKSPACE_API=true REQUIRE_BUCKET_OPERATOR=true REQUIRE_HARBOR=true REQUIRE_PORTAL=true REQUIRE_PDE=true see description REQUIRE_<protection-component> A set of variables that can be used to control which PROTECTION components are deployed by the script, as follows (with defaults): REQUIRE_DUMMY_SERVICE_PROTECTION=false REQUIRE_ADES_PROTECTION=true REQUIRE_RESOURCE_CATALOGUE_PROTECTION=true REQUIRE_DATA_ACCESS_PROTECTION=true REQUIRE_WORKSPACE_API_PROTECTION=true see description MINIKUBE_KUBERNETES_VERSION The Kubernetes version to be used by minikube Note that the EOEPCA development has been conducted primarily using version 1.22.5. v1.22.5 MINIKUBE_MEMORY_AMOUNT Amount of memory to allocate to the docker containers used by minikube to implement the cluster. 12g USE_METALLB Enable use of minikube\u2019s built-in load-balancer. The load-balancer can be used to facilitate exposing services publicly. However, the same can be achieved using minikube\u2019s built-in ingress-controller. Therefore, this option is suppressed by default. false USE_INGRESS_NGINX_HELM Install the ingress-nginx controller using the published helm chart, rather than relying upon the version that is built-in to minikube. By default we prefer the version that is built in to minikube. false USE_INGRESS_NGINX_LOADBALANCER Patch the built-in minikube nginx-ingress-controller to offer a service of type LoadBalancer , rather than the default NodePort . It was initially thought that this would be necessary to achieve public access to the ingress services - but was subsequently found that the default NodePort configuration of the ingress-controller was sufficient. This option is left in case it proves useful. Only applicable for USE_INGRESS_NGINX_HELM=false (i.e. when using the minikube built-in ) false OPEN_INGRESS Create \u2018open\u2019 ingress endpoints that are not subject to authorization protection. For a secure system the open endpoints should be disabled ( false ) and access to resource should be protected via ingress that apply protection false USE_TLS Indicates whether TLS will be configured for service Ingress rules. If not (i.e. USE_TLS=false ), then the ingress-controller is configured to disable ssl-redirect , and TLS_CLUSTER_ISSUER=notls is set. true TLS_CLUSTER_ISSUER The name of the ClusterIssuer to satisfy ingress tls certificates. Out-of-the-box ClusterIssuer instances are configured in the file deploy/cluster/letsencrypt.sh . letsencrypt-staging LOGIN_SERVICE_ADMIN_PASSWORD Initial password for the admin user in the login-service. changeme MINIO_ROOT_USER Name of the \u2018root\u2019 user for the Minio object storage service. eoepca MINIO_ROOT_PASSWORD Password for the \u2018root\u2019 user for the Minio object storage service. changeme HARBOR_ADMIN_PASSWORD Password for the \u2018admin\u2019 user for the Harbor artefact registry service. changeme PROCESSING_MAX_RAM Max RAM allocated to an individual processing job 8Gi PROCESSING_MAX_CORES Max number of CPU cores allocated to an individual processing job 4 STAGEOUT_TARGET Configures the ADES with the destination to which it should push processing results: workspace - via the Workspace API minio - to minio S3 object storage workspace INSTALL_FLUX The Workspace API relies upon Flux CI/CD , and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value false to suppress the Workspace API flux install true CREODIAS_DATA_SPECIFICATION Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services. Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the eodata network. false Openstack Configuration \u2693\ufe0e There are some additional environment variables that configure the BucketOperator with details of the infrastructure Openstack layer. Attention This is only applicable for an Openstack deployment and has only been tested on the CREODIAS. Environment Variables Variable Description Default OS_DOMAINNAME Openstack domain of the admin account in the cloud provider. cloud_XXXXX OS_USERNAME Openstack username of the admin account in the cloud provider. user@cloud.com OS_PASSWORD Openstack password of the admin account in the cloud provider. none OS_MEMBERROLEID ID of a specific role (e.g. the \u2018 member \u2018 role) for operations users (to allow administration), e.g. 7fe2ff9ee5384b1894a90838d3e92bab . none OS_SERVICEPROJECTID ID of a project containing the user identity requiring write access to the created user buckets, e.g. 573916ef342a4bf1aea807d0c6058c1e . none USER_EMAIL_PATTERN Email associated to the created user within the created user project. Note: <name> is templated and will be replaced. eoepca-<name>@platform.com Command-line Arguments \u2693\ufe0e The eoepca.sh script is further configured via command-line arguments\u2026 eoepca.sh <action> <cluster-name> <public-ip> <domain> Arguments Argument Description Default action Action to perform: apply | delete | template . apply makes the deployment delete removes the deployment template outputs generated kubernetes yaml to stdout apply cluster-name The name of the minikube \u2018profile\u2019 for the created minikube cluster eoepca public-ip The public IP address through which the deployment is exposed via the ingress-controller. By default, the value is deduced from the assigned cluster minikube IP address - ref. command minikube ip . <minikube-ip> domain The DNS domain name through which the deployment is accessed. Forms the stem for all service hostnames in the ingress rules - i.e. <service-name>.<domain> . By default, the value is deduced from the assigned cluster minikube IP address, using nip.io to establish a DNS lookup - i.e. <minikube ip>.nip.io . <minikube ip>.nip.io Public Deployment \u2693\ufe0e For simplicity, the out-of-the-box scripts assume a \u2018private\u2019 deployment - with no public IP / DNS and hence no use of TLS for service ingress endpoints. In the case that an external-facing public deployment is desired, then the following configuration selections should be made: public_ip - set to the external-facing public IP of your deployment, e.g. the floating IP of your load-balancer in a cloud deployment domain - set to the domain (as per DNS records) for your deployment Note that the EOEPCA components typically configure their ingress with hostname prefixes applied to this domain . Thus, it is necessary that the DNS record for the domain is established as a wildcard record - i.e. *.<domain> USE_TLS=true - to enable configuration of TLS endpoints in each component service ingress TLS_CLUSTER_ISSUER=<issuer> - should be configured ~ e.g. using the letsencrypt-production or letsencrypt-staging (testing only) Cluster Issuer that are configured by the scripted deployment Deployment \u2693\ufe0e The deployment is initiated by setting the appropriate environment variables and invoking the eoepca.sh script with suitable command-line arguments . You may find it convenient to do so using a wrapper script that customises the environment varaibles according to your cluster, and then invokes the eoepca.sh script. Customised examples are provided for Simple, CREODIAS and Processing deployments. NOTE that if a prior deployment has been attempted then, before redeploying, a clean-up should be performed as described in the Clean-up section below. This is particularly important in the case that the minikube none driver is used, as the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. Initiate the deployment\u2026 ./deploy/eoepca/eoepca.sh apply \"<cluster-name>\" \"<public-ip>\" \"<domain>\" The deployment takes 10+ minutes - depending on the resources of your host/cluster. The progress can be monitored\u2026 kubectl get pods -A The deployment is ready once all pods are either Running or Completed . This can be further confirmed by accessing the login-service web interface at https://auth.<domain>/ and logging in as user admin using the credentials configured via LOGIN_SERVICE_ADMIN_PASSWORD . Default Credentials \u2693\ufe0e Login Service \u2693\ufe0e By default, the Login Service is accessed at the URL https://auth.<domain>/ with the credentials\u2026 username: admin password: Chang3me! \u2026unless the password is overridden via the variable LOGIN_SERVICE_ADMIN_PASSWORD . Minio Object Storage \u2693\ufe0e By default, Minio is accessed at the URL http://console.minio.<domain>/ with the credentials\u2026 username: eoepca password: changeme \u2026unless the username/password are overridden via the variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD . Harbor Container Registry \u2693\ufe0e By default, Harbor is accessed at the URL https://harbor.<domain>/ with the credentials\u2026 username: admin password: changeme \u2026unless the password is overridden via the variable HARBOR_ADMIN_PASSWORD . Protection \u2693\ufe0e The protection of resource server endpoints is applied with the script deploy/eoepca/eoepca-protection.sh . This script should be executed with environment variables and command-line options that are consistent with those of the main deployment (ref. script eoepca.sh ). The script eoepca-protection.sh introduces two users eric and bob to demonstrate the application of authorized access to various service endpoints: ADES, Workspace API and dummy-service (simple endpoint used for debugging). Thus, the users must first be created in the login-service and their unique IDs passed to the protection script. Usage: eoepca-protection.sh <action> <eric-id> <bob-id> <public-ip> <domain> Create Test Users \u2693\ufe0e Access the login-service web interface ( https://auth.<domain>/ ) as user admin using the credentials configured via LOGIN_SERVICE_ADMIN_PASSWORD . Select Users -> Add person to add users eric and bob (dummy details can be used). Note the Inum (unique user ID) for each user for use with the eoepca-protection.sh script. Apply Protection \u2693\ufe0e Apply the protection\u2026 Ensure that the script is executed with the environment variables and command-line options that are consistent with those of the main deployment . ./deploy/eoepca/eoepca-protection.sh apply \"<eric-id>\" \"<bob-id>\" \"<public-ip>\" \"<domain>\" Post-deploy Manual Steps \u2693\ufe0e The scripted deployment has been designed, as far as possible, to automate the configuration of the deployed components. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026 Fix the setting UMA Resource Lifetime - change 2592000 -> 2147483647 secs Fix the setting Authentication method for the Token Endpoint for the ApplicationHub - client_secret_post -> client_secret_basic Add groups group-1 , group-2 , group-3 to ApplicationHub, and add users eric , bob to these groups Create User Workspaces \u2693\ufe0e The protection steps created the test users eric and bob . For completeness we use the Workspace API to create their user workspaces, which hold their personal resources (data, processing results, etc.) within the platform - see Workspace . Using Workspace Swagger UI \u2693\ufe0e The Workspace API provides a Swagger UI that facilitates interaction with the API - at the URL https://workspace-api.<domain>/docs# . Access to The Workspace API is protected, such that the necessary access tokens must be supplied in requests, which is most easily achieved by logging in via the \u2018portal\u2019. The portal is accessed at https://portal.<domain>/ . It is a rudimentary web service that facilitates establishing the appropriate tokens in the user\u2019s browser context. Login to the portal as the admin user, using the configured credentials. Access the Workspace Swagger UI at https://workspace-api.<domain>/docs . Workspaces are created using POST /workspaces (Create Workspace) . Expand the node and select Try it out . Complete the request body, such as\u2026 { \"preferred_name\" : \"eric\" , \"default_owner\" : \"d95b0c2b-ea74-4b3f-9c6a-85198dec974d\" } \u2026where the default_owner is the user ID ( Inum ) for the user - thus protecting the created workspace for the identified user. Using curl \u2693\ufe0e The same can be achieved with a straight http request, for example using curl \u2026 curl -X 'POST' \\ 'https://workspace-api.192-168-49-2.nip.io/workspaces' \\ -H 'accept: application/json' \\ -H 'Content-Type: application/json' \\ -H 'X-User-Id: <admin-id-token>' \\ -d '{ \"preferred_name\": \"<workspace-name>\", \"default_owner\": \"<user-inum>\" }' Values must be provided for: admin-id-token - User ID token for the admin user workspace-name - name of the workspace, typically the username user-inum - the ID of the user for which the created workspace will be protected The ID token for the admin user can be obtained with a call to the token endpoint of the Login Service - supplying the credentials for the admin user and the pre-registered client\u2026 curl -L -X POST 'https://auth.<domain>/oxauth/restv1/token' \\ -H 'Cache-Control: no-cache' \\ -H 'Content-Type: application/x-www-form-urlencoded' \\ --data-urlencode 'scope=openid user_name is_operator' \\ --data-urlencode 'grant_type=password' \\ --data-urlencode 'username=admin' \\ --data-urlencode 'password=<admin-password>' \\ --data-urlencode 'client_id=<client-id>' \\ --data-urlencode 'client_secret=<client-secret>' A json response is returned, in which the field id_token provides the user ID token for the admin user. Using create-workspace helper script \u2693\ufe0e As an aide there is a helper script create-workspace . The script is available in the deployment-guide repository , and can be obtained as follows\u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide The create-workspace helper script requires some command-line arguments\u2026 Usage: create-workspace <domain> <user> <user-inum> [<client-id> <client-secret>] For example\u2026 ./deploy/bin/create-workspace 192 -168-49-2.nip.io eric d95b0c2b-ea74-4b3f-9c6a-85198dec974d The script prompts for the password of the admin user. By default <client-id> and <client-secret> are read from the client.yaml file that is created by the deployment script, which auto-registers a Login Service client. Thus, these args can be ommited to use the default client credentials. Clean-up \u2693\ufe0e Before initiating a fresh deployment, if a prior deployment has been attempted, then it is necessary to remove any persistent artefacts of the prior deployment. This includes\u2026 Minikube cluster Delete the minikube cluster\u2026 minikube delete If necessary specify the cluster (profile)\u2026 minikube -p <profile> delete Persistent Data In the case that the minikube none driver is used, the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. In this case, the minikube standard StorageClass is fulfilled by the hostpath provisioner, whose persistence is removed as follows\u2026 sudo rm -rf /tmp/hostpath-provisioner Client Credentials During the deployment a client of the Authorisation Server is registered, and its credentials stored for reuse in the file client.yaml . Once the cluster has been destroyed, then these client credentials become stale and so should be removed to avoid polluting subsequent deployments\u2026 rm -rf ./deploy/eoepca/client.yaml There is a helper script clean that can be used for steps 2 and 3 above, (the script does not delete the cluster). ./deploy/cluster/clean","title":"Scripted Deployment"},{"location":"quickstart/scripted-deployment/#scripted-deployment","text":"The Scripted Deployment provides a demonstration of an example deployment, and can found in the subdirectory deployment-guide/deploy of the source repository for this guide\u2026 git clone https://github.com/EOEPCA/deployment-guide \\ && cd deployment-guide \\ && ls deploy The script deploy/eoepca/eoepca.sh acts as an entry-point to the full system deployment. In order to tailor the deployment for your target environment, the script is configured through environment variables and command-line arguments. By default the script assumes deployment to a local minikube. Note The scripted deployment assumes that installation of the Prerequisite Tooling has been performed. The following subsections lead through the steps for a full local deployment. Whilst minikube is assumed, minimal adaptions are required to make the deployment to your existing Kubernetes cluster. The deployment follows these broad steps: Configuration Tailoring of deployment options. Deployment Creation of cluster and deployment of eoepca services. Protection Application of protection for authorized access to services. The Protection step is split from Deployment as there are some manual steps to be performed before the Protection can be applied.","title":"Scripted Deployment"},{"location":"quickstart/scripted-deployment/#configuration","text":"The script deploy/eoepca/eoepca.sh is configured by some environment variables and command-line arguments.","title":"Configuration"},{"location":"quickstart/scripted-deployment/#environment-variables","text":"Environment Variables Variable Description Default REQUIRE_<cluster-component> A set of variables that can be used to control which CLUSTER components are deployed by the script, as follows (with defaults): REQUIRE_MINIKUBE=true REQUIRE_INGRESS_NGINX=true REQUIRE_CERT_MANAGER=true REQUIRE_LETSENCRYPT=true REQUIRE_SEALED_SECRETS=false REQUIRE_MINIO=false see description REQUIRE_<eoepca-component> A set of variables that can be used to control which EOEPCA components are deployed by the script, as follows (with defaults): REQUIRE_STORAGE=true REQUIRE_DUMMY_SERVICE=false REQUIRE_LOGIN_SERVICE=true REQUIRE_PDP=true REQUIRE_USER_PROFILE=true REQUIRE_ADES=true REQUIRE_RESOURCE_CATALOGUE=true REQUIRE_DATA_ACCESS=true REQUIRE_WORKSPACE_API=true REQUIRE_BUCKET_OPERATOR=true REQUIRE_HARBOR=true REQUIRE_PORTAL=true REQUIRE_PDE=true see description REQUIRE_<protection-component> A set of variables that can be used to control which PROTECTION components are deployed by the script, as follows (with defaults): REQUIRE_DUMMY_SERVICE_PROTECTION=false REQUIRE_ADES_PROTECTION=true REQUIRE_RESOURCE_CATALOGUE_PROTECTION=true REQUIRE_DATA_ACCESS_PROTECTION=true REQUIRE_WORKSPACE_API_PROTECTION=true see description MINIKUBE_KUBERNETES_VERSION The Kubernetes version to be used by minikube Note that the EOEPCA development has been conducted primarily using version 1.22.5. v1.22.5 MINIKUBE_MEMORY_AMOUNT Amount of memory to allocate to the docker containers used by minikube to implement the cluster. 12g USE_METALLB Enable use of minikube\u2019s built-in load-balancer. The load-balancer can be used to facilitate exposing services publicly. However, the same can be achieved using minikube\u2019s built-in ingress-controller. Therefore, this option is suppressed by default. false USE_INGRESS_NGINX_HELM Install the ingress-nginx controller using the published helm chart, rather than relying upon the version that is built-in to minikube. By default we prefer the version that is built in to minikube. false USE_INGRESS_NGINX_LOADBALANCER Patch the built-in minikube nginx-ingress-controller to offer a service of type LoadBalancer , rather than the default NodePort . It was initially thought that this would be necessary to achieve public access to the ingress services - but was subsequently found that the default NodePort configuration of the ingress-controller was sufficient. This option is left in case it proves useful. Only applicable for USE_INGRESS_NGINX_HELM=false (i.e. when using the minikube built-in ) false OPEN_INGRESS Create \u2018open\u2019 ingress endpoints that are not subject to authorization protection. For a secure system the open endpoints should be disabled ( false ) and access to resource should be protected via ingress that apply protection false USE_TLS Indicates whether TLS will be configured for service Ingress rules. If not (i.e. USE_TLS=false ), then the ingress-controller is configured to disable ssl-redirect , and TLS_CLUSTER_ISSUER=notls is set. true TLS_CLUSTER_ISSUER The name of the ClusterIssuer to satisfy ingress tls certificates. Out-of-the-box ClusterIssuer instances are configured in the file deploy/cluster/letsencrypt.sh . letsencrypt-staging LOGIN_SERVICE_ADMIN_PASSWORD Initial password for the admin user in the login-service. changeme MINIO_ROOT_USER Name of the \u2018root\u2019 user for the Minio object storage service. eoepca MINIO_ROOT_PASSWORD Password for the \u2018root\u2019 user for the Minio object storage service. changeme HARBOR_ADMIN_PASSWORD Password for the \u2018admin\u2019 user for the Harbor artefact registry service. changeme PROCESSING_MAX_RAM Max RAM allocated to an individual processing job 8Gi PROCESSING_MAX_CORES Max number of CPU cores allocated to an individual processing job 4 STAGEOUT_TARGET Configures the ADES with the destination to which it should push processing results: workspace - via the Workspace API minio - to minio S3 object storage workspace INSTALL_FLUX The Workspace API relies upon Flux CI/CD , and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value false to suppress the Workspace API flux install true CREODIAS_DATA_SPECIFICATION Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services. Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the eodata network. false","title":"Environment Variables"},{"location":"quickstart/scripted-deployment/#openstack-configuration","text":"There are some additional environment variables that configure the BucketOperator with details of the infrastructure Openstack layer. Attention This is only applicable for an Openstack deployment and has only been tested on the CREODIAS. Environment Variables Variable Description Default OS_DOMAINNAME Openstack domain of the admin account in the cloud provider. cloud_XXXXX OS_USERNAME Openstack username of the admin account in the cloud provider. user@cloud.com OS_PASSWORD Openstack password of the admin account in the cloud provider. none OS_MEMBERROLEID ID of a specific role (e.g. the \u2018 member \u2018 role) for operations users (to allow administration), e.g. 7fe2ff9ee5384b1894a90838d3e92bab . none OS_SERVICEPROJECTID ID of a project containing the user identity requiring write access to the created user buckets, e.g. 573916ef342a4bf1aea807d0c6058c1e . none USER_EMAIL_PATTERN Email associated to the created user within the created user project. Note: <name> is templated and will be replaced. eoepca-<name>@platform.com","title":"Openstack Configuration"},{"location":"quickstart/scripted-deployment/#command-line-arguments","text":"The eoepca.sh script is further configured via command-line arguments\u2026 eoepca.sh <action> <cluster-name> <public-ip> <domain> Arguments Argument Description Default action Action to perform: apply | delete | template . apply makes the deployment delete removes the deployment template outputs generated kubernetes yaml to stdout apply cluster-name The name of the minikube \u2018profile\u2019 for the created minikube cluster eoepca public-ip The public IP address through which the deployment is exposed via the ingress-controller. By default, the value is deduced from the assigned cluster minikube IP address - ref. command minikube ip . <minikube-ip> domain The DNS domain name through which the deployment is accessed. Forms the stem for all service hostnames in the ingress rules - i.e. <service-name>.<domain> . By default, the value is deduced from the assigned cluster minikube IP address, using nip.io to establish a DNS lookup - i.e. <minikube ip>.nip.io . <minikube ip>.nip.io","title":"Command-line Arguments"},{"location":"quickstart/scripted-deployment/#public-deployment","text":"For simplicity, the out-of-the-box scripts assume a \u2018private\u2019 deployment - with no public IP / DNS and hence no use of TLS for service ingress endpoints. In the case that an external-facing public deployment is desired, then the following configuration selections should be made: public_ip - set to the external-facing public IP of your deployment, e.g. the floating IP of your load-balancer in a cloud deployment domain - set to the domain (as per DNS records) for your deployment Note that the EOEPCA components typically configure their ingress with hostname prefixes applied to this domain . Thus, it is necessary that the DNS record for the domain is established as a wildcard record - i.e. *.<domain> USE_TLS=true - to enable configuration of TLS endpoints in each component service ingress TLS_CLUSTER_ISSUER=<issuer> - should be configured ~ e.g. using the letsencrypt-production or letsencrypt-staging (testing only) Cluster Issuer that are configured by the scripted deployment","title":"Public Deployment"},{"location":"quickstart/scripted-deployment/#deployment","text":"The deployment is initiated by setting the appropriate environment variables and invoking the eoepca.sh script with suitable command-line arguments . You may find it convenient to do so using a wrapper script that customises the environment varaibles according to your cluster, and then invokes the eoepca.sh script. Customised examples are provided for Simple, CREODIAS and Processing deployments. NOTE that if a prior deployment has been attempted then, before redeploying, a clean-up should be performed as described in the Clean-up section below. This is particularly important in the case that the minikube none driver is used, as the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. Initiate the deployment\u2026 ./deploy/eoepca/eoepca.sh apply \"<cluster-name>\" \"<public-ip>\" \"<domain>\" The deployment takes 10+ minutes - depending on the resources of your host/cluster. The progress can be monitored\u2026 kubectl get pods -A The deployment is ready once all pods are either Running or Completed . This can be further confirmed by accessing the login-service web interface at https://auth.<domain>/ and logging in as user admin using the credentials configured via LOGIN_SERVICE_ADMIN_PASSWORD .","title":"Deployment"},{"location":"quickstart/scripted-deployment/#default-credentials","text":"","title":"Default Credentials"},{"location":"quickstart/scripted-deployment/#login-service","text":"By default, the Login Service is accessed at the URL https://auth.<domain>/ with the credentials\u2026 username: admin password: Chang3me! \u2026unless the password is overridden via the variable LOGIN_SERVICE_ADMIN_PASSWORD .","title":"Login Service"},{"location":"quickstart/scripted-deployment/#minio-object-storage","text":"By default, Minio is accessed at the URL http://console.minio.<domain>/ with the credentials\u2026 username: eoepca password: changeme \u2026unless the username/password are overridden via the variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD .","title":"Minio Object Storage"},{"location":"quickstart/scripted-deployment/#harbor-container-registry","text":"By default, Harbor is accessed at the URL https://harbor.<domain>/ with the credentials\u2026 username: admin password: changeme \u2026unless the password is overridden via the variable HARBOR_ADMIN_PASSWORD .","title":"Harbor Container Registry"},{"location":"quickstart/scripted-deployment/#protection","text":"The protection of resource server endpoints is applied with the script deploy/eoepca/eoepca-protection.sh . This script should be executed with environment variables and command-line options that are consistent with those of the main deployment (ref. script eoepca.sh ). The script eoepca-protection.sh introduces two users eric and bob to demonstrate the application of authorized access to various service endpoints: ADES, Workspace API and dummy-service (simple endpoint used for debugging). Thus, the users must first be created in the login-service and their unique IDs passed to the protection script. Usage: eoepca-protection.sh <action> <eric-id> <bob-id> <public-ip> <domain>","title":"Protection"},{"location":"quickstart/scripted-deployment/#create-test-users","text":"Access the login-service web interface ( https://auth.<domain>/ ) as user admin using the credentials configured via LOGIN_SERVICE_ADMIN_PASSWORD . Select Users -> Add person to add users eric and bob (dummy details can be used). Note the Inum (unique user ID) for each user for use with the eoepca-protection.sh script.","title":"Create Test Users"},{"location":"quickstart/scripted-deployment/#apply-protection","text":"Apply the protection\u2026 Ensure that the script is executed with the environment variables and command-line options that are consistent with those of the main deployment . ./deploy/eoepca/eoepca-protection.sh apply \"<eric-id>\" \"<bob-id>\" \"<public-ip>\" \"<domain>\"","title":"Apply Protection"},{"location":"quickstart/scripted-deployment/#post-deploy-manual-steps","text":"The scripted deployment has been designed, as far as possible, to automate the configuration of the deployed components. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026 Fix the setting UMA Resource Lifetime - change 2592000 -> 2147483647 secs Fix the setting Authentication method for the Token Endpoint for the ApplicationHub - client_secret_post -> client_secret_basic Add groups group-1 , group-2 , group-3 to ApplicationHub, and add users eric , bob to these groups","title":"Post-deploy Manual Steps"},{"location":"quickstart/scripted-deployment/#create-user-workspaces","text":"The protection steps created the test users eric and bob . For completeness we use the Workspace API to create their user workspaces, which hold their personal resources (data, processing results, etc.) within the platform - see Workspace .","title":"Create User Workspaces"},{"location":"quickstart/scripted-deployment/#using-workspace-swagger-ui","text":"The Workspace API provides a Swagger UI that facilitates interaction with the API - at the URL https://workspace-api.<domain>/docs# . Access to The Workspace API is protected, such that the necessary access tokens must be supplied in requests, which is most easily achieved by logging in via the \u2018portal\u2019. The portal is accessed at https://portal.<domain>/ . It is a rudimentary web service that facilitates establishing the appropriate tokens in the user\u2019s browser context. Login to the portal as the admin user, using the configured credentials. Access the Workspace Swagger UI at https://workspace-api.<domain>/docs . Workspaces are created using POST /workspaces (Create Workspace) . Expand the node and select Try it out . Complete the request body, such as\u2026 { \"preferred_name\" : \"eric\" , \"default_owner\" : \"d95b0c2b-ea74-4b3f-9c6a-85198dec974d\" } \u2026where the default_owner is the user ID ( Inum ) for the user - thus protecting the created workspace for the identified user.","title":"Using Workspace Swagger UI"},{"location":"quickstart/scripted-deployment/#using-curl","text":"The same can be achieved with a straight http request, for example using curl \u2026 curl -X 'POST' \\ 'https://workspace-api.192-168-49-2.nip.io/workspaces' \\ -H 'accept: application/json' \\ -H 'Content-Type: application/json' \\ -H 'X-User-Id: <admin-id-token>' \\ -d '{ \"preferred_name\": \"<workspace-name>\", \"default_owner\": \"<user-inum>\" }' Values must be provided for: admin-id-token - User ID token for the admin user workspace-name - name of the workspace, typically the username user-inum - the ID of the user for which the created workspace will be protected The ID token for the admin user can be obtained with a call to the token endpoint of the Login Service - supplying the credentials for the admin user and the pre-registered client\u2026 curl -L -X POST 'https://auth.<domain>/oxauth/restv1/token' \\ -H 'Cache-Control: no-cache' \\ -H 'Content-Type: application/x-www-form-urlencoded' \\ --data-urlencode 'scope=openid user_name is_operator' \\ --data-urlencode 'grant_type=password' \\ --data-urlencode 'username=admin' \\ --data-urlencode 'password=<admin-password>' \\ --data-urlencode 'client_id=<client-id>' \\ --data-urlencode 'client_secret=<client-secret>' A json response is returned, in which the field id_token provides the user ID token for the admin user.","title":"Using curl"},{"location":"quickstart/scripted-deployment/#using-create-workspace-helper-script","text":"As an aide there is a helper script create-workspace . The script is available in the deployment-guide repository , and can be obtained as follows\u2026 git clone git@github.com:EOEPCA/deployment-guide cd deployment-guide The create-workspace helper script requires some command-line arguments\u2026 Usage: create-workspace <domain> <user> <user-inum> [<client-id> <client-secret>] For example\u2026 ./deploy/bin/create-workspace 192 -168-49-2.nip.io eric d95b0c2b-ea74-4b3f-9c6a-85198dec974d The script prompts for the password of the admin user. By default <client-id> and <client-secret> are read from the client.yaml file that is created by the deployment script, which auto-registers a Login Service client. Thus, these args can be ommited to use the default client credentials.","title":"Using create-workspace helper script"},{"location":"quickstart/scripted-deployment/#clean-up","text":"Before initiating a fresh deployment, if a prior deployment has been attempted, then it is necessary to remove any persistent artefacts of the prior deployment. This includes\u2026 Minikube cluster Delete the minikube cluster\u2026 minikube delete If necessary specify the cluster (profile)\u2026 minikube -p <profile> delete Persistent Data In the case that the minikube none driver is used, the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. In this case, the minikube standard StorageClass is fulfilled by the hostpath provisioner, whose persistence is removed as follows\u2026 sudo rm -rf /tmp/hostpath-provisioner Client Credentials During the deployment a client of the Authorisation Server is registered, and its credentials stored for reuse in the file client.yaml . Once the cluster has been destroyed, then these client credentials become stale and so should be removed to avoid polluting subsequent deployments\u2026 rm -rf ./deploy/eoepca/client.yaml There is a helper script clean that can be used for steps 2 and 3 above, (the script does not delete the cluster). ./deploy/cluster/clean","title":"Clean-up"},{"location":"quickstart/simple-deployment/","text":"Simple Deployment \u2693\ufe0e A deployment wrapper script has been prepared for a \u2018simple\u2019 deployment - designed to get a core local deployment of the primary servies. The script deploy/simple/simple achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/simple/simple-options . The simple deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Configuration of \u2018open\u2019 interfaces - i.e. service/API endpoints that are not protected and can accessed without authentication. This facilitates experimentation with the services Configuration of ADES stage-out to a local instance of minio , on the assumption that access to CREODIAS buckets for stage-out (via Workspace) is not an option Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/simple/simple See section Deployment for more details regarding the outcome of the scripted deployment. Protection \u2693\ufe0e See section Protection for more details regarding the protection of the deployed services - which, for the simple deployment, is performed via the script deploy/simple/simple-protection \u2026 ./deploy/simple/simple-protection","title":"Simple Deployment"},{"location":"quickstart/simple-deployment/#simple-deployment","text":"A deployment wrapper script has been prepared for a \u2018simple\u2019 deployment - designed to get a core local deployment of the primary servies. The script deploy/simple/simple achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/simple/simple-options . The simple deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Configuration of \u2018open\u2019 interfaces - i.e. service/API endpoints that are not protected and can accessed without authentication. This facilitates experimentation with the services Configuration of ADES stage-out to a local instance of minio , on the assumption that access to CREODIAS buckets for stage-out (via Workspace) is not an option","title":"Simple Deployment"},{"location":"quickstart/simple-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/simple/simple See section Deployment for more details regarding the outcome of the scripted deployment.","title":"Initiate Deployment"},{"location":"quickstart/simple-deployment/#protection","text":"See section Protection for more details regarding the protection of the deployed services - which, for the simple deployment, is performed via the script deploy/simple/simple-protection \u2026 ./deploy/simple/simple-protection","title":"Protection"},{"location":"quickstart/userman-deployment/","text":"User Management Deployment \u2693\ufe0e A deployment wrapper script has been prepared for a \u2018user management\u2019 deployment - that is focused on the Login Service , PDP and User Profile . The script deploy/userman/userman achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/userman/userman-options . The user-management deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: Login Service Policy Decision Point (PDP) User Profile Other eoepca services not deployed Initiate Deployment \u2693\ufe0e Deployment is initiated by invoking the script\u2026 ./deploy/userman/userman The Login Service is accessed at the endpoint auth.<domain> - e.g. auth.192-168-49-2.nip.io .","title":"User Management"},{"location":"quickstart/userman-deployment/#user-management-deployment","text":"A deployment wrapper script has been prepared for a \u2018user management\u2019 deployment - that is focused on the Login Service , PDP and User Profile . The script deploy/userman/userman achieves this by appropriate configuration of the environment variables , before launching the eoepca.sh deployment script . The deployment configuration is captured in the file deploy/userman/userman-options . The user-management deployment applies the following configuration: Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment No TLS for service ingress endpoints Services deployed: Login Service Policy Decision Point (PDP) User Profile Other eoepca services not deployed","title":"User Management Deployment"},{"location":"quickstart/userman-deployment/#initiate-deployment","text":"Deployment is initiated by invoking the script\u2026 ./deploy/userman/userman The Login Service is accessed at the endpoint auth.<domain> - e.g. auth.192-168-49-2.nip.io .","title":"Initiate Deployment"}]}
\ No newline at end of file
diff --git a/current/sitemap.xml.gz b/current/sitemap.xml.gz
index a9817802f11b5051d0edadcd34536daffb597068..11df145162dda72e6905efda8fc35a22a77bff03 100644
GIT binary patch
delta 15
WcmdnWyp@?vzMF$%Wzj~q^^5=|VFa@P

delta 15
WcmdnWyp@?vzMF%?D`X?vdPV>ukpwmX