Skip to content

Commit

Permalink
Deployed a7c7ebb to current with MkDocs 1.5.3 and mike 2.0.0
Browse files Browse the repository at this point in the history
  • Loading branch information
EOEPCA CI committed Apr 3, 2024
1 parent bec732b commit bc54e85
Show file tree
Hide file tree
Showing 4 changed files with 7 additions and 7 deletions.
10 changes: 5 additions & 5 deletions current/eoepca/data-access/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -2115,15 +2115,15 @@ <h4 id="starting-the-harvester">Starting the Harvester<a class="headerlink" href
<p>The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue.</p>
<h5 id="one-off-harvests-via-the-cli">One-off harvests via the CLI<a class="headerlink" href="#one-off-harvests-via-the-cli" title="Permanent link">⚓︎</a></h5>
<p>In order to start a harvest from the CLI, the operator first needs to connect to the kubernetes pod of the harvester. Within that pod, the harvest can be executed like this&hellip;
<div class="language-bash highlight"><pre><span></span><code><span id="__span-4-1"><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a>python3<span class="w"> </span>-m<span class="w"> </span>harvester<span class="w"> </span>harvest<span class="w"> </span>--config-file<span class="w"> </span>/config-run.yaml<span class="w"> </span>--host<span class="w"> </span>data-access-redis-master<span class="w"> </span>--port<span class="w"> </span><span class="m">6379</span><span class="w"> </span>Creodias-Opensearch
<div class="language-bash highlight"><pre><span></span><code><span id="__span-4-1"><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a>python3<span class="w"> </span>-m<span class="w"> </span>harvester<span class="w"> </span>harvest<span class="w"> </span>--config-file<span class="w"> </span>/config-run.yaml<span class="w"> </span>Sentinel2
</span></code></pre></div></p>
<p>This will invoke the Creodias-Opensearch harvester with default arguments. When some values are to be overridden, the &ndash;values switch can be used to pass override values. These values must be a JSON string. The following example adjusts the begin and end times of the query parameters&hellip;
<div class="language-bash highlight"><pre><span></span><code><span id="__span-5-1"><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a>python3<span class="w"> </span>-m<span class="w"> </span>harvester<span class="w"> </span>harvest<span class="w"> </span>--config-file<span class="w"> </span>/config-run.yaml<span class="w"> </span>--host<span class="w"> </span>data-access-redis-master<span class="w"> </span>--port<span class="w"> </span><span class="m">6379</span><span class="w"> </span>Creodias-Opensearch<span class="w"> </span>--values<span class="w"> </span><span class="s1">&#39;{&quot;resource&quot;: {&quot;query&quot;: {&quot;time&quot;: {&quot;begin&quot;: &quot;2020-09-10T00:00:00Z&quot;, &quot;end&quot;: &quot;2020-09-11T00:00:00Z&quot;}}}}&#39;</span>
<p>This will invoke the <code>Sentinel2</code> harvester with default arguments. When some values are to be overridden, the &ndash;values switch can be used to pass override values. These values must be a JSON string. The following example adjusts the begin and end times of the query parameters&hellip;
<div class="language-bash highlight"><pre><span></span><code><span id="__span-5-1"><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a>python3<span class="w"> </span>-m<span class="w"> </span>harvester<span class="w"> </span>harvest<span class="w"> </span>--config-file<span class="w"> </span>/config-run.yaml<span class="w"> </span>Sentinel2<span class="w"> </span>--values<span class="w"> </span><span class="s1">&#39;{&quot;resource&quot;: {&quot;query&quot;: {&quot;time&quot;: {&quot;begin&quot;: &quot;2020-09-10T00:00:00Z&quot;, &quot;end&quot;: &quot;2020-09-11T00:00:00Z&quot;}}}}&#39;</span>
</span></code></pre></div></p>
<h5 id="harvests-via-the-harvest-daemon">Harvests via the harvest daemon<a class="headerlink" href="#harvests-via-the-harvest-daemon" title="Permanent link">⚓︎</a></h5>
<p>The harvester pod runs a service listening on a redis queue. When a message is read from the queue, it will be read as a JSON string, expecting an object with at least a <code>name</code> property. Optionally, it can also have a <code>values</code> property, working in the same way as with CLI <code>--values</code>.</p>
<p>To send a harvesting request via the redis queue, it is necessary to connect to the redis pod and execute the redis-cli there. Then the following command can be used to achieve the same result as above with CLI harvesting&hellip;
<div class="language-bash highlight"><pre><span></span><code><span id="__span-6-1"><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a>redis-cli<span class="w"> </span>LPUSH<span class="w"> </span><span class="s1">&#39;{&quot;name&quot;: &quot;Creodias-Opensearch&quot;, &quot;values&quot;: {&quot;resource&quot;: {&quot;query&quot;: {&quot;time&quot;: {&quot;begin&quot;: &quot;2020-09-10T00:00:00Z&quot;, &quot;end&quot;: &quot;2020-09-11T00:00:00Z&quot;}}}}}&#39;</span>
<div class="language-bash highlight"><pre><span></span><code><span id="__span-6-1"><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a>redis-cli<span class="w"> </span>LPUSH<span class="w"> </span><span class="s1">&#39;{&quot;name&quot;: &quot;Sentinel2&quot;, &quot;values&quot;: {&quot;resource&quot;: {&quot;query&quot;: {&quot;time&quot;: {&quot;begin&quot;: &quot;2020-09-10T00:00:00Z&quot;, &quot;end&quot;: &quot;2020-09-11T00:00:00Z&quot;}}}}}&#39;</span>
</span></code></pre></div></p>
<h4 id="results-of-the-harvesting">Results of the harvesting<a class="headerlink" href="#results-of-the-harvesting" title="Permanent link">⚓︎</a></h4>
<p>The harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created.</p>
Expand Down Expand Up @@ -2285,7 +2285,7 @@ <h3 id="default-harvesting">Default Harvesting<a class="headerlink" href="#defau
<p>At deployment time the <code>harvester</code> helm values include configuration that populates a default harvester configuration, that is prepared in the file <code>/config.yaml</code> in the <code>harvester</code> pod.</p>
<p>The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section <a href="#data-layer-configuration">Data-layer Configuration</a>.</p>
<p>The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the <code>rm/harvester</code> service and executing the command&hellip;
<div class="language-text highlight"><pre><span></span><code><span id="__span-14-1"><a id="__codelineno-14-1" name="__codelineno-14-1" href="#__codelineno-14-1"></a>python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch
<div class="language-text highlight"><pre><span></span><code><span id="__span-14-1"><a id="__codelineno-14-1" name="__codelineno-14-1" href="#__codelineno-14-1"></a>python3 -m harvester harvest --config-file /config-run.yaml Sentinel2
</span></code></pre></div></p>
<h3 id="ad-hoc-harvesting">Ad-hoc Harvesting<a class="headerlink" href="#ad-hoc-harvesting" title="Permanent link">⚓︎</a></h3>
<p>Ad-hoc harvesting can be invoked by provision of a suitable <code>config.yaml</code> into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time.</p>
Expand Down
2 changes: 1 addition & 1 deletion current/quickstart/creodias-deployment/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -1596,7 +1596,7 @@ <h2 id="deployment">Deployment<a class="headerlink" href="#deployment" title="Pe
&hellip;noting that this step is a customised version of that described in section <a href="../scripted-deployment/#deployment">Deployment</a>.</p>
<h2 id="harvest-creodias-data">Harvest CREODIAS Data<a class="headerlink" href="#harvest-creodias-data" title="Permanent link">⚓︎</a></h2>
<p>The harvester can be <a href="../../eoepca/data-access/#harvester-helm-configuration">deployed with a default configuration</a> file at <code>/config.yaml</code>. As described in the <a href="../../eoepca/data-access/#starting-the-harvester">Data Access section</a>, harvesting according to this configuration can be triggered with&hellip;
<div class="language-text highlight"><pre><span></span><code><span id="__span-1-1"><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a>kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch
<div class="language-text highlight"><pre><span></span><code><span id="__span-1-1"><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a>kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml Sentinel2
</span></code></pre></div></p>
<p>See the <a href="#harvester">Harvester</a> section below for an explanation of this harvester configuration.</p>
<p>See <a href="https://creodias.docs.cloudferro.com/en/latest/eodata/EOData-Catalogue-API-Manual-on-Creodias.html">EOData Catalogue API Manual on CREODIAS</a> for details regarding access to the CREODIAS data offering.</p>
Expand Down
2 changes: 1 addition & 1 deletion current/search/search_index.json

Large diffs are not rendered by default.

Binary file modified current/sitemap.xml.gz
Binary file not shown.

0 comments on commit bc54e85

Please sign in to comment.