Merge pull request #44 from plantinformatics/develop

merge down latest self hosting additions

.github/workflows/repo__auto-update-param-branch.yml

@@ -0,0 +1,22 @@
+name: repo__auto-update-param-branch
+
+on:
+  push:
+    branches:
+      - main # Trigger on pushes to main branch
+  workflow_dispatch:  # Allow manual triggering
+
+jobs:
+  auto-merge:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+      - name: Merge develop -> test-space/pk
+        run: |
+          git config user.name github-actions
+          git config user.email [email protected]
+          git checkout develop
+          git rebase origin/develop
+          git push --force-with-lease origin develop

docs/Self-Hosting/Extended-features.md

@@ -0,0 +1,90 @@
+# Extended features
+
+!!! note
+
+    The vast majority of the configuration of Pretzel is done via the docker compose yaml and environment files.
+    As such a lot of the below points reference these files located in our [github](https://github.com/plantinformatics/pretzel/tree/develop/doc/adminGuides/) repo.
+
+## Modifying the Pretzel API
+
+### Deploying older Pretzel API versions
+
+A complete list of released pretzel images can be found on our [dockerhub](https://hub.docker.com/r/plantinformaticscollaboration/pretzel/tags) page.
+
+To change versions just edit the `image:` field in the `docker-compose.yaml` file.
+
+``` yaml title="pretzel.compose.prod.yaml"
+api: # node environment
+...
+image: plantinformaticscollaboration/latest # replace latest with the desired version
+```
+Followed by deploying the new image with the following command:
+
+``` bash title="Deploying the new image"
+docker compose --file docker-compose.prod.yaml --env-file pretzel.compose.prod.env up -d
+```
+### Deploying custom Pretzel API images
+
+If deploying a custom image you will need to build the image from source. See [Building custom Pretzel API images](#building-custom-pretzel-api-images) 
+
+To use the custom image you will need to update the `image:` field in the `docker-compose.yaml` file.
+
+``` yaml title="pretzel.compose.prod.yaml"
+api: # node environment
+...
+image: # Path to the custom image or image tag name
+``` 
+
+Followed by deploying the new image with the following command:
+
+``` bash title="Deploying the new image"
+docker compose --file docker-compose.prod.yaml --env-file pretzel.compose.prod.env up -d
+```
+### Building custom Pretzel API images
+
+In the home directory of the cloned repo run the following commands to build the image:
+
+``` bash title="Building the image"
+mkdir -p ~/log/build/docker # create the log directory if it doesn't exist
+sudo docker build . > ~/log/build/docker/$logDate
+```
+
+The image tag name will be displayed in the log file. Use this tag name in the `image:` field in the `docker-compose.yaml` file. As described in [Deploying custom Pretzel API images](#deploying-custom-pretzel-api-images).
+
+---
+
+## Add a custom landingPage 
+If a directory is provided, this content is displayed in the Pretzel home page before the user logs in. This only supports static html content.
+
+``` env title="pretzel.compose.prod.env"
+landingPage= # directory containing static html content.
+``` 
+
+## Enabling/Disabling email verification
+
+Only enable email verification if you have set up an Amazon SES SMTP email address. Otherwise accounts will be created with no means to verify them.
+
+``` env title="pretzel.compose.prod.env"
+EMAIL_VERIFY= # ADMIN or NONE
+``` 
+### Setting up email verification
+See [Setting an Amazon SES SMTP email address](https://docs.aws.amazon.com/ses/latest/dg/setting-up.html) for more information and for the values to use for the following variables.
+
+``` env title="pretzel.compose.prod.env"
+# [email protected]
+# EMAIL_HOST=email-smtp.<region>.amazonaws.com
+EMAIL_PORT=25
+# EMAIL_PASS=...
+# EMAIL_USER=...
+# [email protected]
+``` 
+
+## Enable results caching
+
+Added in v3.1.0.
+
+The Pretzel API server will cache results for common requests, enabling them to be served more quickly and efficiently.  This directory will be created within the Pretzel API server container, or if it is configured as a shared volume, then the resultsCache will be stored in the directory passed in.  The benefit of this is that when the container is re-created, the resultsCache will be preserved.  This is not essential, but will improve performance in particular for histograms where the chromosomes (Blocks) contain 1e5 - 1e7 features.
+
+``` env title="pretzel.compose.prod.env"
+    resultsCacheDir= # directory for results cache
+``` 

docs/Local-Setup/setup.md → docs/Self-Hosting/Quick-setup.md

@@ -1,4 +1,4 @@
-# Local setup
+# Quick setup
 
 !!! warning
 
@@ -22,33 +22,41 @@ mkdir pretzel-docker-config && cd pretzel-docker-config && touch docker-compose.
 
 Using your text editor of choice create an environment file defining the configuration of directories, names and ports for the servers.
 
-Sample files are shown below
+Sample files are shown below, latest version can be found on our [github](https://github.com/plantinformatics/pretzel/tree/develop/doc/adminGuides/)
 
 ``` env title="pretzel.compose.prod.env"
-# prod
-
-DATA_DIR= # mongoDB directory
-mntData= # blastBD and VCF file directroy
-landingPage= # optional accepts basic HTML and CSS
-
-DB_NAME=admin # used to specify mongo db name
-API_HOST= # URL for pretzel if public, used for email verification
-API_PORT_PROXY=80 # used for email verification
-API_PORT_EXT=3000
+# Prod
+
+# DATA_DIR= # directory for mongoDb database
+# mntData= # directory for Blast and VCF databases
+# landingPage= # directory containing index.html and web page content to display on the home page before the user logs in.
+# This dir maps to /app/node_modules/flat-cache/.cache and contains 1 file : resultsCache
+# resultsCacheDir=/home/ec2-user/home/resultsCache/prod
+#PORT=3010
+
+# Not used ?
+# INSTANCE=agg
+DB_NAME=pretzel
+# API_HOST=agg.plantinformatics.io
+API_PORT_PROXY=80
+API_PORT_EXT=3010
+hostIp=blastserver
+# Flask port is now internal to the compose network, so use fixed (default) 4000; 
+# no need to configure via FLASK_PORT / BLASTSERVER_PORT.
 BLASTSERVER_PORT=4000
 # The value of API_PORT_PROXY is not (currently) a port, it is just defined or undefined.
 MONGO_DEFAULT_PORT=27017
-EMAIL_VERIFY=NONE # can be NONE (no verification) or ADMIN 
-EMAIL_ADMIN= # used if email verification is used
-EMAIL_HOST= # used if email verification is used
+EMAIL_VERIFY=ADMIN
+# EMAIL_ADMIN=user-email[email protected]
+# EMAIL_HOST=email-smtp.<region>.amazonaws.com
 EMAIL_PORT=25
-EMAIL_PASS= # used if email verification is used
-EMAIL_USER= # used if email verification is used
-EMAIL_FROM= # used if email verification is used
+# EMAIL_PASS=...
+# EMAIL_USER=...
+# [email protected]
+# This enables using Feature.value_0 as an index field; all datbases now contain this field and this option can be made to default to true (1).
 use_value_0=1
-# in future use_value_0 will always be true (1) and this variable will be deprecated. Indicates that the database Feautures contain .value_0 which is equal to .value[0] and is used to make location-based search query faster.
-germinate_password= # optional
-handsOnTableLicenseKey=non-commercial-and-evaluation
+# If a handsOnTableLicenseKey is not provided, a prompt message will be displayed in the GUI. Insert your license here; Hands On Table allows non-commercial and evaluation use.
+# handsOnTableLicenseKey=non-commercial-and-evaluation
 
 ```
 
@@ -67,7 +75,8 @@ networks:
 
 services:
   database: # mongo database
-    image: 9b5c4a4fdcb5 # 4.2.24.  library/mongo-4.2.24 # base image off dockerhub
+    # MongoDb up to v5 has been tested OK
+    image: mongo:4.2.24
 #    environment:
 #      - "MONGO_INITDB_ROOT_USERNAME=${DB_USER}"
 #      - "MONGO_INITDB_ROOT_PASSWORD=${DB_PASS}"
@@ -85,13 +94,15 @@ services:
     build:
       context: .
       dockerfile: ./scripts/Dockerfile
-    image: plantinformaticscollaboration/pretzel:v2.17.7b # specifying name for built container
+    image: plantinformaticscollaboration/pretzel:v3.1.0
     command: node /app/lb3app/server/server.js
     environment:
       - "API_HOST=${API_HOST}"
       - "API_PORT_EXT=${API_PORT_EXT}"
       - "API_PORT_PROXY=${API_PORT_PROXY}"
       - "hostIp=${hostIp}"
+      # Flask port is now internal to the compose network, so use fixed (default) 4000; 
+      # no need to configure via FLASK_PORT / BLASTSERVER_PORT.
       - "FLASK_PORT=4000"       # ${BLASTSERVER_PORT}"
       - "DB_HOST=database"
       - "DB_PORT=${MONGO_DEFAULT_PORT}"
@@ -109,24 +120,26 @@ services:
       - "handsOnTableLicenseKey=${handsOnTableLicenseKey}"
     volumes:
       # landingPage
-      # - $landingPage:/app/client/landingPageContent
+      - $landingPage:/app/client/landingPageContent
       # blastVolume
       - $mntData/blast:$mntData/blast
       # vcfVolume
       - $mntData/vcf:$mntData/vcf
+      - ${resultsCacheDir}:/app/node_modules/flat-cache/.cache
+
     ports:
       # match ext / int ports for loopback
       - "${API_PORT_EXT}:${API_PORT_EXT}"
     networks:
       - pretzel-prod
 
-  blastserver: # Python Flask blastn server, used for DNA Sequence Search
-    image: plantinformaticscollaboration/blastserver:latest # based on python
+  blastserver: # Python Flask blastn server, based on a python image, used for DNA Sequence Search
+    image: plantinformaticscollaboration/blastserver:latest
     environment:
       - "FLASK_PORT=${BLASTSERVER_PORT}"
     volumes:
       # mntData=/mnt/data_blast
-      - $mntData/blast:$mntData/blast
+      - $mntData/blast:/mnt/data/blast
       # Enables scripts/blastn_cont.bash to run blastn via docker
       - /usr/bin/docker:/usr/bin/docker
       - /var/run/docker.sock:/var/run/docker.sock

docs/index.md

@@ -1,16 +1,56 @@
+---
+hide:
+  - navigation
+  - toc
+---
+
 # About Pretzel
 
-<img src="https://plantinformatics.io/landingPageContent/pretzel-logo.png" style="float: left; margin-right: 10px;">
+![Image title](https://github.com/user-attachments/assets/e3326edb-1a3c-4bbe-830e-8298ff995e88){ align=left width="600" }
+![Peek 2024-11-01 21-23](https://github.com/user-attachments/assets/41daaab6-7123-4e09-aa42-eafd5180772e){ align=left width="600" }
 
 Pretzel is an interactive web application designed to enable rapid translation of genomics outputs for any species into concrete answers to research and breeding questions
 
-Learn about how Pretzel can connect legacy QTL results and pan genomes to the AGG with [User Story 3 - From SSRs to pan genomes with Pretzel: Two decades of wheat yield QTLs on chromosome 7A](https://docs.plantinformatics.io/User-Stories/User-story-3/).
+<br>
+
+<div class="grid cards" markdown>
+
+-   :material-barley:{ .lg .middle } __Connect and use the latest AGG data__
+
+    ---
+
+    Access the latest AGG data and use it to answer your research questions
+
+    [:octicons-arrow-right-24: Get started now](https://agg.plantinformatics.io/)
+
+-   :material-rocket-launch:{ .lg .middle } __More than just data, we have built in tools__
+
+    ---
+
+    Learn about how Pretzel can connect legacy QTL results and pan genomes to AGG data
+
+    [:octicons-arrow-right-24: Try our worked example](https://docs.plantinformatics.io/User-Stories/User-story-3/)
+
+
+-   :material-pretzel:{ .lg .middle } __Host your own instance of Pretzel__
+
+    ---
+
+    Get started with hosting your own instance of Pretzel in minutes
+
+    [:octicons-arrow-right-24: Quick setup](Self-Hosting/Quick-setup.md)
+
+-   :material-human-greeting:{ .lg .middle } __Have more questions?__
+
+    ---
+
+    We're always happy to help, so please get in touch if you have any questions
+
+    [:octicons-arrow-right-24: Email us now](mailto:[email protected])
 
-![Peek 2024-11-01 21-23](https://github.com/user-attachments/assets/41daaab6-7123-4e09-aa42-eafd5180772e)
+</div>
 
-Check out how we [Identified virus resistant PGRs in field pea using Pretzel](https://docs.plantinformatics.io/User-Stories/User-story-1/).
-
-![enter image description here](https://pretzel-images-public.s3.ap-southeast-2.amazonaws.com/user-stories/user-story-1/us-1-29.png)
+<br>
 
 ## Funding
 Pretzel is currently (2022-) funded as part of the Australian Grains Genebank Strategic Partnership, a $30M joint investment between the Victorian State Government and Grains Research and Development Corporation (GRDC) that aims to unlock the genetic potential of plant genetic resources for the benefit of the Australian grain growers. [https://agriculture.vic.gov.au/crops-and-horticulture/the-australian-grains-genebank](https://agriculture.vic.gov.au/crops-and-horticulture/the-australian-grains-genebank)

mkdocs.yml

@@ -47,8 +47,9 @@ nav:
     - Basic-Functions/Axis-flip-orientation.md
     - Basic-Functions/Feature-search.md
 
-  - Local Setup:
-    - Local-Setup/setup.md
+  - Self Hosting:
+    - Self-Hosting/Quick-setup.md
+    - Self-Hosting/Extended-features.md
 markdown_extensions:
   - toc:
       permalink: true
@@ -63,6 +64,9 @@ markdown_extensions:
       alternate_style: true
   - attr_list
   - md_in_html
+  - pymdownx.emoji:
+      emoji_index: !!python/name:material.extensions.emoji.twemoji
+      emoji_generator: !!python/name:material.extensions.emoji.to_svg
 plugins:
   - search
   - redirects: