Sync staging prod nov 26 (#593)

* Allow API versions in API subpath (#570) * Allow API versions in API subpath * [wri-dev-ingress-fe-internal-api.yaml] handle api/3/action ingress --------- Co-authored-by: Muhammad Ismail Shahzad <[email protected]> * Block web search indexing on backend; add tests (#574) * Block web search indexing on backend; add tests * Fix test URLs * ODP-366 (#576) * ODP-366 * Add retry * Trigger CI * Fix tooltips (#578) * Fix tooltips * Tooltip fixes * Add support for multiple applications as groups (#579) * Add support for multiple applications as groups * Separate applications from groups * Uncomment tests * Trigger CI * Trigger CI * Fix multiple applications and tests * Add missing env variable * Add group scheming env variables to test.ini --------- Co-authored-by: Luccas Mateus <[email protected]> * [ODP-375]: Homepage ui adjustment (#581) * [sitemap]: remove diallow all from sitemap * hompega ui adjustment * Add more layers * Fix url * Fix glad-s2 * Applications as objects (#582) * Applications as objects * Progress * Progress * Progress * Progress * Forgot to commit * Applications as objs frontend * Fix build * Fix tests * Tests * Fix build * Fix build * Fix tests * Fix test * Trigger CI * Rm test * Fix unit tests * Fix application images * Fix typo * Small fixes * [ODP-376] Adjust spacing in dataset callout box (#583) * [sitemap]: remove diallow all from sitemap * update caution section * update line-clamp * update style * [ODP-377] Navigation improvements - breadcrumbs (#585) * [sitemap]: remove diallow all from sitemap * Update Navigation and breadcrumb * Trigger CI * ODP-371 (Hiden topics wihout datesets from homepage) (#584) * ODP-371 (Hiden topics wihout datesets from homepage) * Fix tests * Fix homepage test * Fix tests * Check viewport * Fix tests * Fix test * Add debugging * Fix tests * Fix tests * Enforce sync requests on tests * Fix testx * Fix vulnerabilities (#587) * [main.yml] Enable trivy scanning * [main.yml] Add trivy mirror * Add bots to block list * Remove duplicates * Fix vulnerabilities * Remove location from nginx block list * Fix 375 (#590) * [sitemap]: remove diallow all from sitemap * fix odp-375 issue * update font and then text color to match figma * Add custom harvester to ckanext-wri (#586) * Add custom harvester to ckanext-wri * Add comment to clarify usage of package_show inside package_create * Add harvesting to deployments/dev env * Fix harvesting * More efifcient homepage * Fix vulnerability axios * Logan requests * Upgrade node * Upgrade node * Fix topics page * Fix names * Add private-admin subpath --------- Co-authored-by: Michael Polidori <[email protected]> Co-authored-by: Muhammad Ismail Shahzad <[email protected]> Co-authored-by: Stephen Oni <[email protected]>
wri · Nov 26, 2024 · 8ef158a · 8ef158a
1 parent 83e5bab
commit 8ef158a
Show file tree

Hide file tree

Showing 178 changed files with 15,295 additions and 29,200 deletions.
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -189,6 +189,7 @@ jobs:
         env:
           REGISTRY: ${{ steps.login-ecr.outputs.registry }}
           REPOSITORY: ${{ secrets.ECR_CKAN_REPO }}
+          TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db
         with:
           image-ref: '${{ env.REGISTRY }}/${{ env.REPOSITORY }}:prod-${{ github.sha }}'
           format: 'sarif'
@@ -203,6 +204,7 @@ jobs:
         env:
           REGISTRY: ${{ steps.login-ecr.outputs.registry }}
           REPOSITORY: ${{ secrets.ECR_FRONTEND_REPO }}
+          TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db
         with:
           image-ref: '${{ env.REGISTRY }}/${{ env.REPOSITORY }}:prod-${{ github.sha }}'
           format: 'sarif'

diff --git a/ckan-backend-dev/.env.example b/ckan-backend-dev/.env.example
@@ -84,7 +84,7 @@ DATAPUSHER_REWRITE_RESOURCES=True
 DATAPUSHER_REWRITE_URL=http://ckan-dev:5000
 
 # Extensions
-CKAN__PLUGINS=image_view text_view webpage_view resource_proxy datatables_view datastore datapusher activity s3filestore scheming_datasets scheming_organizations scheming_groups wri auth hierarchy_display hierarchy_form hierarchy_group_form issues envvars
+CKAN__PLUGINS=image_view text_view webpage_view resource_proxy datatables_view datastore datapusher activity s3filestore scheming_datasets scheming_organizations scheming_groups harvest wri wri_harvester auth hierarchy_display hierarchy_form hierarchy_group_form issues envvars
 CKAN__VIEWS__DEFAULT_VIEWS=image_view text_view webpage_view datatables_view
 CKAN__HARVEST__MQ__TYPE=redis
 CKAN__HARVEST__MQ__HOSTNAME=redis
@@ -102,7 +102,7 @@ CKANEXT__S3FILESTORE__HOST_NAME=http://minio:9000
 # scheming
 CKAN___SCHEMING__DATASET_SCHEMAS=ckanext.wri.schema:ckan_dataset.yaml
 CKAN___SCHEMING__ORGANIZATION_SCHEMAS=ckanext.scheming:custom_org_with_address.json
-CKAN___SCHEMING__GROUP_SCHEMAS=ckanext.scheming:custom_group_with_status.json
+CKAN___SCHEMING__GROUP_SCHEMAS=ckanext.wri.schema:wri_application.json
 CKAN___SCHEMING__PRESETS=ckanext.wri.schema:presets.json
 
 # auth

diff --git a/ckan-backend-dev/ckan/Dockerfile.dev b/ckan-backend-dev/ckan/Dockerfile.dev
@@ -30,7 +30,9 @@ RUN pip3 install -e 'git+https://github.com/datopian/ckanext-scheming.git@098373
     pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-s3filestore/wri/cost-splitting-orgs/dev-requirements.txt' && \
     pip3 install -e 'git+https://github.com/datopian/ckanext-auth.git@fd7113f076336c13b9952d6b2161c0f4339e5cc6#egg=ckanext-auth' && \
     pip3 install -e 'git+https://github.com/ckan/ckanext-hierarchy.git@master#egg=ckanext-hierarchy' && \
-    pip3 install -e 'git+https://github.com/datopian/[email protected]#egg=ckanext-issues'
+    pip3 install -e 'git+https://github.com/datopian/[email protected]#egg=ckanext-issues' && \
+    pip3 install -e 'git+https://github.com/ckan/[email protected]#egg=ckanext-harvest' && \
+    pip3 install -r 'https://raw.githubusercontent.com/ckan/ckanext-harvest/refs/tags/v1.6.0/requirements.txt'
 
 # Required by shapely dependency
 RUN apk --update add build-base libxslt-dev python3-dev 

diff --git a/ckan-backend-dev/ckan/scripts/init-extensions.sh b/ckan-backend-dev/ckan/scripts/init-extensions.sh
@@ -4,6 +4,7 @@ docker exec ckan-wri sh -c "ckan -c production.ini issuesdb"
 docker exec ckan-wri sh -c "ckan -c production.ini notificationdb"
 docker exec ckan-wri sh -c "ckan -c production.ini pendingdatasetsdb"
 docker exec ckan-wri sh -c "ckan -c production.ini resourcelocationdb"
+docker exec ckan-wri sh -c "ckan -c production.ini db upgrade -p harvest"
 docker exec ckan-wri sh -c "unset CKAN__DATAPUSHER__API_TOKEN"
 docker cp ./ckan-backend-dev/ckan/scripts/datapusher.sh ckan-wri:/srv/app/datapusher.sh
 echo "Adding +x permissions"

diff --git a/ckan-backend-dev/src/ckanext-wri/README.md b/ckan-backend-dev/src/ckanext-wri/README.md
@@ -58,6 +58,13 @@ This is the WRI Open Data Portal extension for CKAN. It contains CKAN backend cu
 
 For information on the Prefect variables/blocks required for migrations, see the [migration README](../../../migration/README.md).
 
+## Plugins
+
+This extension includes the following plugins:
+- `wri` - The main plugin that loads ckanext-wri.
+- `wri_api_tracking` - A plugin that tracks API usage with Google Analytics.
+- `wri_harvester` - A custom harvester for ingesting datasets between WRI CKAN portals.
+
 ## Notifications Feature
 
 This extension includes a notification feature that utilizes its own database, action endpoints, and custom validators.
@@ -650,6 +657,38 @@ Most fields that are not mapped directly to CKAN are stored in a custom field ca
 }
 ```
 
+## WRI Harvester
+
+This extension includes a custom harvester for the ingesting datasets from other instances of this project (e.g., harvesting datasets from Production to Staging). Most of the customizations are behind-the-scenes, but a new config option was added, and it will need to be set if you want to create new resources within the datasets instead of using links to the original resources.
+
+In the harvest source configuration section, you can add the following option:
+
+```
+{
+    "create_resources": true
+}
+```
+
+Along with the new config option, all of [the existing harvest options](https://github.com/ckan/ckanext-harvest/tree/v1.6.0?tab=readme-ov-file#the-ckan-harvester) should still work as expected, such as specifying organizations to include or exclude, using an API key/token, etc.
+
+For example, let's assume that you want to:
+- Create new resources (full resources, instead of merely metadata pointing to the URL of the original data)
+- Create new organizations to match the source instance (if they don't exist already)
+- Only harvest datasets from two source organizations (`global-forest-watch` and `land-carbon-lab`), and
+- a single source group (`forests`)
+
+Your harvest source configuration would look something like this:
+
+```
+{
+    "remote_orgs": "create",
+    "remote_groups": "create",
+    "create_resources": true,
+    "organizations_filter_include": ["global-forest-watch", "land-carbon-lab"],
+    "groups_filter_include": ["forests"]
+}
+```
+
 ## API Analytics Tracking
 
 **Note**: This plugin requires a Google Analytics account with a Measurement ID and API Secret to work. For more information on how to set up Google Analytics, see the [Google Analytics documentation](https://support.google.com/analytics/answer/9304153?hl=en). Steps to set up reports can be found in the main `docs` directory in this repo (see [`wri-odp/docs/ga-api/README.md`](../docs/ga-api/README.md)).