Merge branch 'main' into issue-890-org-export-import

.github/ISSUE_TEMPLATE/bug.yml

@@ -1,14 +1,14 @@
 name: Bug Report
-description: If something isn't working as expected and you're sure your issue is reproducible, please file this type of issue!
+description: Report a demonstrable problem caused by code in this repo.
 title: "[Bug]: "
 labels: ["bug"]
 body:
   # Version number
   - type: input
     id: version
     attributes:
-      label: Browsertrix Cloud Version
-      description: This can be found in the site footer
+      label: Browsertrix Version
+      description: This can be found in the bottom end of the Browsertrix web app.
       placeholder: "v1.5.0-beta.0-67d0c6a"
     validations:
       required: true
@@ -17,27 +17,38 @@ body:
     attributes:
       label: What did you expect to happen? What happened instead?
       description: |
-        "I was trying to modify the Page Load Timeout value in a saved workflow, however..."
-
-        Please submit any screenshots/videos that can be used to understand how to reproduce the issue. You can attach images by clicking this area to highlight it and then dragging files into the browser window.
+        A clear and concise description of the bug, and what you expected to happen instead.
         
-        If your problem is related to crawling, or something wasn't captured in the way you expect please include a link to the finished crawl/workflow if possible.
+        For issues related to crawling or replay, please include a link to the archived item and workflow when possible.
     validations:
       required: true
   # Step-by-step reproduction instructions
   - type: textarea
     attributes:
-      label: Step-by-step reproduction instructions
+      label: Reproduction instructions
+      description: Step-by-step description of how to reproduce the issue, including the page URL if applicable.
       placeholder: |
         1. Navigate to...
         2. Click on...
         3. See error...
     validations:
       required: true
+  # Screenshots / videos
+  - type: textarea
+    attributes:
+      label: Screenshots / Video
+      description: Please attach any screenshots or screen recordings that demonstrate the bug. You can attach images by clicking this area to highlight it and then dragging files into the browser window.
+  # Environment
+  - type: input
+    attributes:
+      label: Environment
+      description: Please specify your browser if the issue is related to the web app, and provide information on your operating system if you're running Browsertrix locally.
+      placeholder: |
+        Browser:
+        Browser version:
+        OS:
   # Additional details
   - type: textarea
     attributes:
       label: Additional details
-      description: Add any other relevant information here, such as your local environment if you are running Browsertrix Cloud locally.
-    validations:
-      required: false
+      description: Any additional context that helps us investigate the issue. For example, does the issue only happen in a specific browser? Are there forum discussions related to your issue? etc.

.github/ISSUE_TEMPLATE/config.yml

@@ -3,14 +3,14 @@ blank_issues_enabled: true
 
 contact_links:
   - name: Report a replay issue
-    about: Issues related to archived content not displaying properly should be reported in the ReplayWeb.page repo.
+    about: Issues related to an archived item or collection not replaying properly should be reported in the ReplayWeb.page repo.
     url: https://github.com/webrecorder/replayweb.page/issues/new?&labels=replay+bug%2Cbug&projects=&template=replay-bug.yml&title=[Replay+Bug]%3A+
   - name: Report a security vulnerability
-    about: Please do not file an issue and instead email [email protected]. We will follow up with you there!
+    about: Please email [email protected] directly. We will follow up with you there!
     url: https://webrecorder.net/.well-known/security.txt
   - name: Get help on our forum
     url: https://forum.webrecorder.net/
     about: Have a ("how do I...?") question? Not sure if your issue is reproducible? The best way to get help is on our community forum!
   - name: Check out the docs
     url: https://docs.browsertrix.cloud
-    about: Solutions to common questions may be available in the documentation!
+    about: Find solutions to common questions, such as how to install, develop, and deploy Browsertrix.

.github/ISSUE_TEMPLATE/feature-change.yml

@@ -1,49 +1,20 @@
 name: Feature / Change Request
-description: If new things should be added or something that is working as intended should be changed, please file this type of issue!
+description: Request a new feature or change to an existing feature of the app.
 title: "[Feature]: "
 labels: ["enhancement"]
 body:
-  # Context
-  - type: textarea
-    attributes:
-      label: Context
-      description: Describe any prior information that we are taking into account to inform this future development.
-      placeholder: "Now that x is done we should do y to accomplish z."
-    validations:
-      required: true
   # User story sentence
   - type: textarea
     attributes:
       label: What change would you like to see?
-      description: Describe the solution you'd like. If relevant, include ways in which you've tried to solve the issue with the current version.
-      placeholder: "As a user, I want to be able to ____ so that I can ____"
+      description: Describe the improvement or feature you'd like added to Browsertrix.
+      placeholder: I would like to be able to ____________ so that I can ____________.
     validations:
       required: true
-  # Requirements
-  - type: textarea
-    attributes:
-      label: Requirements
-      description: |
-        Intended primarily for use by Webrecorder team, leave blank if unknown.
-        
-        List the outcomes of the feature being implemented without design or implementation details.
-      placeholder: |
-        1. Item metadata should show links to the collections that the item belongs to.
-        2. Items can be added or removed from collections when editing an item.
-    validations:
-      required: false
-  # Todo
+  # Context
   - type: textarea
     attributes:
-      label: Todo
-      description: |
-        Intended primarily for use by Webrecorder team, leave blank if unknown.
-
-        Any other linked issues / tasks to complete to implement this feature.
-      placeholder: |
-        - [ ] Mockups: 
-        - [ ] Design: 
-        - [ ] UI: 
-        - [ ] API: 
+      label: Context
+      description: Any background information that helps us understand the request.
     validations:
-      required: false
+      required: true

.github/workflows/deploy-dev.yaml

@@ -15,20 +15,45 @@ jobs:
         with:
           driver-opts: network=host
 
-      - name: Login to Regsitry
+      - name: Login to Registry
         uses: docker/login-action@v2
         with:
           registry: ${{ secrets.DO_REGISTRY }}
           username: ${{ secrets.DO_API_TOKEN }}
           password: ${{ secrets.DO_API_TOKEN }}
 
-      -
-        name: Set Env Vars
+      - name: Set Env Vars
         run: |
           echo VERSION=`cat version.txt` >> $GITHUB_ENV
           echo GIT_COMMIT_HASH=`git rev-parse --short HEAD` >> $GITHUB_ENV
           echo GIT_BRANCH_NAME=`git rev-parse --abbrev-ref HEAD` >> $GITHUB_ENV
 
+      - name: Checkout values file from ops repo
+        uses: actions/checkout@v4
+        with:
+          repository: "webrecorder/browsertrix-cloud-ops"
+          path: "browsertrix-cloud-ops"
+          ssh-key: ${{ secrets.DEPLOY_KEY_OPS_REPO }}
+          sparse-checkout: |
+            scripts/decrypt-values.py
+            values/btrix-dev-values.yml
+            poetry.lock
+            pyproject.toml
+          sparse-checkout-cone-mode: false
+
+      - name: Install poetry
+        run: pipx install poetry
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: "poetry"
+
+      - name: Install vault decryption dependencies
+        working-directory: browsertrix-cloud-ops
+        run: |
+          poetry install
+
       - name: Build Backend
         uses: docker/build-push-action@v3
         with:
@@ -57,12 +82,10 @@ jobs:
       - name: Get Kubeconfig
         env:
           KUBECONFIG_DATA: ${{ secrets.KUBECONFIG_DATA }}
-          DEV_VALUES: ${{ secrets.DEV_VALUES }}
 
         run: |
           printf "$KUBECONFIG_DATA" >> ./.kubeconfig
           chmod 400 ./.kubeconfig
-          printf "$DEV_VALUES" >> ./dev-values.yaml
 
       - name: Install Kubectl
         uses: azure/setup-kubectl@v3
@@ -72,6 +95,13 @@ jobs:
         with:
           version: 3.10.2
 
+      - name: Decrypt values file
+        env:
+          ANSIBLE_VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }}
+        working-directory: browsertrix-cloud-ops
+        run: |
+          poetry run python scripts/decrypt-values.py values/btrix-dev-values.yml ../dev-values.yaml
+
       - name: Start Cluster with Helm
         run: |
           KUBECONFIG=./.kubeconfig helm upgrade --install -f ./chart/values.yaml -f ./dev-values.yaml btrix ./chart/

.github/workflows/frontend-build-check.yaml

@@ -15,7 +15,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v3
         with:
-          node-version: '16'
+          node-version: '18'
           cache: 'yarn'
           cache-dependency-path: frontend/yarn.lock
       - name: Restore cache
@@ -30,6 +30,14 @@ jobs:
         env:
           HUSKY: 0
         run: yarn install --frozen-lockfile
+      - name: Lint
+        working-directory: frontend
+        run: yarn lint:check
+      - name: Format
+        working-directory: frontend
+        # TODO Reenable when https://github.com/webrecorder/browsertrix-cloud/issues/1618 is addressed
+        # run: yarn format:check
+        run: yarn prettier --list-different .
       - name: Unit tests
         working-directory: frontend
         run: yarn test

.github/workflows/publish-helm-chart.yaml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - "*-release"
 
 jobs:
   package_chart:

.github/workflows/ui-tests-playwright.yml

@@ -22,7 +22,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v3
         with:
-          node-version: '16'
+          node-version: '18'
           cache: 'yarn'
           cache-dependency-path: frontend/yarn.lock
       - name: Install dependencies

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
 - repo: https://github.com/psf/black
-  rev: 22.12.0
+  rev: 24.1.1
   hooks:
     - id: black
       args: ["backend/btrixcloud/"]

.vscode/extensions.json

@@ -2,6 +2,7 @@
   "recommendations": [
     "dbaeumer.vscode-eslint",
     "esbenp.prettier-vscode",
+    "dbaeumer.vscode-eslint",
     "runem.lit-plugin",
     "bradlc.vscode-tailwindcss",
     "redhat.vscode-yaml",

.vscode/settings.json

@@ -20,12 +20,16 @@
     "Browsertrix",
     "btrix",
     "Elems",
+    "favicons",
     "hoverable",
+    "micromark",
     "novnc",
     "profileid",
     "tailwindcss",
     "wacz",
     "Webrecorder",
+    "wysimark",
+    "xstate",
     "zxcvbn"
   ],
   "cSpell.languageSettings": [
@@ -41,5 +45,8 @@
         "css"
       ]
     }
-  ]
+  ],
+  "eslint.workingDirectories": ["./frontend"],
+  "eslint.nodePath": "./frontend/node_modules",
+  "tailwindCSS.experimental.classRegex": ["tw`([^`]*)"]
 }

README.md

@@ -1,45 +1,42 @@
-# Browsertrix Cloud
+<div align="center">
+    <img alt="Browsertrix Logo" src="assets/browsertrix-lockup-color-dynamic.svg" width="90%">
+</div>
 
-<p align="center"><img src="/frontend/assets/btrix-cloud.svg" width="128" height="128"></p>
+&nbsp;
 
-Browsertrix Cloud is an open-source cloud-native high-fidelity browser-based crawling service designed
+Browsertrix is an open-source cloud-native high-fidelity browser-based crawling service designed
 to make web archiving easier and more accessible for everyone.
 
-The service provides an API and UI for scheduling crawls and viewing results,
-and managing all aspects of crawling process. This system provides the orchestration and management around crawling,
-while the actual crawling is performed using
-[Browsertrix Crawler](https://github.com/webrecorder/browsertrix-crawler) containers, which are launched for each crawl.
+The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using [Browsertrix Crawler](https://github.com/webrecorder/browsertrix-crawler) containers, which are launched for each crawl.
 
-See [Browsertrix Cloud](https://browsertrix.com) for a feature overview and information about Browsertrix Cloud Hosting.
+See [browsertrix.com](https://browsertrix.com) for a feature overview and information about Browsertrix hosting.
 
 ## Documentation
 
-The full docs for using, deploying and developing Browsertrix Cloud are available at: [https://docs.browsertrix.cloud](https://docs.browsertrix.cloud)
+The full docs for using, deploying, and developing Browsertrix are available at: [https://docs.browsertrix.cloud](https://docs.browsertrix.cloud)
 
-## Deployment 
+## Deployment
 
 The latest deployment documentation is available at: [https://docs.browsertrix.cloud/deploy](https://docs.browsertrix.cloud/deploy)
 
-The docs cover deploying Browsertrix Cloud in different environments using Kubernetes, from a single-node setup to scalable clusters in the cloud.
+The docs cover deploying Browsertrix in different environments using Kubernetes, from a single-node setup to scalable clusters in the cloud.
 
-Previously, Browsertrix Cloud also supported Docker Compose and podman-based deployment. This is now deprecated due to the complexity
-of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.
+Previously, Browsertrix also supported Docker Compose and podman-based deployment. This has been deprecated due to the complexity of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.
 
-Making deployment of Browsertrix Cloud as easy as possible remains a key goal, and we welcome suggestions for how we can further improve our Kubernetes deployment options.
+Making deployment of Browsertrix as easy as possible remains a key goal, and we welcome suggestions for how we can further improve our Kubernetes deployment options.
 
 If you are looking to just try running a single crawl, you may want to try [Browsertrix Crawler](https://github.com/webrecorder/browsertrix-crawler) first to test out the crawling capabilities.
 
 ## Development Status
 
-Browsertrix Cloud is currently in a beta, though the system and backend API is fairly stable, we are working on many additional features.
+Browsertrix is currently in a beta, though the system and backend API is fairly stable, we are working on many additional features.
 
 Additional developer documentation is available at [https://docs.browsertrix.cloud/develop](https://docs.browsertrix.cloud/develop/)
 
 Please see the GitHub issues and [this GitHub Project](https://github.com/orgs/webrecorder/projects/9) for our current project plan and tasks.
 
-
 ## License
 
-Browsertrix Cloud is made available under the AGPLv3 License.
+Browsertrix is made available under the AGPLv3 License.
 
 Documentation is made available under the Creative Commons Attribution 4.0 International License