Skip to content

Commit

Permalink
Merge branch 'master' into activity-creation
Browse files Browse the repository at this point in the history
  • Loading branch information
brrkrmn committed Feb 26, 2024
2 parents 29a70a1 + 81a6269 commit 138b9c0
Show file tree
Hide file tree
Showing 192 changed files with 25,181 additions and 20,772 deletions.
2 changes: 1 addition & 1 deletion .github/actions/comment_action/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ inputs:
runs:
using: "composite"
steps:
- uses: actions/github-script@v5
- uses: actions/github-script@v7
with:
github-token: ${{ inputs.token }}
script: |
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/build_deploy_frontend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ jobs:

- uses: unstructuredstudio/zubhub/.github/actions/scp_action@master
with:
host: ${{ vars.DO_BACKEND_HOST }}
username: ${{ vars.DO_BACKEND_USERNAME }}
host: ${{ vars.DO_FRONTEND_HOST }}
username: ${{ vars.DO_FRONTEND_USERNAME }}
key: ${{ secrets.DO_SSHKEY }}
source: "."
target: "/home/zubhub-frontend/zubhub"
Expand All @@ -65,7 +65,7 @@ jobs:
script: |
docker system prune -a -f
volumes=$(docker volume ls -q)
# Loop through the volumes
while IFS= read -r volume; do
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/create_destroy_test_vm.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
- name: Get checkout commit ref and repo
run: |
# for context read this https://github.com/Arelle/Arelle/pull/938/files#r1377926250
# for context read this https://github.com/Arelle/Arelle/pull/938/files#r1377926250
merge_commit_sha=${{ github.event.pull_request.merge_commit_sha }}
head_ref=${{ github.event.pull_request.head.ref }}
target_repo=${{ github.repository }}
Expand Down Expand Up @@ -70,7 +70,7 @@ jobs:
script: |
# This script will be executed inside a composite action.
# rename GITHUB_OUTPUT so that it's clear that it's a
# rename GITHUB_OUTPUT so that it's clear that it's a
# composite output and should be handled differently from $GITHUB_OUTPUT
COMPOSITE_OUTPUT=$GITHUB_OUTPUT
NEW_DROPLET_NAME=zubhub-test-${{ github.event.pull_request.number }}
Expand Down Expand Up @@ -279,6 +279,8 @@ jobs:
Test VM is ready ✅✅✅
You can access it with the url:
https://${{env.FRONTEND_DOMAIN}}.unstructured.studio
Default username: dummy
default password: dummy_password
######################################################################################

#################################################################################
Expand All @@ -304,7 +306,7 @@ jobs:
script: |
# This script will be executed inside a composite action.
# rename GITHUB_OUTPUT so that it's clear that it's a
# rename GITHUB_OUTPUT so that it's clear that it's a
# composite output and should be handled differently from $GITHUB_OUTPUT
COMPOSITE_OUTPUT=$GITHUB_OUTPUT
# for events like unlabelled, github.event.pull_request.number is available
Expand Down
87 changes: 87 additions & 0 deletions .github/workflows/pre_commit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: pre-commit
on:
pull_request:

jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Set up Docker Compose
run: |
sudo curl -L "https://github.com/docker/compose/releases/download/v2.23.3/docker-compose-$(uname -s)-$(uname -m)" \
-o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
- name: Get number of PR commits + 1
run: echo "PR_FETCH_DEPTH=$(( ${{ github.event.pull_request.commits }} + 1 ))" >> $GITHUB_ENV

- uses: unstructuredstudio/zubhub/.github/actions/checkout@master
with:
ref: ${{ github.head_ref }}
fetch_depth: ${{ env.PR_FETCH_DEPTH }}

- name: Undo all PR commits and keep changes in staging area
run: |
# some of our pre-commit hooks uses `git diff --name-only --cached` to get the list
# changed files. This step undos the most recent commits in the checkout,
# leaving the changes in the staging area for our `git diff ..` to find.
git reset --soft HEAD~$(( ${{ env.PR_FETCH_DEPTH }} - 1 ))
- run: |
# Running pre-commit on the following files:
target_files=$(git diff --name-only --cached)
echo "Running pre-commit on the following files:"
echo "$target_files"
- uses: actions/setup-python@v3

- uses: actions/setup-node@v4
with:
node-version: 16
cache: 'npm'
cache-dependency-path: zubhub_frontend/zubhub/package-lock.json

- run: npm ci --prefix ./zubhub_frontend/zubhub/

- uses: pre-commit/[email protected]
id: pre_commit
with:
# by default pre-commit/action runs `pre-commit run --all-files ...`.
# Here we use empty extra_args to force it to run `pre-commit run ...` instead.
# this is important because we want it to only run on the files that have changed
extra_args:
continue-on-error: true

- name: Success comment
if: steps.pre_commit.outcome == 'success'
uses: unstructuredstudio/zubhub/.github/actions/comment_action@master
with:
token: ${{ secrets.GITHUB_TOKEN }}
issue_number: ${{ github.event.pull_request.number }}
owner: ${{ github.repository_owner }}
repo: ${{ github.event.repository.name }}
message: |
All pre-commit checks passed ✅✅✅
- name: Failure comment
if: steps.pre_commit.outcome != 'success'
uses: unstructuredstudio/zubhub/.github/actions/comment_action@master
with:
token: ${{ secrets.GITHUB_TOKEN }}
issue_number: ${{ github.event.pull_request.number }}
owner: ${{ github.repository_owner }}
repo: ${{ github.event.repository.name }}
message: |
pre-commit checks failed ❌❌❌
make sure you have pre-commit set up locally (see the README.md for instructions).
If you have set up pre-commit locally, you may need to undo this commit and
run "pre-commit" locally to see what the errors are.
- name: Failure exit
if: steps.pre_commit.outcome != 'success'
run: exit 1
91 changes: 91 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
default_language_version:
python: python3

repos:
- repo: https://github.com/psf/black
rev: 23.1.0
hooks:
- id: black
args: ["-l 88"]
files: zubhub_backend/.*\.py$

- repo: https://github.com/PyCQA/isort
rev: 5.11.5 # do not update this until our CI runner uses Python >=3.8.1
hooks:
- id: isort
files: zubhub_backend/.*\.py$

- repo: https://github.com/PyCQA/flake8
rev: 5.0.4 # do not update this until our CI runner uses Python >=3.8.1
hooks:
- id: flake8
# use same max-line-length as black
# E121, E123, E126, E133, E226, E241, E242, E704 and W503 are default ignores
# E124: closing bracket does not match visual indentation
# E203: non-pep8 compliant brainfart of flake8
args:
[
"--max-line-length=88",
"--ignore=E121,E123,E126,E133,E203,E226,E241,E242,E704,W503,E124",
]
files: zubhub_backend/.*\.py$

# - repo: https://github.com/pre-commit/mirrors-mypy
# rev: v1.0.0
# hooks:
# - id: mypy
# exclude: ^tests/
# verbose: true
# args: [--show-error-codes, --ignore-missing-imports]
# additional_dependencies:
# ["types-requests", "types-PyYAML", "types-flask", "toolforge-weld"]

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-shebang-scripts-are-executable
- id: check-executables-have-shebangs
- id: check-merge-conflict

- repo: https://github.com/pre-commit/mirrors-eslint
rev: v7.32.0
hooks:
- id: eslint
entry: ./zubhub_frontend/zubhub/pre-commit-eslint
pass_filenames: false
always_run: true

- repo: https://github.com/pre-commit/mirrors-prettier
rev: v3.1.0
hooks:
- id: prettier
files: zubhub_frontend/.*\.[jt]sx?$
exclude: ^zubhub_frontend/zubhub/node_modules/
args: ["--config", "./zubhub_frontend/zubhub/.prettierrc.yaml"]

- repo: local
hooks:
- id: zubhub-test
name: zubhub-test
entry: ./zubhub_backend/pre-commit-zubhub-test
language: system
files: zubhub_backend/zubhub/.*\.py$
pass_filenames: false

- id: media-test
name: media-test
entry: ./zubhub_backend/pre-commit-media-test
language: system
files: zubhub_backend/media/.*\.py$
pass_filenames: false

- id: react-test
name: react-test
entry: npm --prefix ./zubhub_frontend/zubhub/ run test -- --coverage --watchAll=false
language: node
files: zubhub_frontend/.*\.[jt]sx?$
exclude: ^zubhub_frontend/zubhub/node_modules/
pass_filenames: false
26 changes: 26 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ Follow the instructions below to set up your local development environment
6. Install make. ( [jump to section](#install-make) )
7. Setup Backend. ( [jump to section](#setup-backend) )
8. Setup Frontend. ( [jump to section](#setup-frontend) )
9. Tests
10. Making changes and Commiting

<br/>
<br/>
Expand Down Expand Up @@ -202,6 +204,30 @@ Visit **localhost:3000** on your browser to access the frontend.
<br/>
<br/>

# Tests
Before you get started making changes and commiting, you should setup pre-commit on your development machine:
- Run `pip install pre-commit`
- Run `pre-commit --version` to verify that pre-commit installation was successful
- Run `cd <path to cloned zubhub repository>`
- Run `pre-commit install` to install the git hook scripts

To verify that your setup was successful, run `pre-commit` in the root of your cloned zubhub repo. You should see this running a number of checks.

## Frontend
You can also test the frontend seperately by going to the root of the frontend project (the place you have package.json) and running `npm run test -- --coverage --watchAll=false`

## Backend
To test the backend seperately,
- Run `cd ./zubhub_backend`
- Run `make zubhub-test` to run the test suites of the web django app
- Run `make media-test` to run the test suites of the media django app

# Making changes and Commiting
Before making changes, make sure that you've set up pre-commit as described in the previous step.
After that just `git add .` and `git commit` your changes. pre-commit will automatically run the different tests and will fail to commit until you fix all the errors.

> **_NOTE:_** If you fail to setup pre-commit on your local machine before making pull requests, our pre-commit action on github will run the same tests on your code and if it fails to pass, your code won't be merged.
# Deployment

ZubHub is currently deployed on its main website using Github Actions that act as our build and deployment tooling. If you are interested in deploying Zubhub on your VM for testing and hosting purposes, follow the [Single VM Deployment](./single_vm_deployment/DEPLOYMENT.md) instructions.
Expand Down
4 changes: 3 additions & 1 deletion zubhub_backend/.gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
media_store/
media/
zubhub/media/
# migrations/
__pycache__/
*.pyc
Expand All @@ -11,3 +11,5 @@ cadvisor.htpasswd
.ssl-data
celerybeat-schedule
celerybeat.pid
venv/
.idea/
8 changes: 6 additions & 2 deletions zubhub_backend/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,13 @@ migrate: ## Run `manage.py migrate`
docker-compose -f docker-compose.yml exec web bash -c "python zubhub/manage.py migrate"
.PHONY: migrate

test: ## Run `manage.py test`
zubhub-test: ## Run `manage.py test` on the web container
docker-compose -f docker-compose.yml exec web bash -c "cd zubhub && python manage.py test"
.PHONY: test
.PHONY: zubhub-test

media-test: ## Run `manage.py test` on the media container
docker-compose -f docker-compose.yml exec media bash -c "cd media && python manage.py test"
.PHONY: media-test

admin-user:
docker-compose -f docker-compose.yml exec web bash -c "python zubhub/manage.py create_dummy_admin_user"
Expand Down
2 changes: 1 addition & 1 deletion zubhub_backend/compose/web/dev/start
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ python /zubhub_backend/zubhub/manage.py collectstatic --noinput
#python /zubhub_backend/zubhub/manage.py makemigrations creators
#python /zubhub_backend/zubhub/manage.py makemigrations projects
#python /zubhub_backend/zubhub/manage.py makemigrations notifications
#python /zubhub_backend/zubhub/manage.py migrate
python /zubhub_backend/zubhub/manage.py migrate

python /zubhub_backend/zubhub/manage.py createcachetable

Expand Down
Loading

0 comments on commit 138b9c0

Please sign in to comment.