Skip to content

Setting up a development environment

Jump to bottom
Adam Hooper edited this page Oct 8, 2020 · 51 revisions

One-time Docker setup

  1. Install Docker. We use Docker because it lets us develop cross-platform.
  2. On Mac or Windows, increase Docker's memory to 4GB. (You don't need this step on Linux.) Mac instructions Windows instructions
  3. Install Git. That's XCode on Mac, GitHub Desktop for Windows, and dnf install git / apt install git on Fedora/Ubuntu Linux.
  4. git clone https://github.com/CJWorkbench/cjworkbench.git to clone this repo

Developer workflow

You'll need a command line. First, get the development server up and running

  1. git pull
  2. bin/dev start to run database migrations, start compiling JavaScript/CSS files, and run Django.
  3. Browse to http://localhost:8000 and register a user.
  4. Django will fake-email you a confirmation code. Check the local_mail/ subdirectory for the email and follow the link within. Then log in.
  5. Press Ctrl+C in the console to kill everything

Creating an admin user

To create a user who has access to the Django admin page:

bin/dev python ./manage.py createsuperuser
# ... respond to the prompts.
# Now you can only log in after confirming the email address:
bin/dev sql -c 'UPDATE account_emailaddress SET verified = TRUE'

Handy commands

  1. bin/dev python -- open a Python console with Workbench's dependencies installed. This is useful for testing Pandas-related code (e.g., import pandas works).
  2. bin/dev python ./manage.py shell -- open a Python console with Django loaded. This is useful for testing Workbench models (e.g., import server.models works).
  3. bin/dev sql -- open a PostgreSQL shell to the development database.
  4. bin/dev npm install PACKAGE -- you may run npm commands that modify package.json and package-lock.json; modules are installed to a Docker volume, not ./node_modules.
  5. bin/dev pipenv install PACKAGE -- you may run pipenv commands that modify Pipfile and Pipfile.lock; packages are installed to a Docker volume.
  6. bin/dev clean -- delete all Python packages, Node modules, the database, and files on disk that the database references.

Running tests

  • bin/dev npm test (~15s) runs JavaScript tests in assets/js/. bin/dev npm test -- --watch watches them for changes. Use bin/dev npm test WfModule to test one module only.
  • bin/dev unittest (~90s) runs Django tests in server/tests/. bin/dev unittest -k server.tests.modules.test_loadurl (~30s first run, ~6s subsequent runs) specifies just one module to test.
  • bin/integration-test (~15min) builds images exactly as we do in production and runs automated integration tests on them. It fails as soon as a single test fails, to save you time. Write tests in integrationtests/*.py; if there's an error, bin/integration-test will fill your screen with helpful advice.

If you test features that send email (such as confirmation email for new accounts) the email will appear in the local_mail folder in your cjworkbench directory.

Using PyCharm

To develop with the integrated tools in PyCharm (run, test, debug, etc.) setup a Python interpreter using the docker-compose.yml file, as described in this documentation.

You will need to tell run the python interpreter inside the frontend container, and launch it with the pipenv-run-python script to make sure it uses the correct virtual environment. This is what your PyCharm Python interpreter settings should look like:

PyCharm docker setup

On Mac, you can get to this dialog through PyCharm -> Preferences -> Project: cjworkbench -> Project Interpreter then pick Show All at the bottom of the Project Interpreter drop-down.

Then your run configuration (for the server, as opposed to tests) needs to use this interpreter, and run on host 0.0.0.0 (not 127.0.0.1) and port 8080. Then, the server will come up on the browser on 127.0.0.1:8000 because of the port remapping in docker-compose.yml

To test social account logins

The "Google Login" and "Facebook Login" buttons are always present (for now -- we're tracking an issue that will hide them). But in your dev instance, if you click one of those buttons, you'll get a 500-error page about DoesNotExist.

To log in on your dev machine with Facebook and Google:

  1. Build an app on Facebook or Google's website. (See the "Google OAuth" section below for instructions. But use a totally different app! Call this app "Workbench Dev Login".)
  2. Browse to http://localhost:8000/admin/socialaccount/socialapp/add/ and add the Social App. Be sure to choose all Sites.
  3. Log out.
  4. Click the "Use Facebook Account" or "Use Google Account" button.

To develop certain features

Google OAuth

To download from Google Drive:

  1. Browse to https://console.developers.google.com
  2. Create a new organization, if wanted
  3. Create a new project within that organization: Workbench Dev
  4. Enable the Google Drive API
  5. Enable the Google Picker API
  6. Add credentials for Google Drive API for a Web server accessing User data: Name GoogleConnect, authorized origin http://localhost:8000, authorized redirect URI http://localhost:8000/oauth, product name Workbench Dev.
  7. Download client_id.json or client_secret_XXXXX.json (they're the same)
  8. Move that file into your Workbench project directory, as client_secret.json. (It's in .gitignore: Google terms dictate it stay private.)
  9. Restart bin/dev start
  10. Create a new workflow and add a Google Drive module.
  11. Sign in.
  12. Pick a file.
  13. Click Update

Intercom OAuth

To download from Intercom:

  1. Browse to https://developers.intercom.com and sign in
  2. Click "Your Apps"
  3. Click "New app"
  4. Enter an App Name (e.g., "Workbench Intercom Module"), choose your Workspace, and select "Internal integration". Click "Create app".
  5. Under "Authentication", click "Edit". Enable OAuth with "Redirect URL" of http://localhost:8000/oauth. Uncheck all permissions except "Read and list users and companies" and "Read tags". Click "Save".
  6. In the left navbar, click "Basic information".
  7. Copy/paste the "Client ID" and "Client secret" values into a new file called intercom_secret.json in the root of the project (we .gitignore it, because it exposes private data) with contents {"client_id":"...","client_secret":"...","redirect_url":"http://localhost:8000/oauth"}.
  8. Restart bin/dev start
  9. Create a new workflow and add an Intercom module.
  10. Sign in.
  11. Click Load

Twitter OAuth

To let users authenticate with Twitter:

  1. Browse to https://apps.twitter.com
  2. Click Create New App. Call it Workbench Dev [random numbers], description Authenticator for CJWorkbench Twitter module, in dev mode only, website http://example.org, callback URL http://localhost:8000/oauth. Read and agree to the developer agreement. Click Create your Twitter application.
  3. Browse to Permissions. Switch to Read only and click Update Settings.
  4. Browse to Keys and Access Tokens. Copy/paste these into a new file called twitter_secret.json in the root of the project (we .gitignore it, because Twitter asks us to keep it private) with contents {"key":"....","secret":".....","redirect_url":"http://localhost:8000/oauth"}.
  5. Restart bin/dev start
  6. Create a new workflow and add a Twitter module.
  7. Sign in.
  8. Pick a search.
  9. Click Update

Stripe

To test Stripe payments:

  1. Log in to https://dashboard.stripe.com (and sign up, if needed)
  2. In the top-left of the Stripe dashboard, either rename your default account or create a new account: call it workbench-dev@home, workbench-dev@laptop or some other descriptive name. Each development environment you use should have its own Stripe account, because the Stripe account must align with the Workbench database. Be sure you're "Viewing test data" in Stripe.
  3. In the Stripe dashboard, go to "Products" -> "Add Product". Call it "Premium Plan".
  4. Create a Price in USD. (Keep "Recurring" and "Monthly".) At the top-right of the page, click "Save product".
  5. Click "Developers" -> "API keys" and click "Reveal test key token". Copy the two tokens "Publishable key" (pk_test_123abc) and the "Secret key" (sk_test_123abc).
  6. In a console, run bin/dev stripe-setup pk_test_123abc sk_test_123abc. (This writes ./stripe.env, copies Stripe Products to our database, and restarts the web server to become aware of Stripe API keys.)
  7. Enable the Customer Portal: browse to https://dashboard.stripe.com/test/settings/billing/portal. Allow customers to cancel subscriptions; Cancel immediately; Prorate cancelled subscriptions; set the links to our public URLs, https://workbenchdata.com/privacy and https://workbenchdata.com/terms-of-service
  8. Whenever you want to interact with Stripe in your dev environment, run bin/dev stripe-setup to re-fetch Products, then run bin/dev stripe-listen and keep it open.
Clone this wiki locally