WorldWide Telescope Constellations: The Backend

This is an ExpressJS web server that communicates with a MongoDB storage backend (location specified with the MONGO_CONNECTION_STRING environment variable), a Keycloak server (specified with KEYCLOAK_URL), and an instance of the WWT Constellations previewer service (specified with CX_PREVIEW_SERVICE_URL), which in turn relies on an Azure Storage server (potentially using a local emulator).

The WWT Constellations frontend server communicates with this backend to create the WWT Constellations app experience. See the frontend README for information on how to start up a development Keycloak server.

Basic Workflow

Make sure to install the dependencies:

yarn install

Build the application for production:

yarn build

Start the server (defaulting to run on http://localhost:7000):

yarn start

Configuration

Environment variables:

• PORT to set the port for the server to listen on; default is 7000.
• MONGO_CONNECTION_STRING to set the path to MongoDB server; must be specified.
  • AZURE_COSMOS_CONNECTIONSTRING has the same effect and higher priority
• KEYCLOAK_URL to set the location of the Keycloak server; default is http://localhost:8080/. If the value of this setting does not end with a slash, the server appends one internally.
• CX_PREVIEW_BASE_URL sets the base used to construct the URLs of social media preview content associated with different scenes.
• CX_PREVIEW_SERVICE_URL sets the URL of the wwt-constellations-previewer service, which creates previews and deposits them in the storage backend. No default, but for local testing the usual setting would be http://localhost:5000.
• CX_SESSION_SECRETS is a space-delimited list of secrets used to hash session cookies. Default is dev-secret. The first secret is used for creating new sessions; any subsequent secrets are used for checking existing sessions, allowing you to rotate the secret periodically.
• CX_SUPERUSER_ACCOUNT_ID sets the Keycloak account ID of an account that can perform some special administrative tasks.
• CX_LOG_LEVEL sets the logging level for the application; default is "info". Valid values, in order, are "verbose", "info", "warning", "error".

Deployment

Merges to the main branch are automatically deployed to a “staging slot” of the WWT web app. To bring the new code into production, you need to “swap” that slot into the production role. You can do this through the Azure Portal UI or using the az CLI tool as follows:

az webapp deployment slot swap -g wwtprod-cxbackend -n wwtprod-cxbe --slot stage --target-slot production

Bootstrapping a Development Environment

Bootstrapping a server for local testing requires a bit of work to set up a complete environment. The following steps use Docker to get everything going. They have been tested on a Linux environment but might work on other operating systems as well.

(We should really set this up in docker-compose but haven't yet done so!)

1. First, you need to set up a Keycloak identity server. It will not be secure, but that's OK for development purposes.
   1. Create and start a long-lived Docker container instance for the server:

      docker create \
        --name cx-keycloak \
        -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=mypass \
        -p 8080:8080 \
        quay.io/keycloak/keycloak start-dev
      
      docker start cx-keycloak
      

   2. Navigate to http://localhost:8080/ to get the Keycloak admin UI
   3. Log in with the admin username and password you specified
   4. Using the top-left dropdown, create a new realm named constellations
   5. In the Clients tab, create a new client
      1. Call it constellations-app
      2. Use http://localhost:3000/ as the "root" and "home" URLs.
      3. Add http://localhost:3000/* as a valid redirect URLs
      4. Add * as allowed web origin
   6. Create another new client
      1. Call it cli-tool
      2. Turn on "Implicit Flow" in its Capability Config. (TODO: check if this is necessary?)
      3. Set * as the redirect URLs.
   7. In the Users tab, create a new user for yourself
      1. Choose a username
      2. After creation, in the Credentials tab of the user, set a password. You can save yourself time later by unchecking the "Temporary" box.
      3. In the "Details" panel, note the client ID. Create a file called .env in the directory containing this README and add a line of the form:

         CX_SUPERUSER_ACCOUNT_ID="<your account id>"
         

2. Now we need to set up a MongoDB database server.
   1. We'll once again use Docker. Although there is a Microsoft Cosmos/Mongo emulator docker image that might mirror what we run in production more closely, it is broken right now (March 2023).

      docker create \
        --name cx-mongodb \
        -p 27017:27017 \
        -e MONGO_INITDB_ROOT_USERNAME=admin \
        -e MONGO_INITDB_ROOT_PASSWORD=mypass \
        mongo:latest
      
      docker start cx-mongodb
      

   2. Add a line to your .env file of the form:

      MONGO_CONNECTION_STRING="mongodb://admin:mypass@localhost:27017/"
      

3. Next, we need an Azure Storage service, which can be emulated with Azurite.
   1. Create or choose a local directory in which your data will be stored.
   2. More Docker:

      docker create \
        --name cx-storage \
        -p 10000:10000 \
        -v {your-data-directory}:/data:rw \
        mcr.microsoft.com/azure-storage/azurite \
        azurite-blob --blobHost 0.0.0.0
      
      docker start cx-storage
      

   3. Export an environment variable named AZURE_STORAGE_CONNECTION_STRING using the default HTTP connection string for the blob service listed in the Azurite README.
   4. Create a storage container in the service named previews. If you use the "az" CLI tool, you can do this with:

      az storage container create --name previews
      

4. After this, you can start an instance of the previewer service. See the previewer README for instructions.
   1. If you build the previewer as a Docker image, you can run it with:

      docker create \
        --name cx-previewer \
        --net=host \
        -e NUXT_PUBLIC_API_URL={your value} \
        -e MONGO_CONNECTION_STRING={your value} \
        -e AZURE_STORAGE_CONNECTION_STRING={your value} \
        aasworldwidetelescope/constellations-previewer:latest
      
      docker start cx-previewer
      

   2. Add a line to your .env file of the form:

      CX_PREVIEW_SERVICE_URL="http://localhost:5000"
      

5. Now we should be able to successfully start the backend server.
   1. Run yarn install (if needed) to fetch dependencies
   2. Run yarn build to build it
   3. Run yarn start to start it
   4. The server should say that it is connected to the database and report that it is running at http://localhost:7000/.
6. Finally, we need to complete the database setup and populate it with some content.
   1. Ensure that the latest version of the wwt_api_client Python package is installed.
   2. In the terminal in which you'll be working, export the environment variable NUXT_PUBLIC_API_URL="http://localhost:7000"
   3. Run the following Python code to test your login. You should be prompted to open a browser window and login to the Keycloak server, where you can use the username and password that you set up for your personal account above:

      from wwt_api_client import constellations as cx
      client = cx.CxClient()
      client._send_and_check("/misc/amisuperuser", http_method="GET").json()

      This should work and return {"result": True}.
   4. Run the following code to do some database setup:

      client._send_and_check("/misc/config-database").json()

      This should return {"error": False}.
   5. Set up the JWST handle:
      1. Download the current JWST WTML file locally with something like:

         curl -fsSL http://www.worldwidetelescope.org/wwtweb/catalog.aspx?W=jwst >jwst.wtml
         

      2. Import content using the scratch/bootstrap_handle.py script:

         python3 scratch/bootstrap_handle.py \
           --display-name "James Webb Space Telescope" \
           --my-account-id {your-account-id} \
           --copyright "Public domain" \
           --license-id CC-PDDC \
           jwst \
           jwst.wtml
         

         (The account ID here has to be the one that your wwt_api_client client is logged in as. As far as I know this value is not directly available to the client, but it's in our .env file.)
   6. Set up other handles as desired.
      1. Using the catalog.aspx URL as above with studiesnoao gives the NOIRLab collection.
      2. eso is ESO
      3. geministudies is old Gemini stuff
      4. studieschandra is mostly old Chandra stuff
      5. studieshubble you can guess
      6. studiesspitzer same
      7. wise same