Skip to content

Latest commit

 

History

History
340 lines (270 loc) · 10.8 KB

new_endpoint_tutorial.md

File metadata and controls

340 lines (270 loc) · 10.8 KB

Tutorial: Adding a new API endpoint

Prerequisite: this guide assumes that you have read the epidata development guide.

In this tutorial we'll create a brand new endpoint for the Epidata API: fluview_meta. At a high level, we'll do the following steps:

  1. understand the data that we want to surface
  2. add the new endpoint to api.php
  3. add the new endpoint to the various client libraries
  4. write an integration test for the new endpoint
  5. update API documentation for the new endpoint
  6. run all unit and integration tests

setup

Follow the backend guide and the epidata guide to install Docker and get your workspace ready for development. Before continuing, your workspace should look something like the following:

tree -L 3 .
.
└── repos
    ├── delphi
    │   ├── delphi-epidata
    │   ├── flu-contest
    │   ├── github-deploy-repo
    │   ├── nowcast
    │   ├── operations
    │   └── utils
    └── undefx
        ├── py3tester
        └── undef-analysis

the data

Here's the requirement: we need to quickly surface the most recent "issue" (epiweek of publication) for the existing fluview endpoint. This is already provided by the existing meta endpoint, however, it's very slow, and it returns a bunch of unrelated data. The goal is extract the subset of metadata pertaining to fluview and return just that data through a new endpoint.

Each row in the fluview table contains a lot of data, but we're particularly interested in the following:

  • latest publication date
  • latest "issue", which is the publication epiweek
  • total size of the table

update the server

Open api.php and navigate to the bottom where we see line like if($source === 'NAME') { ... }. Right below the if block for if($source === 'fluview'), add a new if else block for our new endpoint:

else if($source === 'fluview_meta') {
 // get the data
 $epidata = meta_fluview();
 store_result($data, $epidata);
}

Fortunately, the function meta_fluview() is already defined, so we can just reuse it. (It's used by the meta endpoint as mentioned above.) In general, you will likely need to define a new function named like get_SOURCE(params...), especially if you're reading from a new database table.

update the client libraries

There are currently four client libraries. They all need to be updated to make the new fluview_meta endpoint available to callers. The pattern is very similar for all endpoints, so copy-paste will get you 90% of the way there.

fluview_meta is especially simple as it takes no parameters, and consequently there is no need to validate parameters. In general, it's a good idea to do sanity checks on caller inputs prior to sending the request to the API. See some of the other endpoint implementations (e.g. fluview) for an example of what this looks like.

Here's what we add to each client:

  • delphi_epidata.coffee

    # Fetch FluView metadata
    @fluview_meta: (callback) ->
      # Set up request
      params =
        'source': 'fluview_meta'
      # Make the API call
      _request(callback, params)
  • delphi_epidata.js

    Note that this file can and should be generated from delphi_epidata.coffee. However, for trivial changes, like the addition of this very simple endpoint, it may be slightly faster, though error-prone, to just update the JavaScript manually.

    Epidata.fluview_meta = function(callback) {
      var params;
      params = {
        'source': 'fluview_meta'
      };
      return _request(callback, params);
    };
  • delphi_epidata.py

    Note that this file, unlike the others, is released as a public package, available to install easily though Python's pip tool. That package should be updated once the code is committed, however that is outside of the scope of this tutorial.

    # Fetch FluView metadata
    @staticmethod
    def fluview_meta():
      """Fetch FluView metadata."""
      # Set up request
      params = {
        'source': 'fluview_meta',
      }
      # Make the API call
      return Epidata._request(params)
  • delphi_epidata.R

    # Fetch FluView metadata
    fluview_meta <- function() {
      # Set up request
      params <- list(
        source = 'fluview_meta'
      )
      # Make the API call
      return(.request(params))
    }

    This file requires a second change: updating the list of exported functions. This additional step only applies to this particular client library. At the bottom of the file, inside of return(list(, add the following line to make the function available to callers.

    fluview_meta = fluview_meta,

add an integration test

Now that we've changed several files, we need to make sure that the changes work as intended before submitting code for review or committing code to the repository. Given that the code spans multiple components and languages, this needs to be an integration test. See more about integration testing in Delphi's frontend development guide.

Create an integration test for the new endpoint by creating a new file integrations/server/test_fluview_meta.py. There's a good amount of boilerplate, but fortunately this can be copied almost verbatim from the fluview endpoint integration test.

Include the following pieces:

  • top-level docstring (update name to fluview_meta)
  • the imports section (no changes needed)
  • the test class (update name and docstring for fluview_meta)
  • the methods setUpClass, setUp, and tearDown (no changes needed)

Add the following test method which creates some dummy data, fetches the new fluview_meta endpoint using the Python client library, and asserts that the returned value is what we expect.

def test_round_trip(self):
  """Make a simple round-trip with some sample data."""

  # insert dummy data
  self.cur.execute('''
    insert into fluview values
      (0, "2020-04-07", 202021, 202020, "nat", 1, 2, 3, 4, 3.14159, 1.41421,
        10, 11, 12, 13, 14, 15),
      (0, "2020-04-28", 202022, 202022, "hhs1", 5, 6, 7, 8, 1.11111, 2.22222,
        20, 21, 22, 23, 24, 25)
  ''')
  self.cnx.commit()

  # make the request
  response = Epidata.fluview_meta()

  # assert that the right data came back
  self.assertEqual(response, {
    'result': 1,
    'epidata': [{
       'latest_update': '2020-04-28',
       'latest_issue': 202022,
       'table_rows': 2,
     }],
    'message': 'success',
  })

write documentation

This consists of two steps: add a new document for the fluview_meta endpoint, and add a new entry to the existing table of endpoints.

Create a new file docs/api/fluview_meta.md. Copy as much as needed from other endpoints, e.g. the fluview documentation. Update the description, table of return values, and sample code and URLs as needed.

Edit the table of endpoints in docs/api/README.md, adding the following row in the appropriate place (i.e. next to the row for fluview):

| [`fluview_meta`](fluview_meta.md) | FluView Metadata | Summary data about [`fluview`](fluview.md). | no |

run tests

unit

Finally, we just need to run all new and existing tests. It is recommended to start with the unit tests because they are faster to build, run, and either succeed or fail. Follow the backend development guide. In summary:

# build the image
docker build -t delphi_python \
  -f repos/delphi/operations/dev/docker/python/Dockerfile .

# run epidata unit tests
docker run --rm delphi_python \
  python3 -m undefx.py3tester.py3tester --color \
  repos/delphi/delphi-epidata/tests

If all succeeds, output should look like this:

[...]

✔ All 48 tests passed! 69% (486/704) coverage.

integration

Integration tests require more effort, and take longer to setup and run. However, they allow us to test that various pieces are working together correctly. Many of these pieces we can't test individually with unit tests (e.g. database, and the API server), so integration tests are the only way we can be confident that our changes won't break the API. Follow the epidata development guide. In summary, assuming you have already built the delphi_python image above:

# build web and database images for epidata
docker build -t delphi_web \
  -f repos/delphi/operations/dev/docker/web/Dockerfile .
docker build -t delphi_web_epidata \
  -f repos/delphi/delphi-epidata/dev/docker/web/epidata/Dockerfile .
docker build -t delphi_database \
  -f repos/delphi/operations/dev/docker/database/Dockerfile .
docker build -t delphi_database_epidata \
  -f repos/delphi/delphi-epidata/dev/docker/database/epidata/Dockerfile .

# launch web and database containers in separate terminals
docker run --rm -p 13306:3306 \
  --network delphi-net --name delphi_database_epidata \
  delphi_database_epidata

docker run --rm -p 10080:80 \
  --network delphi-net --name delphi_web_epidata \
  delphi_web_epidata

# wait for the above containers to initialize (~15 seconds)

# run integration tests
docker run --rm --network delphi-net delphi_python \
  python3 -m undefx.py3tester.py3tester --color \
    repos/delphi/delphi-epidata/integrations

If all succeeds, output should look like this. Note also that our new integration test specifically passed.

[...]

delphi.delphi-epidata.integrations.server.test_fluview_meta.FluviewMetaTests.test_round_trip: pass

[...]

✔ All 16 tests passed! 48% (180/372) coverage.

code review and submission

All tests pass, and the changes are working as intended. Now submit the code for review, e.g. by opening a pull request on GitHub. For an example, see the actual pull request for the fluview_meta endpoint created in this tutorial.

Once it's approved, commit the code. Within a short amount of time (usually ~30 seconds), the API will begin serving your new endpoint. Go ahead and give it a try: https://delphi.midas.cs.cmu.edu/epidata/api.php?source=fluview_meta

{
  "result": 1,
  "epidata": [
    {
      "latest_update": "2020-04-24",
      "latest_issue": 202016,
      "table_rows": 957673
    }
  ],
  "message": "success"
}