Prerequisite: this guide assumes that you have read the epidata development guide.
In this tutorial we'll create a brand new endpoint for the Epidata API:
fluview_meta
. At a high level, we'll do the following steps:
- understand the data that we want to surface
- add the new endpoint to
api.php
- add the new endpoint to the various client libraries
- write an integration test for the new endpoint
- update API documentation for the new endpoint
- run all unit and integration tests
Follow the backend guide and the epidata guide to install Docker and get your workspace ready for development. Before continuing, your workspace should look something like the following:
tree -L 3 .
.
└── repos
├── delphi
│ ├── delphi-epidata
│ ├── flu-contest
│ ├── github-deploy-repo
│ ├── nowcast
│ ├── operations
│ └── utils
└── undefx
├── py3tester
└── undef-analysis
Here's the requirement: we need to quickly surface the most recent "issue"
(epiweek of publication) for the existing fluview
endpoint.
This is already provided by the existing meta
endpoint,
however, it's very slow, and it returns a bunch of unrelated data. The goal
is extract the subset of metadata pertaining to fluview
and return just that
data through a new endpoint.
Each row in the fluview
table contains
a lot of data, but we're particularly interested in
the following:
- latest publication date
- latest "issue", which is the publication epiweek
- total size of the table
Open api.php
and navigate to the bottom where we see
line like if($source === 'NAME') { ... }
. Right below the if
block for
if($source === 'fluview')
, add a new if else
block for our new endpoint:
else if($source === 'fluview_meta') {
// get the data
$epidata = meta_fluview();
store_result($data, $epidata);
}
Fortunately, the function meta_fluview()
is already defined, so we can just
reuse it. (It's used by the meta
endpoint as mentioned above.) In general,
you will likely need to define a new function named like
get_SOURCE(params...)
, especially if you're reading from a new database
table.
There are currently four client libraries. They all need to be updated to make
the new fluview_meta
endpoint available to callers. The pattern is very
similar for all endpoints, so copy-paste will get you 90% of the way there.
fluview_meta
is especially simple as it takes no parameters, and consequently
there is no need to validate parameters. In general, it's a good idea to do
sanity checks on caller inputs prior to sending the request to the API. See
some of the other endpoint implementations (e.g. fluview
) for an example of
what this looks like.
Here's what we add to each client:
-
# Fetch FluView metadata @fluview_meta: (callback) -> # Set up request params = 'source': 'fluview_meta' # Make the API call _request(callback, params)
-
Note that this file can and should be generated from
delphi_epidata.coffee
. However, for trivial changes, like the addition of this very simple endpoint, it may be slightly faster, though error-prone, to just update the JavaScript manually.Epidata.fluview_meta = function(callback) { var params; params = { 'source': 'fluview_meta' }; return _request(callback, params); };
-
Note that this file, unlike the others, is released as a public package, available to install easily though Python's
pip
tool. That package should be updated once the code is committed, however that is outside of the scope of this tutorial.# Fetch FluView metadata @staticmethod def fluview_meta(): """Fetch FluView metadata.""" # Set up request params = { 'source': 'fluview_meta', } # Make the API call return Epidata._request(params)
-
# Fetch FluView metadata fluview_meta <- function() { # Set up request params <- list( source = 'fluview_meta' ) # Make the API call return(.request(params)) }
This file requires a second change: updating the list of exported functions. This additional step only applies to this particular client library. At the bottom of the file, inside of
return(list(
, add the following line to make the function available to callers.fluview_meta = fluview_meta,
Now that we've changed several files, we need to make sure that the changes work as intended before submitting code for review or committing code to the repository. Given that the code spans multiple components and languages, this needs to be an integration test. See more about integration testing in Delphi's frontend development guide.
Create an integration test for the new endpoint by creating a new file
integrations/server/test_fluview_meta.py
. There's a good amount of
boilerplate, but fortunately this can be copied almost verbatim from the
fluview
endpoint integration test.
Include the following pieces:
- top-level docstring (update name to
fluview_meta
) - the imports section (no changes needed)
- the test class (update name and docstring for
fluview_meta
) - the methods
setUpClass
,setUp
, andtearDown
(no changes needed)
Add the following test method which creates some dummy data, fetches the new
fluview_meta
endpoint using the Python client library, and asserts that the
returned value is what we expect.
def test_round_trip(self):
"""Make a simple round-trip with some sample data."""
# insert dummy data
self.cur.execute('''
insert into fluview values
(0, "2020-04-07", 202021, 202020, "nat", 1, 2, 3, 4, 3.14159, 1.41421,
10, 11, 12, 13, 14, 15),
(0, "2020-04-28", 202022, 202022, "hhs1", 5, 6, 7, 8, 1.11111, 2.22222,
20, 21, 22, 23, 24, 25)
''')
self.cnx.commit()
# make the request
response = Epidata.fluview_meta()
# assert that the right data came back
self.assertEqual(response, {
'result': 1,
'epidata': [{
'latest_update': '2020-04-28',
'latest_issue': 202022,
'table_rows': 2,
}],
'message': 'success',
})
This consists of two steps: add a new document for the fluview_meta
endpoint,
and add a new entry to the existing table of endpoints.
Create a new file docs/api/fluview_meta.md
. Copy as much as needed from other
endpoints, e.g. the fluview documentation. Update the
description, table of return values, and sample code and URLs as needed.
Edit the table of endpoints in docs/api/README.md
, adding
the following row in the appropriate place (i.e. next to the row for
fluview
):
| [`fluview_meta`](fluview_meta.md) | FluView Metadata | Summary data about [`fluview`](fluview.md). | no |
Finally, we just need to run all new and existing tests. It is recommended to start with the unit tests because they are faster to build, run, and either succeed or fail. Follow the backend development guide. In summary:
# build the image
docker build -t delphi_python \
-f repos/delphi/operations/dev/docker/python/Dockerfile .
# run epidata unit tests
docker run --rm delphi_python \
python3 -m undefx.py3tester.py3tester --color \
repos/delphi/delphi-epidata/tests
If all succeeds, output should look like this:
[...]
✔ All 48 tests passed! 69% (486/704) coverage.
Integration tests require more effort, and take longer to setup and run.
However, they allow us to test that various pieces are working together
correctly. Many of these pieces we can't test individually with unit tests
(e.g. database, and the API server), so integration tests are the only way we
can be confident that our changes won't break the API. Follow the epidata
development guide. In summary, assuming you have
already built the delphi_python
image above:
# build web and database images for epidata
docker build -t delphi_web \
-f repos/delphi/operations/dev/docker/web/Dockerfile .
docker build -t delphi_web_epidata \
-f repos/delphi/delphi-epidata/dev/docker/web/epidata/Dockerfile .
docker build -t delphi_database \
-f repos/delphi/operations/dev/docker/database/Dockerfile .
docker build -t delphi_database_epidata \
-f repos/delphi/delphi-epidata/dev/docker/database/epidata/Dockerfile .
# launch web and database containers in separate terminals
docker run --rm -p 13306:3306 \
--network delphi-net --name delphi_database_epidata \
delphi_database_epidata
docker run --rm -p 10080:80 \
--network delphi-net --name delphi_web_epidata \
delphi_web_epidata
# wait for the above containers to initialize (~15 seconds)
# run integration tests
docker run --rm --network delphi-net delphi_python \
python3 -m undefx.py3tester.py3tester --color \
repos/delphi/delphi-epidata/integrations
If all succeeds, output should look like this. Note also that our new integration test specifically passed.
[...]
delphi.delphi-epidata.integrations.server.test_fluview_meta.FluviewMetaTests.test_round_trip: pass
[...]
✔ All 16 tests passed! 48% (180/372) coverage.
All tests pass, and the changes are working as intended. Now submit the code
for review, e.g. by opening a pull request on GitHub. For an example, see the
actual
pull request for the fluview_meta
endpoint
created in this tutorial.
Once it's approved, commit the code. Within a short amount of time (usually ~30 seconds), the API will begin serving your new endpoint. Go ahead and give it a try: https://delphi.midas.cs.cmu.edu/epidata/api.php?source=fluview_meta
{
"result": 1,
"epidata": [
{
"latest_update": "2020-04-24",
"latest_issue": 202016,
"table_rows": 957673
}
],
"message": "success"
}