-
Notifications
You must be signed in to change notification settings - Fork 16
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improved the client by detailing the failures that can occur on a client.
- Loading branch information
Showing
2 changed files
with
27 additions
and
11 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,12 +1,32 @@ | ||
# Client Documentation | ||
|
||
|
||
## Summary | ||
The clients are their own objects that will request work from the job queue, and perform the work by making calls to [regulations.gov] | ||
(https://www.regulations.gov/) for data downloads, and saves the results. | ||
|
||
## Description | ||
The goal is | ||
that the client will request and complete work in order to download data from | ||
[regulations.gov](https://www.regulations.gov/). | ||
Clients are components of the Mirrulations system responsible for downloading data from Regulations.gov. Unless stopped a client continues to attempt the following steps: | ||
|
||
1. Get work from the job queue | ||
2. Perform the job by downloading data from Regulations.gov | ||
3. Save downloaded regulation data | ||
|
||
To accomplish their task each client interacts externally with Regulations.gov and AWS S3. Internally, each client interacts with the database (Redis) and queue (RabbitMQ). | ||
|
||
## Details | ||
|
||
Every 3.6 seconds a client attempts to get a job, perform its jobs, and save the resulting data. | ||
Download jobs have three fields: `job_id`, `url`, and `job_type`. | ||
|
||
### Getting Work | ||
|
||
A client gets work by removing a job from the queue and updating it's current job in the database. | ||
|
||
If no work is available at the current time the client waits for 3.6 seconds before attempting again. | ||
|
||
### Data Download | ||
|
||
After receiving a job, a client attempts to download the remote resource pointed to by its url. If the `job_type` is a comment, any attachments are also downloaded. The client updates the database after the job is completed. | ||
|
||
If an unrecoverable error occurs while during download, the client marks the job as an invalid job in the database. Invalid jobs will not be retried by other clients. | ||
|
||
### Saving Data | ||
|
||
After downloading data it is saved. By default data is saved to disk and to the `mirrulations` AWS S3 bucket. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,3 @@ | ||
|
||
|
||
|
||
|
||
## Production Environment Documentation | ||
|
||
The system is Dockerized into a number of components: | ||
|