Skip to content

Commit

Permalink
Update client documentation
Browse files Browse the repository at this point in the history
Improved the client by detailing the failures that can occur on a
client.
  • Loading branch information
jpappel committed Oct 4, 2024
1 parent 1a20643 commit d06ce55
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 11 deletions.
34 changes: 27 additions & 7 deletions docs/client.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,32 @@
# Client Documentation


## Summary
The clients are their own objects that will request work from the job queue, and perform the work by making calls to [regulations.gov]
(https://www.regulations.gov/) for data downloads, and saves the results.

## Description
The goal is
that the client will request and complete work in order to download data from
[regulations.gov](https://www.regulations.gov/).
Clients are components of the Mirrulations system responsible for downloading data from Regulations.gov. Unless stopped a client continues to attempt the following steps:

1. Get work from the job queue
2. Perform the job by downloading data from Regulations.gov
3. Save downloaded regulation data

To accomplish their task each client interacts externally with Regulations.gov and AWS S3. Internally, each client interacts with the database (Redis) and queue (RabbitMQ).

## Details

Every 3.6 seconds a client attempts to get a job, perform its jobs, and save the resulting data.
Download jobs have three fields: `job_id`, `url`, and `job_type`.

### Getting Work

A client gets work by removing a job from the queue and updating it's current job in the database.

If no work is available at the current time the client waits for 3.6 seconds before attempting again.

### Data Download

After receiving a job, a client attempts to download the remote resource pointed to by its url. If the `job_type` is a comment, any attachments are also downloaded. The client updates the database after the job is completed.

If an unrecoverable error occurs while during download, the client marks the job as an invalid job in the database. Invalid jobs will not be retried by other clients.

### Saving Data

After downloading data it is saved. By default data is saved to disk and to the `mirrulations` AWS S3 bucket.
4 changes: 0 additions & 4 deletions docs/production.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@




## Production Environment Documentation

The system is Dockerized into a number of components:
Expand Down

0 comments on commit d06ce55

Please sign in to comment.