-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
9f5d719
commit 2552bc0
Showing
7 changed files
with
1,262 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,3 +4,4 @@ data | |
*~ | ||
.ve | ||
.ipynb_checkpoints | ||
process/*/data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,14 +1,32 @@ | ||
# resource-projects-etl | ||
Extract, Transform and Load processes for rp.org | ||
|
||
# Requirements | ||
This repository contains a library for Extract, Transform and Load processes for ResourceProjects.org. | ||
|
||
You can report issues with current transformations, or suggest sources which should be added to this library using the GitHub issue tracker. | ||
|
||
|
||
## Processes | ||
Each process, located in the **process** folder consists of: | ||
|
||
* A README.md file describing the transformation | ||
* An extract.sh or extract.py file to fetch the file | ||
* A data/ subfolder where the extracted data is stored | ||
* A transform.py file which runs the transformations | ||
* A meta.json file, containing the meta-data which transform.py will use | ||
|
||
The output of each process should be written to the /data/ folder, from where it can be loaded onto the ResourceProjects.org platform. | ||
|
||
|
||
|
||
## Requirements | ||
|
||
Python 3 | ||
|
||
# Getting started | ||
### Getting started | ||
|
||
``` | ||
virtualenv .ve --python=/usr/bin/python3 | ||
source .ve/bin/activate | ||
pip install -r requirements.txt | ||
``` | ||
|
Oops, something went wrong.