Create a script to automatically update the data

[anuveyatsu]

Acceptance criteria

• I have a script (preferably python3) to update this dataset.
• I have GH Actions that runs on monthly basis and runs the script to update the data in this repo.

Analysis

Take a look how we are doing it in similar repos:

• https://github.com/datasets/world-cities
• https://github.com/datasets/airport-codes

[gradedSystem]

Hi there! @anuveyatsu I am currently applying for a data engineering position at datopian and it is said to complete the task the given in this thread, I have couple of questions regarding the task, is there a requirement on how data should be presented I mean should we just look through the dataset and check the updates on dataset each month and if data will be updated we should just replace the current data with the new data that will appear in UNData. UNSD Demographic Statistics? And should we submit the task using pull request?

[anuveyatsu]

Hi @gradedSystem thanks for your interest! Would it be OK if I assign to you another similar issue so that your effort will add a lot of value for us? You might have similar questions so feel free to ask them there. Generally we welcome pull requests 👍🏼

Here is the another issue I could assign to you datasets/commodity-prices#4

[judeleonard]

Hi @anuveyatsu, I am also working on this task currently. I want to know if it's something I should proceed with?

[anuveyatsu]

@judeleonard hi, I've just assigned it to you, please, take a look.

[judeleonard]

Hi @anuveyatsu, now I have the script to get the data from the website and then wrangle it to the desired state. But I'm not so sure how you would want this update to happen.

Here are my thoughts:

(1) Overwrite the previous data to include the updated records
(2) Track where the updates happened and insert only the updated records.