rhdx is an R client for the Humanitarian Exchange Data platform.
The Humanitarian Data Exchange platform is the open platform to easily find and analyze humanitarian data.
This package is not on yet on CRAN and to install it, you will need the
remotes package. You can get
rhdx from Gitlab or Github (mirror)
## install.packages("remotes")
remotes::install_gitlab("dickoa/rhdx")
remotes::install_github("dickoa/rhdx")
library("rhdx")
The first step is usually to connect to HDX using the set_rhdx_config
function and check the config using get_rhdx_config
set_rhdx_config(hdx_site = "prod")
get_rhdx_config()
## <HDX Configuration>
## HDX site: prod
## HDX site url: https://data.humdata.org/
## HDX API key:
Now that we are connected to HDX, we can search for dataset using
search_datasets, access resources withini the dataset page with the
get_resources function and finally read the data directly into the R
session using read_resource. magrittr pipes operator are also
supported
library(tidyverse)
search_datasets("ACLED Mali", rows = 2) %>% ## search dataset in HDX, limit the results to two rows
pluck(1) %>% ## select the first dataset
get_resource(1) %>% ## pick the first resource
read_resource() ## read this HXLated data into R
## # A tibble: 2,516 x 30
## data_id iso event_id_cnty event_id_no_cnty event_date year
## * <dbl> <dbl> <chr> <dbl> <date> <dbl>
## 1 2942561 466 MLI2605 2605 2019-01-26 2019
## 2 2942562 466 MLI2606 2606 2019-01-26 2019
## 3 2942557 466 MLI2601 2601 2019-01-25 2019
## 4 2942558 466 MLI2602 2602 2019-01-25 2019
## 5 2942559 466 MLI2603 2603 2019-01-25 2019
## 6 2942560 466 MLI2604 2604 2019-01-25 2019
## 7 2942555 466 MLI2599 2599 2019-01-24 2019
## 8 2942556 466 MLI2600 2600 2019-01-24 2019
## 9 2942553 466 MLI2597 2597 2019-01-23 2019
## 10 2942554 466 MLI2598 2598 2019-01-23 2019
## # … with 2,506 more rows, and 24 more variables:
## # time_precision <dbl>, event_type <chr>, actor1 <chr>,
## # assoc_actor_1 <chr>, inter1 <dbl>, actor2 <chr>,
## # assoc_actor_2 <chr>, inter2 <dbl>, interaction <dbl>,
## # region <chr>, country <chr>, admin1 <chr>, admin2 <chr>,
## # admin3 <chr>, location <chr>, latitude <dbl>,
## # longitude <dbl>, geo_precision <dbl>, source <chr>,
## # source_scale <chr>, notes <chr>, fatalities <dbl>,
## # timestamp <dbl>, iso3 <chr>
read_resource will not work with resources in HDX, so far the
following format are supported: csv, xlsx, xls, json, geojson,
zipped shapefile, kmz, zipped geodatabase and zipped geopackage.
I will consider adding more data types in the future, feel free to file
an issue if it doesn’t work as expected or you want to add a support for
a format.
We can also use pull_dataset to directly read and access a dataset
object.
pull_dataset("acled-data-for-mali") %>%
get_resource(1) %>%
read_resource()
## # A tibble: 3,990 x 31
## data_id iso event_id_cnty event_id_no_cnty event_date year
## <dbl> <dbl> <chr> <dbl> <date> <dbl>
## 1 7173324 466 MLI4111 4111 2020-07-31 2020
## 2 7173322 466 MLI4109 4109 2020-07-29 2020
## 3 7173323 466 MLI4110 4110 2020-07-29 2020
## 4 7173423 466 MLI4107 4107 2020-07-28 2020
## 5 7173761 466 MLI4108 4108 2020-07-28 2020
## 6 7173702 466 MLI4104 4104 2020-07-27 2020
## 7 7173732 466 MLI4103 4103 2020-07-27 2020
## 8 7173319 466 MLI4102 4102 2020-07-27 2020
## 9 7173320 466 MLI4105 4105 2020-07-27 2020
## 10 7173321 466 MLI4106 4106 2020-07-27 2020
## # … with 3,980 more rows, and 25 more variables:
## # time_precision <dbl>, event_type <chr>,
## # sub_event_type <chr>, actor1 <chr>, assoc_actor_1 <chr>,
## # inter1 <dbl>, actor2 <chr>, assoc_actor_2 <chr>,
## # inter2 <dbl>, interaction <dbl>, region <chr>,
## # country <chr>, admin1 <chr>, admin2 <chr>, admin3 <chr>,
## # location <chr>, latitude <dbl>, longitude <dbl>,
## # geo_precision <dbl>, source <chr>, source_scale <chr>,
## # notes <chr>, fatalities <dbl>, timestamp <dbl>, iso3 <chr>
In order to connect to HDX, we can use the set_rhdx_config function
set_rhdx_config(hdx_site = "prod")
Once a server is chosen, we can now search from dataset using the
search_datasets In this case we will limit just to two results (rows
parameter).
list_of_ds <- search_datasets("displaced Nigeria", rows = 2)
list_of_ds
## [[1]]
## <HDX Dataset> 4fbc627d-ff64-4bf6-8a49-59904eae15bb
## Title: Nigeria - Internally displaced persons - IDPs
## Name: idmc-idp-data-for-nigeria
## Date: 01/01/2009-12/31/2016
## Tags (up to 5): displacement, idmc, population
## Locations (up to 5): nga
## Resources (up to 5): displacement_data, conflict_data, disaster_data
## [[2]]
## <HDX Dataset> 4adf7874-ae01-46fd-a442-5fc6b3c9dff1
## Title: Nigeria Baseline Assessment Data [IOM DTM]
## Name: nigeria-baseline-data-iom-dtm
## Date: 01/31/2018
## Tags (up to 5): adamawa, assessment, baseline-data, baseline-dtm, bauchi
## Locations (up to 5): nga
## Resources (up to 5): DTM Nigeria Baseline Assessment Round 21, DTM Nigeria Baseline Assessment Round 20, DTM Nigeria Baseline Assessment Round 19, DTM Nigeria Baseline Assessment Round 18, DTM Nigeria Baseline Assessment Round 17
The result of search_datasets is a list of HDX datasets, you can
manipulate this list like any other list in R. We can use
purrr::pluck to select the element we want in our list, here it is the
first.
ds <- pluck(list_of_ds, 1)
ds
## <HDX Dataset> 4fbc627d-ff64-4bf6-8a49-59904eae15bb
## Title: Nigeria - Internally displaced persons - IDPs
## Name: idmc-idp-data-for-nigeria
## Date: 01/01/2009-12/31/2016
## Tags (up to 5): displacement, idmc, population
## Locations (up to 5): nga
## Resources (up to 5): displacement_data, conflict_data, disaster_data
With our dataset, the next step is to list all the resources. If you are
not familiar with CKAN terminology, resources refer to the actual
files shared in a dataset page and you can download. Each dataset page
contains one or more resources.
get_resources(ds)
## [[1]]
## <HDX Resource> f57be018-116e-4dd9-a7ab-8002e7627f36
## Name: displacement_data
## Description: Internally displaced persons - IDPs (new displacement associated with conflict and violence)
## Size:
## Format: JSON
## [[2]]
## <HDX Resource> 6261856c-afb9-4746-b340-9cf531cbd38f
## Name: conflict_data
## Description: Internally displaced persons - IDPs (people displaced by conflict and violence)
## Size:
## Format: JSON
## [[3]]
## <HDX Resource> b8ff1f4b-105c-4a6c-bf54-a543a486ab7e
## Name: disaster_data
## Description: Internally displaced persons - IDPs (new displacement associated with disasters)
## Size:
## Format: JSON
For this example, we are looking for the displacement data and it’s the
first resource in the dataset page. We can use pluck on the list of
resources or the helper function
get_resource(resource, resource_index) to select the resource we want
to use. The selected resource can be then downloaded and store for
further use or directly read into your R session using the
read_resource function. The resource is a json file and it can be
read directly using jsonlite package, we added a simplify_json
option to get a vector or a data.frame when possible instead of a
list.
idp_nga_rs <- get_resource(ds, 1)
idp_nga_df <- read_resource(idp_nga_rs, simplify_json = TRUE, download_folder = tempdir())
idp_nga_df
## # A tibble: 11 x 7
## ISO3 Name Year `Conflict Stock… `Conflict New D…
## <chr> <chr> <dbl> <dbl> <dbl>
## 1 NGA Nige… 2009 NA 5000
## 2 NGA Nige… 2010 NA 5000
## 3 NGA Nige… 2011 NA 65000
## 4 NGA Nige… 2012 NA 63000
## 5 NGA Nige… 2013 3300000 471000
## 6 NGA Nige… 2014 1075000 975000
## 7 NGA Nige… 2015 2096000 737000
## 8 NGA Nige… 2016 1955000 501000
## 9 NGA Nige… 2017 1707000 279000
## 10 NGA Nige… 2018 2216000 541000
## 11 NGA Nige… 2019 2583000 248000
## # … with 2 more variables: `Disaster New Displacements` <dbl>,
## # `Disaster Stock Displacement` <dbl>
All these operations can be chained using pipes %>% and allow for a
powerful grammar to easily get humanitarian data in R.
library(tidyverse)
set_rhdx_config(hdx_site = "prod")
idp_nga_df <-
search_datasets("displaced Nigeria", rows = 2) %>%
pluck(1) %>%
get_resource(1) %>% ## get the first resource
read_resource(simplify_json = TRUE, download_folder = tempdir()) ## the file will be downloaded in a temporary directory
idp_nga_df
## # A tibble: 11 x 7
## ISO3 Name Year `Conflict Stock… `Conflict New D…
## <chr> <chr> <dbl> <dbl> <dbl>
## 1 NGA Nige… 2009 NA 5000
## 2 NGA Nige… 2010 NA 5000
## 3 NGA Nige… 2011 NA 65000
## 4 NGA Nige… 2012 NA 63000
## 5 NGA Nige… 2013 3300000 471000
## 6 NGA Nige… 2014 1075000 975000
## 7 NGA Nige… 2015 2096000 737000
## 8 NGA Nige… 2016 1955000 501000
## 9 NGA Nige… 2017 1707000 279000
## 10 NGA Nige… 2018 2216000 541000
## 11 NGA Nige… 2019 2583000 248000
## # … with 2 more variables: `Disaster New Displacements` <dbl>,
## # `Disaster Stock Displacement` <dbl>
- Please report any issues or bugs.
- License: MIT
- Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.