merge-trees-to-database #15
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- added a merge step to upload to the database - added the db config files similar to the wtt_server - added command line arguments for the merge statement - the merge statement runs ogr2ogr then calls a stored procedure - still some TODOs for more async code
zoobot
reviewed
Feb 2, 2023
| @@ -12,6 +12,7 @@ | |||
| "convert": "node bin/convert.js", | |||
| "normalize": "node bin/normalize.js", | |||
| "concatenate": "node bin/concatenate.js", | |||
| "merge": "node bin/merge.js", | |||
| "save": "node bin/save.js", | |||
There was a problem hiding this comment.
Merge will get rid of the need for the save step I think. Do you want to handle that or should I assign myself?
| * source: | ||
| * https://github.com/vitaly-t/pg-promise/issues/78#issuecomment-171951303 | ||
| */ | ||
| function camelizeColumns(data) { |
There was a problem hiding this comment.
Just a note, we have this function replicated here: https://github.com/waterthetrees/wtt_server/blob/68e22644d31da834a69b4435373e8b9aeac9ad5e/server/db/pg-promise-config.js
Probably not worth calling between repos tho since its a common utility function.
| console.log(`Running for ${source.destinations.normalized.path}`); | ||
| // FIXME use the async version of exec, but that means a new dependecy | ||
| const command = `ogr2ogr -f "PostgreSQL" PG:"host=${dbConfig.host} user=${dbConfig.user} password=${dbConfig.password} dbname=${dbConfig.database}" ${source.destinations.normalized.path} -nln tree_staging -geomfield geom -append` | ||
| exec(command, async (error, stdout, stderr) => { |
There was a problem hiding this comment.
Prefer spawn here as it has larger buffer for longer running processes. Could use spawnSync.
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a merge step so that we can move trees into the database based on the geojson files. To do this, I've added a
mergestep to the data pipeline. This would run sometime after normalization using the geojsons from that step.The process does two things:
There are still a few TODOs in terms of async code we can add, albeit with additional libraries.
The performance of this step depends on the size of the file.
ogr2ogruploads 20k rows at a time. A file with 600k rows (like NYC) can take on the order of minutes, while a file with 20k rows (like Alameda) is within seconds. This is expected to run monthly, so that is not much of a concern.Not included in here is the
.envfile that will be necessary holding the database info, similar to the wtt_server repo.