Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gdalUtils and gdal 3.0.2 : ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db #24

Open
mgageo opened this issue Dec 9, 2019 · 26 comments

Comments

@mgageo
Copy link

mgageo commented Dec 9, 2019

Hello,
I use gdaUtils with the gdal version from QGIS : https://www.qgis.org/fr/site/forusers/download.html
https://qgis.org/downloads/QGIS-OSGeo4W-3.10.0-2-Setup-x86_64.exe

On a command like

ogr2ogr('rgdal.geojson','rgdal.json',t_srs="EPSG:2154",verbose=TRUE)
Checking gdal_installation...
Scanning for GDAL installations...
Checking the gdalUtils_gdalPath option...
GDAL version 3.0.2
GDAL command being used: "C:\Program Files\QGIS 3.10\bin\ogr2ogr.exe" -t_srs "EPSG:2154" "rgdal.json" "rgdal.geojson"
[1] "ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db" "ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db"
[3] "ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db" "ERROR 1: Failed to process SRS definition: EPSG:2154"

I have an other version of gdal from http://download.gisinternals.com/sdk/downloads/release-1911-x64-gdal-3-0-mapserver-7-4.zip in the directory d:\gdal30

I replace the system call "system(cmde, intern = TRUE)" by a small script, cf https://framagit.org/mgageo/osmdata/blob/master/scripts/ogr2ogr.R and it's good

@lbusett
Copy link

lbusett commented Dec 12, 2019

Hi @mgageo and @jgrn307 ,

I'd guess this is the same problem/related to what I just reported here:

r-spatial/discuss#31

, and that I was going to report also here.

HTH,

Lorenzo

@jgrn307
Copy link
Contributor

jgrn307 commented Jan 16, 2020

If you run this line (from above) from Windows CMD line, what happens?

"C:\Program Files\QGIS 3.10\bin\ogr2ogr.exe" -t_srs "EPSG:2154" "rgdal.json" "rgdal.geojson"

@mgageo
Copy link
Author

mgageo commented Jan 17, 2020

D:\tmp>"C:\Program Files\QGIS 3.10\bin\ogr2ogr.exe" -t_srs "EPSG:2154" "rgdal.json" "rgdal.geojson"
ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db
ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db
ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db
ERROR 1: Failed to process SRS definition: EPSG:2154

D:\tmp>

@lbusett
Copy link

lbusett commented Jan 22, 2020

@mgageo @jgrn307

A way to fix the issue in r-spatial/discuss#31 is to reassign properly (or set in the first place if it is not set) the "PROJ_LIB" environment variable on the fly, before issuing any GDAL command that requires it. I implemented a possible fix (at the moment, only valid for gdalwarp), here:

lbusett@5d6a1ae

@mgageo

Could you try if it works also for you? You can do that by installing my fork from github:

remotes::install_github("lbusett/gdalUtils", ref = "feature/projlibpath")
library(gdalUtils)

and then trying commands:

src_dataset <- system.file("external/tahoe_highrez.tif", package="gdalUtils")
# Command-line gdalwarp call:
gdalwarp(src_dataset,dstfile="tahoe_highrez_utm11.tif",
		t_srs='+proj=utm +zone=11 +datum=WGS84',output_Raster=TRUE,
 		overwrite=TRUE,verbose=TRUE)

and see if it works.

If this seems ok/viable, the fix can be easily extended to other gdalUtils functions.

@jgrn307
Copy link
Contributor

jgrn307 commented Jan 22, 2020

Ok, trying to get a TLDR here -- basically either rgdal or gdalUtils is causing the PROJ_LIB to change, which breaks some of the functionality?

  1. Is this an accurate reading and, if so, which is changing the lib (rgdal, gdalUtils, something else)?
  2. What is the TLDR fix you are proposing? Before the GDAL CMD is run, preserve the existing PROJ_LIB, run the gdal command,, and then set it back to the original PROJ_LIB?
  3. Is this Windows specific?

If so, I think the simplest solution is to modify the GDAL command so this gets pre/post-appended to the command that is actually run (so I can universally change it in gdal_cmd_builder instead of having to tweak each command individually).

--j

@lbusett
Copy link

lbusett commented Jan 23, 2020

Hi @jgrn307 , I'll reply inline:

Ok, trying to get a TLDR here -- basically either rgdal or gdalUtils is causing the PROJ_LIB to change, which breaks some of the functionality?

  1. Is this an accurate reading and, if so, which is changing the lib (rgdal, gdalUtils, something else)?

Yes, that is correct. The "culprit" here is rgdal, which namespace is imported by gdalUtils, so that as soon as the library is loaded (or even a function is called such as in gdalUtils::ogr2ogr(src_datasource_name,...), the environment variable is immediately modified (see https://github.com/cran/rgdal/blob/848067a0251323829dd776cf0789d1410cb77b3b/R/AAA.R#L14).

What is the TLDR fix you are proposing? Before the GDAL CMD is run, preserve the existing PROJ_LIB, run the gdal command,, and then set it back to the original PROJ_LIB?

The fix relies on introducing the new function set_projlibpath to do exactly that, using the on.exit mechanism to restore the previous value after each system call (to avoid breaking rgdal/sf functionality instead)

Is this Windows specific?

I encountered this only on Windows. It is due to the fact that at the moment the Windows rgdal/sf package binaries are linked to GDAL2/PROJ4 while the OSGEO installer/updater provides GDAL3/PROJ6 on new installs/updates. This leads gdalUtils to often rely on GDAL3 binaries, while rgdal forces the use of a PROJ_LIB folder related to a GDAL2 installation.
I do not work on MAC, so I do not know if it is also affected.
Consider however that the proposed solution, though currently being "applied" only on Windows OS, could be used with no problems on all OS, simply removing the Sys.info()[["sysname"]] == "Windows" in the new function.

If so, I think the simplest solution is to modify the GDAL command so this gets pre/post-appended to the command that is actually run (so I can universally change it in gdal_cmd_builder instead of having to tweak each command individually).

This is a good idea. However, consider that this would require moving also the system call within the gdal_cmd_builder function, instead than within the single gdalUtils functions (This because set_projlibpath needs to be called from within the function making the system call, in order to properly re-set the environment variable on exit - see https://yihui.org/en/2017/12/on-exit-parent/). I tested it, and it seems to work. I could prepare a PR if you wish so.

HTH,

Lorenzo

@mgageo
Copy link
Author

mgageo commented Jan 23, 2020

image

@mgageo
Copy link
Author

mgageo commented Jan 23, 2020

and after OK
image

@lbusett
Copy link

lbusett commented Jan 23, 2020

@mgageo

Thaks for testing. Could you try installing again from CRAN, test the following:

src_datasource_name <- system.file("external/tahoe_highrez_training.shp", package="gdalUtils")
dst_datasource_name <- paste(tempfile(),".shp",sep="")
# reproject the input to mercator
ogr2ogr(src_datasource_name,dst_datasource_name,t_srs="EPSG:3395",verbose=TRUE)

, and then re-test after reinstalling again from my fork on github?

@mgageo
Copy link
Author

mgageo commented Jan 23, 2020

The version from CRAN
image

@mgageo
Copy link
Author

mgageo commented Jan 23, 2020

After install from github
image

@lbusett
Copy link

lbusett commented Jan 23, 2020

Strange... it works for me...

could you please share your session info?

devtools::session_info()

@mgageo
Copy link
Author

mgageo commented Jan 23, 2020

devtools::session_info() - Session info -------------------------------------------------------------------------------------------------------- setting value version R version 3.6.2 (2019-12-12) os Windows 10 x64 system x86_64, mingw32 ui RStudio language (EN) collate French_France.1252 ctype French_France.1252 tz Europe/Paris date 2020-01-23 - Packages ------------------------------------------------------------------------------------------------------------ package * version date lib source assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.1) backports 1.1.5 2019-10-02 [1] CRAN (R 3.6.1) callr 3.4.0 2019-12-09 [1] CRAN (R 3.6.2) cli 2.0.1 2020-01-08 [1] CRAN (R 3.6.2) codetools 0.2-16 2018-12-24 [2] CRAN (R 3.6.2) crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.1) desc 1.2.0 2018-05-01 [1] CRAN (R 3.6.1) devtools 2.2.1 2019-09-24 [1] CRAN (R 3.6.1) digest 0.6.23 2019-11-23 [1] CRAN (R 3.6.2) ellipsis 0.3.0 2019-09-20 [1] CRAN (R 3.6.1) fansi 0.4.1 2020-01-08 [1] CRAN (R 3.6.2) foreach 1.4.7 2019-07-27 [1] CRAN (R 3.6.1) fs 1.3.1 2019-05-06 [1] CRAN (R 3.6.1) gdalUtils * 2.0.2.4 2020-01-23 [1] Github (lbusett/gdalUtils@709cc02) glue 1.3.1 2019-03-12 [1] CRAN (R 3.6.1) iterators 1.0.12 2019-07-26 [1] CRAN (R 3.6.1) lattice 0.20-38 2018-11-04 [2] CRAN (R 3.6.2) magrittr 1.5 2014-11-22 [1] CRAN (R 3.6.1) memoise 1.1.0 2017-04-21 [1] CRAN (R 3.6.1) pkgbuild 1.0.6 2019-10-09 [1] CRAN (R 3.6.2) pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.6.1) prettyunits 1.1.0 2020-01-09 [1] CRAN (R 3.6.2) processx 3.4.1 2019-07-18 [1] CRAN (R 3.6.1) ps 1.3.0 2018-12-21 [1] CRAN (R 3.6.1) R.methodsS3 1.7.1 2016-02-16 [1] CRAN (R 3.6.0) R.oo 1.23.0 2019-11-03 [1] CRAN (R 3.6.1) R.utils 2.9.2 2019-12-08 [1] CRAN (R 3.6.1) R6 2.4.1 2019-11-12 [1] CRAN (R 3.6.2) raster 3.0-7 2019-09-24 [1] CRAN (R 3.6.1) Rcpp 1.0.3 2019-11-08 [1] CRAN (R 3.6.1) remotes 2.1.0 2019-06-24 [1] CRAN (R 3.6.1) rgdal 1.4-8 2019-11-27 [1] CRAN (R 3.6.1) rlang 0.4.2 2019-11-23 [1] CRAN (R 3.6.2) rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.6.1) rstudioapi 0.10 2019-03-19 [1] CRAN (R 3.6.1) sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.1) sp 1.3-2 2019-11-07 [1] CRAN (R 3.6.1) testthat 2.3.1 2019-12-01 [1] CRAN (R 3.6.2) usethis 1.5.1 2019-07-04 [1] CRAN (R 3.6.1) withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.1) [1] C:/Users/marcg/Documents/R/win-library/3.6 [2] C:/Program Files/R/R-3.6.2/library
--
 
| >

@lbusett
Copy link

lbusett commented Jan 23, 2020

@mgageo

seems that a reset of the r session is required. Could you retry after restarting the rsession/restarting rstudio, if you did not do that already?

@mgageo
Copy link
Author

mgageo commented Jan 24, 2020

After a restart I have the popup but
image

@lbusett
Copy link

lbusett commented Jan 24, 2020

That should mean that it worked.
What do you get from

file.exists(dst_datasource_name)

?

@mgageo
Copy link
Author

mgageo commented Jan 24, 2020

file.exists(dst_datasource_name)
[1] TRUE
and
image

@lbusett
Copy link

lbusett commented Jan 24, 2020

Seems therefore that the "fix" would work.

@jgrn307 What do you think of the solution introduced in lbusett@709cc02 ?

I's based on moving the system calls within cmd_builder. Implementing it in all gdalUtils functions would be straightforward. I could make a PR if you wish.

Lorenzo

@jgrn307
Copy link
Contributor

jgrn307 commented Jan 24, 2020 via email

@jgrn307
Copy link
Contributor

jgrn307 commented Feb 13, 2020

So this REALLY seems like an rgdal problem the more I look into it. Let me see if Bivand is willing to tweak his code a bit. It seems like if someone runs rgdal for ANY reason before gdalUtils is run, the environment variable is "lost" in a way that is going to make it very hard to recover. Am I missing something?

@rsbivand
Copy link

No, it is a problem for those using Windows or OSX binary installs of CRAN packages, and the same problem will exist for all other packages not building from source locally (I'm not sure which, but rgdal and sf certainly, probably vapour). There is no easy way to provide half-binary Windows builds (OSX the same), depending on DLLs that the user has to install first. Yes, it is a problem, but no, there is no rgdal specific resolution. If we tweak rgdal on Windows to respect an existing PROJ_LIB environment variable, we will very quickly find that say an OSGeo4W PROJ or GDAL is a different version from the version used to build rgdal.dll, so will expect say proj/epsg not proj/proj.db. Protecting all rgdal functions on CRAN binary builds by setting and resetting PROJ_LIB (and the GDAL equivalent) on before and after each .Call() is overkill, I think.

If gdalUtils required https://github.com/rwinlib/gdal2 and that built the binaries too, all the Windows CRAN binary users would be on the same page. Or gdalUtils could wrap the use in sf of the utilities as library functions.

@jgrn307
Copy link
Contributor

jgrn307 commented Feb 13, 2020

Roger, here's a potential very easy tweak you could make that I could easily use with gdalUtils: before you set the environment variable(s), can you simply create/set a new environment variable named something like "OLD_PROJ_LIB" and throw the "old" environment variable in there? This would then let me grab it easily (as long as I know the name of the variable) and I can do a set/reset back. Would you be willing to make this change? Just tack that right above the line you do the environment variable change?

@jgrn307
Copy link
Contributor

jgrn307 commented Feb 13, 2020

@edzer I think this would need to be done with sf also.

@jgrn307
Copy link
Contributor

jgrn307 commented Feb 13, 2020

Let's move this to: r-spatial/discuss#31

@JepsonNomad
Copy link

Full disclosure: I'm not using gdalUtils. Not trying to hijack this discussion, but I've been doing some searching after running into an apparently related issue on Mac and I thought it might be relevant for your open issue. I'm running OSX 10.15.6 Catalina. Using sf::st_wrap_dateline() was causing me problems when rgdal is loaded:

> library(rgdal)
Loading required package: sp
rgdal: version: 1.5-16, (SVN revision 1050)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 2.4.2, released 2019/06/28
Path to GDAL shared files: /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rgdal/gdal
GDAL binary built with GEOS: FALSE 
Loaded PROJ runtime: Rel. 5.2.0, September 15th, 2018, [PJ_VERSION: 520]
Path to PROJ shared files: /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rgdal/proj
Linking to sp version:1.4-2
Overwritten PROJ_LIB was /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rgdal/proj
Warning message:
package ‘rgdal’ was built under R version 3.6.2 
> library(rnaturalearth)
> library(sf)
Linking to GEOS 3.8.1, GDAL 3.1.2, PROJ 7.1.1
> library(ggplot2)
> Sys.getenv("PROJ_LIB")
[1] "/Library/Frameworks/R.framework/Versions/3.6/Resources/library/rgdal/proj"
> sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
       "3.8.1"        "3.1.2"        "7.1.1"         "true"         "true" 
> 
> robin = st_crs("+proj=robin +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs")
> 
> ROI = ne_countries(returnclass = 'sf') %>%
+   st_combine() 
> 
> #Please check the math on this part.
> #KM/earth circumference * degrees in circle
> buffer_in_km <- 500
> buffer_as_arc_degrees<- buffer_in_km/40075*360
> 
> coastalWaters = ROI %>%
+   st_buffer(buffer_as_arc_degrees)  %>% 
+   st_wrap_dateline()
dist is assumed to be in decimal degrees (arc_degrees).
Warning messages:
1: In st_buffer.sfc(., buffer_as_arc_degrees) :
  st_buffer does not correctly buffer longitude/latitude data
2: In st_is_longlat(x) :
  bounding box has potentially an invalid value range for longlat data
> 
> ggplot() +
+   geom_sf(data = coastalWaters, fill = "lightblue", col = "transparent") +
+   geom_sf(data = ROI) +
+   coord_sf(crs = robin)
Error: node stack overflow
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Error during wrapup: node stack overflow

Running warnings() resulted in this:

Warning messages:
1: In CPL_crs_from_input(x) :
  GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db

... with the warning repeated 50 times before exhausting. However, if I repeat exactly the same script as above, but with library(rgdal) commented out, everything runs fine. Note however that Sys.getenv("PROJ_LIB") returns [1] "". My sessionInfo() result is:

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Catalina 10.15.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ggplot2_3.3.0       sf_0.9-5            rnaturalearth_0.1.0 rgdal_1.5-16        sp_1.3-2           

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.3         rstudioapi_0.10    magrittr_1.5       units_0.6-5        munsell_0.5.0     
 [6] tidyselect_0.2.5   colorspace_1.4-1   lattice_0.20-38    R6_2.4.1           rlang_0.4.2       
[11] dplyr_0.8.3        tools_3.6.1        grid_3.6.1         gtable_0.3.0       KernSmooth_2.23-15
[16] e1071_1.7-3        DBI_1.0.0          withr_2.1.2        rgeos_0.5-5        class_7.3-15      
[21] assertthat_0.2.1   lifecycle_0.1.0    tibble_2.1.3       crayon_1.3.4       purrr_0.3.3       
[26] glue_1.3.1         compiler_3.6.1     pillar_1.4.3       scales_1.1.0       classInt_0.4-2    
[31] pkgconfig_2.0.3   

Also possibly relevant, from the Terminal:

JepsonNomad:bin <redacted>$ gdalinfo --version
GDAL 3.1.2, released 2020/07/07
JepsonNomad:bin <redacted>$ proj
Rel. 7.1.1, September 1st, 2020
usage: proj [-bdeEfiIlmorsStTvVwW [args]] [+opt[=arg] ...] [file ...]

@rsbivand
Copy link

@JepsonNomad Post a minimal reproducible example on R-sig-geo after subscribing if need be. This is almost certainly the wrong place to ask.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants