-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Side effects upon loading rgdal/sf affecting GDAL utils functionality on Windows system with GDAL3 #31
Comments
Thanks @lbusett ! I hadn't thought of this option as a possible GDAL3/2 fall out. One way to mitigate this is to use Re: your As of your suggestion for a temporary fix: I'd be happy to submit a "temporary fix" sf PR doing this, but will not write it and also discard it after sf binaries have moved to GDAL3/PROJ6. When these binaries will appear is not entirely clear now: they would require help from @jeroen , and I'm not sure that this is worth the effort now or whether R 4 + the new toolchain are our better short term bet. |
thanks for the reply. Concerning the use of Concerning the possible "fix" on However, although apparently the side effect related with Finally, I think a temporary fix at least for avoiding |
I think a temporary fix in gdalUtils would be useful. I try to use vectortranslate
|
@mgageo you are hijacking the issue really, but you forgot the |
Function needed to avoid problems related to side effects upon loading of rgdal/sf on Windows systems with GDAL 3 installed (see r-spatial/discuss#31) The function automatically resets the PROJ_LIB environment variable to the "correct" folder for a GDAL3 installation, and restores the previous value upon exiting from the caller function (to avoid breaking rgdal/sf functionality). The function needs to be called within each function calling a gdal binary requiring the PROJ_LIB environment variable to be properly set.
So I feel like this is a problem on the rgdal side not resetting back the environment variable. Consider the case where someone does an rgdal command FIRST -- there isn't a clean way I can tell for gdalUtils to figure out what the "correct" environment variable is, since its been changed prior to gdalUtils being given the "chance" to fix it. Given how many unique installations there are of gdalUtils, "fixing" the environment variable seems an incredibly challenging problem if its been lost -- is there some gdal command that lets me "recover" the correct variables? I sent Roger Bivand an email about this, but really if a function is changing a user's environment variable, it should really be setting it back at the end of the run. |
I'm moving this from gearslaboratory/gdalUtils#24 --> this is really an issue with sf and rgdal setting system environment variables -- gdalUtils doesn't do this, but it does need the environment variables "respected". My suggestion is that for sf and rgdal, the easiest solution is if you are going to change a user's system environment variable, you preserve the old variable as a NEW environment variable, like OLD_PROJ_LIB=the_original_lib I am happy to tweak gdalUtils to use this (I can easily set this and then set it back to sf/rgdal's preferred variable) but I need the system variable preserved somehow in the session, and I think it would be easiest to just make an "archived" variable and let me know what the name is. Thoughts? Incidentally, this isn't a Windows-only thing -- this happens with OS X as well which can have > 1 install of GDAL. |
The OLD_PROJ_LIB idea seems sensible, but a bit fragile (someone else may be using it). Can we assume that the packages are loaded in the same R session, and packages setting the environmental variables can (somehow) write to a known file name in |
So I was originally thinking the idea of writing a file like that would be good, but I have repeatedly been told by the CRAN gurus they do not like packages auto-writing to a user's directory. I'd personally be ok with it, but the CRAN gurus may get grumpy. You would also need to set a variable in the user's workspace to point to that file, which can also be fragile. Is it the name you were concerned about (OLD_PROJ_LIB)? We could pick anything, honestly -- that was just a suggestion. As long as the other packages do not mess with it (I wouldn't with gdalUtils) I think it would be ok? Maybe RGDAL_OLD_PROJ_LIB? I'd do this with all the environment variables you are setting just to be on the safe side -- I think only PROJ is causing gdalUtils a headache but I don't know about other programs. |
If this is only about |
But there are many issues here. 1) gdalUtils does not use compiled code; 2) CRAN Windows and OSX binary packages like sf, gdalcubes, lwgeom and rgdal probably shouldn't be happy about mixing PROJ/GDAL versions (binary CRAN packages won't see the extra DLLs, but will see out-of-sync metadata files); 3) and so on. If we can go with a kludge, using |
I didn't know about sf::gdal_utils() --> how does that work? Are the binaries for all the utilities now installed "natively" by sf, or has the API changed so you don't need those standalone executables now? So just as a bit of a history lesson, I actually developed gdalUtils because the format drivers were not all being installed with rgdal -- hdf4 and hdf5 in particular on Windows boxes -- the only way to get a functional gdal with those drivers on Windows was to hand-install GDAL (OsGEO4W originally). The other functionality grew from that simple need. If sf has a complete set of utilities and all of the drivers that e.g. Osgeo4w has, that actually makes my life WAY easier (and would mean I can retire from supporting gdalUtils :) |
Recent windows builds for R packages using GDAL have a pretty extensive set of drivers, see https://github.com/rwinlib/gdal2 @jgrn307 I think we didn't keep you in that loop because you don't use binary linking to GDAL. |
So, to be clear, |
Ok, very cool -- it looks like it might be time to start retiring some of gdalUtils and maybe, if you are interested, moving some of the value-added stuff. So, can I propose, for now, you guys just set an "OLD_PROJ" or whatever environment variable so I can (quickly) tweak gdalUtils to work for existing users, and I can commit to starting to migrate gdalUtils functionality to take advantage of sf:gdal_utils() for backwards compatibility? The migration isn't going to happen quickly, so I was hoping for a quick-fix for now. I like not needing to call an external binary -- this was always a "hack" in my opinion, but one that we needed to get our own work done. It looks like a lot of progress had been made and I'm happy to move towards that solution. |
You could develop |
@edzer that is exactly what I was thinking -- BUT, this is a development angle that is going to take some time that I don't have in the next month, so I was hoping you and @rsbivand would do a quick "tweak" to create that environment variable and I'll do a quick fix with gdalUtils to take advantage of it. |
Before I even think of doing anything to rgdal, is my assumption that rgdal and gdalUtils are running in the same R session correct? I would see:
Something like that? That doesn't work for workflows involving other packages using the same environment variables - does that matter? |
Right, it is harder than you'd think: point 5 doesn't look like a quick fix. |
Personally, I'm more in favor of working towards getting GDAL3/PROJ6 to the CRAN windows and OSX build farms. |
I'll start looking into this so that we can bump the windows stack after the 4.0 release. |
Step five would be an easy fix, actually. Each gdalUtils command is ultimately just a system call, so setting and resetting are approximately two lines of code. |
@edzer and @rsbivand are you ok with the quick-fix on your end so we can get gdalUtils functional for users? I'll build towards a tighter integration with sf over the coming months, but as I said this isn't something I have the bandwidth for in the near-future, so I want to make sure our users can effectively use both for now. |
Draft built source package on: http://spatial.nhh.no/R/rgdal/rgdal_1.4-9.tar.gz. Now no Winbuilder, offline until tomorrow. After installing, look at |
This will not help if other packages setting the environment variables are also in play, but you should have what you need for a quick fix for gdalUtils, except for maybe for Windows OSGeo4W users installing rgdal from source - needs checking. |
Not sure whether this is the right place for following comment. For some reason, a colleague running Windows had a problem with Note that this seems to happen with GDAL 2.2.3 & PROJ.4 4.9.3. > gdalsrsinfo("+init=epsg:31370", as.CRS = TRUE)
Error in CRS(gsub("'", "", cmd_output)) :
PROJ4 argument-value pairs must begin with +: Warning 1: +init=epsg:XXXX syntax is deprecated. It might return a CRS with non-EPSG compliant axis order. +proj=lcc +lat_0=90 +lon_0=4.36748666666667 +lat_1=51.1666672333333 +lat_2=49.8333339 +x_0=150000.013 +y_0=5400088.438 +ellps=intl +towgs84=-106.8686,52.2978,-103.7239,0.3366,-0.457,1.8422,-1.2747 +units=m +no_defs
In addition: Warning messages:
1: In if (!is.na(projargs)) { :
the condition has length > 1 and only the first element will be used
2: In if (!is.na(projargs)) { :
the condition has length > 1 and only the first element will be used
3: In if (is.na(projargs)) uprojargs <- projargs else uprojargs <- paste(unique(unlist(strsplit(projargs, :
Show Traceback
Rerun with Debug
Error in CRS(gsub("'", "", cmd_output)) :
PROJ4 argument-value pairs must begin with +: Warning 1: +init=epsg:XXXX syntax is deprecated. It might return a CRS with non-EPSG compliant axis order. +proj=lcc +lat_0=90 +lon_0=4.36748666666667 +lat_1=51.1666672333333 +lat_2=49.8333339 +x_0=150000.013 +y_0=5400088.438 +ellps=intl +towgs84=-106.8686,52.2978,-103.7239,0.3366,-0.457,1.8422,-1.2747 +units=m +no_defs Environment info of my colleague is below, including external software versions. Environment> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=Dutch_Belgium.1252 LC_CTYPE=Dutch_Belgium.1252 LC_MONETARY=Dutch_Belgium.1252
[4] LC_NUMERIC=C LC_TIME=Dutch_Belgium.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] gdalUtils_2.0.3.2 DBI_1.1.0 odbc_1.2.2 RODBC_1.3-16 kableExtra_1.1.0 git2rdata_0.2.0
[7] n2khab_0.1.0 knitr_1.27 forcats_0.4.0 stringr_1.4.0 dplyr_0.8.3 purrr_0.3.3
[13] readr_1.3.1 tidyr_1.0.2 tibble_2.1.3 ggplot2_3.2.1 tidyverse_1.3.0 sf_0.8-1
loaded via a namespace (and not attached):
[1] httr_1.4.1 jsonlite_1.6 munsell_0.5.0 whisker_0.4 pillar_1.4.3
[6] pkgconfig_2.0.3 webshot_0.5.2 rgeos_0.5-2 R.utils_2.9.2 compiler_3.6.1
[11] lazyeval_0.2.2 hms_0.5.3 httpuv_1.5.2 R6_2.4.1 pkgbuild_1.0.6
[16] yaml_2.2.0 oai_0.3.0 data.table_1.12.8 xtable_1.8-4 tools_3.6.1
[21] promises_1.1.0 mime_0.8 usethis_1.5.1 shiny_1.4.0 modelr_0.1.5
[26] geoaxe_0.1.0 R.oo_1.23.0 reprex_0.3.0 pander_0.6.3 inborutils_0.1.0.9069
[31] git2r_0.26.1 nlme_3.1-143 prettyunits_1.1.1 rgbif_2.1.0 sp_1.3-2
[36] foreach_1.4.8 remotes_2.1.1 digest_0.6.23 KernSmooth_2.23-16 class_7.3-15
[41] codetools_0.2-16 vctrs_0.2.2 later_1.0.0 rvest_0.3.5 withr_2.1.2
[46] xfun_0.12 devtools_2.2.2 tidyselect_0.2.5 testthat_2.3.2 scales_1.1.0
[51] readxl_1.3.1 memoise_1.1.0 stringi_1.4.4 xml2_1.2.2 R.methodsS3_1.7.1
[56] assertthat_0.2.1 units_0.6-5 lattice_0.20-38 bit64_0.9-7 Rcpp_1.0.3
[61] fs_1.3.1 dbplyr_1.4.2 cli_2.0.1 callr_3.4.1 e1071_1.7-3
[66] sessioninfo_1.1.1 grid_3.6.1 classInt_0.4-2 blob_1.2.1 plyr_1.8.5
[71] viridisLite_0.3.0 rprojroot_1.3-2 raster_3.0-12 bookdown_0.17 pkgload_1.0.2
[76] processx_3.4.1 glue_1.3.1 magrittr_1.5 htmlwidgets_1.5.1 curl_4.3
[81] lubridate_1.7.4 parallel_3.6.1 lifecycle_0.1.0 rgdal_1.4-8 htmltools_0.4.0
[86] RSQLite_2.2.0 crayon_1.3.4 leaflet_2.0.3 desc_1.2.0 rstudioapi_0.10
[91] gtable_0.3.0 colorspace_1.4-1 fortunes_1.5-4 ellipsis_0.3.0 cellranger_1.1.0
[96] fansi_0.4.1 assertable_0.2.7 fastmap_1.0.1 ps_1.3.0 bit_1.1-15.2
[101] broom_0.5.4 rmarkdown_2.1 crosstalk_1.0.0 iterators_1.0.12 generics_0.0.2
[106] haven_2.2.0 evaluate_0.14 rlang_0.4.3 backports_1.1.5
>
> sf::sf_extSoftVersion()
GEOS GDAL proj.4 GDAL_with_GEOS USE_PROJ_H
"3.6.1" "2.2.3" "4.9.3" "true" "false"
>
> rgdal::rgdal_extSoftVersion()
GDAL GDAL_with_GEOS PROJ.4 sp
"2.2.3" "TRUE" "4.9.3" "1.3-2" We currently worked around the problem by avoiding a |
Hi all. I'm currently experiencing this issue, and the conversation here is a bit over my head. Would anyone consider providing a simplified explanation of what one should do when experiencing this problem? I'm having this issue when trying to use a function from gdalUtils, and I found another user recently posted the same issue when using RQGIS3. https://stackoverflow.com/questions/62141199/r-package-rqgis3-cannot-find-proj-db |
@Canderson156 a workaround is to @rsbivand Why is rgdal still setting the > Sys.getenv("PROJ_LIB")
[1] "" > library(rgdal)
Loading required package: sp
rgdal: version: 1.5-10, (SVN revision 1006)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.0.4, released 2020/01/28
Path to GDAL shared files: C:/Program Files/R/R-4.0.0/library/rgdal/gdal
GDAL binary built with GEOS: TRUE
Loaded PROJ runtime: Rel. 6.3.1, February 10th, 2020, [PJ_VERSION: 631]
Path to PROJ shared files: C:/Program Files/R/R-4.0.0/library/rgdal/proj
Linking to sp version:1.4-2
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading rgdal. > Sys.getenv("PROJ_LIB")
[1] "C:/Program Files/R/R-4.0.0/library/rgdal/proj" |
For rgdal, please see https://r-forge.r-project.org/scm/viewvc.php/pkg/R/AAA.R?view=markup&root=rgdal, and give me something, a reprex, to go on. If people can halp, I can commit trial versions to R-forge which will be built for Windows (but I'm not sure if R 4.0, if not R 4.0, I can build locally and post here). @jeroen I could try to set the PROJ datadir through the API, but this still needs conditioning on OS, PROJ version, and whether users are running the CRAN binary in an OSGeo setting, that is do not want anything set but want to use the existing PROJ_LIB. |
Yes you need to condition on proj version. I think the best convention is to use the |
As I suspected, use of |
rgdal rev 1008 committed to R-forge; if it builds, The changes implemented are to switch (for now) to the default context for all code using contexts except network code only active for PROJ >= 7, to add a C++ function setting the path to the package proj data directory (not yet exposed to the user) and use it instead of overwriting PROJ_LIB in PROJ 6. Maybe the additional startup message is at the moment unhelpful, but I'm coding on Fedora and only trying Windows separately. Feedback welcome, once R-Forge completes trying to build and check rev. 1008. I can try to put a 4.0-built Windows binary up later if someone wants one. |
rgdal Windows binary 1.5-11.zip online at http://spatial.nhh.no/R/rgdal/rgdal_1.5-11.zip. Feedback essential quickly; no feedback, no adaptation to suit unresponsive users. It is important that standardised use cases be provided, otherwise we can't even check what is or is not going on. Do we know which PROJ thread is set? I've modified rgdal (rev 1008) to only use the default context, I can get and set the path apparently leaving |
I installed the latest rgdal on Windows from source using: remotes::install_github("rforge/rgdal/pkg") And that seems to work. It no longer sets |
Great, thanks! Now what we need is confirmation from the users of RQGIS, etc., that our guess that this works checks out in their use cases. I'm still concerned that there are OSGeo4W users who really want their stack used in R Windows packages to align shared data directories with the versions they are using in other programs, but they need to make a reproducible case. |
I tested the |
The CRAN rgdal is also built with the current https://github.com/rwinlib/gdal3 versions, and no use of gdal2 is expected, so users of rgdal 1.4-8 on Windows should simply update the package. The CRAN Windows binary is using the same PROJ version coincidentally, but once I release 1.5-11 or later, the C API will be used, rather than setting PROJ_LIB for PROJ 6 only. PROJ 7 is harder, and neither sf nor rgdal handle multiple paths yet (see the final part of http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html). |
Dear all, So there's probably still something to fix. I am at your disposal if you would like further information or if you would like me to carry out additional tests to help you find the solution. Please find below an excerpt of the code:
Here is my sessionInfo()
|
@lovalery you should also cite |
@rsbivand below is the requested information. At your disposal, to send you additional information if needed.
|
Thanks. In which order are the packages loaded? With the rgdal release you seem to be running, use is made of |
I'm not sure if I'm answering your question correctly. So feel free to ask me again if there is any information missing. The order the libraries are loaded is as follows:
|
Check |
You will find below the whole sequence: since the start of R, I loaded each library and executed the
|
So none of the attached packages sets |
AFAIK the message does not affect the usage of QGIS algorithms, see r-spatial/RQGIS3#20 |
@jannes-m Thank you for your answer.
Here is the detailed example :
|
But the problem here are not the proj messages. This is a QGIS custom application bug, which we have come across several times and which is hard to debug, since QGIS returns the path to an output file which in fact has never been created. Sometimes it helped to use another file format, e. g. gpkg instead of shp but I don't think this will solve the problem here and was in any case just a crude solution. Sorry that I cannot provide a better answer. |
I confirm that changing .shp to .gpkg does not solve the problem (cf below).
|
I may be wrong , but I seem to remember that in the "old" RQGIS you needed to pass full paths to the input files, while here you are using a relative path to the input raster. Did you try that? Or even tried to pass a "R" raster object rather than a filename? |
@lbusett Thank you very much Lorenzo for these test proposals. I did them but unfortunately none of them succeeded.
|
Hi all,
sorry for the long message, but the problem is a bit complicated to explain.
I noticed this problem because at the moment the “spatial stack” on Windows is in a sorry state, where rgdal/sp/sf binaries are linked to GDAL2/PROJ4 while the OSGEO installer/updater (IMO the “best option” to install spatial libraries on Windows) already provides GDAL3/PROJ6 on new installs, or performs updates on old installs.
This creates problems if a R user/package wants to use both rgdal/sf functionality AND gdal binary utilities (e.g., gdalwarp, ogr2ogr) called either “directly” with
system2
, or through thegdalUtils
package, due to side-effects happening when loading/importingrgdal
/sf
.In particular, there appears to be problems related with
rgdal
andsf
modifying thePROJ_LIB
environment variable on load, happening here and here.I’ll try to clarify with an example:
On a recently updated windows machine with GDAL3/PROJ6 and all R packages up to date, in order for GDAL projection stuff to work the
PROJ_LIB
environment variable needs to be set to the “share/proj” subfolder of the OSgeo install folder.If this is the case, on a “fresh” R session I can use
gdal
binaries from R with no problems:However, as soon as
rgdal
orsf
are loaded we start having problems, because the environment variable is changed so to point to the
rgdal/proj
subfolderof the
gdal
(orsf
) package installation folder:, leading the system calls to unexpectedly error because now
PROJ_LIB
points to the “proj folder” provided byrgdal
, which is currently still PROJ4, while the binaries need to use that related toPROJ6.
The same problem currently prevents the use of
gdalUtils
functionality on Windows machines with GDAL3 installed. Infact, becausegdalUtils
importsrgdal
, even starting from a “fresh” session we get this:because, apparently, as soon as
gdalUtils::ogr2ogr
is called thePROJ_LIB
variable is changed:Now, although I agree with @rsbivand who strongly suggests to have only one “version” of GDAL libraries installed and used, in particular for unexperienced users, I still think that this behaviour is problematic, for two reasons:
On windows, installing from OSGEO is the “way to go” to get GDAL, for both beginners and advanced users. Therefore, until the R binaries start relying on GDAL 3, there may be problems in “mixing” rgdal/gdalUtils/gdal binaries functionalities, and packages exploiting that “mix” could have unexpected breakage.
Advanced users can still “downgrade” their GDAL system installation (although it is not ideal), but I fear that less experieced ones would struggle. Also, I know that packages authors could switch to using
gdalUtilities
orsf::gdal_utils
to avoid the problem (I am already doing that withinMODIStsp
), but 1) unfortunately not all gdal binary utilities are available ingdalUtilities
(see for example https://github.com/r-spatial/sf/issues/1213\#issuecomment-563977401), and 2) that would not solve the issue of packages currently relying on gdalUtils or system calls and unable/unwilling to do the switch.Even when the binaries will catch-up on GDAL3/PROJ6, I think that this kind of side effect should be avoided (and I think it is discouraged by CRAN policies, though I may be mistaken).
What is your take on this? I think a way to solve the problem could be to exploit
Sys.setenv
followed byon.exit
calls in any function “requiring”PROJ_LIB
to be properly set forrgdal
/sf
use, so that the environment variable can be automatically reset to its “former” value. Something like:This seems however a bit cumbersome, because it would need to be replicated several times. Do you see any other way?
Hope this helps, and that I am not missing something obvious here.
Lorenzo
Created on 2019-12-12 by the reprex package (v0.3.0)
The text was updated successfully, but these errors were encountered: