Install error in Docker container based on osgeo/gdal image

[Robinlovelace]

Split-out from rocker-org/geospatial#31 it's not clear why sf is not installing on a Docker container with the latest version of GDAL compiled from source. Reproducible example (takes some time - aim is to get pre-built images that will save time when testing against dev GDAL versions):

cd /tmp
git clone [email protected]:rocker-org/rocker-versioned2
cd rocker-versioned2
docker run --rm -ti -v $(pwd):/home/rstudio/data rocker/verse /bin/bash
bash /home/rstudio/data/scripts/install_gdal_source.sh
Rscript -e 'install.packages("sf")'

That process runs this bash script adapted from this GDAL Dockerfile by @cboettig. It results in the following error message - that suggests it's a question of sqlite3 or PROJ but both seem to be installed:

Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
also installing the dependencies ‘e1071’, ‘classInt’, ‘units’
# ...
* DONE (units)
* installing *source* package ‘classInt’ ...
# ...
* installing *source* package ‘sf’ ...
** package ‘sf’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: gcc
configure: CXX: g++ -std=gnu++11
checking for gdal-config... /usr/bin/gdal-config
checking gdal-config usability... yes
configure: GDAL: 3.3.0
checking GDAL version >= 2.0.1... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking gdal.h usability... yes
checking gdal.h presence... yes
checking for gdal.h... yes
checking GDAL: linking with --libs only... yes
checking GDAL: /usr/share/gdal/pcs.csv readable... no
checking GDAL: checking whether PROJ is available for linking:... yes
checking GDAL: checking whether PROJ is available fur running:... yes
configure: GDAL: 3.3.0
configure: pkg-config proj exists, will use it
configure: using proj.h.
configure: PROJ: 7.2.0
checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
configure: error: libproj or sqlite3 not found in standard or given locations.
ERROR: configuration failed for package ‘sf’
* removing ‘/usr/local/lib/R/site-library/sf’

The downloaded source packages are in
	‘/tmp/RtmprhnVYf/downloaded_packages’
Warning message:
In install.packages("sf") :
  installation of package ‘sf’ had non-zero exit status

[Robinlovelace]

Update: running this command before installing sf solved it, going to show that the process of creating reproducible examples can in some cases lead directly to the solution!

apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -yq --no-install-recommends install sqlite3=3.* && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && mkdir -p /root/db

Source: https://hub.docker.com/r/nouchka/sqlite3/dockerfile

[ranghetti]

Thank you @Robinlovelace for this patch (and many thanks to @cboettig for his work in rocker-org/geospatial#31), but unfortunately it does not solve in my case (which is quite strange, using Docker). Here below my output:

cd /tmp
git clone [email protected]:rocker-org/rocker-versioned2
cd rocker-versioned2
docker run --rm -ti -v $(pwd):/home/rstudio/data rocker/verse /bin/bash
rm -R /rocker_scripts/
ln -s /home/rstudio/data/scripts/ /rocker_scripts
bash /home/rstudio/data/scripts/install_gdal_source.sh
Rscript -e 'install.packages("sf", repos = "https://stat.ethz.ch/CRAN")'

(I added two lines - EDIT: in my previous version of the chunk they were in the wrong position - to update the scripts, in particular to ensure Robinlovelace/rocker-versioned2@54af48e to be present).

* installing *source* package ‘sf’ ...
** package ‘sf’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: gcc
configure: CXX: g++ -std=gnu++11
checking for gdal-config... /usr/bin/gdal-config
checking gdal-config usability... yes
configure: GDAL: 3.3.0
checking GDAL version >= 2.0.1... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking gdal.h usability... yes
checking gdal.h presence... yes
checking for gdal.h... yes
checking GDAL: linking with --libs only... yes
checking GDAL: /usr/share/gdal/pcs.csv readable... no
checking GDAL: checking whether PROJ is available for linking:... yes
checking GDAL: checking whether PROJ is available fur running:... yes
configure: GDAL: 3.3.0
configure: pkg-config proj exists, will use it
configure: using proj.h.
configure: PROJ: 7.2.0
checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
configure: error: libproj or sqlite3 not found in standard or given locations.
ERROR: configuration failed for package ‘sf’
* removing ‘/usr/local/lib/R/site-library/sf’

The downloaded source packages are in
        ‘/tmp/RtmpveuLqc/downloaded_packages’
Warning message:
In install.packages("sf") :
  installation of package ‘sf’ had non-zero exit status

[cboettig]

@Robinlovelace I'm seeing the same issue as @ranghetti

[Robinlovelace]

Hi @cboettig and @ranghetti now I'm seeing a different issue. I get:

cd /tmp
git clone [email protected]:rocker-org/rocker-versioned2
cd rocker-versioned2
docker run --rm -ti -v $(pwd):/home/rstudio/data rocker/verse /bin/bash
bash /home/rstudio/data/scripts/install_gdal_source.sh
install.packages("sf", repos = "https://cloud.r-project.org")
* installing *source* package ‘sf’ ...
** package ‘sf’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: gcc
configure: CXX: g++ -std=gnu++11
checking for gdal-config... no
no
configure: error: gdal-config not found or not executable.
ERROR: configuration failed for package ‘sf’
* removing ‘/usr/local/lib/R/site-library/sf’
* restoring previous ‘/usr/local/lib/R/site-library/sf’

The downloaded source packages are in
        ‘/tmp/Rtmp96aywq/downloaded_packages’

[edzer]

@robin gdal-config is not found, why is that an issue with sf?

[Robinlovelace]

@robin gdal-config is not found, why is that an issue with sf?

I didn't say it was, just reporting the result of what I tried. Initially I thought there was an issue with sf, hence opening the issue. Are you happy for us to continue the conversation here? Can continue it elsewhere but it seems relevant to the package and may help test it against upcoming GDAL versions.

[edzer]

My guess is that you're not focusing exclusively on sf, but on getting an environment for the whole r-spatial stack; for that, an alternative would be to open an issue at https://github.com/r-spatial/discuss.

[Robinlovelace]

My guess is that you're not focusing exclusively on sf, but on getting an environment for the whole r-spatial stack

True that. Suggest conversation on this specific issue continues here (even though the changes that result from the discussion will likely be made in https://github.com/rocker-org/rocker-versioned2, lots of repos + lots of issues!): rocker-org/geospatial#31

There is more general discussion here: r-spatial/discuss#35

[cboettig]

@edzer I'm still seeing the error in sf not being able to install on a ubuntu system with PROJ 7 installed from source, the error I get from sf install script is:

configure: using proj.h.
configure: PROJ: 7.2.0
checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
configure: error: libproj or sqlite3 not found in standard or given locations.
ERROR: configuration failed for package ‘sf’
* removing ‘/usr/local/lib/R/site-library/sf’

I am not clear why sf's configure routines cannot find libproj (installed from source) or sqlite3 (installed from apt-get as usual). What are the 'standard' locations? Can you tell us if those include /usr/local (the standard root path for things installed from source) or is it only looking in /usr/? Any suggestions on where to look?

My build recipe is here: https://github.com/rocker-org/rocker-versioned2/blob/master/scripts/install_gdal_source.sh (referenced install scripts are in the same folder), running on ubuntu:focal-based container with no PPAs or other non-standard repos. The recipe for installing GDAL, PROJ, and GEOS from source is based on the one from OSGEO, as noted in the comment, and appears to be a standard install that would put these libraries in a standard location -- i.e. you can see, it writes PROJ files to the corresponding locations in /usr/local (a bit out of my depth here but I think the system would usually check those paths, at least after ldconfig runs?)

proj seems to run fine by itself, so this does seem to be an issue with sf config and not the PROJ install per se....

[edzer]

If you want me to look into this in detail, you need to tell me how to build the docker image, the script you point to is just part of the whole install I guess.

From your issue and the script you point to I cannot see whether sf was instructed to look at /usr/local, or whether the system has been set up to always look there, or whether there are multiple versions installed of PROJ, or ...

[cboettig]

Hi Edzer, yes apologies that's a much better idea than just guessing where config might be looking!

The easiest way to reproduce is to grab the dev-osgeo tag:

docker run --rm -ti rocker/geospatial:dev-osgeo bash

Once at the bash shell inside the container, just install sf:

install2.r sf

That should reproduce the error.

You could alternatively start a bit farther back by running Docker with the base image:

docker run --rm -ti rocker/verse bash

And then run the gdal install script I mentioned:

/rocker_scripts/install_gda_source.sh

if you need to see the logs etc. After that script runs, once again trying to install sf should create the same error .

[edzer]

I think the clue lies here: https://trac.osgeo.org/gdal/wiki/BuildingOnUnixGDAL25dev

It seems you are trying to install an R spatial stack using some docker setup that was developed to install GDAL properly, but not offer a proper PROJ in addition. In /usr/local/lib of the image we see

root@be33a3844938:/sf# ls -la /usr/local/lib
total 3596
drwxr-xr-x 1 root root    4096 Nov  1 20:35 .
drwxr-xr-x 1 root root    4096 Oct 25 06:38 ..
lrwxrwxrwx 1 root root      25 Oct 25 06:34 libinternalproj.so -> libinternalproj.so.19.1.0
lrwxrwxrwx 1 root root      25 Oct 25 06:34 libinternalproj.so.19 -> libinternalproj.so.19.1.0
-rwxr-xr-x 1 root root 3652912 Oct 25 06:34 libinternalproj.so.19.1.0
lrwxrwxrwx 1 root root      25 Oct 25 06:34 libproj.so.19 -> libinternalproj.so.19.1.0
drwxr-xr-x 1 root root    4096 Oct 25 06:34 pkgconfig
drwxr-xr-x 1 root root    4096 Oct 25 05:34 R

meaning no libproj.so, and that is also not somewhere else. If I create one there using ln -s, and link against libproj.so.19, of course I'm linking to libinternalproj.so and, see gdal wiki ref above, that renamed all symbols so cannot be linked to by anything expecting regular proj symbols. So, essentially, this image comes with a kind of for-GDAL-only proj, and proper PROJ will have to be added to it for anything depending on it (like sf) to work.

[cboettig]

Thanks @edzer , very helpful. Yes, it looks like the OSGeo recipe is doing the essentially same trick that is described on their wiki you point to, building with renamed symbols for an "internal proj". If I've understood the wiki entry correctly, the purpose that approach is to allow users to allow users to also use prebuilt binaries that have been built against other versions of PROJ (?).

That suggested to me that I might be able to simply add a standard PROJ install on top of this image, using vanilla ./configure settings that don't -DPROJ_RENAME_SYMBOLS. Unfortunately and probably no surprise to you, that doesn't work either, as gdal then ends up confused by the two libraries and so sf configure fails.

It does seem that the "best" route suggested by the wiki, rebuilding everything against the standard sources (without renamed symbols) ought to work, but haven't had luck with that route. I can build PROJ from source without renamed symbols, and build GDAL from source binding those libs, but still hit the wall on my sf config, where it appears my gdal installation has not in fact been succesful:

checking GDAL: checking whether PROJ is available for linking:... yes
checking GDAL: checking whether PROJ is available fur running:... double free or corruption (out)
./configure: line 3583: 51349 Aborted                 (core dumped) ./gdal_proj
no
configure: error: OGRCoordinateTransformation() does not return a coord.trans: PROJ not available?
ERROR: configuration failed for package ‘sf’
* removing ‘/usr/local/lib/R/site-library/sf’

Probably not an issue with sf but rather my GDAL install. Anyway, if you have or are aware of existing recipes you'd recommend, let me know, otherwise this will clearly require a bit more digging on my part. appreciate all the help you've already given us.

@Robinlovelace let us continue back over on rocker-org/geospatial#31

[edzer]

FYI, I keep a number of docker files for my own purposes (testing...) here: https://github.com/r-spatial/sf/tree/master/inst/docker