Question: Nix for building docker images

[dorranh]

I recently discovered the Renku project, and it looks super interesting! One question I had was if you all had looked into using nix for generating the renkulab Docker images for greater reproducibility of the renkulab environments (i.e. for stuff like host dependencies)?

[rokroskar]

Hi @dorranh thanks for bringing up this question! We have not considered nix - I really like the idea in principle. We are currently relying on some base images from the Jupyter and R/Rocker ecosystem that might make this a tad more complicated. I might be missing the point, but I think we'd need to rewrite the base images from scratch. Do you have an example of e.g. the Jupyter stack using nix? Would be very keen to see it, and maybe we could use it here?

[dorranh]

Hi @rokroskar, apologies for the late response! Here's a link to a super minimal example on the nixpkgs repo. The thing with Nix is that since it is tracking the entire dependency graph for your application of interest (including transitive dependencies), adding new content to the docker container boils down to just adding it to the list of packages like in the above example (i.e. you just have to worry about packaging the component with Nix, then adding it to a Docker image is trivial).

You can actually use other base images (e.g. one from DockerHub) with Nix, the only thing is that the contents coming from the external base image will not be reproducible (in the build sense, you can still use a tagged image).

For Jupyter + Nix, we currently have a small project providing some common kernels, etc. https://github.com/tweag/jupyterWith. That repo also contains several examples.

I can see two complicating factors with getting RenkuLab set up with Nix. One is that JupyterLab's extension infrastructure is a bit hard to manage under Nix, so I would imagine that we would need to explicitly package the core Jupyter extensions that ship with Renku (at least until JupyterLab 3.0 comes out, which will improve the situation). Two is that it looks like y'all are using the Julia kernel, which we haven't included with jupyterWith yet. That's definitely something we can (and should) do though.

If you're still interested in exploring this option I'd be happy to mock up an example - just let me know!

[rokroskar]

Hey @dorranh thanks a lot for your suggestions! it took me a while but I've finally come around to trying out nix with Renku. My current thinking is that it would be nice to provide nix "building blocks" so that renku components could be attached to any existing (potentially user-provided) image easily. That way we could use the whole jupyter stack of docker images, for example, with a simple nix-build, and all we have to continue to maintain is the renku nix package (Incidentally I'm having quite a lot of trouble installing renku with nix, but I chalk that up to not knowing nix very well.)

I know this isn't optimal in terms of reproducibility - we may take a more nix-centric stance for non-interactive images (e.g. those used only in workflows and not in on-line jupyter sessions). Those will require far less ad-hoc configurations like jupyter configs and user settings/permissions etc. But I think already giving users the option of providing virtually any image they want and putting in the required renku pieces automatically would be a huge win. What do you think?