Pipeline shared volume is mounted with noexec flag, preventing execution of scripts in source code / local dependencies

[joebullard]

Context

I am running Drone 1.0 in Kuberentes-native mode after using Drone 0.8 for 1-2 years.

I noticed that I am unable to execute any scripts within my repository code or any of it's dependencies (e.g. certain packages installed from npm install), despite those files having the proper execute x permissions. This is a necessary part of my workflow and I would assume for many others as well.

My understanding is this:

• For a given pipeline, a shared volume is created and mounted into /drone/src on each step of that pipeline
• This volume is mounted with the noexec flag, which prevents execution of anything in that mountpoint, regardless of the x permission flag on the files.

How to reproduce

Here is a trivial .drone.yml which illustrates the underlying issue (though I discovered this in an uglier way with npm install)

.drone.yml

kind: pipeline
name: default

steps:
  - name: test
    image: node:10.15-alpine
    commands:
      # Display the current mounted filesystems and flags
      - mount
      # Show that we can execute something outside /drone/src (i.e. /bin/sh)
      - /bin/sh -c "echo -e '\nsuccessfully executed binary from outside pipeline dir\n'"
      # Show that we CANNOT execute something inside /drone/src
      - echo -e "#!/bin/sh\necho -e '\nsuccessfully executed script from pipeline dir\n'" > myscript.sh
      - chmod +x myscript.sh
      - ls -l myscript.sh
      - ./myscript.sh

Output

+ mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QPU7W2HS6MGJPSMFTDGN5BGHK4:/var/lib/docker/overlay2/l/R4GNKODYR2M5IFZBNDYDCTUKYR:/var/lib/docker/overlay2/l/2BKTY4MLUID2LNSANILEQNLRFK:/var/lib/docker/overlay2/l/VY5XMZC4RBYFDITRQO2FJM57NY,upperdir=/var/lib/docker/overlay2/1731f5c4b7c41e26dfb70a865d2e8277df6c84909bc494cd1677e00654228e52/diff,workdir=/var/lib/docker/overlay2/1731f5c4b7c41e26dfb70a865d2e8277df6c84909bc494cd1677e00654228e52/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tmpfs on /drone/src type tmpfs (rw,nosuid,nodev,noexec)
/dev/sda1 on /dev/termination-log type ext4 (rw,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/resolv.conf type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/hostname type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/hosts type ext4 (rw,relatime,commit=30,data=ordered)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/sda1 on /usr/drone/bin type ext4 (ro,relatime,commit=30,data=ordered)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,mode=755)
tmpfs on /sys/firmware type tmpfs (ro,relatime)
+ sh -c "echo -e '\nsuccessfully executed binary from outside pipeline dir\n'"

successfully executed binary from outside pipeline dir

+ echo -e "#!/bin/sh\necho -e '\nsuccessfully executed script from pipeline dir\n'" > myscript.sh
+ chmod +x myscript.sh
+ ls -l myscript.sh
-rwxr-xr-x    1 root     root            69 Jun  6 02:21 myscript.sh
/usr/drone/bin/init: line 32: ./myscript.sh: Permission denied
+ ./myscript.sh

You can see that /drone/src mountpoint has the noexec flag set:

+ mount
...
tmpfs on /drone/src type tmpfs (rw,nosuid,nodev,noexec)
...

You can also see that the /bin/sh, which resides in a partition without the noexec flag, is executable, while the dummy script I made within /drone/src is not, even though it has executable permissions on the file.

Ideas?

Is there a simple way to mount this without the noexec flag? A cursory look through the Drone source code and the k8s API Go documentation did not reveal an explicit setting of this flag (though I am a Go noob and have never peaked into the Drone source before this).

It seems though that this may be imposed by k8s itself, since we are using the tmpfs, but I found that in other Docker applications like docker-compose, you can override mount flags. Perhaps the same is possible for the k8s API and someone with more experience would be able to do it easily.

[bradrydzewski]

this seems to be a known issues with kubernetes. I was able to find this issue: https://github.com/kubernetes/kubernetes/issues/48912

I would caution against using the kubernetes runtime in production. The kubernetes runtime is labeled experimental (hopefully the docs convey this) which means it is not production ready and could (hypothetically) be removed in a future version. We are actively considering replacing it with a Tekton runner, for example. I recommend reading #65 and #69 to get a better sense of the state of native kubernetes.

[joebullard]

Thanks @bradrydzewski. I'm currently just experimenting with the k8s-native setup in parallel with our team's server/agent production setup. I guess I'll hold out for the future updates

[bradrydzewski]

the kubernetes implementation in this repository was scrapped for reasons described here. We have a new implementation, created from scratch, that no longer mounts a host machine volume which obsoletes this issue. New codebase can be found here:
https://github.com/drone-runners/drone-runner-kube

I tested to confirm it is fixed:

$ cat .drone.yml
kind: pipeline
type: kubernetes
name: default

clone:
  disable: true

steps:
- name: test
  pull: if-not-exists
  image: alpine
  commands:
  - echo "echo hello" > foo.sh
  - chmod +x foo.sh
  - ./foo.sh

$ drone-runner-kube exec --kubeconfig=$HOME/.kube/config
[test:1] + echo "echo hello" > foo.sh
[test:2] + chmod +x foo.sh
[test:3] + ./foo.sh
[test:4] hello