Using cilium requires patching mounts #5286

Jip-Hop · 2024-11-22T21:55:58Z

Before creating an issue, make sure you've checked the following:

You are running the latest released version of k0s
Make sure you've searched for existing issues, both open and closed
Make sure you've searched for PRs too, a fix might've been merged already
You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

`docker info`

Client: Version: 27.3.1 Context: desktop-linux Debug Mode: false Plugins: ai: Ask Gordon - Docker Agent (Docker Inc.) Version: v0.1.0 Path: /Users/redacted/.docker/cli-plugins/docker-ai buildx: Docker Buildx (Docker Inc.) Version: v0.18.0-desktop.2 Path: /Users/redacted/.docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.30.3-desktop.1 Path: /Users/redacted/.docker/cli-plugins/docker-compose debug: Get a shell into any image or container (Docker Inc.) Version: 0.0.37 Path: /Users/redacted/.docker/cli-plugins/docker-debug desktop: Docker Desktop commands (Alpha) (Docker Inc.) Version: v0.0.15 Path: /Users/redacted/.docker/cli-plugins/docker-desktop dev: Docker Dev Environments (Docker Inc.) Version: v0.1.2 Path: /Users/redacted/.docker/cli-plugins/docker-dev extension: Manages Docker extensions (Docker Inc.) Version: v0.2.27 Path: /Users/redacted/.docker/cli-plugins/docker-extension feedback: Provide feedback, right in your terminal! (Docker Inc.) Version: v1.0.5 Path: /Users/redacted/.docker/cli-plugins/docker-feedback init: Creates Docker-related starter files for your project (Docker Inc.) Version: v1.4.0 Path: /Users/redacted/.docker/cli-plugins/docker-init sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.) Version: 0.6.0 Path: /Users/redacted/.docker/cli-plugins/docker-sbom scout: Docker Scout (Docker Inc.) Version: v1.15.0 Path: /Users/redacted/.docker/cli-plugins/docker-scout

Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 27.3.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.14-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.656GiB
Name: docker-desktop
ID: redacted
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/redacted/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Version

v1.31.2+k0s.0

Sysinfo

`k0s sysinfo`

Total memory: 7.7 GiB (pass)
File system of /var/lib/k0s: ext4 (pass)
Disk space available for /var/lib/k0s: 53.3 GiB (pass)
Relative disk space available for /var/lib/k0s: 91% (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.10.14-linuxkit (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: /sbin/modprobe (pass)
  Executable in PATH: mount: /bin/mount (pass)
  Executable in PATH: umount: /bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: built-in (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: built-in (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: built-in (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: built-in (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: built-in (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: built-in (pass)
      CONFIG_NETFILTER_NETLINK: built-in (pass)
      CONFIG_NF_NAT: built-in (pass)
      CONFIG_IP_SET: IP set support: built-in (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: built-in (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: built-in (pass)
      CONFIG_IP_VS: IP virtual server support: built-in (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: built-in (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: built-in (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: built-in (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: built-in (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: built-in (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: built-in (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: built-in (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: built-in (pass)
      CONFIG_NF_DEFRAG_IPV4: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: built-in (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: built-in (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: built-in (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: built-in (pass)
      CONFIG_NF_DEFRAG_IPV6: built-in (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: built-in (pass)
      CONFIG_LLC: built-in (pass)
      CONFIG_STP: built-in (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

I've tried the latest k0s version with the new entrypoint script by @twz123. I was hoping it would fix the issues I'm having with running k0s + cilium but unfortunately the entrypoint doesn't solve them. The log is full with these errors:

failed to generate spec: path \\\"/sys/fs/bpf\\\" is mounted on \\\"/sys\\\" but it is not a shared mount\"" component=containerd stream=stderr

And the cilium pods never start.

Steps to reproduce

Run this script to reproduce the issue:

#!/bin/bash
set -euo pipefail

KUBE_API_SERVER=https://localhost:6443

docker compose -f - up --build -d <<EOF
services:
  k0s:
    build:
        context: .
        dockerfile_inline: |
            FROM docker.io/k0sproject/k0s:v1.31.2-k0s.0
            # Use the new entrypoint script
            ADD --chmod=0755 https://raw.githubusercontent.com/k0sproject/k0s/18d3545594b8abac7e50aa70720dea44337f25fa/docker-entrypoint.sh /entrypoint.sh
    command: k0s controller --single --disable-components metrics-server --config=/etc/k0s/config.yaml
    volumes:
      - /var/lib/k0s
      - /var/log/pods
      - /lib/modules:/lib/modules:ro # required to get cilium working
    container_name: k0s
    hostname: k0s
    privileged: true
    tmpfs:
      - /run
      - /tmp
    ports:
      - 80:80
      - 443:443
      - 6443:6443
    network_mode: "bridge"
    restart: no
    environment:
      K0S_CONFIG: |-
        apiVersion: k0s.k0sproject.io/v1beta1
        kind: ClusterConfig
        metadata:
          name: k0s
        spec:
          telemetry:
            enabled: false
          network:
            kubeProxy:
              disabled: true
            provider: custom
          api:
            sans:
            - localhost
EOF

while [ ! "$(curl -k -s -o /dev/null -w "%{http_code}" https://localhost:6443)" -eq 401 ]; do
    echo "Sleep..."
    sleep 1
done

KUBECONFIG=$(mktemp -t kubeconfig)
export KUBECONFIG

docker exec k0s k0s kubeconfig admin >"$KUBECONFIG"
kubectl config set clusters.local.server "$KUBE_API_SERVER"

echo "Written kubeconfig to: $KUBECONFIG"

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

cilium install --version 1.16.4 \
    --set k8sServiceHost=localhost \
    --set k8sServicePort=6443 \
    --set kubeProxyReplacement=true \
    --set gatewayAPI.enabled=true \
    --set gatewayAPI.hostNetwork.enabled=true \
    --set envoy.securityContext.capabilities.keepCapNetBindService=true \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup

cilium status --wait --wait-duration=10m

# Apply the echo example from https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/splitting/
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.4/examples/kubernetes/gateway/echo.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.4/examples/kubernetes/gateway/splitting.yaml

# Wait for details deployments to be ready
kubectl rollout status deployment echo-1
kubectl rollout status deployment echo-2

sleep 1

curl --fail -s http://localhost/echo && echo "Cilium working as expected!"

Expected behavior

For cilium pods to come up and the script to continue past cilium status --wait --wait-duration=10m.

Actual behavior

The script times out waiting for cilium pods to come up.

Screenshots and logs

No response

Additional context

This below version of the script has the workarounds in place to make cilium work:

cgroup: host in the compose file
mount --make-rshared / as part of the entrypoint

With just mount --make-rshared / and not using cgroup: host I still run into these errors (and cilium pods don't start):

kubelet.go:1566] \"Failed to start ContainerManager\" err=\"cannot enter cgroupv2 \\\"/sys/fs/cgroup/kubepods\\\" with domain controllers -- it is in an invalid state\"" component=kubelet stream=stderr

Working script:

#!/bin/bash
set -euo pipefail

KUBE_API_SERVER=https://localhost:6443

docker compose -f - up --build -d <<EOF
services:
  k0s:
    cgroup: host # still seems to be required even with the new entrypoint magic
    build:
        context: .
        dockerfile_inline: |
            FROM docker.io/k0sproject/k0s:v1.31.2-k0s.0
            # Use the new entrypoint script
            ADD --chmod=0755 https://raw.githubusercontent.com/k0sproject/k0s/18d3545594b8abac7e50aa70720dea44337f25fa/docker-entrypoint.sh /entrypoint.sh
    command: |-
      sh -c '

        # This command is required in order to fix cilium inside the container
        mount --make-rshared /

        k0s controller --single \
            --disable-components metrics-server \
            --config=/etc/k0s/config.yaml \
      '
    volumes:
      - /var/lib/k0s
      - /var/log/pods
      - /lib/modules:/lib/modules:ro # required to get cilium working
    container_name: k0s
    hostname: k0s
    privileged: true
    tmpfs:
      - /run
      - /tmp
    ports:
      - 80:80
      - 443:443
      - 6443:6443
    network_mode: "bridge"
    restart: no
    environment:
      K0S_CONFIG: |-
        apiVersion: k0s.k0sproject.io/v1beta1
        kind: ClusterConfig
        metadata:
          name: k0s
        spec:
          telemetry:
            enabled: false
          network:
            kubeProxy:
              disabled: true
            provider: custom
          api:
            sans:
            - localhost
EOF

while [ ! "$(curl -k -s -o /dev/null -w "%{http_code}" https://localhost:6443)" -eq 401 ]; do
    echo "Sleep..."
    sleep 1
done

KUBECONFIG=$(mktemp -t kubeconfig)
export KUBECONFIG

docker exec k0s k0s kubeconfig admin >"$KUBECONFIG"
kubectl config set clusters.local.server "$KUBE_API_SERVER"

echo "Written kubeconfig to: $KUBECONFIG"

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

cilium install --version 1.16.4 \
    --set k8sServiceHost=localhost \
    --set k8sServicePort=6443 \
    --set kubeProxyReplacement=true \
    --set gatewayAPI.enabled=true \
    --set gatewayAPI.hostNetwork.enabled=true \
    --set envoy.securityContext.capabilities.keepCapNetBindService=true \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup

cilium status --wait --wait-duration=10m

# Apply the echo example from https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/splitting/
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.4/examples/kubernetes/gateway/echo.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.4/examples/kubernetes/gateway/splitting.yaml

# Wait for details deployments to be ready
kubectl rollout status deployment echo-1
kubectl rollout status deployment echo-2

sleep 1

curl --fail -s http://localhost/echo && echo "Cilium working as expected!"

References:

The text was updated successfully, but these errors were encountered:

Jip-Hop added the bug Something isn't working label Nov 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using cilium requires patching mounts #5286

Using cilium requires patching mounts #5286

Jip-Hop commented Nov 22, 2024 •

edited

Loading

Using cilium requires patching mounts #5286

Using cilium requires patching mounts #5286

Comments

Jip-Hop commented Nov 22, 2024 • edited Loading

Before creating an issue, make sure you've checked the following:

Platform

Version

Sysinfo

What happened?

Steps to reproduce

Expected behavior

Actual behavior

Screenshots and logs

Additional context

Jip-Hop commented Nov 22, 2024 •

edited

Loading