Gateway API on OpenShift 4.13

This doc covers how to deploy the Gateway API on an OpenShift 4.13 Cluster.

Information in this document is not supported by Red Hat.

Versions used:

OpenShift v4.13.0-rc.3
Gateway API v0.6.2
MetalLB v4.12

The end goal is to be able to create HTTPRoutes exposing an application via HTTP/s. In the future, as the gateway implementations mature we can explore other ways of exposing our applications like TCPRoute or TLSRoute, and do cross-namespace references with ReferenceGrants.

Introduction to Gateway API

Gateway API is an open source project managed by the SIG-NETWORK community. It is a collection of resources that model service networking in Kubernetes. These resources - GatewayClass, Gateway, HTTPRoute, TCPRoute, Service, etc - aim to evolve Kubernetes service networking through expressive, extensible, and role-oriented interfaces that are implemented by many vendors and have broad industry support. ^Source

Deploying the Gateway API

For this introduction we are deploying the standard APIs, so we won't get support for TCPRoute, TLSRoute, etc. Only basic support for HTTPRoute will be available.

Deploy the Gateway API CRDs and admission server

oc apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.6.2/standard-install.yaml

If we check the pod created in the gateway-system namespace, we will see that the admission server is on ContainerCreating:

oc -n gateway-system get pods

NAME                                            READY   STATUS              RESTARTS   AGE
gateway-api-admission-server-546bdb8747-mdzsk   0/1     ContainerCreating   0          74s

Pod cannot start because the secret gateway-api-admission is not created. That secret gets created by one job, if we check the jobs we will notice that pods are not running for those jobs:

oc -n gateway-system get jobs

NAME                          COMPLETIONS   DURATION   AGE
gateway-api-admission         0/1           4m3s       4m3s
gateway-api-admission-patch   0/1           4m3s       4m3s

These jobs try to run with a nonroot UID, but in OpenShift it's required access to a valid SecurityContextConstraing (SCC) to be able to do so. We will grant this access to the ServiceAccount that runs the jobs:
```
securityContext:
  runAsNonRoot: true
  runAsUser: 2000
```
```
oc -n gateway-system adm policy add-scc-to-user nonroot-v2 -z gateway-api-admission
```

After a few moments the admission server pod and the job's pod should be okay:

oc -n gateway-system get pods

NAME                                            READY   STATUS      RESTARTS   AGE
gateway-api-admission-patch-r5ghp               0/1     Completed   0          3m12s
gateway-api-admission-server-546bdb8747-mdzsk   1/1     Running     0          7m22s
gateway-api-admission-vbt77                     0/1     Completed   0          3m12s

Now we have the Gateway API running, next step is choosing the Gateway API implementation of our choice. You can find a list of current implementations here.

Deploying the NGinx Gateway Controller

We will be using the NGinx Gateway Controller, you can find the code here.

The instructions to deploy the controller use the commit 918d6506483fb42710a227b4ecb35c9dca43ccc5 which is the content of the main branch as of April 14th.

Create the namespace where the controller will be deployed:

oc apply -f https://raw.githubusercontent.com/nginxinc/nginx-kubernetes-gateway/918d6506483fb42710a227b4ecb35c9dca43ccc5/deploy/manifests/namespace.yaml

Create the njs-modules, this is used by NGinx to implement the data plane:

NOTE: njs is a subset of the JavaScript language that allows extending nginx functionality.

curl -L https://raw.githubusercontent.com/nginxinc/nginx-kubernetes-gateway/918d6506483fb42710a227b4ecb35c9dca43ccc5/internal/nginx/modules/src/httpmatches.js -o /tmp/httpmatches.js
oc -n nginx-gateway create configmap njs-modules --from-file=/tmp/httpmatches.js
rm -f /tmp/httpmatches.js

Create the GatewayClass that will use the nginx controller as backend:

oc apply -f https://raw.githubusercontent.com/nginxinc/nginx-kubernetes-gateway/918d6506483fb42710a227b4ecb35c9dca43ccc5/deploy/manifests/gatewayclass.yaml

Finally, deploy the NGinx Gateway Controller:

INFO: Container nginx-gateway is using ghcr.io/nginxinc/nginx-kubernetes-gateway:edge which at the time of this writing points to ghcr.io/nginxinc/nginx-kubernetes-gateway@sha256:9613836a69fdd527faa3635756959b9884d1017d6f193b82b83fe481d74d235e.
```
oc apply -f https://raw.githubusercontent.com/nginxinc/nginx-kubernetes-gateway/918d6506483fb42710a227b4ecb35c9dca43ccc5/deploy/manifests/nginx-gateway.yaml
```
Again, the pod won't start. That's caused by the busybox init container that requires running as UID 0. Let's fix it:
```
oc -n nginx-gateway adm policy add-scc-to-user anyuid -z nginx-gateway
```

After a few moments, the controller pod will be running:

oc -n nginx-gateway get pods

NAME                             READY   STATUS    RESTARTS   AGE
nginx-gateway-6ddb979dbd-mzffs   2/2     Running   0          2m59s

We need to expose the gateway controller, we will be using a LoadBalancer service for that. In our cluster we are using MetalLB.

INFO: You can read how to deploy MetalLB on the official docs.
```
oc apply -f https://raw.githubusercontent.com/nginxinc/nginx-kubernetes-gateway/918d6506483fb42710a227b4ecb35c9dca43ccc5/deploy/manifests/service/loadbalancer.yaml
```

We should have a Service with an external IP set:

oc -n nginx-gateway get svc

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
nginx-gateway   LoadBalancer   172.30.69.152   10.19.3.55    80:32156/TCP,443:31275/TCP   15s

In order to be able to expose our applications we need proper DNS resolution, we will be configuring a wildcard record in our DNS server. In this case we're creating a record *.apps.gateway-api.test.lab that points to the external IP 10.19.3.55.

NOTE: This wildcard will be used for exposing our apps.
```
dig +short anything.apps.gateway-api.test.lab
```
```
10.19.3.55
```

At this point we are ready to start exposing our applications with the NGinx Gateway.

Exposing applications with the NGinx Gateway

In this section we will go over three scenarios:

Expose a simple application via HTTP.
Blue/Green scenario with weights.
Expose a simple application via HTTPs.

Exposing a simple app via HTTP

Deploy the simple app:

cat <<EOF | oc apply -f -
---
apiVersion: v1
kind: Namespace
metadata:
  name: reverse-words
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reverse-words-blue
  namespace: reverse-words
  labels:
    app: reverse-words-blue
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reverse-words-blue
  template:
    metadata:
      labels:
        app: reverse-words-blue
    spec:
      containers:
      - name: reverse-words
        image: quay.io/mavazque/reversewords:0.27
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: RELEASE
          value: "Blue"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 2
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          timeoutSeconds: 2
          periodSeconds: 15
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: reverse-words-blue
  name: reverse-words-blue
  namespace: reverse-words
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: http
    name: http
  selector:
    app: reverse-words-blue
  type: ClusterIP
EOF

With the app running we will create a Gateway resource pointing to the NGinx Gateway Class created earlier. On top of that, it will only listen for HTTP connections on port 80.

NOTE: When creating HTTPRoutes in the reverse-words namespace we will reference this Gateway and the listener to be used to expose our application. We can have multiple Gateways per namespace, for example: One Gateway that exposes services using NGinx, another one that uses HAProxy, and a third one that uses F5 Big IP.
```
cat <<EOF | oc apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: reverse-words
  labels:
    domain: k8s-gateway.nginx.org
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP
EOF
```

Finally, let's create the HTTPRoute:

INFO: This HTTPRoute uses the Gateway named gateway in this namespace, and will use the listener http to publish the route. The Service being exposed is the reverse-words-blue service on port 8080.

cat <<EOF | oc apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: reversewords
  namespace: reverse-words
spec:
  parentRefs:
  - name: gateway
    sectionName: http
  hostnames:
  - reverse-words.apps.gateway-api.test.lab
  rules:
  - backendRefs:
    - name: reverse-words-blue
      port: 8080
EOF

We can now access our application:

curl http://reverse-words.apps.gateway-api.test.lab

Reverse Words Release: Blue. App version: v0.0.27

Blue/Green Scenario

The goal in this scenario is having two versions of the same service and gradually routing traffic to the newer version.

IMPORTANT: This scenario relies on the application deployed on the previous section.

Deploy the new version of our application (Green)

cat <<EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reverse-words-green
  namespace: reverse-words
  labels:
    app: reverse-words-green
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reverse-words-green
  template:
    metadata:
      labels:
        app: reverse-words-green
    spec:
      containers:
      - name: reverse-words
        image: quay.io/mavazque/reversewords:0.28
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: RELEASE
          value: "Green"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 2
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          timeoutSeconds: 2
          periodSeconds: 15
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: reverse-words-green
  name: reverse-words-green
  namespace: reverse-words
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: http
    name: http
  selector:
    app: reverse-words-green
  type: ClusterIP
EOF

Delete the old HTTPRoute and create a new one with weights:

oc -n reverse-words delete httproute reversewords

NOTE: Below route will send most of the request to the old service (Blue) and a few ones to the new one (Green).

cat <<EOF | oc apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: reversewords
  namespace: reverse-words
spec:
  parentRefs:
  - name: gateway
    sectionName: http
  hostnames:
  - reverse-words.apps.gateway-api.test.lab
  rules:
  - backendRefs:
    - name: reverse-words-blue
      port: 8080
      weight: 90
    - name: reverse-words-green
      port: 8080
      weight: 10
EOF

If we try to access our application this is what we will get:

for i in $(seq 1 10);do curl http://reverse-words.apps.gateway-api.test.lab; done

Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27

We can update the weights, so traffic gets distributed evenly:

oc -n reverse-words patch httproute reversewords -p '{"spec":{"rules":[{"backendRefs":[{"group":"","kind":"Service","name":"reverse-words-blue","port":8080,"weight":50},{"group":"","kind":"Service","name":"reverse-words-green","port":8080,"weight":50}],"matches":[{"path":{"type":"PathPrefix","value":"/"}}]}]}}' --type merge

And check the impact:

for i in $(seq 1 10);do curl http://reverse-words.apps.gateway-api.test.lab; done

Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Blue. App version: v0.0.27
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28

We could remove the old service (Blue) from the balancing:

oc -n reverse-words patch httproute reversewords -p '{"spec":{"rules":[{"backendRefs":[{"group":"","kind":"Service","name":"reverse-words-blue","port":8080,"weight":0},{"group":"","kind":"Service","name":"reverse-words-green","port":8080,"weight":100}],"matches":[{"path":{"type":"PathPrefix","value":"/"}}]}]}}' --type merge

for i in $(seq 1 10);do curl http://reverse-words.apps.gateway-api.test.lab; done

Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28
Reverse Words Release: Green. App version: v0.0.28

Exposing a simple app via HTTPs

We need a TLS cert, let's generate a self-signed one:

openssl req -new -newkey rsa:2048 -sha256 -days 3650 -nodes -x509 -extensions v3_ca -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/C=ES/ST=Valencia/L=Valencia/O=IT/OU=IT/CN=*.apps.gateway-api.test.lab" -addext "subjectAltName = DNS:*.apps.gateway-api.test.lab"

The certificate needs to be stored in a secret:

oc -n reverse-words create secret tls reversewords-gateway-tls --cert=/tmp/tls.crt --key=/tmp/tls.key

The Gateway needs to be updated, a new listener for https protocol will be created on port 443. This listener will be our tls terminator and will use the certificate we just created:

cat <<EOF | oc apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: reverse-words
  labels:
    domain: k8s-gateway.nginx.org
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: reversewords-gateway-tls
        namespace: reverse-words
EOF

Let's remove the old HTTPRoute and create a new one exposing the app via HTTPs:

oc -n reverse-words delete httproute reversewords

cat <<EOF | oc apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: reversewords
  namespace: reverse-words
spec:
  parentRefs:
  - name: gateway
    sectionName: https
  hostnames:
  - reverse-words.apps.gateway-api.test.lab
  rules:
  - backendRefs:
    - name: reverse-words-green
      port: 8080
EOF

We can now access our app via https:

curl -k https://reverse-words.apps.gateway-api.test.lab

Reverse Words Release: Green. App version: v0.0.28

If we try to access the app via http:

curl http://reverse-words.apps.gateway-api.test.lab

INFO: Our HTTPRoute does not expose the app via http, this is expected.

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.23.4</center>
</body>
</html>

Usually you want to redirect users hitting the HTTP endpoint to the HTTPs endpoint, lets configure that:

NOTE: We need to create a new HTTPRoute, but if you take a closer look you will not see any backends being referenced, instead we are applying a RequestRedirect filter to tell the client to go look the HTTPs endpoint. More on filters here.

cat <<EOF | oc apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: reversewords-tls-redirect
  namespace: reverse-words
spec:
  parentRefs:
  - name: gateway
    sectionName: http
  hostnames:
  - reverse-words.apps.gateway-api.test.lab
  rules:
  - filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        port: 443
EOF

If we access the HTTP endpoint, we're told to go somewhere else:

curl http://reverse-words.apps.gateway-api.test.lab

<html>
<head><title>302 Found</title></head>
<body>
<center><h1>302 Found</h1></center>
<hr><center>nginx/1.23.4</center>
</body>
</html>

If we tell curl to follow redirects (-L):

curl -Lk http://reverse-words.apps.gateway-api.test.lab

Reverse Words Release: Green. App version: v0.0.28

Useful Resources

https://gateway-api.sigs.k8s.io/
https://gateway-api.sigs.k8s.io/v1alpha2/guides/
https://gateway-api.sigs.k8s.io/api-types/httproute/
https://gateway-api.sigs.k8s.io/api-types/referencegrant/
https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/docs/installation.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Gateway API on OpenShift 4.13

Introduction to Gateway API

Deploying the Gateway API

Deploying the NGinx Gateway Controller

Exposing applications with the NGinx Gateway

Exposing a simple app via HTTP

Blue/Green Scenario

Exposing a simple app via HTTPs

Useful Resources

Files

README.md

Latest commit

History

README.md

File metadata and controls

Gateway API on OpenShift 4.13

Introduction to Gateway API

Deploying the Gateway API

Deploying the NGinx Gateway Controller

Exposing applications with the NGinx Gateway

Exposing a simple app via HTTP

Blue/Green Scenario

Exposing a simple app via HTTPs

Useful Resources