KNOX-2955 - Knox Readiness Awareness and Notification

[zeroflag]

What changes were proposed in this pull request?

Adding a new endpoint to the health service for checking knox gateway status (weather all topologies which were available at startup are deployed)

How was this patch tested?

Create discovery password:

$ bin/knoxcli.sh create-alias admin --value admin

Adding this to gateway-site.xml

<property>
    <name>gateway.cloudera.manager.descriptors.monitor.interval</name>
    <value>60000</value>
</property>

Dedicated topology for checking gateway status:

$ cat conf/topologies/health.xml

<topology>
    <gateway>
        <provider>
            <role>authentication</role>
            <name>Anonymous</name>
            <enabled>true</enabled>
        </provider>
    </gateway>
    <service>
        <role>HEALTH</role>
    </service>
</topology>

Hadoop xml resource file:

$ cat  conf/descriptors/test.hxr 
<configuration>
    <property>
        <name>custom-desc</name>
        <value>
            discoveryType=ClouderaManager#
            discoveryAddress=http://CM_HOST:7180/#
            discoveryUser=admin#
            discoveryPasswordAlias=admin#
            cluster=Cluster 1#
            providerConfigRef=default-providers#
            app:knoxauth:param1.name=param1.value#
            app:admin-ui#
            WEBHDFS#
            HIVE:url=http://localhost:456#
            HIVE:version=1.0#
            HIVE:httpclient.connectionTimeout=5m#
            HIVE:httpclient.socketTimeout=100m
        </value>
    </property>
</configuration>

Starting gw:

$ bin/gateway.sh restart

Log:

2023-09-06 13:06:14,053  INFO  knox.gateway (AbstractGatewayServices.java:start(60)) - Starting service: org.apache.knox.gateway.services.topology.impl.GatewayStatusChecker
2023-09-06 13:06:15,787  INFO  knox.gateway (GatewayStatusChecker.java:collectTopologies(73)) - Collected topologies for health check: [metadata, manager, knoxsso, sandbox, custom-desc, health, admin, tokenexchange, homepage]
2023-09-06 13:06:15,787  INFO  knox.gateway (GatewayStatusChecker.java:initTopologiesToCheck(66)) - Starting gateway status monitor service. Topologies to check: [metadata, manager, knoxsso, sandbox, custom-desc, health, admin, tokenexchange, homepage]
2023-09-06 13:08:05,770 e10b22f5-c655-450f-a807-1b8b623a9082 INFO  knox.gateway (GatewayStatusChecker.java:status(49)) - Checking gateway status. Deployed topologies: [metadata, manager, knoxsso, custom-desc, sandbox, admin, health, tokenexchange, homepage]. Waiting for: []
2023-09-06 13:12:26,420  INFO  knox.gateway (AbstractGatewayServices.java:stop(72)) - Stopping service: org.apache.knox.gateway.services.topology.impl.GatewayStatusChecker
2023-09-06 13:12:30,403  INFO  knox.gateway (AbstractGatewayServices.java:start(60)) - Starting service: org.apache.knox.gateway.services.topology.impl.GatewayStatusChecker
2023-09-06 13:12:31,770  INFO  knox.gateway (GatewayStatusChecker.java:collectTopologies(73)) - Collected topologies for health check: [metadata, manager, knoxsso, sandbox, custom-desc, health, admin, tokenexchange, homepage]
2023-09-06 13:12:31,770  INFO  knox.gateway (GatewayStatusChecker.java:initTopologiesToCheck(66)) - Starting gateway status monitor service. Topologies to check: [metadata, manager, knoxsso, sandbox, custom-desc, health, admin, tokenexchange, homepage]
2023-09-06 13:12:34,729 50d7c781-826d-4125-8e7e-98cd3a3790b8 INFO  knox.gateway (GatewayStatusChecker.java:status(49)) - Checking gateway status. Deployed topologies: [metadata, manager, knoxsso, sandbox, admin, health, homepage]. Waiting for: [custom-desc, tokenexchange]
...
2023-09-06 13:13:59,917 daa64c86-def1-4144-9932-ac8a26d8b8f7 INFO  knox.gateway (GatewayStatusChecker.java:status(49)) - Checking gateway status. Deployed topologies: [metadata, manager, knoxsso, sandbox, admin, health, tokenexchange, homepage]. Waiting for: [custom-desc]
...
2023-09-06 13:13:59,967 2c08b6d1-f8c0-4505-800f-18b98eec2649 INFO  knox.gateway (GatewayStatusChecker.java:status(49)) - Checking gateway status. Deployed topologies: [metadata, manager, knoxsso, sandbox, custom-desc, admin, health, tokenexchange, homepage]. Waiting for: []

Script for polling health endpoint

#!/bin/bash
while true
do
  echo -n "$(date) Accessing knox.. "
  response=$(curl -k -s "https://localhost:8443/gateway/health/v1/gateway-status")
  echo $response
  if [[ "$response" == "OK" ]] ; then
    curl -vu admin:admin-password -k "https://localhost:8443/gateway/custom-desc/webhdfs/v1?op=LISTSTATUS"
    exit 0
  fi
done

Script Output

...
Wed Sep  6 13:13:59 CEST 2023 Accessing knox.. PENDING
Wed Sep  6 13:13:59 CEST 2023 Accessing knox.. PENDING
Wed Sep  6 13:13:59 CEST 2023 Accessing knox.. OK

[pzampino]

LGTM

[smolnar82]

LGTM