There should be a way to revisit kanister phase's pods

[viveksinghggits]

In the cases where we run a new pod as part of kanister phase, for example kubetask, and the pod was terminated we don't have a way to see the logs of that pod.
In the cases where the failure is in the kubetask pod, debugging becomes a problem because we don't have a way to access the things that have been logged into the kubetask pod.

[ihcsim]

I was able to use the action set podOverride field to change the restartPolicy of the pod, in order to look up the pod logs. Will this be an acceptable solution?

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: backup
  namespace: kanister
actions:
  debug:
    phases:
    - func: KubeTask
      name: debug-task
      args:
        namespace: "{{ .Deployment.Namespace }}"
        image: busybox
        command:
        - sh
        - "-c"
        - |
          echo "test debug task"
          exit 1
---
apiVersion: cr.kanister.io/v1alpha1
kind: ActionSet
metadata:
  generateName: time-logger-debug-
  namespace: kanister
spec:
  actions:
  - name: debug
    blueprint: backup
    object:
      kind: Deployment
      name: time-logger
      namespace: default
    podOverride:
      restartPolicy: Always

$ k get po
NAME                           READY   STATUS             RESTARTS      AGE
kanister-job-vvc6c             0/1     CrashLoopBackOff   5 (29s ago)   3m32s

[ihcsim]

Per discussion during community meeting, using restart to keep the pods around isn’t a good idea because some commands may not be idempotent. We should find ways to capture stdout, stderr outputs on-error before the failed pods are terminated. On clusters where Fluentbit is already running, logs might already be captured.

@pavannd1 has some thoughts on how to address this.

[ihcsim]

Per discussion with @pavannd1 and @viveksinghggits, first, we need a way to stream the command output back to the task/job pod asynchronously, and then decide on where to drain the output to, in order to retain them after task/job pod is terminated. One approach is to append the output to the actionset that triggers the operation.

Ongoing PRs:

• Stream output from KubeExec #1194
• Stream output from KubeExec #1037

[ihcsim]

I think I was able to use kando output to output the logs of the job pod before it was terminated. E.g.,

    phases:
    - name: describeSnapshots
      func: KubeTask
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        image: ghcr.io/kanisterio/postgres-kanister-tools:0.78.0
        command:
        - sh
        - -c
        - |
          logs=$(aws ec2 describe-snapshots --owner-ids self --filters Name=volume-id,Values={{ .ArtifactsIn.EBS.KeyValue.VolumeID }} --output=json 2>&1)
          kando output snapshotList "$logs"

That attached the log outputs to the actionset.

[ihcsim]

Here's another example on how to do it with mongodump:

    - func: KubeTask
      name: mongodump
      args:
        namespace: "{{ .Deployment.Namespace }}"
        image: ghcr.io/kanisterio/mongodb:0.79.0
        command:
          - bash
          - -c
          - |
            svc="{{ .Deployment.Name }}.{{ .Deployment.Namespace }}.svc.cluster.local"
            dbuser='{{ index .Secrets.dbcred.Data "database-user" | toString }}'
            dbpass='{{ index .Secrets.dbcred.Data "database-password" | toString }}'
            dump_cmd="mongodump --gzip --archive --host ${svc} -u ${dbuser} -p ${dbpass}"
            ${dump_cmd} 2>/tmp/out.log | kando location push --profile '{{ toJson .Profile }}' --path '/{{ .Deployment.Name }}/mongodump/{{ toDate "2006-01-02T15:04:05.999999999Z07:00" .Time  | date "2006-01-02T15-04-05" }}/mongodump.gz' -
            kando output logs "$(cat /tmp/out.log)"

The actionset status looks like this:

$ k -n kanister get actionset ${actionset} -oyaml -ojsonpath='{.status.actions[0].artifacts.logs}' | jq .   
{
  "keyValue": {
    "output": "2022-05-12T21:13:21.255+0000\twriting admin.system.users to archive on stdout\n2022-05-12T21:13:21.259+0000\tdone dumping admin.system.users (1 document)\n2022-05-12T21:13:21.260+0000\twriting admin.system.version to archive on stdout\n2022-05-12T21:13:21.262+0000\tdone dumping admin.system.version (2 documents)\n2022-05-12T21:13:21.262+0000\twriting pacman.userstats to archive on stdout\n2022-05-12T21:13:21.280+0000\tdone dumping pacman.userstats (4 documents)"
  }
}

[ihcsim]

Maybe all we need to do is just make sure out is also returned when if err != nil here. I also wonder how output.LogAndParse() knows when the stream's EOF is, assuming the container didn't die; looks like it just waits for ctx to terminate.