Dependent Task after Task with when condition are skipped

[Allda]

Expected Behavior

Hello, I am building a pipeline with a conditional series of tasks that use the same when condition, after that series of tasks there is one or more other tasks that don't have the condition.

Simplified task schema looks like this

A -> B -> C

where A and B use the condition given by the pipeline argument. At the same time, B uses the output of A as an input parameter.

I expect the C is triggered regardless of A or B but as shown bellow this is no the case and if A and B are skipped the C is also skipped.

Actual Behavior

Based on the pipeline argument A and B are skipped as expected, but C is also skipped which is unexpected. Skipped reasons are the following:
A - When Expressions evaluated to false
B - Results were missing < -- This task uses when and should be also skipped the same way as task A
C - Parent Tasks were skipped

Steps to Reproduce the Problem

1. Create the following task and pipeline

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: basic-task
spec:
  params:
    - default: Hello
      name: appName
      type: string
  results:
    - name: my-result
      type: string
  steps:
    - image: registry.redhat.io/ubi7/ubi-minimal
      name: ''
      resources: {}
      script: |

        echo $(inputs.params.appName)

        echo $(inputs.params.appName) > $(results.my-result.path)

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: when
spec:
  params:
    - default: foo
      name: param1
      type: string
  tasks:
    - name: basic-task
      params:
        - name: appName
          value: Hello
      taskRef:
        kind: Task
        name: basic-task
      when:
        - input: $(params.param1)
          operator: in
          values:
            - bar
    - name: basic-task-2
      params:
        - name: appName
          value: $(tasks.basic-task.results.my-result)
      runAfter:
        - basic-task
      taskRef:
        kind: Task
        name: basic-task
      when:
        - input: $(params.param1)
          operator: in
          values:
            - bar
    - name: basic-task-3
      params:
        - name: appName
          value: Hello
      runAfter:
        - basic-task-2
      taskRef:
        kind: Task
        name: basic-task

3. Execute pipeline with default params

Additional Info

Client Version: 4.14.6
Kustomize Version: v5.0.1
Server Version: 4.13.31
Kubernetes Version: v1.26.13+77e61a2

• Tekton Pipeline version:

Client version: 0.33.0
Chains version: v0.16.1
Pipeline version: v0.47.6
Triggers version: v0.24.2
Operator version: v0.67.3

[l-qing]

I once encountered a similar problem, and I suspect it is related to the status "Results were missing" in Phase B.
After all, after skipping Phase A, Phase B still made a judgment on whether to execute.

[l-qing]

I'm not sure if this behavior is by design or a bug.

Relevant code:

pipeline/pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go

Lines 306 to 319 in 123f4a2

	func (t *ResolvedPipelineTask) skip(facts *PipelineRunFacts) TaskSkipStatus {
	var skippingReason v1.SkippingReason
	switch {
	case facts.isFinalTask(t.PipelineTask.Name) || t.isScheduled():
	skippingReason = v1.None
	case facts.IsStopping():
	skippingReason = v1.StoppingSkip
	case facts.IsGracefullyCancelled():
	skippingReason = v1.GracefullyCancelledSkip
	case facts.IsGracefullyStopped():
	skippingReason = v1.GracefullyStoppedSkip
	case t.skipBecauseParentTaskWasSkipped(facts):
	skippingReason = v1.ParentTasksSkip

pipeline/pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go

Lines 367 to 388 in 123f4a2

	// skipBecauseParentTaskWasSkipped loops through the parent tasks and checks if the parent task skipped:
	//
	// if yes, is it because of when expressions?
	// if yes, it ignores this parent skip and continue evaluating other parent tasks
	// if no, it returns true to skip the current task because this parent task was skipped
	// if no, it continues checking the other parent tasks
	func (t *ResolvedPipelineTask) skipBecauseParentTaskWasSkipped(facts *PipelineRunFacts) bool {
	stateMap := facts.State.ToMap()
	node := facts.TasksGraph.Nodes[t.PipelineTask.Name]
	for _, p := range node.Prev {
	parentTask := stateMap[p.Key]
	if parentSkipStatus := parentTask.Skip(facts); parentSkipStatus.IsSkipped {
	// if the parent task was skipped due to its `when` expressions,
	// then we should ignore that and continue evaluating if we should skip because of other parent tasks
	if parentSkipStatus.SkippingReason == v1.WhenExpressionsSkip {
	continue
	}
	return true
	}
	}
	return false
	}

pipeline/pkg/apis/pipeline/v1/pipelinerun_types.go

Lines 550 to 565 in 123f4a2

	// SkippingReason explains why a PipelineTask was skipped.
	type SkippingReason string
	const (
	// WhenExpressionsSkip means the task was skipped due to at least one of its when expressions evaluating to false
	WhenExpressionsSkip SkippingReason = "When Expressions evaluated to false"
	// ParentTasksSkip means the task was skipped because its parent was skipped
	ParentTasksSkip SkippingReason = "Parent Tasks were skipped"
	// StoppingSkip means the task was skipped because the pipeline run is stopping
	StoppingSkip SkippingReason = "PipelineRun was stopping"
	// GracefullyCancelledSkip means the task was skipped because the pipeline run has been gracefully cancelled
	GracefullyCancelledSkip SkippingReason = "PipelineRun was gracefully cancelled"
	// GracefullyStoppedSkip means the task was skipped because the pipeline run has been gracefully stopped
	GracefullyStoppedSkip SkippingReason = "PipelineRun was gracefully stopped"
	// MissingResultsSkip means the task was skipped because it's missing necessary results
	MissingResultsSkip SkippingReason = "Results were missing"

[AlanGreene]

I think this is the relevant section of the docs on when expressions:
https://tekton.dev/docs/pipelines/pipelines/#guarding-a-task-only

When when expressions evaluate to False, the Task will be skipped and:

• The ordering-dependent Tasks will be executed
• The resource-dependent Tasks (and their dependencies) will be skipped because of missing Results from the skipped parent Task.

Specifically the last point.

Also from the example just after that:

slack-msg will be skipped because it is missing the approver Result from manual-approval
dependents of slack-msg would have been skipped too if it had any of them

As far as I understand this matches the behaviour you're describing so is working as designed.

[Allda]

I have spent last several days reading the documentation about how the conditional Tekton task works so I understand why this is happening, but on the other hand if a task depends on the previous one (B depends on A) and B is is not executed due to when condition should this be also blocking and skipping a rest of the pipeline? The missing results from A used in B as params are not needed because the B is skipped.

Would you happen to have any suggestions on how to workaround this? The usecase for this is when a pipeline contains a series of task that depends on each other and are controlled by input parameter. In such a case with a current behavior, the pipeline always skips the rest of the pipeline even though there are no missing data for dependencies.

[Allda]

@AlanGreene @l-qing Do you know if this issue can be workaround somehow until we get any official conclusion of this issue?

[Allda]

I checked the code briefly and I think moving skipBecauseWhenExpressionsEvaluatedToFalse higher in the switch hierarchy and giving it higher priority would resolve this issue. Is there any reason why it is below skipBecauseResultReferencesAreMissing and skipBecauseParentTaskWasSkipped ?

pipeline/pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go

Lines 309 to 332 in 123f4a2

	switch {
	case facts.isFinalTask(t.PipelineTask.Name) || t.isScheduled():
	skippingReason = v1.None
	case facts.IsStopping():
	skippingReason = v1.StoppingSkip
	case facts.IsGracefullyCancelled():
	skippingReason = v1.GracefullyCancelledSkip
	case facts.IsGracefullyStopped():
	skippingReason = v1.GracefullyStoppedSkip
	case t.skipBecauseParentTaskWasSkipped(facts):
	skippingReason = v1.ParentTasksSkip
	case t.skipBecauseResultReferencesAreMissing(facts):
	skippingReason = v1.MissingResultsSkip
	case t.skipBecauseWhenExpressionsEvaluatedToFalse(facts):
	skippingReason = v1.WhenExpressionsSkip
	case t.skipBecausePipelineRunPipelineTimeoutReached(facts):
	skippingReason = v1.PipelineTimedOutSkip
	case t.skipBecausePipelineRunTasksTimeoutReached(facts):
	skippingReason = v1.TasksTimedOutSkip
	case t.skipBecauseEmptyArrayInMatrixParams():
	skippingReason = v1.EmptyArrayInMatrixParams
	default:
	skippingReason = v1.None
	}

[willejs-ec]

@Allda did you get anywhere with this? I think we are facing the same issue.

[Allda]

@willejs-ec Unfortunately I haven't. Due to this limitation, I wasn't able to use any when condition for tasks that have a common when condition and depend on each other. The only workaround I found was to put the condition inside of the task into the actual task code. This solution however makes the pipeline flow harder to read and forces a pipeline to create additional pods that are not doing any useful work and take resources and execution time.

[jameshwc]

We're facing the same issue.

[tommyb82]

We are facing same issue, BTW I think this duplicates stale (but still open) issue that I just commented on before seeing this one: #7029