flaky presence of tekton-generated labels in pipelineRun and taskRun 'Started' cloud events

[gerrnot]

Expected Behavior

As indicated in https://tekton.dev/docs/pipelines/events/, tekton emitted cloud events should always contain tekton-generated k8s labels, e.g.:

• tekton.dev/task (for TaskRuns)
• tekton.dev/pipeline (for PipelineRuns and TaskRuns)
• tekton.dev/pipelineTask (for TaskRuns that are part of a PipelineRun)

Note: Those are just examples, I would really wish to have all labels already present!

Actual Behavior

It seems to depend on some timing (random), whether the labels are already present or not (for all 'Started' cloud events)

Steps to Reproduce the Problem

1. Set kubens to tekton-pipelines (Or if using a different ns, also use a different ns below!)
2. Configure your cloud event sink to point to the EventListener Service that will be created in the next step (e.g. http://el-flaky-labels.tekton-pipelines:8080)
3. Run chmod +x test.sh; test.sh (after extracting the reproduce.zip)
4. The resulting count of PipelineRuns for the Pipeline "flaky-labels-triggered-manually" will be 10 as performed by the script.
   However, the resulting count of PipelineRuns for the Pipeline "flaky-labels-triggered-if-labels-present" will be something like 3, indicating the success rate of "label-presence". If the defect is fixed, it should be 10 as well!

Additional Info

• Kubernetes version:

  Output of kubectl version:

  WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
  Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", 
  GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v5.0.1
  Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.8", GitCommit:"395f0a2fdc940aeb9ab88849e8fa4321decbf6e1", 
  GitTreeState:"clean", BuildDate:"2023-08-24T00:43:07Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}
  

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

  Client version: 0.33.0
  Pipeline version: v0.51.0
  Triggers version: v0.25.0
  Dashboard version: v0.39.0