[CI] Divide CI into gates

[dkijania]

Explain your changes:
Current Buildkite CI does not distinguish on dhall level if we are running nightly or PullRequest tests. However, from deep dive into logic, there is some work done, which can be extended to control which job should run on nightly. By providing export BUILDKITE_PIPELINE_MODE=Stable we can signal Ci that we want to run nightly jobs so Monorepo triage is disabled and all known jobs will be run.

However, it's not satisfactory, as we also want to exclude some jobs from PullRequest stage and run only on Stable. Therefore I introduced new attribute of JobSpec struct : mode which by default is set to PullRequest, but in some occasion can be override to Stable. Then code in monorepo traige job analyze it and prevents this jobs from running.

Stable (a.k.a Nightly ) logic is unchanged.

Then i went a bit further into CI configuration and was experimenting with splitting pipeline into 2-3 pipeline gates.

• 1 Gate only runs fast jobs
• 2 Gate run remaining jobs only if 1 Gate is green.
• 3 Gate only run if 2 Gate is green. It contains only closing jobs like gathering coverage

As a result i come up with tagging mechanism which can be fluently used to choose which subset of jobs should be run. Imagine situation in which we want to run only fast and lints jobs. In my PR i already tag jobs which duration is less than 10 minutes (with tag: Fast) and marked all linting jobs with tag: Lint. Then, I introduced Filter module which creates collection of tags (for user convenience, it's better to use single value: PIPELINE_FILTER: FastAndLints rather than PIPELINE_TAGS: [Fast, Lint]). Now user can run fast and lint jobs by below setup:

steps:
  - commands:
      - "dhall-to-yaml --quoted <<< './buildkite/src/Prepare.dhall' | buildkite-agent pipeline upload"
    label: ":pipeline: Fast"
    agents:
       size: "generic"
    plugins:
      "docker#v3.5.0":
        environment:
          - BUILDKITE_AGENT_ACCESS_TOKEN
          - "BUILDKITE_PIPELINE_MODE=Stable"
          - "BUILDKITE_PIPELINE_FILTER=FastAndLints"
        image: codaprotocol/ci-toolchain-base:v3
        mount-buildkite-agent: false
        propagate-environment: true

But this is not over. We can define multiple stages using filters and wait command between them. Imagine situation in which we would like to have 3 group of jobs for our pipeline:

• Fast (Lints and fast jobs)
• Long (Unit tests/Building dockers etc)
• TearDown (Coverage gathering)

We can reach above goal by setup like below (i omitted not important parameters):

steps:
  - commands:
      - "dhall-to-yaml ..."
        environment:
           - "BUILDKITE_PIPELINE_MODE=Stable"
          - "BUILDKITE_PIPELINE_FILTER=Fast"
  - wait
  - commands:
      - "dhall-to-yaml ..."
          - "BUILDKITE_PIPELINE_MODE=Stable"
          - "BUILDKITE_PIPELINE_FILTER=LongOnly"
  - wait
  - commands:
      - "dhall-to-yaml ..."
          - "BUILDKITE_PIPELINE_MODE=Stable"
          - "BUILDKITE_PIPELINE_FILTER=TearDownOnly"

The above setup can be used for PR verifications. It will first run fast jobs and lints. If any of them is red, fail the build and won't run long jobs, preserving time and cost.

Explain how you tested your changes:

I tested it on fuzzy zkapp tests job example, which is using $NIGHTLY parameter which on script level control if jobs is empty but evergreen on PullRequest or have actual test run on Stable. I removed that logic from script and put Stable as job mode in dhall. Then verified job is not run.

Checklist:

• Dependency versions are unchanged
  • Notify Velocity team if dependencies must change in CI
• Modified the current draft of release notes with details on what is completed or incomplete within this project
• Document code purpose, how to use it
  • Mention expected invariants, implicit constraints
• Tests were added for the new behavior
  • Document test purpose, significance of failures
  • Test names should reflect their purpose
• All tests pass (CI will check this if you didn't)
• Serialized types are in stable-versioned modules
• Does this close issues? List them

• Closes Improved buildkite nightly/stable pipeline configuration #8821

[dkijania]

Effect of my changes can be observed on separate pipeline:

https://buildkite.com/o-1-labs-2/tmp-teardownjob/builds/89

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[stevenplatt]

Looks good. I cannot think of any conflicts of concerns.

One item worth noting is that the Tag and Mode have adjacent functionality to controls available in webhooks settings within the repo, but I don't think they overlap in current form.

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[deepthiskumar]

Approving since the changes are only in the buildkite/CI related files

[deepthiskumar]

!approved-for-mainnet