Investigate GitHub "API rate-limit exceeded" errors

[teor2345]

Motivation

Every few days, multiple PRs will fail with GitHub "API rate-limit exceeded" errors. These errors seem to happen when there are multiple pushes to a PR within a short period of time.

This has happened 3 times in the past 5 days.

The rate-limits reset after an hour or so, but we need to re-queue all the mergify PRs.

Error Messages

This happens on a range of different GitHub actions:

Error: API rate limit exceeded for installation ID 1523599.

https://github.com/ZcashFoundation/zebra/actions/runs/3644366634/jobs/6153541893#step:3:13

This happens when an action logs the full response:

reviewdog: post failed for codespell: fail to parse diff: GET https://api.github.com/repos/ZcashFoundation/zebra/pulls/5784: 403 API rate limit exceeded for installation ID 1523599. [rate reset in 35m43s]

#5784 (comment)

Possible Causes

This has only happened since we got the free open source tier on GitHub.

We might be running too many jobs for every commit.

We might have introduced a workflow, action, or setting that uses up the rate-limit more quickly.

Possible Solutions

Start paying for enterprise GitHub again.
Ask for a rate-limit increase.

Turn jobs into steps (will this help?).
Combine steps into a multi-line script on a single step. (For example, building each crate individually.)

Investigate new workflows, actions, and settings for rate-limit issues. (How?)

Workarounds

One workaround is to push commits to GitHub in a single batch, rather than making many changes to the PR one at a time.

[mpguerra]

Start paying for enterprise GitHub again.

We were never on enterprise github, only the team plan. Now we're just getting the same plan for free as far as I can tell.

[teor2345]

I think we might be running into CI rate-limits:
https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits

Rather than GitHub app rate limits:
https://docs.github.com/en/developers/apps/building-github-apps/rate-limits-for-github-apps

I think the enterprise limit only applies to GitHub apps.

[gustavovalverde]

Updating this ticket as this took me a while and I haven't been able to find the root cause yet, but we can surely track this down to:

• https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/lint.yml
• https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/build-crates-individually.yml

As these are the actions which execute the most demanding queries on GitHub. And I'm leaning towards tj-actions/changed-files.

I'm not sure if there's some analytics around "Requests made to the GitHub API", I tried https://docs.github.com/en/rest/rate-limit with no avail. Maybe we could ask GitHub; not sure if an email or issue might be the best approach here.

[gustavovalverde]

Changing this ticket estimate

[arya2]

Pushing commits to GitHub in a single batch seems like an easy workaround if increasing the limit is non-trivial, I only saw this happen when there was a rapid rate of new commits.

It might help to run a couple short jobs first and have the remainder of CI only start after those are finished and not cancelled but that could increase the wait-time.

Looks like the limit applies to self-hosted runners as well, but combining steps that way could also help.