Use Github actions/cache to cache the go deps in CI

[viveksinghggits]

We should consider caching the go mod dependencies while running tests and go mod download in CI.

[github-actions]

Thanks for opening this issue 👍. The team will review it shortly.

If this is a bug report, make sure to include clear instructions how on to reproduce the problem with minimal reproducible examples, where possible. If this is a security report, please review our security policy as outlined in SECURITY.md.

If you haven't already, please take a moment to review our project's Code of Conduct document.

[viveksinghggits]

By looking at this figure

I thought that go mod is taking too much time (14m) and maybe we should try to cache the deps. But after further looking into that it looks like the go mod (go mod tidy) command is barely taking a minute. But the actual time that is taken in this step is while uploading the artifacts that we do here.

Below image shows the time go mod and upload artifact takes

[viveksinghggits]

Most of the time is being taken by either upload/download-artifact step or while pulling our build image.

- uses: actions/upload-artifact@v3
      with:
        name: src
        path: ./src.tar.gz

We should see if there is a way to run the CI without using uploading and downloading the artifacts. Maybe explore https://github.com/actions/setup-go that support caching as well.

[hairyhum]

The upload artifact was addressed in #2534 without actually using the cache.
We can't cache the build image, which contributes about 40-60 seconds to the pipeline, and switching away from using the build image could allow to use cache, but that needs some changes in the makefiles and build scripts.