Add benchmarks & stress tests

[tallclair]

I think this project is currently lacking benchmarks and stress tests. These tests can be useful for identifying major regressions and issues. Some ideas of tests I'd like to see:

• Single high-throughput connection
  • Agent → Proxy server
  • Agent ← Proxy server
  • Agent ⇆ Proxy server (bidirectional)
• Multiple high-throughput connections
• High proxied connection churn (normal behavior, e.g. multiple webhooks)
• High churn of agent connections (simulating flaky network or agent restarts)

Each case above should collect metrics on CPU, memory, and # of goroutines. Metrics should be measured before, during & after the tests, with quantiles.

• High-throughput cases should also benchmark throughput rates
• Connection churn cases should also look at latency

[justinsb]

This is a great list. If I was adding one scenario, I would add the "unexpected close" scenario. I think this happens in the real world with a webhook if kube-apiserver closes the connection after a timeout. Also (in the other direction) if the webhook crashes I think that should also yield a close at an unexpected place in the message flow.

[tallclair]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tallclair]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tallclair]

/remove-lifecycle stale
/triage accepted

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[tallclair]

/triage accepted

We have much better test coverage, but I think we're still lacking the ability to detect a performance regression.

Some basic benchmarks were added in https://github.com/kubernetes-sigs/apiserver-network-proxy/blob/master/tests/benchmarks_test.go, but we're still missing benchmarks for concurrent requests & throughput, as well as the tooling to run these regularly to detect regressions.