Prometheus is used to monitor CPU and memory utilization in PSM benchmarks.
Prometheus Operator is used to install Prometheus. A prometheus operator is a custom controller that manages the Prometheus instance.
The usual way to install Prometheus Operator is through the Makefile. See the section on Deploying Prometheus in the deployment guide.
There are two configuration files related to Prometheus Operator,
crds.yaml and
install-prometheus-operator.yaml. These
files are taken from
bundle.yaml,
in the Prometheus Operator repository, with the fields namespace and
nodeSelector replaced where applicable.
The files install-prometheus.yaml and servicemonitors.yaml are taken from istio/tools, with configuration related to Istio left out.
The scrape interval (scrape_interval) determines how often Prometheus scrapes
targets. The default value for this interval is
1m.
We change this interval in our configuration to 1s, since our tests typically
last only 30s. See grpc#325 for details.
We add 20s delay before collecting data from Prometheus, in the tests that use Prometheus. This delay allows data to become available (cAdvisor housekeeping interval) and pulled by Prometheus (Prometheus scrape interval). See grpc#330 for details.