Skip to content

Latest commit

 

History

History
96 lines (59 loc) · 3.56 KB

README.md

File metadata and controls

96 lines (59 loc) · 3.56 KB
Raw

gcp-gpu-metrics

CI

Tiny Go binary that aims to export Nvidia GPU metrics to GCP monitoring, based on nvidia-smi.

Requirements ⚓

  • Your machine must be a GCE (Google Compute Engine) instance.
  • The Cloud API access scopes of the instance or Service Account must have the Monitoring Metric Writer permission.
  • You need the nvidia-smi binary installed on your GCE instance.

Protip: You can use a machine learning image provided by GCP as default image.

Install ⏬

If you're root, you can install the latest binary version using the following script:

$ bash < <(curl -sSL https://raw.githubusercontent.com/instadeepai/gcp-gpu-metrics/master/install-latest.sh)

Or, you can download a release/binary from this page and install it manually.

Usage 💻

gcp-gpu-metrics is an UNIX compliant and very simple CLI, you just have to use it as usual:

$ gcp-gpu-metrics

Available flags:

  • --service-account-path string | GCP service account path. (default "")
  • --metrics-interval uint | Fetch metrics interval in seconds. (default 10)
  • --enable-nvidiasmi-pm | Enable persistence mod for nvidia-smi. (default false)
  • --version | Display current version/release and commit hash.

Available env variables:

  • GGM_SERVICE_ACCOUNT_PATH=./service-account.json linked to --service-account-path flag.
  • GGM_METRICS_INTERVAL=10 linked to --metrics-interval flag.
  • GGM_ENABLE_NVIDIASMI_PM=true linked to --enable-nvidiasmi-pm flag.

Priority order is binary flag ➡️ env var ➡️ default value.

Nvidia-smi persistence mod is very useful, the option permits to run nvidia-smi as a daemon in background to prevent 100% of GPU load at each request. Enabling this option requires root.

About logs, they're all located under syslog.

Metrics 📈

There are 6 differents metrics fetched, this number will grow in the future.

  • temperature.gpu as custom.googleapis.com/gpu/temperature_gpu | Core GPU temperature. in degrees C.
  • utilization.gpu as custom.googleapis.com/gpu/utilization_gpu | Percent of time over the past sample period during which one or more kernels were executed on the GPU.
  • utilization.memory as custom.googleapis.com/gpu/utilization_memory | Percent of time over the past sample period during which global (device) memory was being read or written.
  • memory.total as custom.googleapis.com/gpu/memory_total | Total installed GPU memory.
  • memory.free as custom.googleapis.com/gpu/memory_free | Total GPU free memory.
  • memory.used as custom.googleapis.com/gpu/memory_used | Total memory allocated by active contexts.

It creates an amount of time series equal to GPU amount with the label gpu_id + a GPU average.

Here is a list of other labels:

  • bus_id | Identify your GPUs at hardware level.
  • instance_name | Identify instance name.

Example for 2 GPUs with temperature.gpu query, it will create:

gpu_id bus_id instance_name Value
gpu_0 00000000:00:04.0 gcp-gpu-instance 50
gpu_1 00000000:00:05.0 gcp-gpu-instance 60
gpu_avg null gcp-gpu-instance 55

Compile gcp-gpu-metrics ⚙

There is a re command in the Makefile.

$ make re

gcp-gpu-metrics has been tested with go1.15 and use go modules.

Report an issue 📢

Feel free to open a GitHub issue on this project 🚀

License 🔑

See LICENSE.