NVIDIA GPU Prometheus Exporter

This is a Prometheus Exporter for exporting NVIDIA GPU metrics. It uses the NVIDIA Go NVML bindings for NVIDIA Management Library (NVML) which is a C-based API that can be used for monitoring NVIDIA GPU devices. Unlike some other similar exporters, it does not call the nvidia-smi binary.

This Exporter is a fork of https://github.com/mindprince/nvidia_gpu_prometheus_exporter with the following main changes:

• added parsing of /run/gpustat/XX for jobid and uid of the user running on the GPU. Slurm scripts that take advantage of this are available on jobstats website.
• switched from Go bindings to NVIDIA Go NVML bindings
• added support for MIG instance autodetection and stats
• for Hopper and newer GPUs it now also provides GPM based metrics, like SM utilization, occupancy and a few others. To disable these metrics use -disable.gpm option.

Building

E.g.

go build

Running

The exporter requires the following:

• access to NVML library (libnvidia-ml.so.1).
• access to the GPU devices.

To make sure that the exporter can access the NVML libraries, either add them to the search path for shared libraries. Or set LD_LIBRARY_PATH to point to their location.

By default the metrics are exposed on localhost:9445/metrics. The port can be modified using the -web.listen-address flag.

Alerts

The collector allows us to alert on some of the bad states the GPU might be in. The following are ones we currently use:

  - alert: GPUError
    expr: nvidia_gpu_last_error > 0
    for: '0m'
    labels:
      severity: warning
      alerttype: hardware
    annotations:
      summary: 'GPU{{ $labels.minor_number }} error while collecting data on {{ reReplaceAll ":.*" "" $labels.instance }}'
      description: 'Host {{ reReplaceAll ":.*" "" $labels.instance }} had error {{ printf "%.f" $value }} while fetching GPU metrics from GPU{{ $labels.minor_number }}'
  - alert: GPUECCError
    expr: nvidia_gpu_ecc_errors{counter="volatile",error="uncorrected"} > 0
    for: '0m'
    labels:
      severity: warning
      alerttype: hardware
    annotations:
      summary: 'GPU{{ $labels.minor_number }} Uncorrected ECC error on {{ reReplaceAll ":.*" "" $labels.instance }}'
      description: 'Host {{ reReplaceAll ":.*" "" $labels.instance }} has {{ printf "%.f" $value }} uncorrected ECC errors on GPU{{ $labels.minor_number }}'
  - alert: GPUClockError
    expr: nvidia_gpu_clock_event_reason{reason=~".*slowdown.*"} > 31
    for: '5m'
    labels:
      severity: warning
      alerttype: hardware
    annotations:
      summary: 'GPU{{ $labels.minor_number }} Clock Problem {{ $labels.reason }} on {{ reReplaceAll ":.*" "" $labels.instance }}'
      description: 'Host {{ reReplaceAll ":.*" "" $labels.instance }} has {{ $labels.reason }} clock error on GPU{{ $labels.minor_number }}'