---
title: Charmhub | Deploy Hardware Observer using Charmhub - The Open Operator Collection
description: Deploy the latest version of Hardware Observer on any cloud.
url: https://charmhub.io/hardware-observer/docs/metrics-and-alerts-gpu
---

# Hardware Observer

[Canonical BootStack Charmers](https://charmhub.io/publisher/bootstack-charmers "View all packages from Canonical BootStack Charmers")

* [Canonical BootStack Charmers](https://charmhub.io/publisher/bootstack-charmers "View all packages from Canonical BootStack Charmers")

Platform:

26.04

24.04

22.04

20.04

18.04

stable 894

```
juju deploy hardware-observer
```

[Learn to deploy on juju >](https://juju.is/docs/juju/manage-applications)

---

#### Relevant links

* [Homepage](https://github.com/canonical/hardware-observer-operator)

---

#### Contacts

* [Submit a bug](https://github.com/canonical/hardware-observer-operator/issues)

---

Share your thoughts on this charm with the community on discourse.

[Join the discussion](https://discourse.charmhub.io/)

## [Metrics](https://charmhub.io/hardware-observer/docs/metrics-and-alerts-gpu#p-34154-metrics)

The details of the GPU metrics exposed by Hardware Observer using [dcgm-exporter](https://github.com/NVIDIA/dcgm-exporter) and [node-exporter](https://github.com/prometheus/node_exporter) are as follows:

| Metric Name | Description | Labels |
| --- | --- | --- |
| DCGM\_FI\_DEV\_GPU\_TEMP | GPU temperature (in C) | DCGM\_FI\_DEV\_BAR1\_TOTAL, DCGM\_FI\_DEV\_BRAND, DCGM\_FI\_DEV\_CC\_MODE, DCGM\_FI\_DEV\_COMPUTE\_MODE, DCGM\_FI\_DEV\_COUNT, DCGM\_FI\_DEV\_CUDA\_COMPUTE\_CAPABILITY, DCGM\_FI\_DEV\_ECC\_CURRENT, DCGM\_FI\_DEV\_ECC\_INFOROM\_VER, DCGM\_FI\_DEV\_ENFORCED\_POWER\_LIMIT, DCGM\_FI\_DEV\_FB\_TOTAL, DCGM\_FI\_DEV\_GPU\_MAX\_OP\_TEMP, DCGM\_FI\_DEV\_INFOROM\_IMAGE\_VER, DCGM\_FI\_DEV\_MAX\_MEM\_CLOCK, DCGM\_FI\_DEV\_MAX\_SM\_CLOCK, DCGM\_FI\_DEV\_MINOR\_NUMBER, DCGM\_FI\_DEV\_NAME, DCGM\_FI\_DEV\_OEM\_INFOROM\_VER, DCGM\_FI\_DEV\_PERSISTENCE\_MODE, DCGM\_FI\_DEV\_POWER\_MGMT\_LIMIT, DCGM\_FI\_DEV\_POWER\_MGMT\_LIMIT\_MAX, DCGM\_FI\_DEV\_POWER\_MGMT\_LIMIT\_MIN, DCGM\_FI\_DEV\_SERIAL, DCGM\_FI\_DEV\_SHUTDOWN\_TEMP, DCGM\_FI\_DEV\_SLOWDOWN\_TEMP, DCGM\_FI\_DEV\_VBIOS\_VERSION, DCGM\_FI\_DEV\_VIRTUAL\_MODE, DCGM\_FI\_DRIVER\_VERSION, DCGM\_FI\_NVML\_VERSION, Hostname, UUID, device, gpu, modelName, pci\_bus\_id |
| DCGM\_FI\_DEV\_POWER\_USAGE | Power draw (in W) | Same as DCGM\_FI\_DEV\_GPU\_TEMP |
| DCGM\_FI\_DEV\_GPU\_UTIL | GPU utilization (in %) | Same as DCGM\_FI\_DEV\_GPU\_TEMP |
| DCGM\_FI\_DEV\_FAN\_SPEED | Fan speed (in 0-100%) | Same as DCGM\_FI\_DEV\_GPU\_TEMP |
| DCGM\_FI\_DEV\_MEM\_CLOCK | Memory clock frequency (in MHz) | Same as DCGM\_FI\_DEV\_GPU\_TEMP |
| DCGM\_FI\_DEV\_MEM\_COPY\_UTIL | Memory utilization (in %) | Same as DCGM\_FI\_DEV\_GPU\_TEMP |
| DCGM\_FI\_DEV\_CLOCK\_THROTTLE\_REASONS | Throttling reasons bitmask | Same as DCGM\_FI\_DEV\_GPU\_TEMP |
| node\_hwmon\_chip\_names | Annotation metric for human-readable chip names | chip, chip\_name |
| node\_hwmon\_temp\_celsius | Hardware monitor for temperature (input) | chip, sensor |
| node\_hwmon\_power\_average\_watt | Hardware monitor for power usage in watts (average) | chip, sensor |
| node\_hwmon\_freq\_freq\_mhz | Hardware monitor for GPU frequency in MHz | sensor, chip |
| node\_hwmon\_fan\_rpm | Hardware monitor for fan revolutions per minute (input) | sensor, chip |
| node\_hwmon\_fan\_max\_rpm | Hardware monitor for fan revolutions per minute (max) | sensor, chip |
| node\_drm\_card\_info | Card information | card, chip, memory\_vendor, power\_performance\_level, unique\_id |
| node\_drm\_gpu\_busy\_percent | How busy the GPU is as a percentage | card, chip |
| node\_drm\_memory\_vram\_used\_bytes | The used amount of VRAM in bytes | card, chip |
| node\_drm\_memory\_vram\_size\_bytes | The size of VRAM in bytes | card, chip |

**NOTE:** This is the subset of metrics used for alerts and the GPU dashboard. Please see [this file](https://github.com/canonical/hardware-observer-operator/blob/main/src/gpu_metrics/dcgm_metrics.csv) to learn about other DCGM metrics.

**NOTE:** metrics prefixed with `node_` are provided by the [`node_exporter`](https://github.com/prometheus/node_exporter) `DRM` and `HWmon` collectors for any GPU using open-source drivers. [`node_exporter`](https://github.com/prometheus/node_exporter) is deployed by the [grafana-agent charm](https://charmhub.io/grafana-agent), not hardware-observer. The metrics are reported here for convenience.

## [Alerts](https://charmhub.io/hardware-observer/docs/metrics-and-alerts-gpu#p-34154-alerts)

The details of the alerts that Hardware Observer provides for NVIDIA GPUs are as follows:

| Alert Rule Name | Description | Severity |
| --- | --- | --- |
| GPUPowerBrakeThrottle | NVIDIA GPU Hardware Power Brake Slowdown throttling detected | Warning |
| GPUThermalHWThrottle | NVIDIA GPU Hardware Thermal throttling detected | Warning |
| GPUThermalSWThrottle | NVIDIA GPU Software Thermal throttling detected | Warning |
| GPUSyncBoostThrottle | NVIDIA GPU Sync Boost throttling detected | Warning |
| GPUSlowdownThrottle | GPU Hardware Slowdown throttling detected | Warning |
| GPUPowerThrottle | GPU Software Power throttling detected | Warning |

For more details, please see [NVIDIA Clocks Throttle reasons](https://docs.nvidia.com/deploy/nvml-api/group__nvmlClocksThrottleReasons.html).

Throttling detection is currently only available for NVIDIA GPUs.

---