documentation/docs/admin-guide/configuration/monitoring/alloy.md

# Grafana Alloy for environment admins

This guide explains how to enable Grafana Alloy through Hiera and which
settings environment admins can change.

Alloy sends telemetry to the central production observability platform
[grafana.psi.ch](https://grafana.psi.ch):

- host metrics to Mimir;
- the complete local system journal to Loki;
- optional traces to Tempo;
- optional application and Docker telemetry.

Redis, Blackbox, and SNMP telemetry are still work in progress. Their
configuration is documented below for testing and development, but is not yet
available in the production Puppet environment.

Host metrics and journal logs are always enabled when Alloy is enabled. Obtain
approval from the environment owner before enrolling a host because journal
entries can contain sensitive data.

The production backend destinations, package version, service settings, and
basic Alloy configuration are managed centrally. Development hosts can use the
test observability platform by following
[Add Development Hosts to grafana-test](grafana_test.md).

## Quick start

### 1. Obtain credentials

Ask the observability platform operator grafana@psi.ch for the Basic Auth
username and password assigned to your environment or shared organization.

### 2. Encrypt the password

https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets

Encrypt the password with the public key from the same repository:

```bash
eyaml encrypt \
  --pkcs7-public-key ./eyaml_public_key.pem \
  --password \
  --output block
```

The command prompts for the password so it does not enter the shell history.
Never commit the plaintext password.

### 3. Add the credentials to the environment

Add the assigned username and encrypted password to `<environment>.yaml` in
the `data-<environment>` repository:

```yaml
alloy::backend::username: 'assigned-collector'
alloy::backend::password: >
  ENC[PKCS7,...]
```

Use the EYAML output without modifying the encrypted content. A value
encrypted with another environment's public key cannot be decrypted by this
environment's Puppet server key.

### 4. Enable one canary host

Add the following to the canary's node-level YAML file:

```yaml
base::enable_alloy: true
```

```bash
sudo puppet agent -t
```

### 5. Verify the canary

Check the service and recent warnings:

```bash
sudo systemctl status alloy --no-pager
sudo journalctl -u alloy --since '-10 min' --no-pager
sudo /usr/bin/alloy validate /etc/alloy/config.alloy
```

Alloy logs at `warn`, so a healthy service can have very little journal output.

In Grafana, check for the host with queries such as:

```promql
node_uname_info{host="<fqdn>"}
```

```logql
{host="<fqdn>"}
```

Metrics normally appear after the next scrape, although backend delivery can
add more delay. Journal logs are forwarded continuously.

### 6. Roll out the environment

After the canary is accepted, move the enable setting to
`<environment>.yaml`:

```yaml
base::enable_alloy: true
```

This enrolls every eligible host in the Hiera environment unless a more
specific group, subgroup, or node file overrides the value.

RHEL 7 hosts ignore Alloy enrollment. The current repository supports RHEL 8,
9, and 10.

## Enable, disable, or leave unmanaged

`base::enable_alloy` has three states:

| Value | Result |
| --- | --- |
| `true` | Install, configure, enable, and run Alloy. |
| `false` | Stop and disable Alloy, then remove its package and managed configuration. |
| absent or `null` | Do not manage Alloy. This does **not** stop an existing installation. |

Use `false` to remove Alloy from a host or scope. Do not delete a previous
`true` setting and assume that Alloy will stop. Puppet deliberately does not
remove `/var/lib/alloy` and does not explicitly remove the service user or
directories; package uninstall behavior can still vary.

## Fixed baseline

Enabling Alloy always collects:

- Unix host metrics;
- every entry from the local system journal.

The code-owned baseline host collector selection is:

```text
cpu, diskstats, filefd, filesystem, loadavg, meminfo, netclass,
netdev, netstat, os, pressure, selinux, sockstat, softnet, stat, time,
timex, uname, vmstat, xfs
```
[Available collectors](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.unix/#collectors-list)

Alloy's own internal metrics are not forwarded.

Every metric and log entry receives centrally derived Puppet and SysDB
identity where applicable:

- `host`;
- `puppet_group`;
- `puppet_role`;
- `puppet_subgroup`;
- `sysdb_env`.

These values come from SysDB and cannot be replaced with Hiera labels.

## Host metrics

### Available settings

| Hiera key | Type | Default | Purpose |
| --- | --- | --- | --- |
| `alloy::host_metrics::scrape_interval` | duration string | `'60s'` | How often host metrics are collected. |
| `alloy::host_metrics::scrape_timeout` | duration string | `'10s'` | Maximum duration of one host scrape. |
| `alloy::host_metrics::include_exporter_metrics` | Boolean | `false` | Include operational metrics from the embedded Unix exporter. |
| `alloy::host_metrics::disable_collectors` | list of strings | `[]` | Remove collectors from the fixed selection. |
| `alloy::host_metrics::enable_collectors` | list of strings | `[]` | Add collectors to the fixed selection. |

For example, this removes the XFS collector and adds InfiniBand metrics:

```yaml
alloy::host_metrics::disable_collectors:
  - 'xfs'

alloy::host_metrics::enable_collectors:
  - 'infiniband'
```

Collector names are passed to the Alloy Unix exporter. Puppet does not
maintain a separate collector allowlist. Avoid placing the same collector in
both lists.

Setting `include_exporter_metrics` does not enable Alloy's own internal
metrics. It only controls metrics produced by the embedded Unix exporter.

## Journal logs

Journal collection cannot be disabled while Alloy is enabled.

### Available settings

| Hiera key | Type | Default | Purpose |
| --- | --- | --- | --- |
| `alloy::journal::max_age` | duration string | `'1h'` | Do not read journal entries older than this limit. |
| `alloy::journal::format_as_json` | Boolean | `false` | Send the journal entry as JSON instead of only its message. |
| `alloy::journal::labels` | string map | `{}` | Add static labels to journal entries. |

Example:

```yaml
alloy::journal::max_age: '30m'
alloy::journal::format_as_json: false

alloy::journal::labels:
  application_area: 'controls'
```

All service units receive a `systemd_unit` label. Instances of templated units
are collapsed to their template name, for example `worker@123.service` becomes
`worker@.service`. Transient `run-*.service` units are grouped as
`systemd-run.service`. The exact unit name and the syslog identifier are stored
as structured metadata named `systemd_unit_exact` and `syslog_identifier`.

These fields are managed centrally and cannot be configured through Hiera.
Static journal labels named `systemd_unit`, `systemd_unit_exact`, or
`syslog_identifier` are discarded.

## Additional Prometheus endpoints

Use a named Prometheus endpoint for an application that exposes Prometheus or
OpenMetrics data over HTTP.

Minimal example:

```yaml
alloy::components:
  prometheus_endpoints:
    gitea:
      enable: true
      address: '127.0.0.1:3000'
```

Complete example:

```yaml
alloy::components:
  prometheus_endpoints:
    application_api:
      enable: true
      address: '127.0.0.1:9400'
      scheme: 'http'
      metrics_path: '/internal/metrics'
      scrape_interval: '30s'
      scrape_timeout: '10s'
      labels:
        service: 'application-api'
```

### Endpoint settings

| Field | Required | Default | Purpose |
| --- | --- | --- | --- |
| `enable` | yes | none | Enable or disable this named endpoint. |
| `address` | when enabled | none | Target in `host:port` form, not a complete URL. |
| `scheme` | no | `'http'` | `http` or `https`. |
| `metrics_path` | no | `'/metrics'` | HTTP path containing the metrics. |
| `scrape_interval` | no | `'60s'` | How often to scrape this endpoint. |
| `scrape_timeout` | no | `'10s'` | Maximum duration of one scrape. |
| `labels` | no | `{}` | Static labels for this endpoint's metrics. |

The endpoint name becomes its `job` and `target` label. Use a unique name and
do not use the managed names `psi_host` or `psi_docker`.

Disable an endpoint inherited from a broader Hiera level with:

```yaml
alloy::components:
  prometheus_endpoints:
    gitea:
      enable: false
```

Endpoint authentication, custom HTTP headers, and custom certificate
authorities are not currently exposed. Prefer a loopback listener when the
application supports it.

## Redis metrics

!!! warning "Work in progress"
    Redis telemetry is not available in the production Puppet environment yet.
    The configuration in this section is intended only for development and
    testing.

Redis metrics are disabled by default. The component connects to Redis on the
same host at `localhost:6379`.

Enable it with:

```yaml
alloy::components:
  redis:
    enable: true
```

Redis-specific address, authentication, labels, and scrape settings are not
configurable through Hiera. Alloy scrapes Redis every 60 seconds with a
10-second timeout. The metric namespace is generated from the host's Puppet
certificate name (Hostname) and ends in `_redis`.

Set `enable: false` at a more specific Hiera level to disable an inherited
`true` value.

## Blackbox probes

!!! warning "Work in progress"
    Blackbox probing is not available in the production Puppet environment
    yet. The configuration in this section is intended only for development
    and testing.

Blackbox probing is disabled by default. Both the component and every target
must be explicitly enabled.

The following example checks an SSH banner:

```yaml
alloy::components:
  blackbox:
    enable: true
    modules:
      ssh_banner:
        prober: tcp
        timeout: '5s'
        tcp:
          query_response:
            - expect: '^SSH-2.0-'
            - send: "SSH-2.0-blackbox-ssh-check\r"
    targets:
      gpu001:
        enable: true
        address: 'gpu001.merlin7.psi.ch:22'
        module: 'ssh_banner'
        hostname: 'gpu001'
        labels:
          service: 'ssh'
```

### Blackbox settings

| Field | Required | Default | Purpose |
| --- | --- | --- | --- |
| `enable` | no | `false` | Enable Blackbox probing on this Alloy host. |
| `modules` | for enabled targets | `{}` | Native Blackbox exporter module definitions, keyed by module name. |
| `targets` | for probes | `{}` | Named Blackbox targets. Each target is opt-in. |

### Blackbox target settings

| Field | Required | Default | Purpose |
| --- | --- | --- | --- |
| `enable` | no | `false` | Probe this named target. |
| `address` | when enabled | none | Address expected by the selected module, such as `host:port` for TCP. |
| `module` | when enabled | none | Name of a module defined under `modules`. |
| `hostname` | when enabled | none | Hostname used for the target's `instance` label. |
| `labels` | no | `{}` | Additional string labels for the probe metrics. |

The target map key identifies the Alloy target. The selected module becomes
part of the `job` label, and `hostname` becomes the `instance` label. A target
without `enable: true` is ignored. Disable an inherited target by setting its
`enable` field to `false` at a more specific Hiera level. Enabled targets are
scraped every 30 seconds.

Module definitions use the native Blackbox exporter format. Encrypt sensitive
module values with EYAML as described in the
[Hiera secrets guide](https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets).
Never commit plaintext secrets.

## SNMP metrics

!!! warning "Work in progress"
    SNMP telemetry is not available in the production Puppet environment yet.
    The configuration in this section is intended only for development and
    testing.

SNMP collection is disabled by default. Alloy includes the standard SNMP
exporter modules and allows Hiera to define authentication profiles and named
targets. Both the component and every target must be explicitly enabled.

For example:

```yaml
alloy::components:
  snmp:
    enable: true
    auths:
      site_v2:
        community: >
          ENC[PKCS7,...]
        version: 2
    targets:
      switch01:
        enable: true
        address: 'switch01.psi.ch'
        module: 'if_mib'
        auth: 'site_v2'
        labels:
          location: 'building-001'
```

### SNMP settings

| Field | Required | Default | Purpose |
| --- | --- | --- | --- |
| `enable` | no | `false` | Enable SNMP collection on this Alloy host. |
| `auths` | no | `{}` | Native SNMP exporter authentication profiles, keyed by profile name. |
| `targets` | for collection | `{}` | Named SNMP targets. Each target is opt-in. |

### SNMP target settings

| Field | Required | Default | Purpose |
| --- | --- | --- | --- |
| `enable` | no | `false` | Collect metrics from this named target. |
| `address` | when enabled | none | Hostname or address of the SNMP device. |
| `module` | when enabled | none | Embedded SNMP exporter module or comma-separated modules to use. |
| `auth` | when enabled | none | Authentication profile defined under `auths` or supplied by Alloy. |
| `labels` | no | `{}` | Additional string labels for this target's metrics. |

The target map key becomes its `target` label, and its address becomes the
`instance` label. A target without `enable: true` is ignored. Disable an
inherited target by setting its `enable` field to `false`. Enabled targets are
scraped every 60 seconds with a 50-second timeout.

Encrypt SNMP communities, authentication passwords, and privacy passwords
with EYAML. Never commit them as plaintext. Custom SNMP modules, walk
parameters, contexts, and scrape settings are not currently exposed through
Hiera.

## Docker metrics and logs

Docker telemetry is disabled by default. Enable it with:

```yaml
alloy::components:
  docker:
    enable: true
```

Set `enable: false` at a more specific Hiera level to disable an inherited
`true` value.

This single switch enables both:

- cAdvisor metrics for local containers;
- logs from all local containers through the Docker socket.

Metrics and logs cannot currently be enabled separately. The host must have a
working Docker socket. The managed Alloy service runs as `root` while this
component is enabled so that cAdvisor can read Docker layer metadata and Alloy
can access the Docker socket. Without Docker telemetry, Alloy runs as the
unprivileged `alloy` user.

Access to the Docker socket is effectively privileged. Enable this component
only on approved Docker hosts. Runtime access can also depend on the storage
driver, cgroup setup, and SELinux policy.

If Docker uses the journald logging driver, the same container output can be
collected once through the mandatory journal source and again through the
Docker log source.

## OTLP traces

Trace reception is disabled by default. Enable it with:

```yaml
alloy::components:
  otlp_traces:
    enable: true
```

Set `enable: false` at a more specific Hiera level to disable an inherited
`true` value.

Applications on the same host can then send traces to:

- OTLP/gRPC on `127.0.0.1:4317`;
- OTLP/HTTP on `127.0.0.1:4318`.

The receivers are loopback-only and accept traces only. Receiver addresses,
batch settings, authentication, and sampling are not configurable through
Hiera. Applications are responsible for trace sampling.

Alloy adds the centrally derived host and environment resource attributes.
The general `alloy::labels` map does not apply to traces.

## Additional labels

Add stable labels shared by metrics and logs with:

```yaml
alloy::labels:
  application: 'gitea'
  lifecycle: 'production'
```

These labels apply to host and additional Prometheus metrics as well as journal
and Docker logs. They do not apply to traces. Once the WIP Redis, Blackbox, and
SNMP components become available, the labels will apply to their metrics too.

Use labels only for stable, low-cardinality values. Do not use usernames,
request IDs, timestamps, container IDs, or other values that continually
change.

More specific label options are available for journal logs and individual
Prometheus endpoints. The WIP Blackbox and SNMP components also provide
target-specific labels:

```yaml
alloy::journal::labels:
  log_source: 'operating-system'

alloy::components:
  prometheus_endpoints:
    application_api:
      enable: true
      address: '127.0.0.1:9400'
      labels:
        service: 'application-api'
```

The centrally derived Puppet/SysDB identity and fixed component labels take
precedence over conflicting admin-provided labels. Static journal labels named
`systemd_unit`, `systemd_unit_exact`, or `syslog_identifier` are discarded.

## Hiera priority and merging

Admin-managed Hiera data is applied from most specific to least specific:

1. node;
2. subgroup;
3. group;
4. Hiera environment.

Scalar settings use the first value found, so the most-specific value takes
precedence. The enable and disable collector lists use Hiera's unique array
merge. Values from node, subgroup, group, and SysDB environment levels are
combined, with duplicates removed.

The following maps deep-merge across the hierarchy:

- `alloy::components`;
- `alloy::labels`;
- `alloy::journal::labels`.

This makes it possible to define common environment settings and modify one
component or label for a group or host.

Use the option names exactly as shown in this guide. Puppet intentionally does
not validate unknown component or field names. Some mistakes fail while
rendering or validating the Alloy configuration, but an unused unknown key can
be ignored. For example, `scrape_intervall` is ignored and leaves the default
`60s` interval in effect.

Write Boolean values as the unquoted YAML values `true` and `false`. Quoted
values are strings, not Booleans.

## Settings that admins cannot change

The following are owned by the Puppet code:

- the production Mimir, Loki, and Tempo destinations, except for the
  documented development-host test override;
- the Alloy package version;
- the baseline host collector list, which admins can adjust with the documented
  enable and disable arrays;
- mandatory host metric and journal collection;
- the local Alloy HTTP listener and service hardening;
- OTLP listener addresses and batching;
- once available, the WIP Redis address, namespace pattern, and scrape settings;
- once available, the WIP Blackbox and SNMP scrape settings and embedded SNMP
  module set;
- Docker component internals;
- raw or custom Alloy configuration.

For normal production hosts, the only backend settings that admins need to
provide in Hiera are:

```yaml
alloy::backend::username: 'assigned-collector'
alloy::backend::password: >
  ENC[PKCS7,...]
```

Development hosts may override all three backend URLs together as documented
in [Add Development Hosts to grafana-test](grafana_test.md). Partial test
overrides and destinations other than the centrally managed production and
test platforms are not supported.

## Troubleshooting

### Alloy configuration validation failed

Puppet validates the generated configuration before replacing the active
file. Check addresses, duration strings, collector names, and component names
in the Hiera change. You can validate the installed configuration manually:

```bash
sudo /usr/bin/alloy validate /etc/alloy/config.alloy
```

### Docker component does not start

Confirm that the host has a `docker` group, that Docker is running, and that
`/var/run/docker.sock` exists. Then inspect:

```bash
sudo journalctl -u alloy --since '-10 min' --no-pager
```

### No data in Grafana

Confirm that Alloy is active and look for authentication or remote-write
warnings:

```bash
sudo systemctl is-active alloy
sudo journalctl -u alloy --since '-30 min' --no-pager
```

Also verify that the assigned Basic Auth username is mapped to the intended
tenant by the observability platform operator.