pombas_n/documentation

0 Fork 0

forked from linux/documentation

Files

T

bruhn_b bc3a9c1473 Update docs/admin-guide/configuration/monitoring/alloy.md

2026-09-15 15:18:56 +02:00

20 KiB

Raw Permalink Blame History

Grafana Alloy for environment admins

This guide explains how to enable Grafana Alloy through Hiera and which settings environment admins can change.

Alloy sends telemetry to the central production observability platform grafana.psi.ch:

host metrics to Mimir;
the complete local system journal to Loki;
optional traces to Tempo;
optional application and Docker telemetry.

Redis, Blackbox, and SNMP telemetry are still work in progress. Their configuration is documented below for testing and development, but is not yet available in the production Puppet environment.

Host metrics and journal logs are always enabled when Alloy is enabled. Obtain approval from the environment owner before enrolling a host because journal entries can contain sensitive data.

The production backend destinations, package version, service settings, and basic Alloy configuration are managed centrally. Development hosts can use the test observability platform by following Add Development Hosts to grafana-test.

Quick start

1. Obtain credentials

Ask the observability platform operator grafana@psi.ch for the Basic Auth username and password assigned to your environment or shared organization.

2. Encrypt the password

https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets

Encrypt the password with the public key from the same repository:

eyaml encrypt \
  --pkcs7-public-key ./eyaml_public_key.pem \
  --password \
  --output block

The command prompts for the password so it does not enter the shell history. Never commit the plaintext password.

3. Add the credentials to the environment

Add the assigned username and encrypted password to <environment>.yaml in the data-<environment> repository:

alloy::backend::username: 'assigned-collector'
alloy::backend::password: >
  ENC[PKCS7,...]

Use the EYAML output without modifying the encrypted content. A value encrypted with another environment's public key cannot be decrypted by this environment's Puppet server key.

4. Enable one canary host

Add the following to the canary's node-level YAML file:

base::enable_alloy: true

sudo puppet agent -t

5. Verify the canary

Check the service and recent warnings:

sudo systemctl status alloy --no-pager
sudo journalctl -u alloy --since '-10 min' --no-pager
sudo /usr/bin/alloy validate /etc/alloy/config.alloy

Alloy logs at warn, so a healthy service can have very little journal output.

In Grafana, check for the host with queries such as:

node_uname_info{host="<fqdn>"}

{host="<fqdn>"}

Metrics normally appear after the next scrape, although backend delivery can add more delay. Journal logs are forwarded continuously.

6. Roll out the environment

After the canary is accepted, move the enable setting to <environment>.yaml:

base::enable_alloy: true

This enrolls every eligible host in the Hiera environment unless a more specific group, subgroup, or node file overrides the value.

RHEL 7 hosts ignore Alloy enrollment. The current repository supports RHEL 8, 9, and 10.

Enable, disable, or leave unmanaged

base::enable_alloy has three states:

Value	Result
`true`	Install, configure, enable, and run Alloy.
`false`	Stop and disable Alloy, then remove its package and managed configuration.
absent or `null`	Do not manage Alloy. This does not stop an existing installation.

Use false to remove Alloy from a host or scope. Do not delete a previous true setting and assume that Alloy will stop. Puppet deliberately does not remove /var/lib/alloy and does not explicitly remove the service user or directories; package uninstall behavior can still vary.

Fixed baseline

Enabling Alloy always collects:

Unix host metrics;
every entry from the local system journal.

The code-owned baseline host collector selection is:

cpu, diskstats, filefd, filesystem, loadavg, meminfo, netclass,
netdev, netstat, os, pressure, selinux, sockstat, softnet, stat, time,
timex, uname, vmstat, xfs

Available collectors

Alloy's own internal metrics are not forwarded.

Every metric and log entry receives centrally derived Puppet and SysDB identity where applicable:

host;
puppet_group;
puppet_role;
puppet_subgroup;
sysdb_env.

These values come from SysDB and cannot be replaced with Hiera labels.

Host metrics

Available settings

Hiera key	Type	Default	Purpose
`alloy::host_metrics::scrape_interval`	duration string	`'60s'`	How often host metrics are collected.
`alloy::host_metrics::scrape_timeout`	duration string	`'10s'`	Maximum duration of one host scrape.
`alloy::host_metrics::include_exporter_metrics`	Boolean	`false`	Include operational metrics from the embedded Unix exporter.
`alloy::host_metrics::disable_collectors`	list of strings	`[]`	Remove collectors from the fixed selection.
`alloy::host_metrics::enable_collectors`	list of strings	`[]`	Add collectors to the fixed selection.

For example, this removes the XFS collector and adds InfiniBand metrics:

alloy::host_metrics::disable_collectors:
  - 'xfs'

alloy::host_metrics::enable_collectors:
  - 'infiniband'

Collector names are passed to the Alloy Unix exporter. Puppet does not maintain a separate collector allowlist. Avoid placing the same collector in both lists.

Setting include_exporter_metrics does not enable Alloy's own internal metrics. It only controls metrics produced by the embedded Unix exporter.

Journal logs

Journal collection cannot be disabled while Alloy is enabled.

Available settings

Hiera key	Type	Default	Purpose
`alloy::journal::max_age`	duration string	`'1h'`	Do not read journal entries older than this limit.
`alloy::journal::format_as_json`	Boolean	`false`	Send the journal entry as JSON instead of only its message.
`alloy::journal::labels`	string map	`{}`	Add static labels to journal entries.

Example:

alloy::journal::max_age: '30m'
alloy::journal::format_as_json: false

alloy::journal::labels:
  application_area: 'controls'

All service units receive a systemd_unit label. Instances of templated units are collapsed to their template name, for example worker@123.service becomes worker@.service. Transient run-*.service units are grouped as systemd-run.service. The exact unit name and the syslog identifier are stored as structured metadata named systemd_unit_exact and syslog_identifier.

These fields are managed centrally and cannot be configured through Hiera. Static journal labels named systemd_unit, systemd_unit_exact, or syslog_identifier are discarded.

Additional Prometheus endpoints

Use a named Prometheus endpoint for an application that exposes Prometheus or OpenMetrics data over HTTP.

Minimal example:

alloy::components:
  prometheus_endpoints:
    gitea:
      enable: true
      address: '127.0.0.1:3000'

Complete example:

alloy::components:
  prometheus_endpoints:
    application_api:
      enable: true
      address: '127.0.0.1:9400'
      scheme: 'http'
      metrics_path: '/internal/metrics'
      scrape_interval: '30s'
      scrape_timeout: '10s'
      labels:
        service: 'application-api'

Endpoint settings

Field	Required	Default	Purpose
`enable`	yes	none	Enable or disable this named endpoint.
`address`	when enabled	none	Target in `host:port` form, not a complete URL.
`scheme`	no	`'http'`	`http` or `https`.
`metrics_path`	no	`'/metrics'`	HTTP path containing the metrics.
`scrape_interval`	no	`'60s'`	How often to scrape this endpoint.
`scrape_timeout`	no	`'10s'`	Maximum duration of one scrape.
`labels`	no	`{}`	Static labels for this endpoint's metrics.

The endpoint name becomes its job and target label. Use a unique name and do not use the managed names psi_host or psi_docker.

Disable an endpoint inherited from a broader Hiera level with:

alloy::components:
  prometheus_endpoints:
    gitea:
      enable: false

Endpoint authentication, custom HTTP headers, and custom certificate authorities are not currently exposed. Prefer a loopback listener when the application supports it.

Redis metrics

!!! warning "Work in progress" Redis telemetry is not available in the production Puppet environment yet. The configuration in this section is intended only for development and testing.

Redis metrics are disabled by default. The component connects to Redis on the same host at localhost:6379.

Enable it with:

alloy::components:
  redis:
    enable: true

Redis-specific address, authentication, labels, and scrape settings are not configurable through Hiera. Alloy scrapes Redis every 60 seconds with a 10-second timeout. The metric namespace is generated from the host's Puppet certificate name (Hostname) and ends in _redis.

Set enable: false at a more specific Hiera level to disable an inherited true value.

Blackbox probes

!!! warning "Work in progress" Blackbox probing is not available in the production Puppet environment yet. The configuration in this section is intended only for development and testing.

Blackbox probing is disabled by default. Both the component and every target must be explicitly enabled.

The following example checks an SSH banner:

alloy::components:
  blackbox:
    enable: true
    modules:
      ssh_banner:
        prober: tcp
        timeout: '5s'
        tcp:
          query_response:
            - expect: '^SSH-2.0-'
            - send: "SSH-2.0-blackbox-ssh-check\r"
    targets:
      gpu001:
        enable: true
        address: 'gpu001.merlin7.psi.ch:22'
        module: 'ssh_banner'
        hostname: 'gpu001'
        labels:
          service: 'ssh'

Blackbox settings

Field	Required	Default	Purpose
`enable`	no	`false`	Enable Blackbox probing on this Alloy host.
`modules`	for enabled targets	`{}`	Native Blackbox exporter module definitions, keyed by module name.
`targets`	for probes	`{}`	Named Blackbox targets. Each target is opt-in.

Blackbox target settings

Field	Required	Default	Purpose
`enable`	no	`false`	Probe this named target.
`address`	when enabled	none	Address expected by the selected module, such as `host:port` for TCP.
`module`	when enabled	none	Name of a module defined under `modules`.
`hostname`	when enabled	none	Hostname used for the target's `instance` label.
`labels`	no	`{}`	Additional string labels for the probe metrics.

The target map key identifies the Alloy target. The selected module becomes part of the job label, and hostname becomes the instance label. A target without enable: true is ignored. Disable an inherited target by setting its enable field to false at a more specific Hiera level. Enabled targets are scraped every 30 seconds.

Module definitions use the native Blackbox exporter format. Encrypt sensitive module values with EYAML as described in the Hiera secrets guide. Never commit plaintext secrets.

SNMP metrics

!!! warning "Work in progress" SNMP telemetry is not available in the production Puppet environment yet. The configuration in this section is intended only for development and testing.

SNMP collection is disabled by default. Alloy includes the standard SNMP exporter modules and allows Hiera to define authentication profiles and named targets. Both the component and every target must be explicitly enabled.

For example:

alloy::components:
  snmp:
    enable: true
    auths:
      site_v2:
        community: >
          ENC[PKCS7,...]
        version: 2
    targets:
      switch01:
        enable: true
        address: 'switch01.psi.ch'
        module: 'if_mib'
        auth: 'site_v2'
        labels:
          location: 'building-001'

SNMP settings

Field	Required	Default	Purpose
`enable`	no	`false`	Enable SNMP collection on this Alloy host.
`auths`	no	`{}`	Native SNMP exporter authentication profiles, keyed by profile name.
`targets`	for collection	`{}`	Named SNMP targets. Each target is opt-in.

SNMP target settings

Field	Required	Default	Purpose
`enable`	no	`false`	Collect metrics from this named target.
`address`	when enabled	none	Hostname or address of the SNMP device.
`module`	when enabled	none	Embedded SNMP exporter module or comma-separated modules to use.
`auth`	when enabled	none	Authentication profile defined under `auths` or supplied by Alloy.
`labels`	no	`{}`	Additional string labels for this target's metrics.

The target map key becomes its target label, and its address becomes the instance label. A target without enable: true is ignored. Disable an inherited target by setting its enable field to false. Enabled targets are scraped every 60 seconds with a 50-second timeout.

Encrypt SNMP communities, authentication passwords, and privacy passwords with EYAML. Never commit them as plaintext. Custom SNMP modules, walk parameters, contexts, and scrape settings are not currently exposed through Hiera.

Docker metrics and logs

Docker telemetry is disabled by default. Enable it with:

alloy::components:
  docker:
    enable: true

Set enable: false at a more specific Hiera level to disable an inherited true value.

This single switch enables both:

cAdvisor metrics for local containers;
logs from all local containers through the Docker socket.

Metrics and logs cannot currently be enabled separately. The host must have a working Docker socket. The managed Alloy service runs as root while this component is enabled so that cAdvisor can read Docker layer metadata and Alloy can access the Docker socket. Without Docker telemetry, Alloy runs as the unprivileged alloy user.

Access to the Docker socket is effectively privileged. Enable this component only on approved Docker hosts. Runtime access can also depend on the storage driver, cgroup setup, and SELinux policy.

If Docker uses the journald logging driver, the same container output can be collected once through the mandatory journal source and again through the Docker log source.

OTLP traces

Trace reception is disabled by default. Enable it with:

alloy::components:
  otlp_traces:
    enable: true

Set enable: false at a more specific Hiera level to disable an inherited true value.

Applications on the same host can then send traces to:

OTLP/gRPC on 127.0.0.1:4317;
OTLP/HTTP on 127.0.0.1:4318.

The receivers are loopback-only and accept traces only. Receiver addresses, batch settings, authentication, and sampling are not configurable through Hiera. Applications are responsible for trace sampling.

Alloy adds the centrally derived host and environment resource attributes. The general alloy::labels map does not apply to traces.

Additional labels

Add stable labels shared by metrics and logs with:

alloy::labels:
  application: 'gitea'
  lifecycle: 'production'

These labels apply to host and additional Prometheus metrics as well as journal and Docker logs. They do not apply to traces. Once the WIP Redis, Blackbox, and SNMP components become available, the labels will apply to their metrics too.

Use labels only for stable, low-cardinality values. Do not use usernames, request IDs, timestamps, container IDs, or other values that continually change.

More specific label options are available for journal logs and individual Prometheus endpoints. The WIP Blackbox and SNMP components also provide target-specific labels:

alloy::journal::labels:
  log_source: 'operating-system'

alloy::components:
  prometheus_endpoints:
    application_api:
      enable: true
      address: '127.0.0.1:9400'
      labels:
        service: 'application-api'

The centrally derived Puppet/SysDB identity and fixed component labels take precedence over conflicting admin-provided labels. Static journal labels named systemd_unit, systemd_unit_exact, or syslog_identifier are discarded.

Hiera priority and merging

Admin-managed Hiera data is applied from most specific to least specific:

node;
subgroup;
group;
Hiera environment.

Scalar settings use the first value found, so the most-specific value takes precedence. The enable and disable collector lists use Hiera's unique array merge. Values from node, subgroup, group, and SysDB environment levels are combined, with duplicates removed.

The following maps deep-merge across the hierarchy:

alloy::components;
alloy::labels;
alloy::journal::labels.

This makes it possible to define common environment settings and modify one component or label for a group or host.

Use the option names exactly as shown in this guide. Puppet intentionally does not validate unknown component or field names. Some mistakes fail while rendering or validating the Alloy configuration, but an unused unknown key can be ignored. For example, scrape_intervall is ignored and leaves the default 60s interval in effect.

Write Boolean values as the unquoted YAML values true and false. Quoted values are strings, not Booleans.

Settings that admins cannot change

The following are owned by the Puppet code:

the production Mimir, Loki, and Tempo destinations, except for the documented development-host test override;
the Alloy package version;
the baseline host collector list, which admins can adjust with the documented enable and disable arrays;
mandatory host metric and journal collection;
the local Alloy HTTP listener and service hardening;
OTLP listener addresses and batching;
once available, the WIP Redis address, namespace pattern, and scrape settings;
once available, the WIP Blackbox and SNMP scrape settings and embedded SNMP module set;
Docker component internals;
raw or custom Alloy configuration.

For normal production hosts, the only backend settings that admins need to provide in Hiera are:

alloy::backend::username: 'assigned-collector'
alloy::backend::password: >
  ENC[PKCS7,...]

Development hosts may override all three backend URLs together as documented in Add Development Hosts to grafana-test. Partial test overrides and destinations other than the centrally managed production and test platforms are not supported.

Troubleshooting

Alloy configuration validation failed

Puppet validates the generated configuration before replacing the active file. Check addresses, duration strings, collector names, and component names in the Hiera change. You can validate the installed configuration manually:

sudo /usr/bin/alloy validate /etc/alloy/config.alloy

Docker component does not start

Confirm that the host has a docker group, that Docker is running, and that /var/run/docker.sock exists. Then inspect:

sudo journalctl -u alloy --since '-10 min' --no-pager

No data in Grafana

Confirm that Alloy is active and look for authentication or remote-write warnings:

sudo systemctl is-active alloy
sudo journalctl -u alloy --since '-30 min' --no-pager

Also verify that the assigned Basic Auth username is mapped to the intended tenant by the observability platform operator.

20 KiB Raw Permalink Blame History

Grafana Alloy for environment admins

Quick start

1. Obtain credentials

2. Encrypt the password

3. Add the credentials to the environment

4. Enable one canary host

5. Verify the canary

6. Roll out the environment

Enable, disable, or leave unmanaged

Fixed baseline

Host metrics

Available settings

Journal logs

Available settings

Additional Prometheus endpoints

Endpoint settings

Redis metrics

Blackbox probes

Blackbox settings

Blackbox target settings

SNMP metrics

SNMP settings

SNMP target settings

Docker metrics and logs

OTLP traces

Additional labels

Hiera priority and merging

Settings that admins cannot change

Troubleshooting

Alloy configuration validation failed

Docker component does not start

No data in Grafana

20 KiB

Raw Permalink Blame History