diff --git a/docs/admin-guide/configuration/monitoring/alloy.md b/docs/admin-guide/configuration/monitoring/alloy.md new file mode 100644 index 00000000..399cc40d --- /dev/null +++ b/docs/admin-guide/configuration/monitoring/alloy.md @@ -0,0 +1,461 @@ +# Grafana Alloy for environment admins + +This guide explains how to enable Grafana Alloy through Hiera and which +settings environment admins can change. + +Alloy sends telemetry to the central observability platform grafana.psi.ch: + +- host metrics to Mimir; +- the complete local system journal to Loki; +- optional traces to Tempo; +- optional application and Docker telemetry. + +Host metrics and journal logs are always enabled when Alloy is enabled. Obtain approval from the environment owner +before enrolling a host because journal entries can contain sensitive data. + +The backend destination, package version, service settings, and basic Alloy +configuration are managed centrally. Admins cannot redirect Alloy to another +observability stack through Hiera. + +## Quick start + +### 1. Obtain credentials + +Ask the observability platform operator grafana@psi.ch for the Basic Auth username and +password assigned to your environment or shared organization. +### 2. Encrypt the password + +https://linux.psi.ch/documentation/admin-guide/puppet/hiera/?h=eyaml#secrets + +Encrypt the password with the public key from the same repository: + +```bash +eyaml encrypt \ + --pkcs7-public-key ./eyaml_public_key.pem \ + --password \ + --output block +``` + +The command prompts for the password so it does not enter the shell history. +Never commit the plaintext password. + +### 3. Add the credentials to the environment + +Add the assigned username and encrypted password to `.yaml` in +the `data-` repository: + +```yaml +alloy::backend::username: 'assigned-writer' +alloy::backend::password: > + ENC[PKCS7,...] +``` + +Use the EYAML output without modifying the encrypted content. A value +encrypted with another environment's public key cannot be decrypted by this +environment's Puppet server key. + +### 4. Enable one canary host + +Add the following to the canary's node-level YAML file: + +```yaml +base::enable_alloy: true +``` + +```bash +sudo puppet agent -t --environment alloy +``` + +### 5. Verify the canary + +Check the service and recent warnings: + +```bash +sudo systemctl status alloy --no-pager +sudo journalctl -u alloy --since '-10 min' --no-pager +sudo /usr/bin/alloy validate /etc/alloy/config.alloy +``` + +Alloy logs at `warn`, so a healthy service can have very little journal output. + +In Grafana, check for the host with queries such as: + +```promql +node_uname_info{host=""} +``` + +```logql +{host=""} +``` + +Metrics normally appear after the next scrape, although backend delivery can +add more delay. Journal logs are forwarded continuously. + +### 6. Roll out the environment + +After the canary is accepted, move the enable setting to +`.yaml`: + +```yaml +base::enable_alloy: true +``` + +This enrolls every eligible host in the SysDB environment unless a more +specific group, subgroup, or node file overrides the value. + +RHEL 7 hosts ignore Alloy enrollment. The current repository supports RHEL 8, +9, and (10). + +## Enable, disable, or leave unmanaged + +`base::enable_alloy` has three states: + +| Value | Result | +| --- | --- | +| `true` | Install, configure, enable, and run Alloy. | +| `false` | Stop and disable Alloy, then remove its package and managed configuration. | +| absent or `null` | Do not manage Alloy. This does **not** stop an existing installation. | + +Use `false` to remove Alloy from a host or scope. Do not delete a previous +`true` setting and assume that Alloy will stop. Puppet deliberately does not +remove `/var/lib/alloy` and does not explicitly remove the service user or +directories; package uninstall behavior can still vary. + +## Fixed baseline + +Enabling Alloy always collects: + +- Unix host metrics; +- every entry from the local system journal. + +The code-owned baseline host collector selection is: + +```text +cpu, diskstats, filefd, filesystem, loadavg, meminfo, netclass, +netdev, netstat, os, pressure, sockstat, stat, time, timex, +uname, vmstat +``` + +Alloy's own internal metrics are not forwarded. + +Every metric and log entry receives centrally derived Puppet and SysDB +identity where applicable: + +- `host`; +- `puppet_group`; +- `puppet_role`; +- `puppet_subgroup`; +- `sysdb_env`. + +These values come from SysDB and cannot be replaced with Hiera labels. + +## Host metrics + +### Available settings + +| Hiera key | Type | Default | Purpose | +| --- | --- | --- | --- | +| `alloy::host_metrics::scrape_interval` | duration string | `'60s'` | How often host metrics are collected. | +| `alloy::host_metrics::scrape_timeout` | duration string | `'10s'` | Maximum duration of one host scrape. | +| `alloy::host_metrics::include_exporter_metrics` | Boolean | `false` | Include operational metrics from the embedded Unix exporter. | +| `alloy::host_metrics::disable_collectors` | list of strings | `[]` | Remove collectors from the fixed selection. | +| `alloy::host_metrics::enable_collectors` | list of strings | `[]` | Add collectors to the fixed selection. | + +For example, this removes the pressure collector and adds XFS and InfiniBand +metrics: + +```yaml +alloy::host_metrics::disable_collectors: + - 'pressure' + +alloy::host_metrics::enable_collectors: + - 'xfs' + - 'infiniband' +``` + +Collector names are passed to the Alloy Unix exporter. Puppet does not +maintain a separate collector allowlist. Avoid placing the same collector in +both lists. + +The collector lists do not merge across Hiera levels. A node-level list +replaces the complete environment-, group-, or subgroup-level list. + +Setting `include_exporter_metrics` does not enable Alloy's own internal +metrics. It only controls metrics produced by the embedded Unix exporter. + +## Journal logs + +Journal collection cannot be disabled while Alloy is enabled. + +### Available settings + +| Hiera key | Type | Default | Purpose | +| --- | --- | --- | --- | +| `alloy::journal::max_age` | duration string | `'1h'` | Do not read journal entries older than this limit. | +| `alloy::journal::format_as_json` | Boolean | `false` | Send the journal entry as JSON instead of only its message. | +| `alloy::journal::labels` | string map | `{}` | Add static labels to journal entries. | +| `alloy::journal::unit_labels` | Boolean map | central defaults | Promote selected systemd units to a Loki label. | + +Example: + +```yaml +alloy::journal::max_age: '30m' +alloy::journal::format_as_json: false + +alloy::journal::labels: + application_area: 'controls' + +alloy::journal::unit_labels: + 'my-application.service': true + 'docker.service': false +``` + +`unit_labels` does not select or filter logs. All journal entries are still +sent. A `true` entry only promotes that stable unit name to the +`systemd_unit` Loki label. A `false` entry removes an inherited promotion. + +Use unit labels only for stable service names. Do not promote sessions, +transient scopes, generated instance names, or other high-cardinality values. + +The common configuration already promotes several frequently used services, +including Alloy, Chrony, Docker, Gitea, HTTP servers, SSH, and Slurm services. + +## Additional Prometheus endpoints + +Use a named Prometheus endpoint for an application that exposes Prometheus or +OpenMetrics data over HTTP. + +Minimal example: + +```yaml +alloy::components: + prometheus_endpoints: + gitea: + enable: true + address: '127.0.0.1:3000' +``` + +Complete example: + +```yaml +alloy::components: + prometheus_endpoints: + application_api: + enable: true + address: '127.0.0.1:9400' + scheme: 'http' + metrics_path: '/internal/metrics' + scrape_interval: '30s' + scrape_timeout: '10s' + labels: + service: 'application-api' +``` + +### Endpoint settings + +| Field | Required | Default | Purpose | +| --- | --- | --- | --- | +| `enable` | yes | none | Enable or disable this named endpoint. | +| `address` | when enabled | none | Target in `host:port` form, not a complete URL. | +| `scheme` | no | `'http'` | `http` or `https`. | +| `metrics_path` | no | `'/metrics'` | HTTP path containing the metrics. | +| `scrape_interval` | no | `'60s'` | How often to scrape this endpoint. | +| `scrape_timeout` | no | `'10s'` | Maximum duration of one scrape. | +| `labels` | no | `{}` | Static labels for this endpoint's metrics. | + +The endpoint name becomes its `job` and `target` label. Use a unique name and +do not use the managed names `psi_host` or `psi_docker`. + +Disable an endpoint inherited from a broader Hiera level with: + +```yaml +alloy::components: + prometheus_endpoints: + gitea: + enable: false +``` + +Endpoint authentication, custom HTTP headers, and custom certificate +authorities are not currently exposed. Prefer a loopback listener when the +application supports it. + +## Docker metrics and logs + +Docker telemetry is disabled by default. Enable it with: + +```yaml +alloy::components: + docker: + enable: true +``` + +Set `enable: false` at a more specific Hiera level to disable an inherited +`true` value. + +This single switch enables both: + +- cAdvisor metrics for local containers; +- logs from all local containers through the Docker socket. + +Metrics and logs cannot currently be enabled separately. The host must have a +working Docker socket and `docker` group. The managed systemd service receives +`SupplementaryGroups=docker` while this component is enabled; Puppet does not +revoke any Unix group membership configured elsewhere. + +Access to the Docker socket is effectively privileged. Enable this component +only on approved Docker hosts. Runtime access can also depend on the storage +driver, cgroup setup, and SELinux policy. + +If Docker uses the journald logging driver, the same container output can be +collected once through the mandatory journal source and again through the +Docker log source. + +## OTLP traces + +Trace reception is disabled by default. Enable it with: + +```yaml +alloy::components: + otlp_traces: + enable: true +``` + +Set `enable: false` at a more specific Hiera level to disable an inherited +`true` value. + +Applications on the same host can then send traces to: + +- OTLP/gRPC on `127.0.0.1:4317`; +- OTLP/HTTP on `127.0.0.1:4318`. + +The receivers are loopback-only and accept traces only. Receiver addresses, +batch settings, authentication, and sampling are not configurable through +Hiera. Applications are responsible for trace sampling. + +Alloy adds the centrally derived host and environment resource attributes. +The general `alloy::labels` map does not apply to traces. + +## Additional labels + +Add stable labels shared by metrics and logs with: + +```yaml +alloy::labels: + application: 'gitea' + lifecycle: 'production' +``` + +These labels apply to host metrics, additional Prometheus metrics, journal +logs, and Docker logs. They do not apply to traces. + +Use labels only for stable, low-cardinality values. Do not use usernames, +request IDs, timestamps, container IDs, or other values that continually +change. + +More specific label options are available for journal logs and individual +Prometheus endpoints: + +```yaml +alloy::journal::labels: + log_source: 'operating-system' + +alloy::components: + prometheus_endpoints: + application_api: + enable: true + address: '127.0.0.1:9400' + labels: + service: 'application-api' +``` + +The centrally derived Puppet/SysDB identity and fixed component labels take +precedence over conflicting admin-provided labels. A static `systemd_unit` +journal label is discarded; use `alloy::journal::unit_labels` instead. + +## Hiera priority and merging + +Admin-managed Hiera data is applied from most specific to least specific: + +1. node; +2. subgroup; +3. group; +4. SysDB environment. + +Scalar values and collector lists use the first value found. A more specific +value therefore replaces the broader value or complete list. + +The following maps deep-merge across the hierarchy: + +- `alloy::components`; +- `alloy::labels`; +- `alloy::journal::labels`; +- `alloy::journal::unit_labels`. + +This makes it possible to define common environment settings and modify one +component, label, or systemd unit for a group or host. + +Use the option names exactly as shown in this guide. Puppet intentionally does +not validate unknown component or field names. Some mistakes fail while +rendering or validating the Alloy configuration, but an unused unknown key can +be ignored. For example, `scrape_intervall` is ignored and leaves the default +`60s` interval in effect. + +## Settings that admins cannot change + +The following are owned by the Puppet code: + +- the Mimir, Loki, and Tempo destinations; +- the Alloy package version; +- the baseline host collector list, which admins can adjust with the documented + enable and disable arrays; +- mandatory host metric and journal collection; +- the local Alloy HTTP listener and service hardening; +- OTLP listener addresses and batching; +- Docker component internals; +- raw or custom Alloy configuration. + +The only backend settings available in Hiera are: + +```yaml +alloy::backend::username: 'assigned-writer' +alloy::backend::password: > + ENC[PKCS7,...] +``` + +Keys such as `alloy::backend::tenant_id`, +`alloy::backend::metrics_url`, `alloy::backend::logs_url`, or +`alloy::backend::otlp_base_url` are not supported. + +## Troubleshooting + +### Alloy configuration validation failed + +Puppet validates the generated configuration before replacing the active +file. Check addresses, duration strings, collector names, and component names +in the Hiera change. You can validate the installed configuration manually: + +```bash +sudo /usr/bin/alloy validate /etc/alloy/config.alloy +``` + +### Docker component does not start + +Confirm that the host has a `docker` group, that Docker is running, and that +`/var/run/docker.sock` exists. Then inspect: + +```bash +sudo journalctl -u alloy --since '-10 min' --no-pager +``` + +### No data in Grafana + +Confirm that Alloy is active and look for authentication or remote-write +warnings: + +```bash +sudo systemctl is-active alloy +sudo journalctl -u alloy --since '-30 min' --no-pager +``` + +Also verify that the assigned Basic Auth username is mapped to the intended +tenant by the observability platform operator.