diff --git a/docs/admin-guide/configuration/monitoring/alloy.md b/docs/admin-guide/configuration/monitoring/alloy.md index 152f1de4..38eba3ca 100644 --- a/docs/admin-guide/configuration/monitoring/alloy.md +++ b/docs/admin-guide/configuration/monitoring/alloy.md @@ -1,28 +1,32 @@ # Grafana Alloy for environment admins This guide explains how to enable Grafana Alloy through Hiera and which -settings environment admins can change. +settings environment admins can change. -Alloy sends telemetry to the central observability platform grafana-test.psi.ch: +Alloy sends telemetry to the central production observability platform +[grafana.psi.ch](https://grafana.psi.ch): - host metrics to Mimir; - the complete local system journal to Loki; - optional traces to Tempo; -- optional application and Docker telemetry. +- optional application, Redis, Blackbox, SNMP, and Docker telemetry. -Host metrics and journal logs are always enabled when Alloy is enabled. Obtain approval from the environment owner -before enrolling a host because journal entries can contain sensitive data. +Host metrics and journal logs are always enabled when Alloy is enabled. Obtain +approval from the environment owner before enrolling a host because journal +entries can contain sensitive data. -The backend destination, package version, service settings, and basic Alloy -configuration are managed centrally. Admins cannot redirect Alloy to another -observability stack through Hiera. +The production backend destinations, package version, service settings, and +basic Alloy configuration are managed centrally. Development hosts can use the +test observability platform by following +[Add Development Hosts to grafana-test](grafana_test.md). ## Quick start ### 1. Obtain credentials -Ask the observability platform operator grafana@psi.ch for the Basic Auth username and -password assigned to your environment or shared organization. +Ask the observability platform operator grafana@psi.ch for the Basic Auth +username and password assigned to your environment or shared organization. + ### 2. Encrypt the password https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets @@ -75,6 +79,13 @@ Add the following to the canary's node-level YAML file: base::enable_alloy: true ``` +<<<<<<< HEAD +======= +```bash +sudo puppet agent -t +``` + +>>>>>>> 7ec8b2f6 (update alloy docu, move grafana-test to different file) ### 5. Verify the canary Check the service and recent warnings: @@ -140,9 +151,9 @@ Enabling Alloy always collects: The code-owned baseline host collector selection is: ```text - 'cpu', 'diskstats', 'filefd', 'filesystem', 'loadavg', 'meminfo', - 'netclass', 'netdev', 'netstat', 'os', 'selinux', 'sockstat', - 'softnet', 'stat', 'time', 'timex', 'uname', 'vmstat', 'xfs' +cpu, diskstats, filefd, filesystem, loadavg, meminfo, netclass, +netdev, netstat, os, selinux, sockstat, softnet, stat, time, timex, +uname, vmstat, xfs ``` Alloy's own internal metrics are not forwarded. @@ -170,15 +181,13 @@ These values come from SysDB and cannot be replaced with Hiera labels. | `alloy::host_metrics::disable_collectors` | list of strings | `[]` | Remove collectors from the fixed selection. | | `alloy::host_metrics::enable_collectors` | list of strings | `[]` | Add collectors to the fixed selection. | -For example, this removes the pressure collector and adds XFS and InfiniBand -metrics: +For example, this removes the XFS collector and adds InfiniBand metrics: ```yaml alloy::host_metrics::disable_collectors: - - 'pressure' + - 'xfs' alloy::host_metrics::enable_collectors: - - 'xfs' - 'infiniband' ``` @@ -223,7 +232,8 @@ sent. A `true` entry only promotes that stable unit name to the Use unit labels only for stable service names. Do not promote sessions, transient scopes, generated instance names, or other high-cardinality values. -The code-owned defaults promote Chrony, NetworkManager, SSH, and SSSD service units. +The code-owned defaults promote Chrony, NetworkManager, SSH, and SSSD service +units. ## Additional Prometheus endpoints @@ -269,7 +279,8 @@ alloy::components: | `labels` | no | `{}` | Static labels for this endpoint's metrics. | The endpoint name becomes its `job` and `target` label. Use a unique name and -do not use the managed names `psi_host` or `psi_docker`. +do not use the managed names `psi_host`, `psi_redis`, `blackbox`, `psi_snmp`, +or `psi_docker`. Disable an endpoint inherited from a broader Hiera level with: @@ -284,6 +295,140 @@ Endpoint authentication, custom HTTP headers, and custom certificate authorities are not currently exposed. Prefer a loopback listener when the application supports it. +## Redis metrics + +Redis metrics are disabled by default. The component connects to Redis on the +same host at `localhost:6379`. + +Enable it with: + +```yaml +alloy::components: + redis: + enable: true +``` + +Redis-specific address, authentication, labels, and scrape settings are not +configurable through Hiera. Alloy scrapes Redis every 60 seconds with a +10-second timeout. The metric namespace is generated from the host's Puppet +certificate name and ends in `_redis`. + +Set `enable: false` at a more specific Hiera level to disable an inherited +`true` value. + +## Blackbox probes + +Blackbox probing is disabled by default. Both the component and every target +must be explicitly enabled. + +The following example checks an SSH banner: + +```yaml +alloy::components: + blackbox: + enable: true + modules: + ssh_banner: + prober: tcp + timeout: '5s' + tcp: + query_response: + - expect: '^SSH-2.0-' + - send: "SSH-2.0-blackbox-ssh-check\r" + targets: + gpu001: + enable: true + address: 'gpu001.merlin7.psi.ch:22' + module: 'ssh_banner' + hostname: 'gpu001' + labels: + service: 'ssh' +``` + +### Blackbox settings + +| Field | Required | Default | Purpose | +| --- | --- | --- | --- | +| `enable` | no | `false` | Enable Blackbox probing on this Alloy host. | +| `modules` | for enabled targets | `{}` | Native Blackbox exporter module definitions, keyed by module name. | +| `targets` | for probes | `{}` | Named Blackbox targets. Each target is opt-in. | + +### Blackbox target settings + +| Field | Required | Default | Purpose | +| --- | --- | --- | --- | +| `enable` | no | `false` | Probe this named target. | +| `address` | when enabled | none | Address expected by the selected module, such as `host:port` for TCP. | +| `module` | when enabled | none | Name of a module defined under `modules`. | +| `hostname` | when enabled | none | Hostname used for the target's `instance` label. | +| `labels` | no | `{}` | Additional string labels for the probe metrics. | + +The target map key identifies the Alloy target. The selected module becomes +part of the `job` label, and `hostname` becomes the `instance` label. A target +without `enable: true` is ignored. Disable an inherited target by setting its +`enable` field to `false` at a more specific Hiera level. Enabled targets are +scraped every 30 seconds. + +Module definitions use the native Blackbox exporter format. Encrypt sensitive +module values with EYAML as described in the +[Hiera secrets guide](https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets). +Never commit plaintext secrets. + +## SNMP metrics + +SNMP collection is disabled by default. Alloy includes the standard SNMP +exporter modules and allows Hiera to define authentication profiles and named +targets. Both the component and every target must be explicitly enabled. + +For example: + +```yaml +alloy::components: + snmp: + enable: true + auths: + site_v2: + community: > + ENC[PKCS7,...] + version: 2 + targets: + switch01: + enable: true + address: 'switch01.psi.ch' + module: 'if_mib' + auth: 'site_v2' + labels: + location: 'building-001' +``` + +### SNMP settings + +| Field | Required | Default | Purpose | +| --- | --- | --- | --- | +| `enable` | no | `false` | Enable SNMP collection on this Alloy host. | +| `auths` | no | `{}` | Native SNMP exporter authentication profiles, keyed by profile name. | +| `targets` | for collection | `{}` | Named SNMP targets. Each target is opt-in. | + +### SNMP target settings + +| Field | Required | Default | Purpose | +| --- | --- | --- | --- | +| `enable` | no | `false` | Collect metrics from this named target. | +| `address` | when enabled | none | Hostname or address of the SNMP device. | +| `module` | when enabled | none | Embedded SNMP exporter module or comma-separated modules to use. | +| `auth` | when enabled | none | Authentication profile defined under `auths` or supplied by Alloy. | +| `labels` | no | `{}` | Additional string labels for this target's metrics. | + +The target map key becomes its `target` label, and its address becomes the +`instance` label. A target without `enable: true` is ignored. Disable an +inherited target by setting its `enable` field to `false`. Enabled targets are +scraped every 60 seconds with a 50-second timeout. + +Encrypt SNMP communities, authentication passwords, and privacy passwords +with EYAML. Never commit them as plaintext. Custom SNMP modules, walk +parameters, contexts, and scrape settings are not currently exposed through +Hiera. + ## Docker metrics and logs Docker telemetry is disabled by default. Enable it with: @@ -350,15 +495,15 @@ alloy::labels: lifecycle: 'production' ``` -These labels apply to host metrics, additional Prometheus metrics, journal -logs, and Docker logs. They do not apply to traces. +These labels apply to host, Redis, Blackbox, SNMP, and additional Prometheus +metrics as well as journal and Docker logs. They do not apply to traces. Use labels only for stable, low-cardinality values. Do not use usernames, request IDs, timestamps, container IDs, or other values that continually change. -More specific label options are available for journal logs and individual -Prometheus endpoints: +More specific label options are available for journal logs, individual +Prometheus endpoints, Blackbox targets, and SNMP targets: ```yaml alloy::journal::labels: @@ -386,8 +531,15 @@ Admin-managed Hiera data is applied from most specific to least specific: 3. group; 4. Hiera environment. +<<<<<<< HEAD Scalar settings use the first value found, so the most-specific value takes precedence. The enable and disable collector lists use Hiera’s unique array merge. Values from node, subgroup, group, and Hiera environment levels are combined, with duplicates removed. +======= +Scalar settings use the first value found, so the most-specific value takes +precedence. The enable and disable collector lists use Hiera's unique array +merge. Values from node, subgroup, group, and SysDB environment levels are +combined, with duplicates removed. +>>>>>>> 7ec8b2f6 (update alloy docu, move grafana-test to different file) The following maps deep-merge across the hierarchy: @@ -405,21 +557,28 @@ rendering or validating the Alloy configuration, but an unused unknown key can be ignored. For example, `scrape_intervall` is ignored and leaves the default `60s` interval in effect. +Write Boolean values as the unquoted YAML values `true` and `false`. Quoted +values are strings, not Booleans. + ## Settings that admins cannot change The following are owned by the Puppet code: -- the Mimir, Loki, and Tempo destinations; +- the production Mimir, Loki, and Tempo destinations, except for the + documented development-host test override; - the Alloy package version; - the baseline host collector list, which admins can adjust with the documented enable and disable arrays; - mandatory host metric and journal collection; - the local Alloy HTTP listener and service hardening; - OTLP listener addresses and batching; +- the Redis address, namespace pattern, and scrape settings; +- Blackbox and SNMP scrape settings and the embedded SNMP module set; - Docker component internals; - raw or custom Alloy configuration. -The only backend settings available in Hiera are: +For normal production hosts, the only backend settings that admins need to +provide in Hiera are: ```yaml alloy::backend::username: 'assigned-writer' @@ -427,9 +586,10 @@ alloy::backend::password: > ENC[PKCS7,...] ``` -Keys such as `alloy::backend::tenant_id`, -`alloy::backend::metrics_url`, `alloy::backend::logs_url`, or -`alloy::backend::otlp_base_url` are not supported. +Development hosts may override all three backend URLs together as documented +in [Add Development Hosts to grafana-test](grafana_test.md). Partial test +overrides and destinations other than the centrally managed production and +test platforms are not supported. ## Troubleshooting diff --git a/docs/admin-guide/configuration/monitoring/grafana_test.md b/docs/admin-guide/configuration/monitoring/grafana_test.md new file mode 100644 index 00000000..46976ecb --- /dev/null +++ b/docs/admin-guide/configuration/monitoring/grafana_test.md @@ -0,0 +1,80 @@ +# Add Development Hosts to grafana-test + +Grafana Alloy sends telemetry to the production observability platform by +default. Individual development hosts can instead send their metrics, logs, +and any enabled traces to +[grafana-test.psi.ch](https://grafana-test.psi.ch). + +This changes the destination of the managed Alloy instance; it does not +install a second instance. Alloy still collects the complete local system +journal, so obtain approval from the environment owner before enrolling a +host. + +## Obtain test credentials + +Ask the observability platform operator at grafana@psi.ch for a writer username +and password for the required test tenant. Encrypt the password with EYAML as +described in the +[Hiera secrets guide](https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets). + +## Configure the development host + +Add the following values to the host's node-level YAML file in the +`data-` repository: + +```yaml +base::enable_alloy: true + +alloy::backend::metrics_url: 'https://mimir-test.psi.ch/api/v1/push' +alloy::backend::logs_url: 'https://loki-test.psi.ch/loki/api/v1/push' +alloy::backend::otlp_base_url: 'https://tempo-test.psi.ch' +alloy::backend::username: 'assigned-test-writer' +alloy::backend::password: > + ENC[PKCS7,...] +``` + +Override all three backend URLs together, even when the host does not +currently receive traces. This prevents one host's telemetry from being split +between the production and test platforms. + +Keep these overrides at node level unless every host in a broader Hiera scope +is intentionally a development host. + +## Apply and verify + +Run Puppet: + +```bash +sudo puppet agent -t +``` + +Check the service and validate the generated configuration: + +```bash +sudo systemctl status alloy --no-pager +sudo journalctl -u alloy --since '-10 min' --no-pager +sudo /usr/bin/alloy validate /etc/alloy/config.alloy +``` + +Then open [grafana-test.psi.ch](https://grafana-test.psi.ch) and look for the +host: + +```promql +node_uname_info{host=""} +``` + +```logql +{host=""} +``` + +The general component and label settings are documented in +[Grafana Alloy for environment admins](alloy.md). + +## Return the host to production + +Remove the three test backend URL overrides and the test credentials from the +node-level YAML file. Add production credentials there only when they are not +inherited from a broader Hiera scope, then run Puppet again. + +Do not set `base::enable_alloy: false` unless Alloy should be removed from the +host entirely. diff --git a/docs/services/gitea/_static/link_package_to_repository.png b/docs/services/gitea/_static/link_package_to_repository.png index 80f3826d..c2ff3766 100644 Binary files a/docs/services/gitea/_static/link_package_to_repository.png and b/docs/services/gitea/_static/link_package_to_repository.png differ