forked from linux/documentation
3 files changed
+268
-28
No files matched your search
@@ -1,28 +1,32 @@
|
||||
# Grafana Alloy for environment admins
|
||||
|
||||
This guide explains how to enable Grafana Alloy through Hiera and which
|
||||
settings environment admins can change.
|
||||
settings environment admins can change.
|
||||
|
||||
Alloy sends telemetry to the central observability platform grafana-test.psi.ch:
|
||||
Alloy sends telemetry to the central production observability platform
|
||||
[grafana.psi.ch](https://grafana.psi.ch):
|
||||
|
||||
- host metrics to Mimir;
|
||||
- the complete local system journal to Loki;
|
||||
- optional traces to Tempo;
|
||||
- optional application and Docker telemetry.
|
||||
- optional application, Redis, Blackbox, SNMP, and Docker telemetry.
|
||||
|
||||
Host metrics and journal logs are always enabled when Alloy is enabled. Obtain approval from the environment owner
|
||||
before enrolling a host because journal entries can contain sensitive data.
|
||||
Host metrics and journal logs are always enabled when Alloy is enabled. Obtain
|
||||
approval from the environment owner before enrolling a host because journal
|
||||
entries can contain sensitive data.
|
||||
|
||||
The backend destination, package version, service settings, and basic Alloy
|
||||
configuration are managed centrally. Admins cannot redirect Alloy to another
|
||||
observability stack through Hiera.
|
||||
The production backend destinations, package version, service settings, and
|
||||
basic Alloy configuration are managed centrally. Development hosts can use the
|
||||
test observability platform by following
|
||||
[Add Development Hosts to grafana-test](grafana_test.md).
|
||||
|
||||
## Quick start
|
||||
|
||||
### 1. Obtain credentials
|
||||
|
||||
Ask the observability platform operator grafana@psi.ch for the Basic Auth username and
|
||||
password assigned to your environment or shared organization.
|
||||
Ask the observability platform operator grafana@psi.ch for the Basic Auth
|
||||
username and password assigned to your environment or shared organization.
|
||||
|
||||
### 2. Encrypt the password
|
||||
|
||||
https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets
|
||||
@@ -75,6 +79,13 @@ Add the following to the canary's node-level YAML file:
|
||||
base::enable_alloy: true
|
||||
```
|
||||
|
||||
<<<<<<< HEAD
|
||||
=======
|
||||
```bash
|
||||
sudo puppet agent -t
|
||||
```
|
||||
|
||||
>>>>>>> 7ec8b2f6 (update alloy docu, move grafana-test to different file)
|
||||
### 5. Verify the canary
|
||||
|
||||
Check the service and recent warnings:
|
||||
@@ -140,9 +151,9 @@ Enabling Alloy always collects:
|
||||
The code-owned baseline host collector selection is:
|
||||
|
||||
```text
|
||||
'cpu', 'diskstats', 'filefd', 'filesystem', 'loadavg', 'meminfo',
|
||||
'netclass', 'netdev', 'netstat', 'os', 'selinux', 'sockstat',
|
||||
'softnet', 'stat', 'time', 'timex', 'uname', 'vmstat', 'xfs'
|
||||
cpu, diskstats, filefd, filesystem, loadavg, meminfo, netclass,
|
||||
netdev, netstat, os, selinux, sockstat, softnet, stat, time, timex,
|
||||
uname, vmstat, xfs
|
||||
```
|
||||
|
||||
Alloy's own internal metrics are not forwarded.
|
||||
@@ -170,15 +181,13 @@ These values come from SysDB and cannot be replaced with Hiera labels.
|
||||
| `alloy::host_metrics::disable_collectors` | list of strings | `[]` | Remove collectors from the fixed selection. |
|
||||
| `alloy::host_metrics::enable_collectors` | list of strings | `[]` | Add collectors to the fixed selection. |
|
||||
|
||||
For example, this removes the pressure collector and adds XFS and InfiniBand
|
||||
metrics:
|
||||
For example, this removes the XFS collector and adds InfiniBand metrics:
|
||||
|
||||
```yaml
|
||||
alloy::host_metrics::disable_collectors:
|
||||
- 'pressure'
|
||||
- 'xfs'
|
||||
|
||||
alloy::host_metrics::enable_collectors:
|
||||
- 'xfs'
|
||||
- 'infiniband'
|
||||
```
|
||||
|
||||
@@ -223,7 +232,8 @@ sent. A `true` entry only promotes that stable unit name to the
|
||||
Use unit labels only for stable service names. Do not promote sessions,
|
||||
transient scopes, generated instance names, or other high-cardinality values.
|
||||
|
||||
The code-owned defaults promote Chrony, NetworkManager, SSH, and SSSD service units.
|
||||
The code-owned defaults promote Chrony, NetworkManager, SSH, and SSSD service
|
||||
units.
|
||||
|
||||
## Additional Prometheus endpoints
|
||||
|
||||
@@ -269,7 +279,8 @@ alloy::components:
|
||||
| `labels` | no | `{}` | Static labels for this endpoint's metrics. |
|
||||
|
||||
The endpoint name becomes its `job` and `target` label. Use a unique name and
|
||||
do not use the managed names `psi_host` or `psi_docker`.
|
||||
do not use the managed names `psi_host`, `psi_redis`, `blackbox`, `psi_snmp`,
|
||||
or `psi_docker`.
|
||||
|
||||
Disable an endpoint inherited from a broader Hiera level with:
|
||||
|
||||
@@ -284,6 +295,140 @@ Endpoint authentication, custom HTTP headers, and custom certificate
|
||||
authorities are not currently exposed. Prefer a loopback listener when the
|
||||
application supports it.
|
||||
|
||||
## Redis metrics
|
||||
|
||||
Redis metrics are disabled by default. The component connects to Redis on the
|
||||
same host at `localhost:6379`.
|
||||
|
||||
Enable it with:
|
||||
|
||||
```yaml
|
||||
alloy::components:
|
||||
redis:
|
||||
enable: true
|
||||
```
|
||||
|
||||
Redis-specific address, authentication, labels, and scrape settings are not
|
||||
configurable through Hiera. Alloy scrapes Redis every 60 seconds with a
|
||||
10-second timeout. The metric namespace is generated from the host's Puppet
|
||||
certificate name and ends in `_redis`.
|
||||
|
||||
Set `enable: false` at a more specific Hiera level to disable an inherited
|
||||
`true` value.
|
||||
|
||||
## Blackbox probes
|
||||
|
||||
Blackbox probing is disabled by default. Both the component and every target
|
||||
must be explicitly enabled.
|
||||
|
||||
The following example checks an SSH banner:
|
||||
|
||||
```yaml
|
||||
alloy::components:
|
||||
blackbox:
|
||||
enable: true
|
||||
modules:
|
||||
ssh_banner:
|
||||
prober: tcp
|
||||
timeout: '5s'
|
||||
tcp:
|
||||
query_response:
|
||||
- expect: '^SSH-2.0-'
|
||||
- send: "SSH-2.0-blackbox-ssh-check\r"
|
||||
targets:
|
||||
gpu001:
|
||||
enable: true
|
||||
address: 'gpu001.merlin7.psi.ch:22'
|
||||
module: 'ssh_banner'
|
||||
hostname: 'gpu001'
|
||||
labels:
|
||||
service: 'ssh'
|
||||
```
|
||||
|
||||
### Blackbox settings
|
||||
|
||||
| Field | Required | Default | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `enable` | no | `false` | Enable Blackbox probing on this Alloy host. |
|
||||
| `modules` | for enabled targets | `{}` | Native Blackbox exporter module definitions, keyed by module name. |
|
||||
| `targets` | for probes | `{}` | Named Blackbox targets. Each target is opt-in. |
|
||||
|
||||
### Blackbox target settings
|
||||
|
||||
| Field | Required | Default | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `enable` | no | `false` | Probe this named target. |
|
||||
| `address` | when enabled | none | Address expected by the selected module, such as `host:port` for TCP. |
|
||||
| `module` | when enabled | none | Name of a module defined under `modules`. |
|
||||
| `hostname` | when enabled | none | Hostname used for the target's `instance` label. |
|
||||
| `labels` | no | `{}` | Additional string labels for the probe metrics. |
|
||||
|
||||
The target map key identifies the Alloy target. The selected module becomes
|
||||
part of the `job` label, and `hostname` becomes the `instance` label. A target
|
||||
without `enable: true` is ignored. Disable an inherited target by setting its
|
||||
`enable` field to `false` at a more specific Hiera level. Enabled targets are
|
||||
scraped every 30 seconds.
|
||||
|
||||
Module definitions use the native Blackbox exporter format. Encrypt sensitive
|
||||
module values with EYAML as described in the
|
||||
[Hiera secrets guide](https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets).
|
||||
Never commit plaintext secrets.
|
||||
|
||||
## SNMP metrics
|
||||
|
||||
SNMP collection is disabled by default. Alloy includes the standard SNMP
|
||||
exporter modules and allows Hiera to define authentication profiles and named
|
||||
targets. Both the component and every target must be explicitly enabled.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
alloy::components:
|
||||
snmp:
|
||||
enable: true
|
||||
auths:
|
||||
site_v2:
|
||||
community: >
|
||||
ENC[PKCS7,...]
|
||||
version: 2
|
||||
targets:
|
||||
switch01:
|
||||
enable: true
|
||||
address: 'switch01.psi.ch'
|
||||
module: 'if_mib'
|
||||
auth: 'site_v2'
|
||||
labels:
|
||||
location: 'building-001'
|
||||
```
|
||||
|
||||
### SNMP settings
|
||||
|
||||
| Field | Required | Default | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `enable` | no | `false` | Enable SNMP collection on this Alloy host. |
|
||||
| `auths` | no | `{}` | Native SNMP exporter authentication profiles, keyed by profile name. |
|
||||
| `targets` | for collection | `{}` | Named SNMP targets. Each target is opt-in. |
|
||||
|
||||
### SNMP target settings
|
||||
|
||||
| Field | Required | Default | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `enable` | no | `false` | Collect metrics from this named target. |
|
||||
| `address` | when enabled | none | Hostname or address of the SNMP device. |
|
||||
| `module` | when enabled | none | Embedded SNMP exporter module or comma-separated modules to use. |
|
||||
| `auth` | when enabled | none | Authentication profile defined under `auths` or supplied by Alloy. |
|
||||
| `labels` | no | `{}` | Additional string labels for this target's metrics. |
|
||||
|
||||
The target map key becomes its `target` label, and its address becomes the
|
||||
`instance` label. A target without `enable: true` is ignored. Disable an
|
||||
inherited target by setting its `enable` field to `false`. Enabled targets are
|
||||
scraped every 60 seconds with a 50-second timeout.
|
||||
|
||||
Encrypt SNMP communities, authentication passwords, and privacy passwords
|
||||
with EYAML. Never commit them as plaintext. Custom SNMP modules, walk
|
||||
parameters, contexts, and scrape settings are not currently exposed through
|
||||
Hiera.
|
||||
|
||||
## Docker metrics and logs
|
||||
|
||||
Docker telemetry is disabled by default. Enable it with:
|
||||
@@ -350,15 +495,15 @@ alloy::labels:
|
||||
lifecycle: 'production'
|
||||
```
|
||||
|
||||
These labels apply to host metrics, additional Prometheus metrics, journal
|
||||
logs, and Docker logs. They do not apply to traces.
|
||||
These labels apply to host, Redis, Blackbox, SNMP, and additional Prometheus
|
||||
metrics as well as journal and Docker logs. They do not apply to traces.
|
||||
|
||||
Use labels only for stable, low-cardinality values. Do not use usernames,
|
||||
request IDs, timestamps, container IDs, or other values that continually
|
||||
change.
|
||||
|
||||
More specific label options are available for journal logs and individual
|
||||
Prometheus endpoints:
|
||||
More specific label options are available for journal logs, individual
|
||||
Prometheus endpoints, Blackbox targets, and SNMP targets:
|
||||
|
||||
```yaml
|
||||
alloy::journal::labels:
|
||||
@@ -386,8 +531,15 @@ Admin-managed Hiera data is applied from most specific to least specific:
|
||||
3. group;
|
||||
4. Hiera environment.
|
||||
|
||||
<<<<<<< HEAD
|
||||
Scalar settings use the first value found, so the most-specific value takes precedence.
|
||||
The enable and disable collector lists use Hiera’s unique array merge. Values from node, subgroup, group, and Hiera environment levels are combined, with duplicates removed.
|
||||
=======
|
||||
Scalar settings use the first value found, so the most-specific value takes
|
||||
precedence. The enable and disable collector lists use Hiera's unique array
|
||||
merge. Values from node, subgroup, group, and SysDB environment levels are
|
||||
combined, with duplicates removed.
|
||||
>>>>>>> 7ec8b2f6 (update alloy docu, move grafana-test to different file)
|
||||
|
||||
The following maps deep-merge across the hierarchy:
|
||||
|
||||
@@ -405,21 +557,28 @@ rendering or validating the Alloy configuration, but an unused unknown key can
|
||||
be ignored. For example, `scrape_intervall` is ignored and leaves the default
|
||||
`60s` interval in effect.
|
||||
|
||||
Write Boolean values as the unquoted YAML values `true` and `false`. Quoted
|
||||
values are strings, not Booleans.
|
||||
|
||||
## Settings that admins cannot change
|
||||
|
||||
The following are owned by the Puppet code:
|
||||
|
||||
- the Mimir, Loki, and Tempo destinations;
|
||||
- the production Mimir, Loki, and Tempo destinations, except for the
|
||||
documented development-host test override;
|
||||
- the Alloy package version;
|
||||
- the baseline host collector list, which admins can adjust with the documented
|
||||
enable and disable arrays;
|
||||
- mandatory host metric and journal collection;
|
||||
- the local Alloy HTTP listener and service hardening;
|
||||
- OTLP listener addresses and batching;
|
||||
- the Redis address, namespace pattern, and scrape settings;
|
||||
- Blackbox and SNMP scrape settings and the embedded SNMP module set;
|
||||
- Docker component internals;
|
||||
- raw or custom Alloy configuration.
|
||||
|
||||
The only backend settings available in Hiera are:
|
||||
For normal production hosts, the only backend settings that admins need to
|
||||
provide in Hiera are:
|
||||
|
||||
```yaml
|
||||
alloy::backend::username: 'assigned-writer'
|
||||
@@ -427,9 +586,10 @@ alloy::backend::password: >
|
||||
ENC[PKCS7,...]
|
||||
```
|
||||
|
||||
Keys such as `alloy::backend::tenant_id`,
|
||||
`alloy::backend::metrics_url`, `alloy::backend::logs_url`, or
|
||||
`alloy::backend::otlp_base_url` are not supported.
|
||||
Development hosts may override all three backend URLs together as documented
|
||||
in [Add Development Hosts to grafana-test](grafana_test.md). Partial test
|
||||
overrides and destinations other than the centrally managed production and
|
||||
test platforms are not supported.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# Add Development Hosts to grafana-test
|
||||
|
||||
Grafana Alloy sends telemetry to the production observability platform by
|
||||
default. Individual development hosts can instead send their metrics, logs,
|
||||
and any enabled traces to
|
||||
[grafana-test.psi.ch](https://grafana-test.psi.ch).
|
||||
|
||||
This changes the destination of the managed Alloy instance; it does not
|
||||
install a second instance. Alloy still collects the complete local system
|
||||
journal, so obtain approval from the environment owner before enrolling a
|
||||
host.
|
||||
|
||||
## Obtain test credentials
|
||||
|
||||
Ask the observability platform operator at grafana@psi.ch for a writer username
|
||||
and password for the required test tenant. Encrypt the password with EYAML as
|
||||
described in the
|
||||
[Hiera secrets guide](https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets).
|
||||
|
||||
## Configure the development host
|
||||
|
||||
Add the following values to the host's node-level YAML file in the
|
||||
`data-<environment>` repository:
|
||||
|
||||
```yaml
|
||||
base::enable_alloy: true
|
||||
|
||||
alloy::backend::metrics_url: 'https://mimir-test.psi.ch/api/v1/push'
|
||||
alloy::backend::logs_url: 'https://loki-test.psi.ch/loki/api/v1/push'
|
||||
alloy::backend::otlp_base_url: 'https://tempo-test.psi.ch'
|
||||
alloy::backend::username: 'assigned-test-writer'
|
||||
alloy::backend::password: >
|
||||
ENC[PKCS7,...]
|
||||
```
|
||||
|
||||
Override all three backend URLs together, even when the host does not
|
||||
currently receive traces. This prevents one host's telemetry from being split
|
||||
between the production and test platforms.
|
||||
|
||||
Keep these overrides at node level unless every host in a broader Hiera scope
|
||||
is intentionally a development host.
|
||||
|
||||
## Apply and verify
|
||||
|
||||
Run Puppet:
|
||||
|
||||
```bash
|
||||
sudo puppet agent -t
|
||||
```
|
||||
|
||||
Check the service and validate the generated configuration:
|
||||
|
||||
```bash
|
||||
sudo systemctl status alloy --no-pager
|
||||
sudo journalctl -u alloy --since '-10 min' --no-pager
|
||||
sudo /usr/bin/alloy validate /etc/alloy/config.alloy
|
||||
```
|
||||
|
||||
Then open [grafana-test.psi.ch](https://grafana-test.psi.ch) and look for the
|
||||
host:
|
||||
|
||||
```promql
|
||||
node_uname_info{host="<fqdn>"}
|
||||
```
|
||||
|
||||
```logql
|
||||
{host="<fqdn>"}
|
||||
```
|
||||
|
||||
The general component and label settings are documented in
|
||||
[Grafana Alloy for environment admins](alloy.md).
|
||||
|
||||
## Return the host to production
|
||||
|
||||
Remove the three test backend URL overrides and the test credentials from the
|
||||
node-level YAML file. Add production credentials there only when they are not
|
||||
inherited from a broader Hiera scope, then run Puppet again.
|
||||
|
||||
Do not set `base::enable_alloy: false` unless Alloy should be removed from the
|
||||
host entirely.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 67 KiB After Width: | Height: | Size: 130 B |
Reference in new issue
Block a user