forked from linux/documentation
documentation fragment for alloy
Signed-off-by: Basil Bruhn <basil.bruhn@psi.ch>
This commit is contained in:
1 file changed
+461
@@ -0,0 +1,461 @@
|
||||
# Grafana Alloy for environment admins
|
||||
|
||||
This guide explains how to enable Grafana Alloy through Hiera and which
|
||||
settings environment admins can change.
|
||||
|
||||
Alloy sends telemetry to the central observability platform grafana.psi.ch:
|
||||
|
||||
- host metrics to Mimir;
|
||||
- the complete local system journal to Loki;
|
||||
- optional traces to Tempo;
|
||||
- optional application and Docker telemetry.
|
||||
|
||||
Host metrics and journal logs are always enabled when Alloy is enabled. Obtain approval from the environment owner
|
||||
before enrolling a host because journal entries can contain sensitive data.
|
||||
|
||||
The backend destination, package version, service settings, and basic Alloy
|
||||
configuration are managed centrally. Admins cannot redirect Alloy to another
|
||||
observability stack through Hiera.
|
||||
|
||||
## Quick start
|
||||
|
||||
### 1. Obtain credentials
|
||||
|
||||
Ask the observability platform operator grafana@psi.ch for the Basic Auth username and
|
||||
password assigned to your environment or shared organization.
|
||||
### 2. Encrypt the password
|
||||
|
||||
https://linux.psi.ch/documentation/admin-guide/puppet/hiera/?h=eyaml#secrets
|
||||
|
||||
Encrypt the password with the public key from the same repository:
|
||||
|
||||
```bash
|
||||
eyaml encrypt \
|
||||
--pkcs7-public-key ./eyaml_public_key.pem \
|
||||
--password \
|
||||
--output block
|
||||
```
|
||||
|
||||
The command prompts for the password so it does not enter the shell history.
|
||||
Never commit the plaintext password.
|
||||
|
||||
### 3. Add the credentials to the environment
|
||||
|
||||
Add the assigned username and encrypted password to `<environment>.yaml` in
|
||||
the `data-<environment>` repository:
|
||||
|
||||
```yaml
|
||||
alloy::backend::username: 'assigned-writer'
|
||||
alloy::backend::password: >
|
||||
ENC[PKCS7,...]
|
||||
```
|
||||
|
||||
Use the EYAML output without modifying the encrypted content. A value
|
||||
encrypted with another environment's public key cannot be decrypted by this
|
||||
environment's Puppet server key.
|
||||
|
||||
### 4. Enable one canary host
|
||||
|
||||
Add the following to the canary's node-level YAML file:
|
||||
|
||||
```yaml
|
||||
base::enable_alloy: true
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo puppet agent -t --environment alloy
|
||||
```
|
||||
|
||||
### 5. Verify the canary
|
||||
|
||||
Check the service and recent warnings:
|
||||
|
||||
```bash
|
||||
sudo systemctl status alloy --no-pager
|
||||
sudo journalctl -u alloy --since '-10 min' --no-pager
|
||||
sudo /usr/bin/alloy validate /etc/alloy/config.alloy
|
||||
```
|
||||
|
||||
Alloy logs at `warn`, so a healthy service can have very little journal output.
|
||||
|
||||
In Grafana, check for the host with queries such as:
|
||||
|
||||
```promql
|
||||
node_uname_info{host="<fqdn>"}
|
||||
```
|
||||
|
||||
```logql
|
||||
{host="<fqdn>"}
|
||||
```
|
||||
|
||||
Metrics normally appear after the next scrape, although backend delivery can
|
||||
add more delay. Journal logs are forwarded continuously.
|
||||
|
||||
### 6. Roll out the environment
|
||||
|
||||
After the canary is accepted, move the enable setting to
|
||||
`<environment>.yaml`:
|
||||
|
||||
```yaml
|
||||
base::enable_alloy: true
|
||||
```
|
||||
|
||||
This enrolls every eligible host in the SysDB environment unless a more
|
||||
specific group, subgroup, or node file overrides the value.
|
||||
|
||||
RHEL 7 hosts ignore Alloy enrollment. The current repository supports RHEL 8,
|
||||
9, and (10).
|
||||
|
||||
## Enable, disable, or leave unmanaged
|
||||
|
||||
`base::enable_alloy` has three states:
|
||||
|
||||
| Value | Result |
|
||||
| --- | --- |
|
||||
| `true` | Install, configure, enable, and run Alloy. |
|
||||
| `false` | Stop and disable Alloy, then remove its package and managed configuration. |
|
||||
| absent or `null` | Do not manage Alloy. This does **not** stop an existing installation. |
|
||||
|
||||
Use `false` to remove Alloy from a host or scope. Do not delete a previous
|
||||
`true` setting and assume that Alloy will stop. Puppet deliberately does not
|
||||
remove `/var/lib/alloy` and does not explicitly remove the service user or
|
||||
directories; package uninstall behavior can still vary.
|
||||
|
||||
## Fixed baseline
|
||||
|
||||
Enabling Alloy always collects:
|
||||
|
||||
- Unix host metrics;
|
||||
- every entry from the local system journal.
|
||||
|
||||
The code-owned baseline host collector selection is:
|
||||
|
||||
```text
|
||||
cpu, diskstats, filefd, filesystem, loadavg, meminfo, netclass,
|
||||
netdev, netstat, os, pressure, sockstat, stat, time, timex,
|
||||
uname, vmstat
|
||||
```
|
||||
|
||||
Alloy's own internal metrics are not forwarded.
|
||||
|
||||
Every metric and log entry receives centrally derived Puppet and SysDB
|
||||
identity where applicable:
|
||||
|
||||
- `host`;
|
||||
- `puppet_group`;
|
||||
- `puppet_role`;
|
||||
- `puppet_subgroup`;
|
||||
- `sysdb_env`.
|
||||
|
||||
These values come from SysDB and cannot be replaced with Hiera labels.
|
||||
|
||||
## Host metrics
|
||||
|
||||
### Available settings
|
||||
|
||||
| Hiera key | Type | Default | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `alloy::host_metrics::scrape_interval` | duration string | `'60s'` | How often host metrics are collected. |
|
||||
| `alloy::host_metrics::scrape_timeout` | duration string | `'10s'` | Maximum duration of one host scrape. |
|
||||
| `alloy::host_metrics::include_exporter_metrics` | Boolean | `false` | Include operational metrics from the embedded Unix exporter. |
|
||||
| `alloy::host_metrics::disable_collectors` | list of strings | `[]` | Remove collectors from the fixed selection. |
|
||||
| `alloy::host_metrics::enable_collectors` | list of strings | `[]` | Add collectors to the fixed selection. |
|
||||
|
||||
For example, this removes the pressure collector and adds XFS and InfiniBand
|
||||
metrics:
|
||||
|
||||
```yaml
|
||||
alloy::host_metrics::disable_collectors:
|
||||
- 'pressure'
|
||||
|
||||
alloy::host_metrics::enable_collectors:
|
||||
- 'xfs'
|
||||
- 'infiniband'
|
||||
```
|
||||
|
||||
Collector names are passed to the Alloy Unix exporter. Puppet does not
|
||||
maintain a separate collector allowlist. Avoid placing the same collector in
|
||||
both lists.
|
||||
|
||||
The collector lists do not merge across Hiera levels. A node-level list
|
||||
replaces the complete environment-, group-, or subgroup-level list.
|
||||
|
||||
Setting `include_exporter_metrics` does not enable Alloy's own internal
|
||||
metrics. It only controls metrics produced by the embedded Unix exporter.
|
||||
|
||||
## Journal logs
|
||||
|
||||
Journal collection cannot be disabled while Alloy is enabled.
|
||||
|
||||
### Available settings
|
||||
|
||||
| Hiera key | Type | Default | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `alloy::journal::max_age` | duration string | `'1h'` | Do not read journal entries older than this limit. |
|
||||
| `alloy::journal::format_as_json` | Boolean | `false` | Send the journal entry as JSON instead of only its message. |
|
||||
| `alloy::journal::labels` | string map | `{}` | Add static labels to journal entries. |
|
||||
| `alloy::journal::unit_labels` | Boolean map | central defaults | Promote selected systemd units to a Loki label. |
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
alloy::journal::max_age: '30m'
|
||||
alloy::journal::format_as_json: false
|
||||
|
||||
alloy::journal::labels:
|
||||
application_area: 'controls'
|
||||
|
||||
alloy::journal::unit_labels:
|
||||
'my-application.service': true
|
||||
'docker.service': false
|
||||
```
|
||||
|
||||
`unit_labels` does not select or filter logs. All journal entries are still
|
||||
sent. A `true` entry only promotes that stable unit name to the
|
||||
`systemd_unit` Loki label. A `false` entry removes an inherited promotion.
|
||||
|
||||
Use unit labels only for stable service names. Do not promote sessions,
|
||||
transient scopes, generated instance names, or other high-cardinality values.
|
||||
|
||||
The common configuration already promotes several frequently used services,
|
||||
including Alloy, Chrony, Docker, Gitea, HTTP servers, SSH, and Slurm services.
|
||||
|
||||
## Additional Prometheus endpoints
|
||||
|
||||
Use a named Prometheus endpoint for an application that exposes Prometheus or
|
||||
OpenMetrics data over HTTP.
|
||||
|
||||
Minimal example:
|
||||
|
||||
```yaml
|
||||
alloy::components:
|
||||
prometheus_endpoints:
|
||||
gitea:
|
||||
enable: true
|
||||
address: '127.0.0.1:3000'
|
||||
```
|
||||
|
||||
Complete example:
|
||||
|
||||
```yaml
|
||||
alloy::components:
|
||||
prometheus_endpoints:
|
||||
application_api:
|
||||
enable: true
|
||||
address: '127.0.0.1:9400'
|
||||
scheme: 'http'
|
||||
metrics_path: '/internal/metrics'
|
||||
scrape_interval: '30s'
|
||||
scrape_timeout: '10s'
|
||||
labels:
|
||||
service: 'application-api'
|
||||
```
|
||||
|
||||
### Endpoint settings
|
||||
|
||||
| Field | Required | Default | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `enable` | yes | none | Enable or disable this named endpoint. |
|
||||
| `address` | when enabled | none | Target in `host:port` form, not a complete URL. |
|
||||
| `scheme` | no | `'http'` | `http` or `https`. |
|
||||
| `metrics_path` | no | `'/metrics'` | HTTP path containing the metrics. |
|
||||
| `scrape_interval` | no | `'60s'` | How often to scrape this endpoint. |
|
||||
| `scrape_timeout` | no | `'10s'` | Maximum duration of one scrape. |
|
||||
| `labels` | no | `{}` | Static labels for this endpoint's metrics. |
|
||||
|
||||
The endpoint name becomes its `job` and `target` label. Use a unique name and
|
||||
do not use the managed names `psi_host` or `psi_docker`.
|
||||
|
||||
Disable an endpoint inherited from a broader Hiera level with:
|
||||
|
||||
```yaml
|
||||
alloy::components:
|
||||
prometheus_endpoints:
|
||||
gitea:
|
||||
enable: false
|
||||
```
|
||||
|
||||
Endpoint authentication, custom HTTP headers, and custom certificate
|
||||
authorities are not currently exposed. Prefer a loopback listener when the
|
||||
application supports it.
|
||||
|
||||
## Docker metrics and logs
|
||||
|
||||
Docker telemetry is disabled by default. Enable it with:
|
||||
|
||||
```yaml
|
||||
alloy::components:
|
||||
docker:
|
||||
enable: true
|
||||
```
|
||||
|
||||
Set `enable: false` at a more specific Hiera level to disable an inherited
|
||||
`true` value.
|
||||
|
||||
This single switch enables both:
|
||||
|
||||
- cAdvisor metrics for local containers;
|
||||
- logs from all local containers through the Docker socket.
|
||||
|
||||
Metrics and logs cannot currently be enabled separately. The host must have a
|
||||
working Docker socket and `docker` group. The managed systemd service receives
|
||||
`SupplementaryGroups=docker` while this component is enabled; Puppet does not
|
||||
revoke any Unix group membership configured elsewhere.
|
||||
|
||||
Access to the Docker socket is effectively privileged. Enable this component
|
||||
only on approved Docker hosts. Runtime access can also depend on the storage
|
||||
driver, cgroup setup, and SELinux policy.
|
||||
|
||||
If Docker uses the journald logging driver, the same container output can be
|
||||
collected once through the mandatory journal source and again through the
|
||||
Docker log source.
|
||||
|
||||
## OTLP traces
|
||||
|
||||
Trace reception is disabled by default. Enable it with:
|
||||
|
||||
```yaml
|
||||
alloy::components:
|
||||
otlp_traces:
|
||||
enable: true
|
||||
```
|
||||
|
||||
Set `enable: false` at a more specific Hiera level to disable an inherited
|
||||
`true` value.
|
||||
|
||||
Applications on the same host can then send traces to:
|
||||
|
||||
- OTLP/gRPC on `127.0.0.1:4317`;
|
||||
- OTLP/HTTP on `127.0.0.1:4318`.
|
||||
|
||||
The receivers are loopback-only and accept traces only. Receiver addresses,
|
||||
batch settings, authentication, and sampling are not configurable through
|
||||
Hiera. Applications are responsible for trace sampling.
|
||||
|
||||
Alloy adds the centrally derived host and environment resource attributes.
|
||||
The general `alloy::labels` map does not apply to traces.
|
||||
|
||||
## Additional labels
|
||||
|
||||
Add stable labels shared by metrics and logs with:
|
||||
|
||||
```yaml
|
||||
alloy::labels:
|
||||
application: 'gitea'
|
||||
lifecycle: 'production'
|
||||
```
|
||||
|
||||
These labels apply to host metrics, additional Prometheus metrics, journal
|
||||
logs, and Docker logs. They do not apply to traces.
|
||||
|
||||
Use labels only for stable, low-cardinality values. Do not use usernames,
|
||||
request IDs, timestamps, container IDs, or other values that continually
|
||||
change.
|
||||
|
||||
More specific label options are available for journal logs and individual
|
||||
Prometheus endpoints:
|
||||
|
||||
```yaml
|
||||
alloy::journal::labels:
|
||||
log_source: 'operating-system'
|
||||
|
||||
alloy::components:
|
||||
prometheus_endpoints:
|
||||
application_api:
|
||||
enable: true
|
||||
address: '127.0.0.1:9400'
|
||||
labels:
|
||||
service: 'application-api'
|
||||
```
|
||||
|
||||
The centrally derived Puppet/SysDB identity and fixed component labels take
|
||||
precedence over conflicting admin-provided labels. A static `systemd_unit`
|
||||
journal label is discarded; use `alloy::journal::unit_labels` instead.
|
||||
|
||||
## Hiera priority and merging
|
||||
|
||||
Admin-managed Hiera data is applied from most specific to least specific:
|
||||
|
||||
1. node;
|
||||
2. subgroup;
|
||||
3. group;
|
||||
4. SysDB environment.
|
||||
|
||||
Scalar values and collector lists use the first value found. A more specific
|
||||
value therefore replaces the broader value or complete list.
|
||||
|
||||
The following maps deep-merge across the hierarchy:
|
||||
|
||||
- `alloy::components`;
|
||||
- `alloy::labels`;
|
||||
- `alloy::journal::labels`;
|
||||
- `alloy::journal::unit_labels`.
|
||||
|
||||
This makes it possible to define common environment settings and modify one
|
||||
component, label, or systemd unit for a group or host.
|
||||
|
||||
Use the option names exactly as shown in this guide. Puppet intentionally does
|
||||
not validate unknown component or field names. Some mistakes fail while
|
||||
rendering or validating the Alloy configuration, but an unused unknown key can
|
||||
be ignored. For example, `scrape_intervall` is ignored and leaves the default
|
||||
`60s` interval in effect.
|
||||
|
||||
## Settings that admins cannot change
|
||||
|
||||
The following are owned by the Puppet code:
|
||||
|
||||
- the Mimir, Loki, and Tempo destinations;
|
||||
- the Alloy package version;
|
||||
- the baseline host collector list, which admins can adjust with the documented
|
||||
enable and disable arrays;
|
||||
- mandatory host metric and journal collection;
|
||||
- the local Alloy HTTP listener and service hardening;
|
||||
- OTLP listener addresses and batching;
|
||||
- Docker component internals;
|
||||
- raw or custom Alloy configuration.
|
||||
|
||||
The only backend settings available in Hiera are:
|
||||
|
||||
```yaml
|
||||
alloy::backend::username: 'assigned-writer'
|
||||
alloy::backend::password: >
|
||||
ENC[PKCS7,...]
|
||||
```
|
||||
|
||||
Keys such as `alloy::backend::tenant_id`,
|
||||
`alloy::backend::metrics_url`, `alloy::backend::logs_url`, or
|
||||
`alloy::backend::otlp_base_url` are not supported.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Alloy configuration validation failed
|
||||
|
||||
Puppet validates the generated configuration before replacing the active
|
||||
file. Check addresses, duration strings, collector names, and component names
|
||||
in the Hiera change. You can validate the installed configuration manually:
|
||||
|
||||
```bash
|
||||
sudo /usr/bin/alloy validate /etc/alloy/config.alloy
|
||||
```
|
||||
|
||||
### Docker component does not start
|
||||
|
||||
Confirm that the host has a `docker` group, that Docker is running, and that
|
||||
`/var/run/docker.sock` exists. Then inspect:
|
||||
|
||||
```bash
|
||||
sudo journalctl -u alloy --since '-10 min' --no-pager
|
||||
```
|
||||
|
||||
### No data in Grafana
|
||||
|
||||
Confirm that Alloy is active and look for authentication or remote-write
|
||||
warnings:
|
||||
|
||||
```bash
|
||||
sudo systemctl is-active alloy
|
||||
sudo journalctl -u alloy --since '-30 min' --no-pager
|
||||
```
|
||||
|
||||
Also verify that the assigned Basic Auth username is mapped to the intended
|
||||
tenant by the observability platform operator.
|
||||
Reference in new issue
Block a user