2.4 KiB
Add Development Hosts to grafana-test
Grafana Alloy sends telemetry to the production observability platform by default. Individual development hosts can instead send their metrics, logs, and any enabled traces to grafana-test.psi.ch.
This changes the destination of the managed Alloy instance; it does not install a second instance. Alloy still collects the complete local system journal, so obtain approval from the environment owner before enrolling a host.
Obtain test credentials
Ask the observability platform operator at grafana@psi.ch for a writer username and password for the required test tenant. Encrypt the password with EYAML as described in the Hiera secrets guide.
Configure the development host
Add the following values to the host's node-level YAML file in the
data-<environment> repository:
base::enable_alloy: true
alloy::backend::metrics_url: 'https://mimir-test.psi.ch/api/v1/push'
alloy::backend::logs_url: 'https://loki-test.psi.ch/loki/api/v1/push'
alloy::backend::otlp_base_url: 'https://tempo-test.psi.ch'
alloy::backend::username: 'assigned-test-writer'
alloy::backend::password: >
ENC[PKCS7,...]
Override all three backend URLs together, even when the host does not currently receive traces. This prevents one host's telemetry from being split between the production and test platforms.
Keep these overrides at node level unless every host in a broader Hiera scope is intentionally a development host.
Apply and verify
Run Puppet:
sudo puppet agent -t
Check the service and validate the generated configuration:
sudo systemctl status alloy --no-pager
sudo journalctl -u alloy --since '-10 min' --no-pager
sudo /usr/bin/alloy validate /etc/alloy/config.alloy
Then open grafana-test.psi.ch and look for the host:
node_uname_info{host="<fqdn>"}
{host="<fqdn>"}
The general component and label settings are documented in Grafana Alloy for environment admins.
Return the host to production
Remove the three test backend URL overrides and the test credentials from the node-level YAML file. Add production credentials there only when they are not inherited from a broader Hiera scope, then run Puppet again.
Do not set base::enable_alloy: false unless Alloy should be removed from the
host entirely.