# Add Development Hosts to grafana-test

Grafana Alloy sends telemetry to the production observability platform by
default. Individual development hosts can instead send their metrics, logs,
and any enabled traces to
[grafana-test.psi.ch](https://grafana-test.psi.ch).

This changes the destination of the managed Alloy instance; it does not
install a second instance. Alloy still collects the complete local system
journal, so obtain approval from the environment owner before enrolling a
host.

## Obtain test credentials

Ask the observability platform operator at grafana@psi.ch for a writer username
and password for the required test tenant. Encrypt the password with EYAML as
described in the
[Hiera secrets guide](https://linux.psi.ch/documentation/admin-guide/puppet/hiera/#secrets).

## Configure the development host

Add the following values to the host's node-level YAML file in the
`data-<environment>` repository:

```yaml
base::enable_alloy: true

alloy::backend::metrics_url: 'https://mimir-test.psi.ch/api/v1/push'
alloy::backend::logs_url: 'https://loki-test.psi.ch/loki/api/v1/push'
alloy::backend::otlp_base_url: 'https://tempo-test.psi.ch'
alloy::backend::username: 'assigned-test-writer'
alloy::backend::password: >
  ENC[PKCS7,...]
```

Override all three backend URLs together, even when the host does not
currently receive traces. This prevents one host's telemetry from being split
between the production and test platforms.

Keep these overrides at node level unless every host in a broader Hiera scope
is intentionally a development host.

## Apply and verify

Run Puppet:

```bash
sudo puppet agent -t
```

Check the service and validate the generated configuration:

```bash
sudo systemctl status alloy --no-pager
sudo journalctl -u alloy --since '-10 min' --no-pager
sudo /usr/bin/alloy validate /etc/alloy/config.alloy
```

Then open [grafana-test.psi.ch](https://grafana-test.psi.ch) and look for the
host:

```promql
node_uname_info{host="<fqdn>"}
```

```logql
{host="<fqdn>"}
```

The general component and label settings are documented in
[Grafana Alloy for environment admins](alloy.md).

## Return the host to production

Remove the three test backend URL overrides and the test credentials from the
node-level YAML file. Add production credentials there only when they are not
inherited from a broader Hiera scope, then run Puppet again.

Do not set `base::enable_alloy: false` unless Alloy should be removed from the
host entirely.