more Icinga2 internals

infrastructure-guide/icinga2.md

@@ -15,6 +15,14 @@ We want to support monitoring of the Linux machines in Icinga2. The Icinga2 infr
 
 ## Automated Host Configuration
 
+The Linux part of the Icinga2 Master configuration is manged using Ansible in the [`icinga_master` role in the `bootstrap` repo](https://git.psi.ch/linux-infra/bootstrap/-/tree/prod/ansible/roles/icinga_master).
+
+For Puppet managed nodes there is an automated import pipeline using the Director. For the central infrastructure itself there is a predefined Configuration Basket snapshot which is installed by manual Ansible run.
+
+Configuration which is shared and used by both type of systems are found in the [`awi-lx-basic` Director Configuration Basket](https://git.psi.ch/linux-infra/bootstrap/-/blob/prod/ansible/roles/icinga_master/files/etc/icingaweb2/psi/lx-core/Director-Basket_awi-lx-basic.json)
+
+### Puppet Managed Nodes
+
 The individual host configuration is automatically generated using already known information sources like
 - Hiera
 - Puppet Facts
@@ -22,7 +30,33 @@ The individual host configuration is automatically generated using already known
 
 TODO: diagram, details how this is achieved
 
+The Icinga2 Director import pipeline is provides as [Configuration Basket template `awi-lx-sysdb`](https://git.psi.ch/linux-infra/bootstrap/-/blob/prod/ansible/roles/icinga_master/templates/Director-Basket_awi-lx-sysdb.json)
+
+#### Import of Hiera Data to Sysdb
+
+#### Import of Puppet Facts to Sysdb
+
+#### Import of NetOps Data
+
+### Ansible Managed Central Infrastructure (e.g. Puppet Server)
+
+## Development of Director Import Pipeline
+
+The base are always the Configuration Basket snapshots (JSON files) which we have in Git. For changes either change them directly or change them in the Director web UI and then create a new snapshot of the according Configuration Basket, download it, modify if necessary:
+- if it is templated as for the Sysdb import pipeline
+- fix the definition of the Configuration Basket itself which is stringified JSON and should be plain JSON ([bug](https://github.com/Icinga/icingaweb2-module-director/issues/2774))
+and then commit it to the git repo.
+
+The rollout into production is then done with the bootstrap Ansible role for the Icinga2 Master nodes.
+Note that it will only attempt to import the Configuration Basket snapshot as provided from git when the file changes on disk. So if there is an failure during the import, best delete them on the Icinga2 Master:
+```
+rm /etc/icingaweb2/psi/lx-core/*
+```
+
+Further there is an issue with updated Sync Rules in the Configuration Basket snapshot. There is a [bug which makes their property list not updated on import](https://github.com/Icinga/icingaweb2-module-director/issues/2779). To work around you need to delete the Sync Rule manually in the Director UI. They cannot be deleted from shell with `icingacli director` ([feature request](https://github.com/Icinga/icingaweb2-module-director/issues/2706)).
+
+
+
 
-## Automated Setup
 
 

infrastructure-guide/infrastructure_systems.md

@@ -17,9 +17,15 @@ __Core Infrastructure:__
 
 __Additional Infrastructure__
 
-Monitoring:
+Sysdb Access:
+
 * [lxsup00](lxsup00) - 129.129.190.24 - Shell for linux support, primarily to run bob
 
+
+Monitoring:
+
+* [Icinga2](icinga2) - autmoatic integration into Icinga2
+
 * [influx00](influx00) - 129.129.190.225 - Influx database server
 
 * [metrics00](metrics00) - 129.129.190.226 - Grafana frontend for Influx