document how host data gets imported

infrastructure-guide/icinga2.md

@@ -23,23 +23,64 @@ Configuration which is shared and used by both type of systems are found in the
 
 ### Puppet Managed Nodes
 
-The individual host configuration is automatically generated using already known information sources like
-- Sysdb
-- Hiera
-- Puppet Facts
-- NetOps
+The individual host configuration is automatically generated using already known information from
+- Sysdb (inventory of nodes)
+- Hiera (configuration and customization of the nodes)
+- Puppet Facts (OS version, attached networks, partitions)
+- NetOps (security levels of networks, needed for selecting the correct Icinga2 Satellite)
 
 ![high level idea of Linux computer data import to Icinga2](icinga2/icinga2_import_big_picture.png)
 
-The Icinga Director import pipeline is provides as [Configuration Basket template `awi-lx-sysdb`](https://git.psi.ch/linux-infra/bootstrap/-/blob/prod/ansible/roles/icinga_master/templates/Director-Basket_awi-lx-sysdb.json)
-
 #### Import of Hiera Data to Sysdb
 
+TODO
+
 #### Import of Puppet Facts to Sysdb
 
+TODO
+
 #### Import of NetOps Data
 
+TODO
+
+#### Import into Director
+What in the overview diagram is one arrow, is a bit more complicated when implemented:
+
+![high level idea of Linux computer data import to Icinga2](icinga2/icinga2_import.png)
+
+It contains two parts:
+- the import of host information (a bit simpler)
+- the import of notification users (a bit more complex).
+
+The import as such is triggered every 10 minutes by the `sysdb-director-import` timer.
+
+To avoid interference with manual Director change operations the import script checks first if there are no undeployed objects. If so it it will not run.
+
+A Director import pipeline needs first a Import Source which imports the data from an external source. The second element is the Sync Rule which then creates actual Director objects out of the data provided by one or more Import Sources
+
+##### Director Import of Host Information
+
+The Icinga Director import pipeline is provides as [Configuration Basket template `awi-lx-sysdb`](https://git.psi.ch/linux-infra/bootstrap/-/blob/prod/ansible/roles/icinga_master/templates/Director-Basket_awi-lx-sysdb.json).
+
+The pipeline for host groups imports the list of sysdb host groups and the filter to select the hosts where the group name is listed in the `host.vars.host_groups` array of the host.
+
+The host import contains all the special configuration for a host. But there we need two sync rules, where `awi-lx-sysdb-host-sync-rule` creates and fills the host object.
+
+The second `awi-lx-sysdb-service-override-sync-rule` just fills `host.vars._override_servicevars` which the Director does not touch in the normal import.
+The reason is that this variable stores usually manual serice override changes made in Director, an those should survive normal imports. But as we also manage these in Hiera, we need a `Update only` sync rule which allows also to set this variable.
+
+##### Director Import of Notification Users
+Here the actual user information comes from the AD. But as we do not import all users, but only those who want to be notified, we also need to modify what is being imported.
+
+An LDAP filter selects these users specificly. The filter has three sections:
+- users we do not want (service users, etc)
+- departments we want
+- notification users for Linux machines
+
+The LDAP filter is created on one side out of `ignore_list.conf` and `department_list.conf` which are managed by AIT/Icinga2 Team. The second part is read from the Sysdb API (`user_list.conf`). Together the Configuration Basket `Director-Basket_icinga-notification-user-import.json` which contains the full notification user import pipeline (import source `icinga-notification-user-import` and sync rule `icinga-notification-user-sync-rule` which also get updated always on an import run, before they are then actually triggered.
+
 ### Ansible Managed Central Infrastructure (e.g. Puppet Server)
+TODO
 
 ## Development of Icinga Director Import Pipeline
 
@@ -67,6 +108,7 @@ The Icinga2 infrastructure is maintained and prepared by AIT. Following items ne
   ```
 - in `roles.ini` have a `Generic User Role` with read/monitoring-only permissions
 - the `/etc/icingaweb2/psi/merge-roles-ini.py` script provided by AIT to be able to merge in roles via Ansible/Sysdb API
+- access to the AD with and LDAP import source named `LDAP PSI`
 
 From our side we need the following manual setup
 - prepare the Scheduled Downtime `Generic Linux Alert Suppression` (cannot be imported with Configuration Basket, see [feature request](https://github.com/Icinga/icingaweb2-module-director/issues/2795)) with
@@ -94,5 +136,3 @@ From our side we need the following manual setup
   ```
   ansible-playbook -i inventory.yaml -i inventory_dmz.yaml --vault-pass-file ./vault-pass prepare_icinga_master.yaml
   ```
-
-

infrastructure-guide/icinga2/icinga2_import.puml

@@ -1,7 +1,7 @@
 @startuml
 
 node "Icinga2 Master" as master {
-    control "sysdb-director-import.timer" as import_timer
+    control "= sysdb-director-import.timer" as import_timer #line.bold
     file "sysdb-director-import" as import_script
     component "Director" as director {
         component "awi-lx-sysdb-hostgroup-import" as hostgroup_import
@@ -9,24 +9,51 @@ node "Icinga2 Master" as master {
         component "awi-lx-sysdb-host-import" as host_import
         component "awi-lx-sysdb-host-sync-rule" as host_sync
         component "awi-lx-sysdb-service-override-sync-rule" as service_override_sync
+        component "icinga-notification-user-import" as user_import
+        component "icinga-notification-user-sync-rule" as user_sync
     }
+    together {
+        file "user_and_group_list.conf" as user_and_group_list
+        file "department_list.conf" as department_list
+        file "ignore_list.conf" as ignore_list
+    }
+    file "generate_icinga_notification_user_import_basket.py" as notification_user_basket_generator
+    file "Director-Basket_icinga-notification-user-import.json" as notification_user_basket
+    note top of notification_user_basket : adapted LDAP filter
+    ' make it stand out, = is for large text
+    folder "= Icinga2 configuration" as icinga2_configuration #line.bold
 }
 node "Sysdb Server" as sysdb_server {
     component "Sysdb API" as sysdb_api {
-        component "icinga2/v1/roles" as sysdb_api_roles
+        ' component "icinga2/v1/roles" as sysdb_api_roles
         component "icinga2/v1/hostgroups" as sysdb_api_hostgroups
         component "icinga2/v1/hosts" as sysdb_api_hosts
         component "icinga2/v1/ad_notification_users" as sysdb_api_roles_ad_notification_users
     }
 }
-import_timer -[dashed]-> import_script :  "triggers"
-import_script -[dashed]-> director : triggers imports, sync rules and deployment
+database "AD" as ad
 
-sysdb_api_hostgroups --> hostgroup_import
+import_timer -[dashed]down-> import_script :  "triggers"
+import_script -[dashed]down-> director : triggers imports, sync rules and deployment
+import_script -[dashed]up-> user_and_group_list : downloads
+import_script -[dashed]right-> notification_user_basket_generator : runs
+
+sysdb_api_hostgroups ==> hostgroup_import
 hostgroup_import --> hostgroup_sync
 
-sysdb_api_hosts --> host_import
+sysdb_api_hosts ==> host_import
 host_import --> host_sync
 host_import --> service_override_sync
 
+sysdb_api_roles_ad_notification_users ==> user_and_group_list
+notification_user_basket_generator -[dashed]-> notification_user_basket : generates
+user_and_group_list --> notification_user_basket
+department_list --> notification_user_basket
+ignore_list --> notification_user_basket
+notification_user_basket --> user_import
+ad ==> user_import
+user_import --> user_sync
+
+director ==> icinga2_configuration
+
 @enduml