From 328334d2ba6f6fb3a76a9b47bb5636b372a6892a Mon Sep 17 00:00:00 2001 From: ebner Date: Thu, 8 Aug 2024 10:22:08 +0200 Subject: [PATCH] reshuffle installation topics --- admin-guide/deployment.md | 4 + admin-guide/deployment/basic_installation.md | 148 +++++++++++++++++++ admin-guide/deployment/consoles.md | 36 +++++ admin-guide/deployment/dmz.md | 78 ++++++++++ admin-guide/deployment/partitioning.md | 38 +++++ admin-guide/deployment/reinstall.md | 15 ++ 6 files changed, 319 insertions(+) create mode 100644 admin-guide/deployment.md create mode 100644 admin-guide/deployment/basic_installation.md create mode 100644 admin-guide/deployment/consoles.md create mode 100644 admin-guide/deployment/dmz.md create mode 100644 admin-guide/deployment/partitioning.md create mode 100644 admin-guide/deployment/reinstall.md diff --git a/admin-guide/deployment.md b/admin-guide/deployment.md new file mode 100644 index 00000000..cb7ead05 --- /dev/null +++ b/admin-guide/deployment.md @@ -0,0 +1,4 @@ +# Installation + +```{tableofcontents} +``` diff --git a/admin-guide/deployment/basic_installation.md b/admin-guide/deployment/basic_installation.md new file mode 100644 index 00000000..afe6f6fe --- /dev/null +++ b/admin-guide/deployment/basic_installation.md @@ -0,0 +1,148 @@ +# Installation + +# Example Installation + +In this section we describe the deployment of a new server from +scratch. +All the operations will be performed using `bob`, the command-line +client for interacting with `sysdb`. + +The informations we need are: + + - the server name: `logging-ra.psi.ch` + - the server MAC address: `23:3d:ef:33:11:22` + - optional static or dynamic IP: `static` (RHEL7 & RHEL8 only) + - the OS installer: `rhel8install` + - the server role: `role::logging_server` + - optional the server environment in puppet: `prod` + - the sysdb deployment environment: `daas` + - optional the group inside the sysdb environment (`default` if not specified): `logging` + - optional the subgroup inside the group: `collector` + +## Setting up bob + +The `bob` tool is already set up on `lxsup.psi.ch`. You might connect with `ssh` and use it from there. + +Altenatively you many have on your workstation a local copy of `bob`. This can be done by installing the RPM from the pli-misc repository - https://repos.psi.ch/rhel8/latest/pli-misc/ + +You might need to set a `PSI_BOB_USER` environment variable (`export PSI_BOB_USER=`) or use the option `--user` if your local user is different from the one on sysdb. + +## Sysdb configuration + +We add the node to the environment, specifing `netboot` as +the boot action +```bash +bob node add logging-ra.psi.ch daas netboot +``` + +We setup its MAC address: +```bash +bob node add-mac logging-ra.psi.ch 23:3d:ef:33:11:22 +``` + +Next decide how the IP address is configured (`dhcp` or `static`): +```bash +bob node set-attr logging-ra.psi.ch network=static +``` + +And which installer is used: +```bash +bob node set-attr logging-ra.psi.ch ipxe_installer=rhel8install +``` + +And we finally setup the puppet role and puppet environment: +```bash +bob node set-attr logging-ra.psi.ch puppet_role=role::log_server +bob node set-attr logging-ra.psi.ch puppet_env=prod +bob node set-attr logging-ra.psi.ch puppet_group=logging +bob node set-attr logging-ra.psi.ch puppet_subgroup=collector +``` + +## Example + +Full example: + +```bash +bob node add lx-test-02.psi.ch lx netboot +bob node add-mac lx-test-02 00:50:56:9d:19:76 +bob node set-attr lx-test-02.psi.ch ipxe_installer=rhel8install puppet_role=role::server +bob node netboot lx-test-02.psi.ch +bob node list -v lx-test-02 +``` + +Ensure that a potential previous puppet certificate for this server is deleted on the puppet server: https://puppet.psi.ch + + +## Special Settings + +### Custom Kernel Commandline Arguments +For custom kernel commandline arguments for the installer (e.g. to provide drivers) the sysdb attribute `kernel_cmdline` can be used: + +```bash +bob node set-attr lx-test-02.psi.ch kernel_cmdline=inst.dd=https://linuxsoft.cern.ch/elrepo/dud/el8/x86_64/dd-megaraid_sas-07.725.01.00-1.el8_9.elrepo.iso +``` + +### Custom Partitioning + +Please check out the [Partitioning page](partitioning). + + + + +Linux systems are generally deployed using PXE and Kickstart. We use iPXE, which +can retrieve its configuration via HTTP(S) and therefore gives us a lot of +flexibility, because the iPXE configuration can be generated on the fly with the +specific settings needed by the booting client. In the same way the Kickstart +configuration for the RHEL installer is auto-generated. + +The information used for generating the iPXE and Kickstart configurations is +stored in the sysdb (see below). The sysdb is accessible through a web api, and +a command line client is available. + +When PXE boot is not an option, e.g. in restricted networks, it is possible to +start iPXE from a USB stick or other media. + +The general process for a deployment/installation is: + +1. Register the system with sysdb (only for new systems). +2. Tell sysdb to perform an installation on the next boot. +3. Reboot the system and trigger a PXE boot (usually by pressing F12 during + POST). + +The easiest way to interact with sysdb is to use `bob`, a command-line client, set up ready to use on `lxsup.psi.ch`. +Alternatively it is possible to use the web API directly. + +First, add the new node: +```bash +bob node add $FQDN $ENV netboot +``` + + +To be able to PXE boot we need to configure at least one MAC address for the new +node: +```bash +bob node add-mac $FQDN 00:50:56:aa:fe:9b +``` + + +Finally we need to configure the installer to use, and the Puppet-related +parameters: +```bash +bob node set-attr $FQDN ipxe_installer=rhel8install +bob node set-attr $FQDN puppet_role=role::server +``` + +and optional: +```bash +bob node set-attr $FQDN puppet_group=cluster +bob node set-attr $FQDN puppet_subgroup=compute +bob node set-attr $FQDN network=static +bob node set-attr $FQDN puppet_env=prod +``` + +## Redeployment + +After the initial installation the boot mode has been reset from netboot to local so it will then always boot from the local disk. For a redeployment the netboot needs to be set anew (on UEFI based systems netboot also needs to be always selected by the UEFI menu)): +``` +bob node netboot $FQDN +``` diff --git a/admin-guide/deployment/consoles.md b/admin-guide/deployment/consoles.md new file mode 100644 index 00000000..0f8a9d9c --- /dev/null +++ b/admin-guide/deployment/consoles.md @@ -0,0 +1,36 @@ +# Console Deployment + +## Overview +A console is a multi user system (ideally running on standard hardware) with a graphical desktop. The individual users do not have admin rights on the system and all configuration and packages must be deployed by puppet (ensuring reproducibility and fast re-installation in case of hardware failures, etc.) + +Consoles are, for example, used at experimental stations, beamlines, endstations. + +The standard naming of a console is: __<group>-cons-<two digit number>__ +Due to various reasons these systems __must__ have a static IP assigned. + +## Installation Workflow + +1. Register a static ip in https://qip.psi.ch/qip for the console +2. Create the necessary bob entries for the machine: +```bash +bob node add netboot +bob node add-mac xx:xx:xx:xx:xx:xx +bob node set-attr network=static +bob node set-attr ipxe_installer=rhel8install + +bob node set-attr puppet_role=role::console +bob node set-attr puppet_env=prod +bob node set-attr puppet_group=default # replace default if needed + +# Optional +bob node set-attr puppet_subgroup=collector +``` + +3. Create a host specific file (`.yaml`) in the respective hiera repository/directory with the following content: + +```yaml +networking::setup: auto_static_ip +``` + +4. Ensure that the UEFI/BIOS is set to netboot +5. Kickstart the machine \ No newline at end of file diff --git a/admin-guide/deployment/dmz.md b/admin-guide/deployment/dmz.md new file mode 100644 index 00000000..eeba3630 --- /dev/null +++ b/admin-guide/deployment/dmz.md @@ -0,0 +1,78 @@ +# DMZ Deployment + +The deployment in the DMZ ist the basically the same as [internaly](sample), but there are a few points to consider: + +- a firewall rule for puppet is needed +- the commissioning can only be done in the special DMZ commissioning network + +Because of this commissioning network we suggest that the DMZ VM gets for commissioning two interfaces, a "front-door" to the actual network where it will finally provide its service and the "back-door" in the commissioning network. After successful setup that interface will be removed. + +## Preparation + +- get static IP addresss for "front-door" interface +- For Puppet you need to [order a firewall rule](https://psi.service-now.com/psisp?id=psi_new_sc_cat_item&sys_id=faccb8644fe58f8422b0119f0310c7f7) from your machine to `puppet01.psi.ch` using TCP port 8140. +- (let) the VM be set up with to interfaces, the first one in the final network ("front-door") and the second one attached to `172.23.206.0/24` ("back-door") +- get both MAC addresses +- prepare the node in Sysdb/`bob` with the "back-door" MAC address +- in Hiera following network configuration is suggested which keeps the "front-door" interface disabled for the start: +``` +networking::setup: managed + +networking::connections: + - dmz_network + - commissioning_network + +networking::connection::dmz_network: + mac_address: '00:50:56:9d:47:eb' + ipv4_method: 'disabled' + ipv6_method: 'disabled' + +networking::connection::commissioning_network: + mac_address: '00:50:56:9d:c7:fe' + ipv4_method: 'auto' + ipv6_method: 'disabled' +``` +## Commissioning/Kickstart + +- commission/kickstart the node via network boot +- for SSH access get assigned IP address from VMWare or Puppet facts or QIP +- at the moment puppet will fail, provide the IP address to your fellow friendly Core Linux Team member to manually finish the first boot +- if the configuration is fully ready, configure the "front-door" interface: +``` +networking::setup: managed + +networking::connections: + - dmz_network + - commissioning_network + +networking::connection::dmz_network: + mac_address: '00:50:56:9d:47:eb' + ipv4_method: 'manual' + ipv4_address: '192.33.120.60/24' + ipv4_gateway: '192.33.120.1' + ipv6_method: 'disabled' + +networking::connection::commissioning_network: + mac_address: '00:50:56:9d:c7:fe' + ipv4_method: 'auto' + ipv6_method: 'disabled' +``` + +## Cleanup + +- check if you still have management access (`ssh`) over the front door interface +- remove the configuration of the "back-door" interface: +``` +networking::setup: managed + +networking::connections: + - dmz_network + +networking::connection::dmz_network: + mac_address: '00:50:56:9d:47:eb' + ipv4_method: 'manual' + ipv4_address: '192.33.120.60/24' + ipv4_gateway: '192.33.120.1' + ipv6_method: 'disabled' +``` +- remove the "back-door" interface from the VM diff --git a/admin-guide/deployment/partitioning.md b/admin-guide/deployment/partitioning.md new file mode 100644 index 00000000..8fa14766 --- /dev/null +++ b/admin-guide/deployment/partitioning.md @@ -0,0 +1,38 @@ +# Partitioning + +Partitions system are configured with a standard schema using LVM, so that they can be possibly changed afterwards. + +By default the whole space available on the first block device is used and any existing partition is removed. + +Alternatively you might set the sysdb attribute `system_disk` with the device name of the disk which should be used instead:: + +```bash +bob node set-attr $FQDN system_disk=md126 +``` + +It is also possible to customize the partitioning by using the `partitions` attribute on sysdb. See https://git.psi.ch/linux-infra/bob for more details. + + +## RHEL7 +The default partition schema for RHEL7 is: + +- create one primary ``/boot`` partition of 1Gb; +- create the ``vg_root`` Volume Group that uses the rest of the disk; +- on ``vg_root`` create the following logical volumes: + - ``lv_root`` of 12 Gb size for ``/root``; + - ``lv_var`` of 8 Gb size for ``/var``; + - ``lv_var_log`` of 2 Gb size for ``/var/log``; + - ``lv_tmp`` of 2 Gb size for ``/tmp``. + +## RHEL8 +The default partition schema for RHEL8 is: + +- create one primary ``/boot`` partition of 1Gb; +- create the ``vg_root`` Volume Group that uses the rest of the disk; +- on ``vg_root`` create the following logical volumes: + - ``lv_root`` of 14 Gb size for ``/root``; + - ``lv_home`` of 2 Gb size for ``/home``; + - ``lv_var`` of 8 Gb size for ``/var``; + - ``lv_var_log`` of 3 Gb size for ``/var/log``; + - ``lv_var_tmp`` of 2 Gb size for ``/var/log``; + - ``lv_tmp`` of 2 Gb size for ``/tmp``. diff --git a/admin-guide/deployment/reinstall.md b/admin-guide/deployment/reinstall.md new file mode 100644 index 00000000..2476d1df --- /dev/null +++ b/admin-guide/deployment/reinstall.md @@ -0,0 +1,15 @@ +# How to reinstall a machine + +Generally speaking, a reinstall can be done without doing anything other than doing the PXE boot, but there are some caveats to consider: + +- the puppet server certificate is saved on the puppet server + +- the puppet client certificate is saved by the kickstart script (which obviously can only happen, if the machine is reinstalled to the same drive with an intact file-system) + +- if you do a new install to a blank drive, but the puppet server has a certificate saved for the host, the client will generate a new cert, but the server will not, so the certificates saved on the 2 sides, will not match and will never work. In this case both sides need to be cleaned up before a new puppet run is attempted. + +- somewhat unrelated to the other points, but a similar case is the ssh server keys, which are stored on the puppet server and are put in place by puppet agent, so they remain unchanged under all reinstall scenarios + +Puppet server certs can be deleted at https://puppet01.psi.ch/ and on that page, the command to delete the client cert is specified. + +To access https://puppet01.psi.ch one needs to authenticate with your username/password. The server uses a invalid https certificate that is not accepted by modern safari/chrome any more. Use Firefox as a workaround.