documentation/docs/engineering-guide/repo_server.md

Repository Server

The repository server is lx-repos-01.psi.ch, usually accessed with the alias repos.psi.ch. For the DMZ we have lx-repos-dmz-01.psi.ch with the alias repos-dmz.psi.ch.

Prepare a new RHEL Version for Installation

Download the iso image on lx-sync-01.psi.ch from https://id-sat-prd-03.ethz.ch or https://access.redhat.com/downloads/content/rhel and add it to the /packages/rhel\[8/9]/iso directory

The service restart mounts the iso as a loop device into a directory of the same name.

Then the ipxe and grub templates of the sysdb have to be edited, to add the new version:

https://gitea.psi.ch/lx/sysdb/src/branch/prod/sysdb/ipxe_template.py

https://gitea.psi.ch/lx/sysdb/src/branch/prod/sysdb/grub_template.py

Once the change is committed, the changes have to be pulled on sysdb.psi.ch:

[root@sysdb ~]# cd /var/www/sysdb/app/
[root@sysdb ~]# git pull
[root@sysdb ~]# systemctl restart httpd

The changes only come live after a restart of the httpd.

Upgrade a Repository Server to a new RHEL version

This is a bit tricky because the automated process wants to load from ISO and packages from the old server. But it works well with a few manual interactions:

1. rename the VM to *-old and create a new one
2. update MAC address in https://gitea.psi.ch/lx/bootstrap/src/branch/prod/generate-pxe-uefi
3. run generate-pxe-uefi and update the result in https://gitea.psi.ch/lx/network-boot or https://gitea.psi.ch/lx/network-boot-dmz
4. ensure the kickstart files is also regenerated an available on lx-sync-01.psi.ch:/packages/kickstart
5. on lx-sync-01.psi.ch:/packages/kickstart modify the respective kickstart file by removing all manual IP configuration and changing to --bootproto=dhcp to make it get an dynamic IP which does not conflict with the one used by the old server
6. boot the new VM and let it install the OS
7. get the dynamcially assigned IP from vSphere
8. make your ssh temporarily go for the domain name to the new IP, eg. by adjuting your local .ssh/config or /etc/hosts or mgt-gw.psi.ch:/etc/hosts (but do not change the hostname in the inventory file)
9. let the Ansible playbook run until it fails because it cannot mount /packages
10. login to the new server
11. run dnf update
12. on the new server edit /etc/NetworkManager/system-connections/ens*.nmconnection and set there the IP configuration of the old server
13. shutdown old server
14. reboot new server
15. remove temporary ssh redirection to dynamic IP of new server
16. run the Ansible playbook again, it should not fail any more