From 229ec5045e3b42ec8e6f2c479441b6cfa83950e2 Mon Sep 17 00:00:00 2001 From: buchel_k Date: Tue, 14 May 2024 09:15:29 +0200 Subject: [PATCH] software update policy --- _toc.yml | 4 +- admin-guide/software.md | 82 -------------------------------- admin-guide/software/updates.rst | 70 --------------------------- admin-guide/updates.md | 27 +++++++++++ 4 files changed, 28 insertions(+), 155 deletions(-) delete mode 100644 admin-guide/software.md delete mode 100644 admin-guide/software/updates.rst create mode 100644 admin-guide/updates.md diff --git a/_toc.yml b/_toc.yml index e341247d..05ef5e39 100644 --- a/_toc.yml +++ b/_toc.yml @@ -97,9 +97,7 @@ chapters: - glob: admin-guide/puppet/components/* - file: admin-guide/puppet/development - file: admin-guide/selinux - - file: admin-guide/software - sections: - - file: admin-guide/software/updates + - file: admin-guide/updates - file: admin-guide/mgmt-tools sections: - file: admin-guide/mgmt-tools/sysdb diff --git a/admin-guide/software.md b/admin-guide/software.md deleted file mode 100644 index 8cbb5efc..00000000 --- a/admin-guide/software.md +++ /dev/null @@ -1,82 +0,0 @@ -# Software and Licenses - -```{tableofcontents} -``` - - -## Licenses - -Our Red Hat Enterprise Linux subscriptions are provided by ETHZ, other licenses -are usually managed by Roland Blättler. - -The RHEL repositories can be accessed on the [ETHZ Satellite server](https://id-sat-prd-02.ethz.ch). It is only accessible from within PSI and a -special account is needed, which can be requested at -. - -One thing to keep in mind is that several groups at PSI use the Satellite server -directly, so not all PSI hosts known to the Satellite belong to the central -Linux environment. In particular the network team has a number of systems there. - -Normally only certain infrastructure systems are registered with the Satellite. - -About once a year we report the total number of systems to ETHZ, so they can -track subscription usage. When we started using the ETHZ subscriptions in 2016 -we provided an estimate of 2000 systems max. - - -## Distribution and deployment - -All software is distributed as RPMs with the following exceptions: - -- some software is provided via AFS/NFS instead -- configuration files and scripts that can be considered configuration, e.g. the - files in `/etc/cron.daily` -- server applications can be deployed by cloning a Git repository (e.g. sysdb) - -There are a number of advantages to deploying software as RPM: - -- automatic installation/protection of dependencies -- inventory of installed software (`rpm -qa` etc) including version - information -- integrity checking (`rpm -q --verify`) -- ownership tracking (`rpm -qf`, `yum provides`) -- no network dependency, which is useful for laptops and increases reliability - on other systems - - -## Repositories - -We maintain an internal mirror for every repository that we use, or at least a -local repository containing the specific packages. We never point `yum.conf` -(or any other package manager) to an external repository directly. - -The repository server is `repos.psi.ch`. - -Currently we maintain the following repositories: - -1. RHEL 7 (almost all channels) -2. EPEL for RHEL 7 -3. Puppetlabs PC1 -4. Google Chrome -5. OpenAFS (PSI) -6. GPFS (PSI) -7. OpenHPC - -The mirroring is done using :manpage:`reposync(1)` with custom :manpage:`yum.conf` files. - - -The script to run a sync of all the repositories is `/opt/pli/sbin/psi-mirror-yum` but is -currently not executed automatically. - -To add a new repository to the list the files `/opt/pli/etc/mirror/yum.conf` and -`/opt/pli/etc/mirror/repolist` should be edited. The first file contains the repositories -definitions, the second one the list of repositories to mirror. - - - -## Packaging - -All packaging information for in-house packages must be tracked in Git. The -repositories containing packaging information should be in the -[linux-packages](https://git.psi.ch/groups/linux-packages) group on the -Gitlab server. diff --git a/admin-guide/software/updates.rst b/admin-guide/software/updates.rst deleted file mode 100644 index 3be1a5fc..00000000 --- a/admin-guide/software/updates.rst +++ /dev/null @@ -1,70 +0,0 @@ -Updates -======= - -There are two major aspects to software updates on PSI Linux systems: - -1. The configured repositories. -2. The Update policy - -Generally the assumption is that ``/etc/yum.conf`` points the system at -repositories which, for each package, contain the version of that package which -should be installed on the system, and possibly older versions. This way, a -simple ``yum update`` would always bring a system to the desired state, even -though it is not recommended to actually run ``yum update`` directly (see -below). - -The policy answers the following questions: - -1. When should updates be installed? -2. Which types of updates should be installed? I.e. security fixes, security and - bug fixes, or all updates? -3. When should the system reboot? I.e. always (offline updates, recommended), - only when there is a new kernel, or never? -4. For how long can updates be pending before an alert is triggered? - - -Policy ------- - -Parameters: -- timing (never, timespec) -- severity -- reboot behaviour (always, only if necessary (kernel update or needs-restarting), never) - -Example of fully automatic updates with mandatory reboots:: - - update::schedule: Thu, 02:00 -- 04:00 - update::severity: 'security' - update::reboot: 'always' - - -Example of manually triggered, full updates with reboots if necessary:: - - update::schedule: 'never' - update::severity: 'all' - update::reboot: 'if-necessary' - - - -Procedure ---------- - -Updates should be performed by starting the ``psi-update`` service. Using this -service instead of running ``yum`` directly has a number of benefits: - -1. All update-related log output can be retrieved from the journal easily using - ``journalctl -u psi-update.service``. -2. The service will take into account the update policy even when in manual - mode, e.g. it will only install the desired types of updates. -3. The service will automatically notify monitoring systems if a reboot is - necessary, avoiding false positives. -4. It will take of any cleanup actions necessary. E.g. Splunk, a commercial - logging product requires to accept its license again after updates (by - running something like ``splunk --accept-license``). - - -Monitoring ----------- - -There will be a separate mechanism, e.g. a Nagios check, to alert on systems -that are not compliant with their update policy. diff --git a/admin-guide/updates.md b/admin-guide/updates.md new file mode 100644 index 00000000..2740b9cb --- /dev/null +++ b/admin-guide/updates.md @@ -0,0 +1,27 @@ +# Software Update Policy + +## Responsibility +It is in the responsibilty of the owner/administrator of a system to care about the software update policy and its application. + +From the regulatory side there is the "Weisung" [AA-9500-142 "Handling of Software Updates"](https://DLS01P.PSI.CH/documents/jsp/qv?pri=PSIcgc&ft=cgcDocument@STORE_MAIN_CGC&q_cgcId=ID22023011616445798386). It states that for security related updates "must be applied in a mandatory and timely manner". Exceptions need to be agreed with IT Security. + +The Linux Core Group on the other side is reponsible to make the latest upstream Linux software updates available inside PSI. + + +## Automatic Updates + +By default once a week (in the night from Sunday to Monday) security updates are automatically applied. Other updates, including Kernel updates, need to be installed manually. + +This is [configurable](configuration/package_updates), you may switch it off completely, make it run daily or make it install all updates. + +Reboots are never done automatically. + +Also for software which have been installed from other sources than RPM package repositories (like `pip` or manual install) there is no automatic update procedure. + +## Snapshots + +On specially protected systems where stability is more important than being up-to-date, there is the option to freeze the provided RPM package version to a specified date. Also this can be [configured in Hiera](configuration/package_repositories#using-specific-package-repository-snapshot). If such a system is set by such a "Repo Tag" to a specific snapshot, the update procedure cannot get newer than the given state. + +Again, this should only be done for nodes in protected networks, e.g. with access restrictions through an [ssh gateway](../services-admin-guide/ssh_gateways) and requires consent with IT Security. + +