Controls/gitea-pages
0
2
Fork 0
Code Pull Requests Actions Activity

latest updates regarding repo mirroring

Browse Source
This commit is contained in:
Daniela Metzler
2022-05-07 17:26:12 +02:00
parent bf14c62028
commit 2672c7c62f
1 changed files with 46 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure-guide/repo01.md
+46 -93
View File
@@ -1,125 +1,78 @@
# repo01
# repo01.psi.ch
This machine is a RHEL8 system **not** under Puppet control. This machine has no AFS dependencies.
The basic service provided by this system is: httpd
This machine acts as a mirror for the RHEL8 iso images.
The installer iso images are downloaded from https://id-sat-prd.ethz.ch/pub/isos/ and put into /var/www/html/iso manually
Repo01 is connected to the Satelitte server from ETH Zürich. The Red Hat repositories are synced from there.
The iso images in `/var/www/html/iso` then (automatically) mounted as loop devices by the `pli-mount-iso-images.service`. This is an enabled service and it runs once on system boot automatically. Otherwise changes are not monitored, if one puts an iso there and wants it mounted, a manual restart of the service is required for anything to happen.
The installer iso images are downloaded from https://access.redhat.com/downloads and put into /var/www/html/iso manually
The iso images in /var/www/html/iso are then (automatically) mounted as loop devices by the pli-mount-iso-images.service under /etc/systemd/system/. Otherwise changes are not monitored, if a new iso is put there, a manual restart of the service is required for anything to happen.
df -kh | grep /var/www
/dev/loop0 11G 11G 0 100% /var/www/html/iso/rhel85 /dev/loop1 6.7G 6.7G 0 100% /var/www/html/iso/rhel80
```
[root@repo01 ~]# df -kh | grep /var/www
/dev/mapper/vg_repo-repofiles 1.4T 1003G 373G 73% /var/www/html
/dev/loop2 7.1G 7.1G 0 100% /var/www/html/iso/HP.SPP.2020.03
/dev/loop3 903M 903M 0 100% /var/www/html/iso/IP330.2019_0207.248
/dev/loop4 12M 12M 0 100% /var/www/html/iso/Memtest86-7.5
/dev/loop6 158M 158M 0 100% /var/www/html/iso/MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.4-x86_64
/dev/loop7 163M 163M 0 100% /var/www/html/iso/MLNX_OFED_LINUX-4.2-1.0.0.0-rhel7.4-x86_64
/dev/loop9 163M 163M 0 100% /var/www/html/iso/MLNX_OFED_LINUX-4.2-1.2.0.0-rhel7.4-x86_64
/dev/loop10 275M 275M 0 100% /var/www/html/iso/MLNX_OFED_LINUX-4.7-1.0.0.1-rhel7.6-x86_64
/dev/loop11 275M 275M 0 100% /var/www/html/iso/MLNX_OFED_LINUX-4.7-1.0.0.1-rhel7.7-x86_64
/dev/loop12 5.5G 5.5G 0 100% /var/www/html/iso/P03093_001_spp-Gen8.1-SPPGen81.4
/dev/loop13 5.7G 5.7G 0 100% /var/www/html/iso/P14481_001_spp-2019.03.0-SPP2019030.2019_0206.85
/dev/loop14 5.8G 5.8G 0 100% /var/www/html/iso/P19473_001_spp-2019.09.0-SPP2019090.2019_0905.39
/dev/loop15 7.0G 7.0G 0 100% /var/www/html/iso/P26228_001_spp-2019.12.0-SPP2019120.2019_1209.4
/dev/loop17 7.9G 7.9G 0 100% /var/www/html/iso/rhel-8.2-x86_64-dvd
/dev/loop18 8.9G 8.9G 0 100% /var/www/html/iso/rhel-8.3-x86_64-dvd
/dev/loop19 3.8G 3.8G 0 100% /var/www/html/iso/rhel-server-7.4-x86_64-dvd
/dev/loop20 4.4G 4.4G 0 100% /var/www/html/iso/rhel-server-7.5-x86_64-dvd
/dev/loop21 4.2G 4.2G 0 100% /var/www/html/iso/rhel-server-7.6-x86_64-dvd
/dev/loop22 4.2G 4.2G 0 100% /var/www/html/iso/rhel-server-7.7-x86_64-dvd
/dev/loop23 4.3G 4.3G 0 100% /var/www/html/iso/rhel-server-7.8-x86_64-dvd
/dev/loop24 4.3G 4.3G 0 100% /var/www/html/iso/rhel-server-7.9-x86_64-dvd
```
Under /opt/pli/libexec exist five shell scripts that run with systemctl timer.
The pli-repo-mirror runs a daily sync (at 21:30) via /etc/systemd/system/pli-repo-mirror.timer, which pulls the latest repos into /var/www/html/el8/sources.
The gen-snapshot-page is the basic script to generate a snapshot.
From the above, a weekly snapshot (at Sun 23:30) is taken by the pli-repo-snapshot via the pli-repo-snapshot.timer
From there the snapshot_is_prodable checks wheter the new snapshot is ready for a new prod tag.
It will check if the latest yfs kernel module corresponds with the latest rhel8 kernel.
If the kernel and the yfs module version correspond a prod snapshot is made under /var/www/html/el8/tags
The pli-repo-zoom is run via the pli-repo-zoom.timer. It maintains the zoom repo at /var/www/html/zoom/
The pli-repo-yfs script is run via the pli-repo-yfs.timer. It syncs the auristor repo and apparently also create tags.
In case of emergency, you can run a snapshot by force. Then you have to add the version before you can run the script $./pli-repo-snapshot el8
Under /etc/systemd/system/ we have all the pli-repo* timer and service configurations.
In order to start a new timer/service, the service and timer have to be a enabled.
The `pli-repo-mirror.timer` runs a daily sync (at 21:30), which pulls the repos into `/var/www/html/el7/sources` . The name is misleading, these are actually all the latest repos.
Under /opt/pli/etc/mirror/ you find the repolist, cuda and yum.conf
From the above, a weekly snapshot (at Sun 23:30) is taken by the `pli-repo-snapshot.timer`.
Under /etc/yum.repos.d/ you find the redhat.repo. This are the Red Hat repositories that are synced from ETH.
The `/opt/pli/libexec/pli-repo-zoom.sh` is run via the `pli-repo-zoom.timer` timer, it maintains the zoom repo at /var/www/html/zoom/
Under /var/www/html/el8/keys/ you find the rpm-gpg-keys
Under /var/www/html/el8/manual/ you find the repos for nxserver. There is no automatic synchronisation for the rpms, they will be downloaded manually from the responsible person.
Under /var/www/html/el8/snapshots/ you will find all the preprod snapshots
Under /var/www/html/el8/sources you will find the packages and repodata from synced rpms.
Under /var/www/html/el8/tags/ you will find all prod tags
The `/opt/pli/libexec/pli-repo-yfs` script is run via the `pli-repo-yfs.timer` timer. It syncs the auristor repo and apparently also create tags.
The scripts and files in /opt/pli (as well as a copy of the systemd files) can be found in this repository:
https://git.psi.ch/linux-infra/repo01_pli-scripts
Under /var/www/html/ipxe is for testing purposes
Under /var/www/html/iso you find the rhel-8.0 and the rhel-8.5 iso image.
Under /var/www/html/ks-test you find the new kickstart directory, for testing. If all works this can be deleted.
The URI is https://repo01.psi.ch/el8/sources/
Provided http services:
```
[root@repo01 ~]# netstat -tulnp | grep http
tcp6 0 0 :::80 :::* LISTEN 11278/httpd
tcp6 0 0 :::443 :::* LISTEN 11278/httpd
[root@repo01 ~]#
```
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3634/httpd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3634/httpd
The httpd configuration can be found in /etc/httpd/conf.d
```
[root@repo01 ~]# ls -l /etc/httpd/conf.d/
total 12
-rw-r--r--. 1 root root 694 Apr 9 2019 25-repo01.psi.ch_non_ssl.conf
-rw-r--r--. 1 root root 1131 Apr 9 2019 25-repo01.psi.ch_ssl.conf
-rw-r--r--. 1 root root 366 Oct 9 2020 README
[root@repo01 ~]#
```
The ssh certificate is located in `/etc/pki/tls/`
# Directory Structure / Services
![](repo01_overview.drawio.svg)
# Questions / TODO
- I added the /opt/pli directory under git control, the repo is https://git.psi.ch/linux-infra/repo01_pli-scripts. Ideally the pli-* service files in /etc/systemd/system should be replaced with links to the /opt/pli/systemd/pli* files. Could you please do that and test whether things still work.
The ssl certificate is located in /etc/pki/tls/certs
- SELinux is enforcing, this will not work.
For the certificate renewal the CSR configuration is under /root/certs
- Can you explain a little bit more the structure of the /var/www/html/ directory (what is where, who is responsible for certain directories, what are they needed for, ...). The content of the web directory:
```
[root@repo01 ~]# ls -la /var/www/html/
total 56
drwxr-xr-x. 11 root root 4096 Mar 29 11:32 .
drwxr-xr-x. 4 root root 31 Oct 9 2020 ..
drwxr-xr-x. 7 root root 71 Apr 12 2019 el7
drwxr-xr-x. 3 root root 16 Sep 21 2020 fcos
drwxr-xr-x. 3 root root 4096 Apr 24 2020 HP.FW.RPMs
drwxr-xr-x. 23 root root 4096 Apr 12 14:27 iso
-rw-r--r--. 1 root root 8605 Jun 11 2019 lxdev00.ks
-rw-r--r--. 1 root root 8604 Jun 13 2019 lxdev01.ks
drwxr-xr-x. 5 root root 4096 Oct 30 2018 mt86
drwxr-xr-x. 2 root root 87 Aug 31 2020 ppc
drwxr-xr-x. 5 root root 69 Apr 24 2020 rhcos
-rw-r--r--. 1 root root 356 Feb 18 13:58 rhel7_hashes.txt
-rw-r--r--. 1 root root 211 Nov 27 2018 rhel8.ipxe
drwxr-xr-x. 25 root root 4096 Nov 21 2019 yum
drwxr-xr-x. 3 root root 4096 Apr 12 23:23 zoom
```
* el7 - where the automated mirroring and snapshotting is done
* iso - where the images are placed and mounted
* zoom - zoom repo
The rest were put there by hand. Much of it is probably not needed, but wouldn't know who needed them.
- Is there any additional documentation on how this system was set up? Where can I find this? If not, could you add here some more details which packages and configs are important (beside the /opt/pli scripts/services)
- I know of no further documentation and it was set up by Kai, years ago. It would take quite a bit of trial and error to reproduce.
- Is the mentiond httpd config everything that is needed, who is taking care of this certificate, how is it installed? how is the expiration monitored?
- I don't think anything further is needed. The cert is requested from SWITCH and placed here manually. It is not monitored. The owner/admin of this server must take care of this.
- Could you replace the files in /etc/httpd/conf.d/25* with a link to /opt/pli/httpd/25* and see whether things still work (this way also the httpd config would be versioned).
- No, SELinux.
- Who belongs this repo? https://repo01.psi.ch/mt86/ (I guess mt86 is a person short code - unfortunately I cannot find this code in the phonebook)
- It's memory test for x86 systems, not a person at PSI.
https://git.psi.ch/linux-infra/repo01_pli-scripts