zimoch/Controls-docs
0
0
Fork 0
forked from Controls/gitea-pages
Code Pull Requests Actions Activity

"merged" admin-guide

Browse Source
This commit is contained in:
ebner
2021-05-05 14:24:27 +02:00
parent 296ecb3f26
commit c488e8de62
182 changed files with 17246 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
admin-guide/deployment/infrastructure.rst
+94
View File
@@ -0,0 +1,94 @@
============================
Deploying the infrastructure
============================
Introduction
============
The deployment infrastructure is composed of:
- one sysdb server;
- one puppet server;
- one repository server.
We currently have two infrastructures in place, one for testing and the
other for production, with the following hostnames:
+--------------------+----------------+-----------------+
| Role | testing host | production host |
+====================+================+=================+
| sysdb server | boot00-test | boot00 |
+--------------------+----------------+-----------------+
| puppet server | puppet00-test | puppet00 |
+--------------------+----------------+-----------------+
| repository server | repo00-test | repo00 |
+--------------------+----------------+-----------------+
The Sysdb Server
----------------
The sysdb server is a server running HTTP(S) services needed by
other components of the deployment infrastructure, specifically to:
- setup (add/remove/modify) the hosts managed by the system;
- get the iPXE file for the managed hosts;
- get the Kickstart file for the managed hosts.
The code for these services is avaiable at `<https://git.psi.ch/linux-infra-software/sysdb>`_.
the Puppet Server
-----------------
The puppet server is a rather standard puppet master, configured to
use ``sysdb`` informations as ENC.
The Repo Server
---------------
The service that provides clones of different repositories (RHEL,
Puppetlabs, Google Chrome) plus some internally developed ones for
specific products (eg. GPFS).
You can reach the production one at: `<http://repo00.psi.ch/el7/>`_.
Deployment of an infrastructure server
======================================
The procedure is not fully automated and completely documented.
The reference repo server is `<https://git.psi.ch/linux-infra/bootstrap>`_.
The installation of an infrastructure server uses:
- ETH repositories (since the repo server is not assumed to be available)
- PXE only and static kickstart file (since the boot server is not supposed to be available)
After the installation, inside ``/root/bootstrap/instcode`` there is a script for each
server, called ``{puppet,boot,repo}-server``.
The script should be run manually and some user interaction is needed.
Before to install you need to have:
- a copy of the ssh root user keys and ssh server keys
- a Red Hat Satellite Key. You can get it via the RHN Satellite Account at ETH
So to install eg. ``puppet00-test`` the procedure would be:
- ensure on the Satellite web site that the system is not present (Systems-> Overview page)
- copy the kickstart on ``/afs/psi.ch/project/linux/www/kickstart/configs/linux-infra/``
- copy the pxe in ``/afs/psi.ch/service/linux/tftpboot/pxelinux.cfg/``
- install the server booting from network
- after the installation the server will stop
- reboot it
- login on the server
- copy the ssh key
- enter the ``/root/bootstrap/instcode`` directory
- optionally change the branch
- run ``./puppet-server``
Please note that in some cases the ETH repo used during the
installation is not properly working.
In the case you are not redeploying the repo server, then it can be
used instead of the ETH one, changing properly the PXE and the
kickstart file.
admin-guide/deployment/ipxe.rst
+61
View File
@@ -0,0 +1,61 @@
PXE-booting with iPXE
=====================
UEFI
----
iPXE supports UEFI and so do we. This requires the ``ipxe.efi`` boot image.
Process
-------
When a system PXE boots, the DHCP server will boot either pxelinux in legacy
mode or grub on UEFI. It is still possible to load ipxe from either of these.
With the right option, the boot process is pointed to ``boot00.psi.ch`` and one
of the following boot images:
- ``ipxe.efi`` for systems using UEFI
- ``ipxe.lkrn`` for all other systems
The system downloads the image and executes it.
The image contains a small iPXE script, which makes iPXE retrieve its actual
configuration from a web service running (usually) on ``boot00.psi.ch``.
Specifically, it queries the URL
``https://boot00.psi.ch/ipxe/v1/config?mac=<MAC>``, where ``<MAC>`` is the MAC
address of the interface used by iPXE.
The web service on ``boot00.psi.ch`` will generate the iPXE configuration on the
fly, depending on whether the system is supposed to be reinstalled and if so,
which distribution it is supposed to use. The menu offers other options as well,
e.g. an interactive iPXE shell and a memory test.
Building the iPXE boot image
----------------------------
The steps for building an iPXE image are
1. Clone ``git@git.psi.ch:linux-infra/ipxe-build.git``
2. Change into the new directory
3. Run the ``refresh-ipxe.sh`` script. This will check out the iPXE source code
mirror from ``git.psi.ch``.
4. Run the ``build.sh`` script. This will use the named configurations in the
``ipxe-build`` repository to compile two iPXE images: ``src/bin/ipxe.lkrn``
(legacy boot) and ``src/bin-x86_64-efi/ipxe.efi`` (UEFI).
As described above, we use an iPXE boot image with an embedded script looking
roughly like the following::
#!ipxe
dhcp && goto dhcp_succeeded
shell
#prompt for network info
:dhcp_succeeded
chain http://boot00.psi.ch/ipxe/v1/config?mac=${netX/mac}
admin-guide/deployment/kickstart.rst
+11
View File
@@ -0,0 +1,11 @@
Kickstart
=========
The auto-generated Grub or iPXE configuration will, when installing RHEL, instruct
the RHEL installer to download the Kickstart configuration from
``https://boot00.psi.ch/kickstart/v1/config?fqdn=<FQDN>&instrepo=<INSTREPO>``,
where ``<FQDN>`` is the FQDN of the host to be installed, and ``<INSTREPO>`` is
the installation repository to be used.
The web service will auto-generate the Kickstart configuration for the client
based on information in sysdb, e.g. the release to be installed.
admin-guide/deployment/partitioning.rst
+30
View File
@@ -0,0 +1,30 @@
Partitioning
============
Partitions system are configured with a standard schema using LVM, so
that they can be possibly changed afterwards.
By default the whole space available on the first block device is used
and any existing partition is removed.
The default partition schema is:
- create one primary ``/boot`` partition of 1Gb;
- create the ``vg_root`` Volume Group that uses the rest of the disk;
- on ``vg_root`` create the following logical volumes:
- ``lv_root`` of 12 Gb size for ``/root``;
- ``lv_var`` of 8 Gb size for ``/var``;
- ``lv_var_log`` of 2 Gb size for ``/var/log``;
- ``lv_tmp`` of 2 Gb size for ``/tmp``.
Custom Partitioning
-------------------
It is possible to customize the partitioning by using the ``partitions`` attribute on sysdb.
The ``partitions`` attribute can take two values:
- ``manual``: this will not partition the disks and the installation process will wait for manual partitioning;
- ``<URL>``: it is assumed to be a file containing the partitioning commands as used in kickstart.
If it is not possible to download the given URL the system fails-back to manual at installation time.
Please also note that no check is done on the URL content.
admin-guide/deployment/sample.rst
+60
View File
@@ -0,0 +1,60 @@
Sample deployment of a new server
=================================
In this section we describe the deployment of a new server from
scratch.
All the operations will be performed using `bob`, the command-line
client for interacting with `sysdb`.
The informations we need are:
- the server name: ``logging-ra.psi.ch``
- the server MAC address: ``23:3d:ef:33:11:22``
- the server role: ``role::logging_server``
- the server environment in puppet: ``production``
- the sysdb environment: ``daas``
On our laptop, where we have a local copy of `bob`, we first setup
some environment variables to have shorter commands: ::
export PSI_BOB_URL=https://boot00-test.psi.ch/sysdb
export PSI_BOB_USER=talamo_i
The $PSI_BOB_USER is needed because our local user is different from
the one on sysdb. It could also be specified on the command line or
let `bob` take the local username.
We then create a deployment environment::
bob env add daas talamo_i daas-admins 'DaaS Cluster'
We add the node to the environment, specifing ``netboot`` as
the boot action ::
bob node add logging-ra.psi.ch daas netboot
We setup its MAC address: ::
bob node add-mac logging-ra.psi.ch 23:3d:ef:33:11:22
And we finally setup the puppet role and puppet environment: ::
bob node set-attr syslog-ra.psi.ch puppet_role=role::log_server
bob node set-attr syslog-ra.psi.ch puppet_env=production
PXE server-side configuration
-----------------------------
In case the node is configured to use legacy BIOS mode (and
not UEFI mode) the following has to be run on the PXE server to make
the node boot the ipxe image: ::
cd /afs/psi.ch/service/linux/tftpboot/pxelinux.cfg/
ln -sf lxprod logging-ra
MAC='the::node::mac::address'
MACFILENAME="01-`echo $MAC|tr 'A-Z' 'a-z'| sed 's/:/-/g'`"
ln -sf logging-ra $MACFILENAME
Otherwise the node will boot with the old pxe configuration. In such
case is possible to boot ipxe typing `lxprod` on the PXE command line.
admin-guide/deployment/workflow.rst
+165
View File
@@ -0,0 +1,165 @@
===================================
Deploying New Environment: Workflow
===================================
Introduction
============
Deploying a new environment requires the following:
1. Configuring the environment in **bob**
2. Configuring the environment in GIT (``git.psi.ch``)
- Environment format: ``data-<environment_name>``
- And change permissions accordingly
3. Configuring the environment in Puppet (``puppet00.psi.ch``)
Configuring the environment in sysdb
------------------------------------
Bob allows to create a new environment in ``sysdb`` by using the ``bob
env`` option. You must have permissions to do that.
You must belong to the ``sysdb-admins`` group that is actually
configured on the local ``/etc/group`` file. A migration to Active
Directory should be done for that group.
To list current defined environments run::
bob env list
To add a new environment, run::
bob env add <environment_name> <owner> <admin_group> "<description>"
In example, for the MeG cluster::
bob env add meg caubet_m unx-hpc_adm "MeG Cluster"
Test new environment in BOB
~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to test that environment was successfully created::
bob env list | grep <environment_name>
In example::
caubet_m@caubet-laptop:~/GIT/admin-guide/deployment$ bob env list | grep meg
meg caubet_m unx-hpc_adm MeG Cluster
Configuring the environment in GIT
----------------------------------
Current **GIT** server at **PSI** is ``git.psi.ch``. Every new environment should in principle belong to the **linux-infra** project.
You must belong to the ``puppet_env`` Active Directory group in order to be able to create new projects.
In order to create a new environment in **GIT**, you should access and login in the following link: https://git.psi.ch/linux-infra/. Here you can see
the different environments and **GIT** projects belonging to the **linux-infra** group.
The steps to create and configure a new **GIT** project are:
1. Create a new project (environment). It can be done here: https://git.psi.ch/projects/new
- Go to ``[Blank project] Tab`` (which is the *default* tab)
- Change ``[Project Path]`` as follows:
- https://git.psi.ch/ + ``linux-infra``
- Define ``[Project Name]``, which *must* have the following format:
- ``data-<environment_name>`` where ``<environment_name>`` is the one defined in **Bob**
- *[Optional]* Specify ``[Project description]``
- Specify ``[Visibility Level]``:
- Should be ``Internal``
2. Configure *project* permissions as follows:
- ``[data-<environment_name]->[Settings]->[Repository]``, or directly: https://git.psi.ch/linux-infra/data-(environment_name)/settings/repository
- ``[Deploy Keys]`` -> select ``root@puppet00.psi.ch'`` -> click on ``'Enable'``
- ``[data-<environment_name]->[Settings]->[Members]``, or directly: https://git.psi.ch/linux-infra/data-(environment_name)/project_members
- Setup specific permissions for specific users or groups. In example:
- Set project ``Master``:
- ``[Select members to invite]`` (``caubet_m``) + ``[Choose a role permission]`` (``Master``) + ``[Add to project]``
- Set other roles:
- ``[Select members to invite]`` (``ozerov_d``) + ``[Choose a role permission]`` (``Developer``) + ``[Add to project]``
- ``[data-<environment_name]->[Settings]->[Integrations]``, or directly: https://git.psi.ch/linux-infra/data-(environment_name)/settings/integrations
- Add WebHook as follows:
- ``[URL]``: http://puppet00.psi.ch/events/dataupdate
- (Checked) ``[Push events]``. Uncheck the rest.
- ``[SSL verification]`` -> (uncheck) ``[Enable SSL verification]``
- Confirm information from above, and click on ``[Add webhook]`` to add the new WebHook.
Test new environment in GIT
~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to test that environment was successfully created::
git clone git@git.psi.ch:linux-infra/data-<environment_name>.git
In example::
caubet_m@caubet-laptop:~/GIT$ git clone git@git.psi.ch:linux-infra/data-meg.git
Cloning into 'data-meg'...
X11 forwarding request failed
warning: You appear to have cloned an empty repository.
Configuring the environment in Puppet
-------------------------------------
Current *test* **Puppet** server is ``puppet00-test.psi.ch``. Current *production* **Puppet** server is ``puppet00.psi.ch``. This documentation will take as an example an environment
deployed in the *production* server ``puppet00.psi.ch``.
You must have ``root`` access to the Puppet server in order to be able to configure it.
Steps are:
1. Login in to the Puppet server::
ssh root@puppet00.psi.ch
2. Clone the already created **GIT** project to the ``/srv/puppet/data`` path, which contains all the different projects (Bob environments)::
git clone git@git.psi.ch:linux-infra/data-<environment_name>.git /srv/puppet/data/<environment_name>
# In example:
git clone git@git.psi.ch:linux-infra/data-meg.git /srv/puppet/data/meg
3. Change permissions to ``plidata.puppet`` for the new directory ``/srv/puppet/data/<environment_name>``::
chown -R plidata.puppet /srv/puppet/data/<environment_name>
# In example:
chown -R plidata.puppet /srv/puppet/data/meg
At this point, everything should be configured and we can proceed to test that it works.
Test new environment in Puppet
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to test that environment was successfully created::
git clone git@git.psi.ch:linux-infra/data-<environment_name>.git
Add a new file <environment_name>.yaml in to the project::
cd data-<environment_name>
touch <environment_name>.yaml
git add <environment_name>.yaml
git commit -a -m "Added first empty file"
git push
After a few seconds (needs time to trigger the change), check in ``puppet00.psi.ch:/srv/puppet/data/<environment_name>`` that file was successfully triggered (copied) to the puppet server from **GIT**::
ssh root@puppet00.psi.ch ls /srv/puppet/data/<environment_name>/<environment_name>.yaml
Full real example::
git clone git@git.psi.ch:linux-infra/data-meg.git
cd data-meg
touch meg.yaml
git add meg.yaml
git commit -a -m "Added first empty file"
git push
sleep 5
ssh root@puppet00.psi.ch ls /srv/puppet/data/meg/meg.yaml