Controls/gitea-pages
0
2
Fork 0
Code Pull Requests Actions Activity

update hiera topic

Browse Source
This commit is contained in:
ebner
2024-08-08 14:17:31 +02:00
parent 8f11d2cf1d
commit d6011a97e8
7 changed files with 178 additions and 250 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_toc.yml
+3 -3
View File
@@ -85,6 +85,7 @@ chapters:
- file: admin-guide/troubleshooting/filesystem
- file: admin-guide/troubleshooting/processes
- file: admin-guide/troubleshooting/pcie_bus_error
- file: admin-guide/troubleshooting/puppet
- file: admin-guide/virtual_machines
- file: admin-guide/container
- file: admin-guide/certificates
@@ -93,10 +94,9 @@ chapters:
- file: admin-guide/puppet
sections:
- file: admin-guide/puppet/general
- file: admin-guide/puppet/client
- file: admin-guide/puppet/puppet-master
- file: admin-guide/puppet/puppet_client
- file: admin-guide/puppet/puppet_environments
- file: admin-guide/puppet/hiera
- file: admin-guide/puppet/development
- file: infrastructure-guide/index
sections:
admin-guide/puppet/hiera.md
+127
View File
@@ -0,0 +1,127 @@
# Hiera
Please refer to `here <https://docs.puppet.com/hiera/>`_ for a general Hiera
introduction.
Our current hierarchy has seven levels (first will be considered first
during value lookup):
- nodes (FQDN)
- subgroup (optional, ``puppet_subgroup`` attribute in sysdb)
- group (``puppet_group`` attribute in sysdb)
- sysdb environments
- Puppet server specific
- global
- common
The first four layers can be edited by the admin in the respective hiera git repository. The common layer (default values) and the server specific layer (differences between test and prod) are part of the Puppet code repository. Finally the global layer contains a few configurations which are managed by the Core Linux Group outside of the normal Puppet release process, eg. for license management.
The values can be stored as classical YAML values or with [encrypted yaml](https://github.com/TomPoulton/hiera-eyaml) for secrets.
The filesystem structure is as follows (the last 3 cannot be controlled by a common admin):
1. ``%{::sysdb_env}/%{::group}/%{::fqdn}.yaml`` or ``%{::sysdb_env}/%{::group}/%{::subgroup}/%{::fqdn}.yaml``
2. ``%{::sysdb_env}/%{::group}/%{::subgroup}.yaml``
3. ``%{::sysdb_env}/%{::group}.yaml``
4. ``%{::sysdb_env}/%{::sysdb_env}.yaml``
5. ``%{::environment}/data/server_%{server_facts.servername}.yaml``
6. ``/srv/puppet/data/global/global.yaml``
7. ``%{::environment}/data/common.yaml``
Depending if a subgroup is defined, the node specific YAML is at a different level in the filesysystem hierarchy.
The ``%{variable}`` notation is hiera specific.
## Repositories
Hiera data are organized in different repositories. These repositories are located at: https://git.psi.ch/linux-infra/hiera
Each __sysdb environment__ has a dedicated hiera repository, called ``data-<sydbenv>``, eg. [data-hpc]( https://git.psi.ch/linux-infra/hiera/data-hpc).
The first 4 levels of the filesystem structure shown before are actually the files inside this kind of repositories.
Any change to the repo will automatically trigger a redeployment of the new version of its content on the puppet master within a few seconds from the push.
## Configuration
### Secrets
Secrets and clear-text values can be mixed inside the same yaml file, eg.::
```yaml
ntp_client::servers:
- pstime1.psi.ch
- pstime2.psi.ch
- pstime3.psi.ch
secret_key: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAY/9V1S0VAMrRX1B4V06AgsbHPHdONFCQ4RiWfTrhV02rL5gSL4LAdqOuvGPY8YZZv8Mp06/FARlvP1aOfEx7avqSBy11IoUGkeajKZFzJV3OJsfhso4wroQ4JmfBaVKICnQZwCdpke+PHPRkwTgHcjmY2FeBnhvOlrGiQMQU3JzCjLePOa7UvlIIin3xOU/TdetzhfvoNGRhsz7+XRPD+mTT8efJ+OslJmqU7hEqMbs9CmhPJWqsjsQUp8jsM10Dk2Rv4v+zYeJd1ZLRGK3Z56G4NrlLyYua+/yyPbUP4+1bEuisDg9bfQHp3R491/kN0W558oQ+85rsRVXCp1Hb6TBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBB2x9awGQnxAJsxIHA9OiM2gCBFvgxIR4SJZPrrQ/UlhKU39yYSkEmuKE/ou+yeIe5AMA==]
```
The encrypted values can be decrypted transparently from Hiera (on a host having the proper hiera key):
```bash
[root]# hiera secret_key
this is a secret value
```
You can edit secure data inside any yaml file with the command `/opt/puppetlabs/puppet/bin/eyaml edit common.yaml`. In this case secure data will appear in clear-text inside the editor.
### Encrypt Data
To encrypting data you have to use following public key:
```
-----BEGIN CERTIFICATE-----
MIIC2TCCAcGgAwIBAgIBATANBgkqhkiG9w0BAQUFADAAMCAXDTE2MTAyNDE0NTY1
N1oYDzIwNjYxMDEyMTQ1NjU3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA2eykSgS7VJEXrWYkQMV48ZkUVcHMbCEo2gZXD4vIJsOdJu77F7tA53Ay
NxdKnJTftsj+R7yFP9Z2XllA9Our0Ypphj40rNstRg5O4IoSkAqitJchlfGL9jZ3
CB4dJqFitzOkxxCWZjQpjBd3dMJc6U3us6IDWohCjYqyjMZIVwU5EflzJKV4haEy
Y9qHkVt938RM9UohEvia5/1lZxuZQmDpYqCw9gmBK/dVKZ7abZGkujTKAg5cjD/X
vuexLMCGrjnPdrsblwBh+yfu6cEo9nfvfj6EA0FxPHIvQ3fv1yJZ+90OA9eUJnqQ
ED66OGPATAJIqhWlgb8a760xPQFQQQIDAQABo1wwWjAPBgNVHRMBAf8EBTADAQH/
MB0GA1UdDgQWBBSF05r9TYDiAmkdguCVcDzmYR8Q6TAoBgNVHSMEITAfgBSF05r9
TYDiAmkdguCVcDzmYR8Q6aEEpAIwAIIBATANBgkqhkiG9w0BAQUFAAOCAQEAWAER
CTGsOFUkCfvqke75PmIkxKBp/2eJbavWzPkbA/mwAGS4lQc5oyS8FMkUFxATo1k/
WIb2B3WJIMHfCzMNxTlQLjJiSyvWAlEBHDW4H2XekzKSbj96l+/nirmOq3QkEKTK
omexF5zYSPkBVA/S2m2wae3g2kubH1p42+REKQUvt1+xaecHBYD6eXzBWChnMMnq
FbXoayTibn0p9Roo8HClGGJpjPZUTMf+VGUqKWPfvaKl48Y0yrc/4BzZT6Sbzeou
ZSiHwa62rTV7ia7m2SILZU5b65JUVkFH/2r6qkxCr0Ep+oaxSNXtAXLCbnXmdOeK
B40J8ePbbmmGE24+zQ==
-----END CERTIFICATE-----
```
On the puppet server this key can be found at `/etc/puppetlabs/keys/eyaml/public_key.pkcs7.pem`.
Beside this key you also need to have `hiera-eyaml` tool installed on your system.
Assuming the public key is saved in a file (e.g. ``~/eyaml_key.pub``), that the file path has been put into the environment varialbe ``EYAML_PUB_KEY``, then a string can be encripted with::
```bash
eyaml encrypt --pkcs7-public-key=$EYAML_PUB_KEY -s secret_string
```
While a complete file can be encrypted with:
```bash
eyaml encrypt --pkcs7-public-key=$EYAML_PUB_KEY -f secret_file
```
#### Example
To encrypting password for a system you can go about like this:
```bash
# openssl passwd -6 | eyaml encrypt --pkcs7-public-key=$EYAML_PUB_KEY --stdin
Password:
Verifying - Password:
string: ENC[PKCS7,MIIBxxxxxxxx...xxxxxxxx]
OR
block: >
ENC[PKCS7,MIIBxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
...
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
#
```
and place either the string or the block at the required place in your Hiera YAML.
admin-guide/puppet/hiera.rst
-188
View File
@@ -1,188 +0,0 @@
Hiera
=====
Look `here <https://docs.puppet.com/hiera/3.1/>`_ for a general Hiera
introduction.
The current hierarchy has seven levels (first will be considered first
during value lookup):
- nodes (FQDN)
- subgroup (optional, ``puppet_subgroup`` attribute in sysdb)
- group (``puppet_group`` attribute in sysdb)
- sysdb environments
- Puppet server specific
- global
- common
The first four layers can be edited by the admin in the respective hiera git repository.
The common layer (default values) and the server specific layer (differences between test and prod) are part of the Puppet code repository.
Finally the global layer contains a few configurations which are managed by the Core Linux Group outside of the normal Puppet release process, eg. for license management.
The values can be stored as classical YAML values or with `encrypted yaml
<https://github.com/TomPoulton/hiera-eyaml>`_ for secrets.
The filesystem structure is as follows:
1. ``%{::sysdb_env}/%{::group}/%{::fqdn}.yaml`` or ``%{::sysdb_env}/%{::group}/%{::subgroup}/%{::fqdn}.yaml``
2. ``%{::sysdb_env}/%{::group}/%{::subgroup}.yaml``
3. ``%{::sysdb_env}/%{::group}.yaml``
4. ``%{::sysdb_env}/%{::sysdb_env}.yaml``
5. ``%{::environment}/data/server_%{server_facts.servername}.yaml``
6. ``/srv/puppet/data/global/global.yaml``
7. ``%{::environment}/data/common.yaml``
Depending if a subgroup is defined, the node specific YAML is at a different level in the filesysystem hierarchy.
The ``%{variable}`` notation is hiera specific.
Hiera repositories
------------------
Hiera data are organized in different repositories.
Sysdb environments data
^^^^^^^^^^^^^^^^^^^^^^^
Each sysdb environment has a dedicated hiera repository, called ``data-<sydbenv>``,
eg. `data-hpc <https://git.psi.ch/linux-infra/hiera/data-hpc>`_
and `data-sls <https://git.psi.ch/linux-infra/hiera/data-sls>`_.
The first three levels of the filesystem structure shown before are actually the
files inside this kind of repositories.
Any change to the repo will automatically trigger a redeployment of the new version
of its content on the puppet master within a few seconds from the push.
This choice has been made to allow groups to change their hiera data independently of
the linux infrastructure admins. Furthermore there is no way to influence other sysdb
environments data.
Common data
^^^^^^^^^^^
The last element in the hierarchy (``common.yaml``) is instead defined inside the main puppet repository
(the one containing also the real puppet code). It is important to notice that the version
of the ``common.yaml`` used for a specific host will depend on the puppet environment it
is running on, while for the sysdb environements data are the same, whatever the puppet
environment of the host.
The common part is kept under the control of the linux infrastructure admins
since a change on this can have an impact on a much larger set of hosts and all the changes
on this file are discussed and approved through a longer process.
Example
-------
Assuming two sysdb environments ``hpc`` and ``sls``, as well as:
- group ``merlin4`` in ``hpc`` with ``merlin4l`` and in subgroup ``compute`` ``merlinc10`` and ``merlinc11``
- group ``mx`` in ``sls`` with ``mxcn-1`` and ``mxcn-2``
- host ``xbl-gateway`` in no explicit group (will take the implicit ``default``)
the Hiera structure would look like this::
data/hpc/merlin4/merlin4l.psi.ch.yaml
data/hpc/merlin4/compute/merlinc10.psi.ch.yaml
data/hpc/merlin4/compute/merlinc11.psi.ch.yaml
data/hpc/merlin4.yaml
data/hpc.yaml
data/sls/mx/mxcn-1.psi.ch.yaml
data/sls/mx/mxcn-2.psi.ch.yaml
data/sls/mx.yaml
data/sls/default/xbl-gateway.psi.ch.yaml
data/sls.yaml
code/{prod,preprod}/server_$SERVERNAME.yaml
data/global/global.yaml
code/{prod,preprod}/common.yaml
While the output of bob would be something like (some unneeded attributes have been removed)::
merlin4l.psi.ch hpc local puppet_group=merlin4
merlinc10.psi.ch hpc local puppet_group=merlin4 puppet_subgroup=compute
merlinc11.psi.ch hpc local puppet_group=merlin4 puppet_subgroup=compute
mxcn-1.psi.ch sls local puppet_group=mx
mxcn-2.psi.ch sls local puppet_group=mx
xbl-gateway.psi.ch sls local
Secret values
-------------
Secrets and clear-text values can be mixed inside the same yaml file, eg.::
ntp_client::servers:
- pstime1.psi.ch
- pstime2.psi.ch
- pstime3.psi.ch
secret_key: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAY/9V1S0VAMrRX1B4V06AgsbHPHdONFCQ4RiWfTrhV02rL5gSL4LAdqOuvGPY8YZZv8Mp06/FARlvP1aOfEx7avqSBy11IoUGkeajKZFzJV3OJsfhso4wroQ4JmfBaVKICnQZwCdpke+PHPRkwTgHcjmY2FeBnhvOlrGiQMQU3JzCjLePOa7UvlIIin3xOU/TdetzhfvoNGRhsz7+XRPD+mTT8efJ+OslJmqU7hEqMbs9CmhPJWqsjsQUp8jsM10Dk2Rv4v+zYeJd1ZLRGK3Z56G4NrlLyYua+/yyPbUP4+1bEuisDg9bfQHp3R491/kN0W558oQ+85rsRVXCp1Hb6TBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBB2x9awGQnxAJsxIHA9OiM2gCBFvgxIR4SJZPrrQ/UlhKU39yYSkEmuKE/ou+yeIe5AMA==]
The encrypted values can be decrypted transparently from Hiera (on a host having the proper hiera key)::
[root]# hiera secret_key
this is a secret value
You can edit secure data inside any yaml file with the command
``/opt/puppetlabs/puppet/bin/eyaml edit common.yaml``. In this case secure data
will appear in clear-text inside the editor.
Encrypting data with the public key
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The eyaml public key is::
-----BEGIN CERTIFICATE-----
MIIC2TCCAcGgAwIBAgIBATANBgkqhkiG9w0BAQUFADAAMCAXDTE2MTAyNDE0NTY1
N1oYDzIwNjYxMDEyMTQ1NjU3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA2eykSgS7VJEXrWYkQMV48ZkUVcHMbCEo2gZXD4vIJsOdJu77F7tA53Ay
NxdKnJTftsj+R7yFP9Z2XllA9Our0Ypphj40rNstRg5O4IoSkAqitJchlfGL9jZ3
CB4dJqFitzOkxxCWZjQpjBd3dMJc6U3us6IDWohCjYqyjMZIVwU5EflzJKV4haEy
Y9qHkVt938RM9UohEvia5/1lZxuZQmDpYqCw9gmBK/dVKZ7abZGkujTKAg5cjD/X
vuexLMCGrjnPdrsblwBh+yfu6cEo9nfvfj6EA0FxPHIvQ3fv1yJZ+90OA9eUJnqQ
ED66OGPATAJIqhWlgb8a760xPQFQQQIDAQABo1wwWjAPBgNVHRMBAf8EBTADAQH/
MB0GA1UdDgQWBBSF05r9TYDiAmkdguCVcDzmYR8Q6TAoBgNVHSMEITAfgBSF05r9
TYDiAmkdguCVcDzmYR8Q6aEEpAIwAIIBATANBgkqhkiG9w0BAQUFAAOCAQEAWAER
CTGsOFUkCfvqke75PmIkxKBp/2eJbavWzPkbA/mwAGS4lQc5oyS8FMkUFxATo1k/
WIb2B3WJIMHfCzMNxTlQLjJiSyvWAlEBHDW4H2XekzKSbj96l+/nirmOq3QkEKTK
omexF5zYSPkBVA/S2m2wae3g2kubH1p42+REKQUvt1+xaecHBYD6eXzBWChnMMnq
FbXoayTibn0p9Roo8HClGGJpjPZUTMf+VGUqKWPfvaKl48Y0yrc/4BzZT6Sbzeou
ZSiHwa62rTV7ia7m2SILZU5b65JUVkFH/2r6qkxCr0Ep+oaxSNXtAXLCbnXmdOeK
B40J8ePbbmmGE24+zQ==
-----END CERTIFICATE-----
On the puppet server it found at ``/etc/puppetlabs/keys/eyaml/public_key.pkcs7.pem``.
Then you need to have ``hiera-eyaml`` tool installed, either from the package manager of your distribution or from the `source <https://github.com/TomPoulton/hiera-eyaml>`_.
Assuming the public key is saved in a file (e.g. ``~/eyaml_key.pub``), that the file path has been put into the environment varialbe ``EYAML_PUB_KEY``, then a string can be encripted with::
eyaml encrypt --pkcs7-public-key=$EYAML_PUB_KEY -s secret_string
While a complete file can be encripted with::
eyaml encrypt --pkcs7-public-key=$EYAML_PUB_KEY -f secret_file
Example: Encrypting password
----------------------------
First prepare the public key and the shell as explaned in above chapter.
Then::
# openssl passwd -6 | eyaml encrypt --pkcs7-public-key=$EYAML_PUB_KEY --stdin
Password:
Verifying - Password:
string: ENC[PKCS7,MIIBxxxxxxxx...xxxxxxxx]
OR
block: >
ENC[PKCS7,MIIBxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
...
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
#
and place either the string or the block at the required place in your Hiera YAML.
admin-guide/puppet/puppet-master.rst
-37
View File
@@ -1,37 +0,0 @@
Puppet Master
=============
Installation
------------
Installation of a puppet master is in some way more difficult than all
the other system, since no pre-existant infrastructure is supposed to
be present. It comes natural that the Puppet master is the first
server to be deployed in the current infrastructure.
This `Git repository <https://git.psi.ch/linux-infra/bootstrap>`_
contains the code to deploy a new puppet master.
The procedure to install it are the following steps:
1. install the server, using ``puppet master`` as profile. See `this <http://boh.dont.know>`_ for more informations;
2. after the server reboot, you are be able to login with your `authorized ssh key <http://dont.know>`_;
3. copy the ssh root key to the right location with the right
permissions, so that the server is able to authenticate itself
against the git repository: ::
PUPPETMMASTER=puppetmaster.psi.ch
scp ssh_rsa_key* root@$PUPPETMASTER:/root/.ssh
ssh root@$PUPPETMASTER
chmod 400 /etc/ssh/ssh_rsa_key
chmod 444 /etc/ssh/ssh_rsa_key.pub
4. You can now deploy the puppet server with the following code: ::
git clone https://git.psi.ch/linux-infra/bootstrap
cd bootstrap/instcode
./puppetmaster.sh
admin-guide/puppet/puppet_client.md
+37
View File
@@ -0,0 +1,37 @@
# Puppet Client
The Puppet Agent updates the configuration on the node. It automatically runs regulary, for configuration details check the [configuration guide](../configuration/puppet_agent).
The puppet client can be manually triggered/run by calling following command on the client:
```bash
puppet agent -t
```
To only check what the client would actually do, without actually changing the systems configuration/files/etc. you can use the `--noop` option:
```bash
puppet agent -t --noop
```
To instruct the puppet client to use a specific puppet environment use:
```bash
puppet agent -t --environment my_environment
```
The client will keep this environment until
- either the puppet environment is removed from the server (i.e. the client will do a fallback to the default environment)
- you manually change the environment again
## Attach Node to Different Puppet Server
For testing purpose you might want to change the Puppet server to which a test node is attached to. To do so do, change the ``server`` in the ``[main]`` section of ``/etc/puppetlabs/puppet/puppet.conf`` accordingly.
Then delete the current host and CA certficate and the CRL:
#. ``rm /etc/puppetlabs/puppet/ssl/certs/*``
#. ``rm /etc/puppetlabs/puppet/ssl/crl.pem``
Finally run the puppet agent again.
admin-guide/puppet/development.md → admin-guide/puppet/puppet_environments.md
+9 -15
View File
@@ -1,8 +1,14 @@
# Development Environments
# Puppet Environments
An Puppet environment is a particular version of the Puppet code/configuration. In our infrastructure an Puppet environment always corresponds to a git branch in the main puppet codebase: https://git.psi.ch/linux-infra/puppet
To all time our Puppet server provides two environments, __prod__ and __preprod__.
## Development Environments
Beside the __prod__ and __preprod__ environments, development environments for feature branches of the puppet git repository are automatically put on and removed from the puppet server. To create a new feature branch and test environment simply:
ON YOUR COMPUTER!
__ON YOUR COMPUTER!__
1. Clone the puppet repository https://git.psi.ch/linux-infra/puppet
2. Create a new branch with a name that matches following regex: `^[a-z]+[a-z,0-9,_]+$` (e.g. my_test_branch)
@@ -24,20 +30,8 @@ The following video shows all the necessary steps in detail. (The left terminal
Once you are done with the develoment and testing create a merge request for the branch to the __preprod__ branch.
## Development Environment Names
### Development Environment Names
The name of an branch/environment must match the following regex expression: `^[a-z]+[a-z,0-9,_]+$`
If the name is not compliant with this rule, a push of the branch to the git server will not be possible.
# Attach Node to Different Puppet Server
For testing purpose you might change the Puppet server to which a test node is attached to. To do so do, change the ``server`` in the ``[main]`` section of ``/etc/puppetlabs/puppet/puppet.conf`` accordingly.
Then delete the current host and CA certficate and the CRL:
#. ``rm /etc/puppetlabs/puppet/ssl/certs/*``
#. ``rm /etc/puppetlabs/puppet/ssl/crl.pem``
Finally run the puppet agent again.
admin-guide/puppet/client.md → admin-guide/troubleshooting/puppet.md
+2 -7
View File
@@ -1,11 +1,9 @@
# Puppet Client
The Puppet Agent updates the configuration on the node. It automatically runs regulary, for configuration details check the [configuration guide](../configuration/puppet_agent).
# Puppet
## Manually Retrieve Node Information From Puppet Server
To manually check the node information on the Puppet server for given host, do
```
```bash
FQDN=$(hostname --fqdn)
curl \
--cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem \
@@ -27,6 +25,3 @@ To figure out what part of the catalog is concerned, look at `/opt/puppetlabs/pu
Then search for `\u` which starts a binary (non-utf8) character in PSON in a way that is still valid JSON.
Please improve the Puppet code wherever such non-utf8 strings are used. Be aware that not only resources, but also arguments to profiles, classes, etc. end up in the catalog.