documentation/docs/admin-guide/configuration/files/distribute_files.md

Distribute Files

With Hiera it is possible to download files and git repositories as well as to create directories and symlinks.

Create Directories

The files::directories hash specifies directories to be created. The keys of the hash are the absolute pathnames of the directories, the optional value a hash with:

• owner: file owner (optional, default root)
• group: file owner (optional, default root)
• mode: file permissions (optional, default 0775)

Parent directories are automatically created with default settings. If that is not desired, a custom definition for each parent directory is required.

Example:

files::directories:
  /etc/test1:
  /etc/test2/foo/bar:
    owner: 'buchel_k'
    group: 'unx-nogroup'
    mode: '755'

Create Symlinks

The files::symlinks hash is used to configure symlinks. The keys of the hash are the absolute pathnames of the symlinks, the values of the hash are the corresponding symlink targets.

Example:

files::symlinks:
  '/opt/foo': '/var/lib/foo'

Before RHEL9 by default the symlink definitions are not merged over the full Hiera hierarchy, only the most specific definition is used. To allow merge, set

files::symlinks::merge: true

On RHEL9 and later this is already the default.

Then also existing files and symlinks are not overwritten. Also this can be changed with

files::symlinks::force: true

but this is then applies for all symlink definitions.

Create Files with Specific Content

Textfiles with specific content can be created as follows:

files::files:
  /testdir/test/three:
    content: |
      hello world
      this is a test

Binary files you may provide with base64 encoding:

base64 < $FILE_PATH

and using the base64_content key:

files::files:
  /test/zero:
    base64_content: AAAAAAAAAAAAAA==

If the binary file has been encrypted with eyaml, then this detour is not needed, you can set the encrypted String directly with the content key. It might then look like this:

files::files:
  /var/lib/gssproxy/clients/8547.keytab:
    mode: '0400'
    content: >
      ENC[PKCS7,MIIBuQYJKoZIhvcNAQcDoIIBqjCCAaYCAQAxggEhMIIBHQIBAD
      ...
      2KJw==]

It is also possible to provide files directly from your Hiera data repository if you configure the path of the desired file within that repo with the hiera_file key:

files::files:
  /etc/logo.png:
    hiera_file: files/logo.png

Here none of the usual decryption, inheritance or interpolation functionality of Hiera is applied, the file is just copied as is.

Additional file parameters:

• owner: file owner (optional, default root)
• group: file owner (optional, default root)
• mode: file permissions (optional, default 0664)

If you want to encrypt sensitive parts of the file content, then check out the chapter "Encrypt Part of File Content/Download URL in Hiera" more below.

Set Capabilities on Files

Capabilities can be set on files as follows:

files::files:
  '/tmp/test':
    capabilities: "cap_sys_nice=p"

Download Git Repositories

To synchronize a git repository to the host you might list them in the files::git hash. The key is the destination directory and the value is a hash with following keys (all optional):

• url: URL of the git repository to clone, without it will create a central repository
• revision: what branch, tag or commit-hash should be checked out, if not set it will use HEAD on creation and after leave it as is
• ensure: with the values
  • present: the git repository is cloned according to the other options, i.e. check out or switch to local copy of specified revision if provided
  • latest: the checked out branch is up to date with upstream
  • bare: creates a bare repository
  • mirror: creates a mirroring repository
  • absent: removes the repository
• owner: what user owns the files in the repository
• group: what group owns the files in the repository
• user: what user shall be used to run the git commands
• umask: umask to use for all file operations
• ... for all options see the vcsrepo documentation

Example:

files::git:
  /var/test/container-images:
    ensure: latest
    url: 'https://gitea.psi.ch/linux/documentation.git'
    revision: 'main'

Note that submodules are automatically initialized, if you wish different set submodules: false.

If you need to use a token to access the repository (how to get a token), you can

• include it into the URL (which makes to token visible inside the checkout as it is part of the remote configuration)
• rewrite the URL in ~/.gitconfig of user (which then will cover all git operations with the respective URL prefix):

[url "https://a84t...4ksf@gitea.psi.ch/"]
    insteadOf = https://gitea.psi.ch/

See the eyaml guide for how to best deal with encrypting the token, or below full example:

gitea::token::lx-gitea: 'ENC[PKCS7,MIIBmQYJKoZI...dvkM=]'

files::git:
  /srv/pupped/code/prod:
    url: 'https://gitea.psi.ch/puppet/puppet.git'
    ensure: latest
    branch: prod
  /srv/pupped/code/preprod:
    url: 'https://gitea.psi.ch/puppet/puppet.git'
    ensure: latest
    branch: preprod
  /home/lx-test/puppet:
    url: "https://%{lookup('gitea::token::lx-gitea')}@gitea.psi.ch/puppet/puppet.git"
    user: lx-test

files::files:
  '/root/.gitconfig':
    mode: '0400'
    content: |
      [url "https://%{lookup('gitea::token::lx-gitea')}@gitea.psi.ch/"]
          insteadOf = https://gitea.psi.ch/

Download Files

It is possible to download files from any webserver (including our Git forge gitea.psi.ch).

Note that once downloaded files are kept on the Puppet server. On each Puppet run a new download is made, but upon failure the already existing file is delivered. Still there is a failure message printed, but the Puppet run as such does not fail.

Single Downloads

"Simple" downloads can go with the hash files::download where you can map destination file paths to URLs or with more detailed configuration including token and destination file permissions:

files::download:
  '/tmp/dummy.pdf': 'https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf'
  '/tmp/dummy.png':
    url: 'https://www.filesampleshub.com/download/image/png/sample1.png'
    owner: 'buchel_k'
    group: 'unx-nogroup'
    mode: '0444'

Download Token

If a token is needed to access the web or git server, you may provide it with the token parameter.

files::download:
  '/tmp/dummy.png':
    url: 'https://www.filesampleshub.com/download/image/png/sample1.png'
    token: 'token: 'ENC[PKCS7......]'

If you need a token for gitea.psi.ch see how to get a token.

Download Lists

For more complex configurations you can do "file download lists" which can be prepared e.g. for all hiera environment and then selected on lower groups or single nodes.

The list configuration allows to set the default values and an url-prefix which is then used for all downloaded files if not specified in more detail for the individual files:

files::download_list::test:
  url-prefix: 'https://gitea.psi.ch/linux/documentation/raw/branch/master/'
  token: ENC[PKCS7,MIIBmQ...Lg=]
  files:
    '/tmp/index.md':
      url-end: 'index.md'
    '/tmp/Readme.md':
      url-end: 'Readme.md'
      owner: 'buchel_k'
      group: 'unx-nogroup'
      mode: '0444'
    '/tmp/index.html': 'https://linux.psi.ch/'

files::download_lists:
  - 'test'

As you see in above example the actual download URL is combined with url-prefix from the list parameters and the url-end parameter for individual files. Or alternatively a complete URL may be set on file level instead with the url parameter or just as plain string in place of the named parameter list.

Encrypt Part of File Content/Download URL in Hiera

If you want to provide a secret like a password or a token inside the file content or as part of an URL, store the eyaml encrypted encrypt secret in an else unused Hiera key and include it into the actual place using string interpolation and the lookup function:

my-project::my-user::my-password: 'ENC[PKCS7,MIIBmQYJKoZI...dvkM=]'

files::files:
  /etc/my-software.conf:
    mode: '0400'
    content: |
      db-user = my-user
      db-password = %{lookup('my-project::my-user::my-password')}

Delete Files/Directories

Individual files and directories can be deleted as follows:

files::files:
  /testdir/test/two:
    ensure: absent
    force: true

The option force: true is only needed for directories. For a directory, if absent, the directory will not be deleted (even if it is empty)!

Known Problems

Downloading TAR File Fails

Following warning/error happened with an uncompressed tar file being distributed:

Warning: Using a checksum in a file's "content" property is deprecated. The ability to use a checksum to retrieve content from the filebucket using the "content" property will be removed in a future release. The literal value of the "content" property will be written to the file. The checksum retrieval functionality is being replaced by the use of static catalogs. See https://puppet.com/docs/puppet/latest/static_catalogs.html for more information.
...
Error: /Stage[main]/Profile::Files/File[/test/test.tar]: Could not evaluate: invalid byte sequence in UTF-8

Somehow Puppet wrongly interprets the content. The workaround is to send a compressed file (e.g. test.tar.bz2).

Hints how to disable this feature are welcome!