pixi-deploy/autodeploy/TODO.md

TODO

• ## Make envs available in jupytera

There's pixi-kernel but it is specifically "per directory", which does not fit our use case.

However, Pixi envs are compatible with nb_conda_kernels, which Leo installs on jupytera already.

The only "problem" is in how jupyter shows the env names:

• prefix/envs/envname => prefix-envname

... which means that all pixi envs would show as .pixi-envname, which is weird. This can be solved by adding a symlink:

• /sf/alvra/envs -> /sf/alvra/config/python/.pixi/envs

... so we'd get alvra-envname. As a bonus, the users may now add very intuitive paths to their .condarc.

Note: if the envs folder is called differently, no prefix is shown at all... This might be an alternative idea.

Solution: setup.sh creates/updates the symlink as part of the procedure.

• ## Shared features

Pixi supports "features" (i.e., sets of packages that allow the user to do XYZ). This would be very useful for, e.g., a feature like "be able to plot on jupytera", which gives the correct versions of ipympl, matplotlib, etc.

We would need this to be a central file that is somehow made available and/or merged into everyone's individual files.

Pixi does not seem to have this option (yet? see below). TOML does not allow to link in further files out of the box. However, there is tomlincl.

Opened a ticket and the devs proposed a solution. Need to test!

• ## Running the setup script

Currently, there runs a deploy script as part of the deployment. We do NOT want to replace that script, but run after it.

Can this be changed such that the deploy script runs our script at the end? Where would our script live?

Solution: Simon calls /sf/daq/code/snek/autodeploy/setup.sh at the end of the deploy script with /sf/alvra/config/python as working dir.

• ## Timeout

Currently the timeout of the autodeployer is rather short as the expectation is that there are only "filesystem copy tasks" performed.

Conda env creation can take a few minutes even with Pixi.

Can the timeout be made that long without negative implications? Can this, for instance, block other tasks?

Solution: Simon set up a separate docker container with increased time out of 5 mins (to be confirmed this is enough).

• ## Trigger only on change of specific file

In case of changes to the env, the lock file (pixi.lock) should be committed at the end of the setup script to have a history and reproducibility.

Currently, this will trigger an additional autodeployer run which does nothing. This is a bit ugly.

Can we instead trigger the autodeployer only if pixi.yml changes?

gitlab webhooks are not that fine grained. Filter on specific file would need to happen in Simon's webhook handler web server. Might be too complicated a change. Can we stay with the extra do-nothing run?

• ## Commit from the Autodeployer

In case of changes to the env, the lock file (pixi.lock) should be committed at the end of the setup script to have a history and reproducibility.

We need to allow the autodeployer to make commits.

Options:

1. One Project Access Token per repo. Potentially more complicated since each repo needs its own token and we'd manage a long list of them in the docker container.
2. Personal Access Token of a special account (like the DevOps account). The special account only needs to be added as developer to each repo. Potentially easier since it's a single token that needs to be added to the docker container. There's a mechanism we can use in form of ~/.netrc (with chmod 600 ):

machine git.psi.ch
  login oauth2
  password personal-access-token-for-special-account

Solution: Simon has set it up such that the Autodeployer can deploy. Not sure which mechanism is used.

• ## Mounts

Problem

At creation time, conda envs hard code their location to several files.

From Simon:

The autodeployer runs in a podman container which maps:

• /root/target (in container world)

to:

• /net (in autodeployer machine host world)

Then, the autodeployer machine host has a diskmount that maps:

• /net

to:

• /gfa05

This means that references in the autodeployer config file such as the following:

• gfa05/export/sf/alvra/config/python

result in writes to the combined path:

• /net/gfa05/export/sf/alvra/config/python

Possible solution

Temporarily bind the autodeployer path to the console path:

sudo mkdir -p /sf/alvra/config/python
sudo mount --bind /root/target/gfa05/export/sf/alvra/config/python /sf/alvra/config/python
# install conda envs into /sf/alvra/config/python
sudo umount /sf/alvra/config/python
sudo rmdir -p /sf/alvra/config/python

Solution: bind mounts do not work in unprivileged containers. Simon now mounts the console path when starting the docker container.