CSCS/Waldur authentication with reusable credentials
project NH allocation and quarterly usage reporting
optional storage reporting
project team and role information
detailed, table, colon-delimited CSV and JSON output
underused NH allocation email notifications with preview and per-project approval

Files

cscs_portal/auth.py – secure Chromium login and system keyring cache
cscs_portal/client.py – authenticated Waldur requests client
cscs_portal/project.py – projects, NH allocations, quarterly usage, storage
cscs_portal/team.py – project membership, role mapping and selectable fields
cscs_portal/output.py – detailed, fixed-width table and colon-delimited CSV
cscs_portal/notifications.py – notification calculation, YAML templating and SMTP
project_info.py – project reporting CLI, including JSON output
project_notify.py – underused NH project selection, notification preview and per-project approval
authenticate.py, logout.py – credential management
templates/underused_nh.yaml – editable notification email subject, body and optional SMTP defaults
tests/test_reporting.py – offline reporting regression tests
tests/test_notifications.py – offline notification tests
tests/test_smtp_config.py – SMTP/header precedence and security tests

Setup

Create and activate the Conda environment:

conda env create -n cscs-portal -f environment.yml
conda activate cscs-portal

All required Python dependencies, including the notification and keyring packages, are included in environment.yml.

Install the Playwright browser:

playwright install

Authentication uses Playwright-bundled Chromium only. The CSCS password/MFA remain in Chromium. The resulting Waldur token is held in an OS keyring and reused and validated on later invocations.

Authenticate explicitly with:

python authenticate.py

Clear the locally cached Waldur token with:

python logout.py

Keyring backend

If the operating system does not provide an existing keyring backend, a message such as the following may appear:

Unable to read OS keyring: No recommended backend was available.

The Conda environment includes keyrings.cryptfile, which can provide an encrypted file-based keyring backend when a suitable OS keyring such as gnome-keyring is not available.

Authentication may then open CSCS/Waldur authentication in Chromium and request CSCS IAM/MFA as needed.

Project reporting

Examples

Default output includes NH allocation, usage, remaining NH and usage percentage, plus PI name, UID and email. Storage and team status are opt-in.

python project_info.py psi psigpu psi01 --quarter 2026-Q3 --table

Include storage information:

python project_info.py psi psigpu psi01 --quarter 2026-Q3 --table --storage

Show team status explicitly:

python project_info.py psi psigpu psi01 --quarter 2026-Q3 --table --team-status

Use a custom authentication window, in seconds:

python project_info.py psi01 --quarter 2026-Q3 --login-timeout 900
python authenticate.py --login-timeout 900

The default browser-login timeout is 28800 seconds.

Also report Deputy PI names, UIDs and emails:

python project_info.py psi psigpu psi01 --quarter 2026-Q3 --table --deputy-pi

Report Project Member login UIDs, but not member emails:

python project_info.py psi psigpu psi01 --quarter 2026-Q3 --table --member-uids

Report member emails:

python project_info.py psi psigpu psi01 --quarter 2026-Q3 --table --member-emails

PI information is always included unless --no-team is used.

Enable all optional team field groups, including additional discovered roles:

python project_info.py psi psigpu psi01 --quarter 2026-Q3 --all-team --csv > projects.csv

Select individual groups in colon-separated CSV:

python project_info.py psi psigpu psi01 --quarter 2026-Q3 --csv \
    --deputy-pi --member-uids --member-emails > projects.csv

Detailed output is the default. JSON can be selected explicitly, or team lookups can be disabled entirely:

python project_info.py psi01 --quarter 2026-Q3
python project_info.py psi01 --quarter 2026-Q3 --json --member-uids
python project_info.py psi01 --quarter 2026-Q3 --no-team --table

--csv, --table and --json are mutually exclusive.

--deputy-pi, --member-uids and --member-emails can be combined freely. --all-team enables all groups and exposes newly discovered roles. --no-team is an explicit opt-out and cannot be combined with other team-selection options.

UIDs are the CSCS login names in Waldur's user_username field, not numeric Unix UIDs or user UUIDs.

Output defaults

By default, detailed, table, colon-delimited CSV and JSON output show NH compute figures and the currently assigned PI.

Use --storage to include:

storage allocation
storage used
signed remaining storage
storage usage percentage
billing month
reporting status
inodes

Not requesting storage avoids the per-month storage usage API queries.

Both NH and storage percentages are calculated as:

used / allocated * 100

and rounded to two decimal places.

Table and detailed output include the % suffix. CSV percentages are numeric without the suffix and retain two decimal places. JSON contains percentage numbers rounded to two decimals.

Missing percentage values, or values with zero allocation, are represented as unknown:

- in tables
empty in CSV
null in JSON

Team status is hidden by default. Use --team-status to include it. The detailed view still reports a membership access error when relevant even if the status field itself is hidden.

--no-team disables member lookups and cannot be combined with --team-status.

The browser-login timeout is 28800 seconds by default. Override it with --login-timeout SECONDS on project_info.py or authenticate.py. The selected timeout is reused if API authentication needs refreshing after a 401. A valid cached credential skips interactive authentication.

For current-quarter storage, missing monthly data is not converted to zero. A last-reported-month reading is labelled accordingly. This logic runs only when --storage is used.

Examples:

# Compact NH + PI overview
python project_info.py psi psigpu psi01 --quarter 2026-Q4 --table --deputy-pi

# Include storage and show team status explicitly
python project_info.py psi psigpu psi01 --quarter 2026-Q4 \
    --table --deputy-pi --storage --team-status

# Colon-separated CSV with optional storage fields
python project_info.py psi psigpu psi01 --quarter 2026-Q4 \
    --csv --storage --deputy-pi > projects.csv

# JSON with storage but not team status
python project_info.py psi01 --quarter 2026-Q4 --json --storage

CSCS role mapping

The CSCS deployment's observed role identifiers are mapped as follows:

Actual role name	Report label
`PROJECT.MANAGER`	PI
`PROJECT.ADMIN`	PI
`PROJECT.DEPUTY`	Deputy PI
`PROJECT.MEMBER`	Project Member

This intentionally differs from the generic role mapping in older published CSCS documentation: the observed membership data puts the actual PI in PROJECT.MANAGER, while PROJECT.DEPUTY contains deputy PIs.

Review CSCS_ROLE_LABELS in cscs_portal/team.py if the deployment changes.

Unknown roles remain available unmodified under --all-team. Members with multiple roles appear in the corresponding groups. Missing identity fields are left empty rather than guessed.

CSV format

CSV uses a colon (:) as the field separator.

Inside each role-specific column, multiple member names, usernames or emails are separated with commas. Commas inside member lists therefore do not create additional CSV fields.

The implementation uses:

csv.writer(delimiter=':')

and quotes values containing a colon or newline.

Import the resulting file with Python:

import csv

with open('projects.csv', newline='', encoding='utf-8') as handle:
    rows = list(csv.DictReader(handle, delimiter=':'))

or pandas:

pandas.read_csv('projects.csv', sep=':')

Data notes

Team membership is current, even if NH or storage are reported for a past quarter.
Team membership does not include pending invitations.
The project team is not queried when --no-team is used.
403/404 team-access errors set team_status=unavailable instead of claiming that there are no members.
Summary CSV/table output shows the first compute component and, when --storage is enabled, the first storage component.
Warnings about multiple components go to stderr; detailed and JSON output expose all selected components.
NH units are normalized to NH in all formats.
Negative remaining NH values remain negative.
Historical storage allocation is the currently exposed Waldur limit.
The stored billing_period describes the usage month, not the exact measurement time.
CSV and JSON may contain personal identities. Share and store exports accordingly.

Underused NH allocation notifications

project_notify.py identifies projects whose NH utilization is below a configured threshold, prepares an email for each eligible project, displays a complete preview and optionally sends each message after explicit approval.

It uses the existing authentication, Waldur client, project information and team-role mapping used by the reporting functionality.

Preview projects below the threshold

Preview projects below 50% NH usage without sending:

python project_notify.py psi01 psi02 psi05 --dry-run

Use a custom threshold, for example strictly less than 35% used:

python project_notify.py psi01 psi02 psi05 --threshold 35 --dry-run

Include Deputy PI email addresses as additional To recipients:

python project_notify.py psi01 psi02 --deputy-pi --dry-run

Inspect a historical quarter:

python project_notify.py psi01 psi02 --quarter 2026-Q3 --dry-run

For each project, NH allocations are summed across every compute component before utilization is calculated.

The comparison with --threshold uses the unrounded value:

NH used / NH allocated * 100

The preview displays the percentage rounded to two decimal places.

Projects are skipped when they have:

missing usage data
zero NH allocation
inaccessible team data
no PI email address

Projects named psi and psigpu are skipped by default because they are shared allocation pools. Use --include-shared-projects to include them.

Notification recipients

The To header contains all current PI email addresses.

When --deputy-pi is enabled, current Deputy PI email addresses are also included.

CC defaults to:

psi-hpc-at-cscs-admin@lists.psi.ch

It can be configured through the environment, YAML template or --cc-address.

Duplicate email addresses are removed.

PI names come from the current CSCS Waldur project team, not a historical membership snapshot. The notification code does not guess the PI's surname or academic title.

Notification template

The email content is defined in:

templates/underused_nh.yaml

Both subject and body are required. The email and smtp mappings are optional.

For example:

subject: "CSCS NH allocation review - $PROJECT_ID ($QUARTER)"

body: |
  Dear $PI_GREETING,
  # ...notification body...

email:
  from_address: 'marc.caubet@psi.ch'
  cc: 'psi-hpc-at-cscs-admin@lists.psi.ch'
  reply_to: 'psi-hpc-at-cscs-admin@lists.psi.ch'

smtp:
  host: 'smtpint.psi.ch'
  port: 587
  security: none
  username: null

The distributed template leaves smtp.host pointing to localhost. With this configuration, it is assumed that the host where the script is being executed is able to send emails.

Otherwise, if this is a PSI Puppet-managed machine, one can grant a system access to send emails via Mail Gateway by using the corresponding PSI Service Now: Register E-Mail Sender form.

Alternatively, one can set a smtp.host which is reachable and capable for sending emails (e.g. merlindssmgmt01.psi.ch, which currently only supports email relay from 129.129.186.0/24 and 192.168.194.0/24).

SMTP passwords must not be stored in the YAML template. The template loader rejects smtp.password and email.password. These values must be configured through the environment, CLI or template before sending. Dry runs do not require them.

The template is loaded with yaml.safe_load; it is data rather than executable code.

Template placeholders

The following forms can be used as ${PLACEHOLDER} or $PLACEHOLDER:

Variable	Meaning
`$PROJECT_ID`	Waldur project name
`$QUARTER`	Quarter, for example `2026-Q4`
`$QUARTER_END`	Quarter end date, for example `2026-12-31`
`$AS_OF_DATE`	Date on which the draft is prepared
`$PI_GREETING`	PI name(s), without a guessed title
`$PI_NAMES`	Comma-separated PI full names
`$NH_ALLOCATED`	NH allocation formatted to two decimals
`$NH_USED`	Consumed NH formatted to two decimals
`$NH_REMAINING`	Signed remaining NH formatted to two decimals
`$NH_PERCENTAGE`	Unrounded utilization ratio displayed to two decimals

Use $$ for a literal $.

The script rejects undefined placeholders, missing required template fields, multiline subjects and malformed recipient email addresses.

The suggested wording is deliberately forward-looking. It does not assume that resources can be transferred automatically and does not presume an academic title such as "Dr.". It asks the PI to estimate unused NH and acknowledges that reallocation is subject to CSCS processes.

SMTP configuration

The host environment is consulted before the YAML template.

There is no portable OS-wide setting that reliably identifies the appropriate SMTP relay on every Linux system, so an SMTP relay or sender address is not guessed. An existing local MTA can instead be configured explicitly using localhost SMTP.

Configuration precedence is:

CLI option
    ↓
PSI_SMTP_* environment variable
    ↓
SMTP_* environment variable
    ↓
YAML setting
    ↓
built-in default

For example (note that the examples are only for reference):

export PSI_SMTP_HOST='merlindssmgmt01.psi.ch'
export PSI_SMTP_PORT='587'
export PSI_SMTP_SECURITY='none'
export PSI_SMTP_FROM='marc.caubet@psi.ch'

# Optional
export PSI_SMTP_CC='psi-hpc-at-cscs-admin@lists.psi.ch'
export PSI_SMTP_REPLY_TO='psi-hpc-at-cscs-admin@lists.psi.ch'
export PSI_SMTP_USER='caubet_m'

The generic equivalents can also be used:

SMTP_HOST
SMTP_PORT
SMTP_SECURITY
SMTP_FROM
SMTP_CC
SMTP_REPLY_TO
SMTP_USER

The PSI_SMTP_* forms take priority when both forms are defined.

When SMTP authentication is required, the program asks for the password on the first approved send.

Alternatively, PSI_SMTP_PASSWORD or SMTP_PASSWORD can be supplied through an existing secure secret provider.

Passwords are never printed in previews or stored in YAML.

SMTP security

Remote SMTP delivery uses either:

STARTTLS, port 587 by default
implicit SSL, port 465 by default

Normal TLS certificate verification is used. The port can be overridden.

To submit through a local MTA:

smtp:
  host: '127.0.0.1'
  port: 25
  security: none

Unencrypted none mode is permitted only for:

localhost
127.0.0.1
::1

SMTP authentication is not permitted in this mode.

Sending notifications

Use configured environment or template defaults:

python project_notify.py psi01 psi02 psi05 --threshold 50

Or supply one-off command-line overrides:

python project_notify.py psi01 \
    --smtp-host smtpint.psi.ch \
    --smtp-port 25 \
    --smtp-security none \
    --from-address marc.caubet@psi.ch \
    --cc-address psi-hpc-at-cscs-admin@lists.psi.ch

Preview only, without connecting to SMTP:

python project_notify.py psi01 psi02 --dry-run

For every eligible project, the program displays:

To
CC
From, when configured
optional Reply-To
Subject
complete message body

It then asks:

Send this email? [y/N]

Only y or yes sends that individual email. Pressing Enter or providing any other response skips it.

A delivery error is reported but does not stop processing subsequent projects.

Other options include:

--smtp-user
--reply-to
--template
--login-timeout SECONDS

The default login timeout is 28800 seconds when supported by the client.

Historical-quarter safety

--quarter can be used to select usage from a past quarter, but historical requests are preview-only by default. Sending historical-quarter notifications requires:

--allow-historical-send

in addition to the normal per-email confirmation.

This feature is in general useless when using the default template for sending emails, but it is kept in case one needs to generate new templates and notify information corresponding to previous quarters.

Tests

Run the complete offline test suite:

python -m unittest discover -s tests -v

The notification tests mock Waldur and SMTP. They do not verify the organization's SMTP configuration and do not send messages.

Before processing multiple projects, use --dry-run and then test a single real project with the normal per-email confirmation.

README.md Unescape Escape

CSCS portal helper