Benjamin LabrecqueandClaude Opus 5
cfd3ecad6c
Deploy-time placeholders were spread across four independent
implementations: sed in the Gitea deploy script, ansible.builtin.replace
in two playbooks, a hardcoded suffix in bin/sls_hla_launch_gui.sh, and
another in agebd/pv.py. Each defined its own values, so nothing kept them
in sync and a missed substitution failed silently.
Step 1 -- one source of truth:
* config/environments.yml declares every deploy-time token, its value
per environment, and where tokens are allowed to live. An environment
that is missing a token, or defines an undeclared one, is a load-time
error.
* agebd.subs loads it and renders. Rendering is a strict regex over
declared token names rather than a template engine: the rendered
files are EPICS db/substitution files and Qt XML, and an unknown
token must raise rather than survive into the output.
* scripts/ci/check_subs.py enforces the contract in CI: tokens in a
rendered path are declared, declared tokens only appear where they
are rendered or generated, and no pre-migration delimiters survive.
Step 2 -- separate the two substitution stages by delimiter:
* Creation-time (copier, once per new service) keeps {{ token }}.
Deploy-time (every deploy, per environment) becomes @@token@@.
* This drops the {{ '{{ token }}' }} escaping from templates/, which
existed only because one Jinja pass had to emit syntax for a second.
* Angle brackets were tried first and rejected: << >> is not
well-formed XML, and cli parses qt/*.ui with ElementTree.
* The existing sed and ansible consumers are repointed at the new
delimiters so this stays behaviour-preserving. Routing them through
agebd.subs is step 3.
Also fixes, as a side effect of having one implementation:
* The sed pattern '{{ *agebd_env_suffix_upper* }}' quantified 'r'
rather than the space, so it only matched one exact spacing.
* agebd.subs.read_ioc_port compares service names case-insensitively;
ansible/tasks/utils/get-ioc-port.yml still compares CamelCase
registry names against a lowercase input and never matches.
qt/*.ui is declared as a render path but nothing renders it yet -- the
Qt deploy step is still a plain rsync. Wiring that up is step 4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
132 lines
4.4 KiB
Python
132 lines
4.4 KiB
Python
"""
|
|
Guard the deploy-time substitution contract.
|
|
|
|
Every ``@@token@@`` in this repo must be declared in config/environments.yml,
|
|
and must live somewhere that actually gets rendered. Three things are checked:
|
|
|
|
1. Tokens inside a render path are declared.
|
|
Catches typos, which would otherwise ship the literal placeholder.
|
|
|
|
2. Declared tokens only appear in a render, passthrough or reference path.
|
|
Catches a placeholder added to a file nobody renders -- exactly the
|
|
regression that put a suffix token into qt/*.ui while the Qt deploy
|
|
step was a plain rsync.
|
|
|
|
3. No pre-migration ``{{ token }}`` form of a deploy-time token survives,
|
|
in the paths that carry tokens. Catches files missed by the switch to
|
|
the ``@@token@@`` delimiters. Reference paths are exempt: Ansible and
|
|
copier own ``{{ }}`` in their own files, and some of their variables
|
|
share a name with our tokens.
|
|
|
|
Files tracked by git plus new, non-ignored files are scanned, so build and
|
|
cache directories are skipped for free.
|
|
"""
|
|
|
|
import sys
|
|
|
|
import git
|
|
from agebd.subs import (
|
|
SubstitutionsConfig,
|
|
find_legacy_tokens,
|
|
find_tokens,
|
|
)
|
|
from agebd.utils import get_git_root
|
|
|
|
REPO_ROOT = get_git_root(__file__)
|
|
|
|
|
|
def iter_tracked_text_files(repo_root=REPO_ROOT):
|
|
"""
|
|
Yield (relative_path, contents) for every UTF-8 decodable file that is
|
|
tracked, or new and not gitignored. New files are included so that a bad
|
|
token is reported before it is committed rather than after.
|
|
"""
|
|
repo = git.Repo(repo_root)
|
|
listing = repo.git.ls_files("--cached", "--others", "--exclude-standard")
|
|
for rel_path in sorted(set(listing.splitlines())):
|
|
path = repo_root / rel_path
|
|
if not path.is_file():
|
|
continue
|
|
try:
|
|
yield rel_path, path.read_text()
|
|
except (UnicodeDecodeError, OSError):
|
|
continue
|
|
|
|
|
|
def check_declared(config, rel_path, contents):
|
|
"""1. Tokens inside a render path must be declared."""
|
|
if not config.is_render_path(rel_path):
|
|
return []
|
|
|
|
return [
|
|
f"{rel_path}: '@@{token}@@' is not declared under 'tokens' in "
|
|
f"{config.environments_file.name}"
|
|
for token in sorted(find_tokens(contents))
|
|
if token not in config.tokens
|
|
]
|
|
|
|
|
|
def carries_tokens(config, rel_path):
|
|
"""
|
|
True for paths whose files carry deploy-time tokens as data: they are
|
|
either rendered, or copied into something that is.
|
|
"""
|
|
return config.is_render_path(rel_path) or config.is_passthrough_path(rel_path)
|
|
|
|
|
|
def check_location(config, rel_path, contents):
|
|
"""2. Declared tokens must sit in a render, passthrough or reference path."""
|
|
if carries_tokens(config, rel_path) or config.is_reference_path(rel_path):
|
|
return []
|
|
|
|
return [
|
|
f"{rel_path}: '@@{token}@@' is in a path that is never rendered. In "
|
|
f"{config.environments_file.name}, add the path to 'locations.render' if it "
|
|
f"should be substituted at deploy time, or to 'locations.references' if it is "
|
|
f"part of the substitution machinery. Otherwise remove the token."
|
|
for token in sorted(find_tokens(contents) & set(config.tokens))
|
|
]
|
|
|
|
|
|
def check_no_legacy_delimiters(config, rel_path, contents):
|
|
"""
|
|
3. No deploy-time token may still use the creation-time delimiters, in the
|
|
paths that carry tokens. Elsewhere {{ }} belongs to whichever tool owns
|
|
the file.
|
|
"""
|
|
if not carries_tokens(config, rel_path):
|
|
return []
|
|
|
|
return [
|
|
f"{rel_path}: '{{{{ {token} }}}}' uses the creation-time delimiters for a "
|
|
f"deploy-time token. Write it as '@@{token}@@'."
|
|
for token in sorted(find_legacy_tokens(contents) & set(config.tokens))
|
|
]
|
|
|
|
|
|
CHECKS = (check_declared, check_location, check_no_legacy_delimiters)
|
|
|
|
|
|
def check_subs(repo_root=REPO_ROOT):
|
|
"""
|
|
Raise AssertionError listing every violation, or return silently.
|
|
"""
|
|
config = SubstitutionsConfig.load(repo_root / "config")
|
|
|
|
errors = []
|
|
for rel_path, contents in iter_tracked_text_files(repo_root):
|
|
for check in CHECKS:
|
|
errors.extend(check(config, rel_path, contents))
|
|
|
|
if errors:
|
|
raise AssertionError("Deploy-time substitution problems found:\n\n " + "\n ".join(errors))
|
|
|
|
|
|
if __name__ == "__main__":
|
|
try:
|
|
check_subs()
|
|
except AssertionError as e:
|
|
print(e, file=sys.stderr)
|
|
sys.exit(1)
|
|
print("All deploy-time tokens are declared and live in a rendered path.")
|