Files
muSRFlow/docs/WEB_USAGE_COLLECTION.md

8.0 KiB

Web usage collection for AI fitting models

Status: design / not yet implemented. This document records the agreed approach so a future implementation can follow it directly. It is deliberately short on prose and long on code anchors and schema, because the goal is to become the implementation brief.

1. Goal

Collect, from real web usage, the mapping between

  • feature: the fit configuration a user assembled in the GUI (instrument, runs, model components, parameter table, ranges, minimizer), and
  • label: the outcome musrfit actually produced (converged?, statistic, fitted parameter values),

so the resulting corpus can later train AI-fitting models that propose/set initial guesses and model structures from a data set.

The share link already encodes the feature side (see §4). The label side only exists inside the CGI job. Both are joined server-side.

2. Non-goals / boundaries

  • No user identity, no IP logging, no absolute file paths or data-root paths.
  • Nothing is ever written unless two gates both hold at request time:
    1. the CGI was started with MUSRFIT_COLLECTION_ENABLED=1, and
    2. the client declared consent for the request (usage.consent === true).
  • Collection is a best-effort side effect. It never blocks, fails, or changes a fit/plot outcome. If the usage write fails, log to stderr and continue.
  • The site presents a clear, one-time disclosure before the first captured fit ("We anonymously record fit configurations and outcomes to improve the fit assistant. See [docs]. You can disable this at any time."), then enables collection.
  • A Usage / Privacy toggle lives in the help/settings UI; when checked off, the client stops sending the usage field and (optionally) purges its stored session id.
  • A per-browser state persisted in localStorage under musrflowUsageConsent ("1"/"0").
  • The server-side env gate means a fresh or private deployment collects nothing until its operator explicitly enables collection, irrespective of the client flag.

The web client already owns this machinery:

  • buildSharedConfiguration() (in muSRFitlib.js) assembles the full GUI configuration object.
  • encodeWebSharedConfiguration(config) (muSRFitweb.js:322) wraps it into the share-link format via encodeSharedConfigurationLink(... gzipText) (muSRFitlib.js:186).
  • decodeWebSharedConfiguration() is the matching decode side.

Implementation note: the client does not need to copy the share-link code. It reuses buildSharedConfiguration() and sends the plain JSON object (not the gzipped URL fragment) as a usage form field on fit/plot requests:

usage := JSON({
  consent: true,
  session: "<anonymous uuid persisted under musrflowUsageSession>",
  config: buildSharedConfiguration()      // run numbers, not paths
})

The config object contains run numbers and instrument/model settings — path free. Before sending, assert the serialized object does not contain MUSRFIT_DATA_ROOT/absolute paths (test guard).

5. Server-side capture (single_musrfit.cgi)

Hook point: after a fit/plot action has produced its outcome and just before the final respond_json(...). The CGI has everything needed in scope:

  • the job directory $TMPROOT/$job_name;
  • the input MSR text staged for this run;
  • the resolved musrfit executable version (a --version probe already exists in the health path, cgi/~line 90);
  • the outcome (exit status, converged flag, statistic).

Behavior:

if ($ENV{MUSRFIT_COLLECTION_ENABLED} && $usage && $usage->{consent}) {
    usage_append_row(\%row);   # never throws
}

usage_append_row:

  • computes a per-row "id" (UUID) — do not reuse $job_name so job directories can still be aged out;
  • opens $USAGE_ROOT/<yyyy-mm>.jsonl with open(">>", ...), grabs flock(LOCK_EX), writes one JSON line, unlocks, closes;
  • on failure, warn "usage collection: $!" and continue.

6. Storage layout

$USAGE_ROOT        # env MUSRFIT_USAGE_ROOT; default MUSRFIT_DATA_ROOT/../usage
  2026-09.jsonl            # one JSON object per captured fit (appended)
  2026-09.jsonl.gz         # rotated+compressed previous month
  purge_marker
  • Rotation: at first write of a new month, compress the prior month (gzip) and remove it from the hot path. A cron-style aging routine (same pattern as age_out_web_jobs, cgi/~line 1517) can enforce a retention cap (MUSRFIT_USAGE_RETENTION_MONTHS, default 24) and a disk cap.
  • Atomicity: flock-append is sufficient; rows are independent.

7. Row schema (JSONL, "embed MSR + concise outcome")

One line per captured fit. Self-contained example:

{
  "schema": 1,
  "id": "7f9c…",
  "ts": "2026-09-11T12:34:56Z",
  "session": "f3a2…",
  "app_version": "1.0.0-rc.1",
  "musrfit_version": "1.10.14",
  "transport": "web",
  "action": "fit",
  "instrument": "LEM",
  "year": 2025,
  "runs": [2532, 2534],
  "config": { "…share-link payload…" },
  "input_msr": "…full generated MSR text…",
  "outcome": {
    "ok": true,
    "converged": true,
    "statistic": 1.012,
    "series": [ { "run": 2532, "found": true, "converged": true, "chisq": 1.011 },
                { "run": 2534, "found": true, "converged": true, "chisq": 1.013 } ]
  }
}

Field notes:

  • runs/instrument/year are denormalized copies of config for cheap filtering without parsing the whole payload.
  • input_msr embeds the exact MSR text the fit ran on — the primary feature.
  • outcome is the concise label: converged flag + statistic (single fit) or the per-run series summary. Full fitted parameter values are scored by the goal but deliberately deferred: adding par.dat requires an artifact subdir, which is a documented future step (§10).
  • schema/app_version/musrfit_version enable reproducible splits and shift detection.

8. Downstream export (scripts/usage_export.py)

A read-only Python script (stdlib + optional pandas/pyarrow):

  • walks $USAGE_ROOT, yields rows (streaming, gzip-aware);
  • optional filters: time range, instrument, converged == true, schema==1;
  • normalizes config and outcome into flat training records;
  • writes training/tune/test splits as JSONL or parquet, with a dataset.json manifest recording source files and versions for provenance.

9. Privacy operations

  • scripts/usage_export.py never emits session unless --with-session is given (default: drop it).
  • scripts/usage_purge.py --before <date> deletes and rewrites the affected monthly files (used for deletion/retention enforcement).
  • No IP/username ever enters the row; session is a random UUID with nothing linking it to a person or machine account.

10. Future work / open decisions

  • Persist par.dat (full per-parameter fitted values/errors) next to the JSONL as artifact files when the corpus size justifies it.
  • Decide whether to capture plot-only actions (config without an outcome); the row schema already supports "action": "plot".
  • Series action bookkeeping (which runs were reuse/refit/skip) belongs in config and can be surfaced later.
  • GDPR wording and a deletion-request form if deployment reaches EU users.

11. Implementation checklist

  • muSRFitweb.js: consent toggle (help/settings) + musrflowUsageConsent in localStorage + anonymous session uuid.
  • muSRFitweb.js: attach usage: JSON.stringify({consent, session, config: buildSharedConfiguration()}) to fit/plot requests; path-leak guard.
  • single_musrfit.cgi: read MUSRFIT_COLLECTION_ENABLED + usage field; usage_append_row + $USAGE_ROOT layout with flock-append + monthly rotation.
  • single_musrfit.cgi: age-out/retention respecting MUSRFIT_USAGE_RETENTION_MONTHS.
  • scripts/usage_export.py and scripts/usage_purge.py.
  • Tests: client never leaks an absolute path; CGI writes exactly one valid row and never breaks a fit when $USAGE_ROOT is unwritable; consent=off and env-disabled both record nothing.