Files
salman 01df50c3ac Clarify installation guides and retire obsolete CGI parity references
Put Windows and Linux setup, upgrades, and troubleshooting in dedicated manual chapters linked directly from the README. Separate user installation from maintainer packaging instructions and refresh generated HTML/PDF manuals.

Remove stale checklist references and completed legacy gap descriptions while preserving relevant future workflow improvements and historical release notes.
2026-10-28 17:22:53 +01:00

26 KiB

muSRFlow Roadmap and TODO

Current version: 1.0.0-rc.5. Roadmap reviewed: 2026-09-23; documentation and obsolete checklist references cleaned up: 2026-10-28.

The 1.0 target is a stable set of documented, regression-tested workflows. Work is grouped below by release gate and by the larger workflow features needed for routine analysis. Historical delivery details are in CHANGELOG.md.

1. Version 1.0 release path

Supported 1.0 scope

  • Single-run Asymmetry and SingleHist fits are reliable. (Reference fixtures, real-musrfit integration tests, fitted statistic + non-convergence reporting, title handling, t0/background and data-range fields, and portable RUN naming. tests/reference-fixtures.test.js, tests/integration-musrfit.test.js.)
  • Multi-run/global fits preserve parameter sharing and run ordering. (Parameter-table sharing flags, template/global workflow, per-run series action chips, and independent-series runs with a per-run statistics view. tests/series-fit-*.test.js, tests/msr-generation.test.js.)
  • Fit, Plot, FFT, and fitted-parameter plots work in Electron and on the web. (Native musrview in Electron, ROOT/JSROOT canvases on the web, FFT toggle/rerun, floating parameter-plot window with scan-variable detection and declarative templates. tests/root-plot-clean.test.js, tests/parameter-plot.test.js.)
  • Template and result/global MSRs can be inspected and edited safely. (Independent source-of-truth, dirty/lock/stale states, editor ownership authority flow, and sync import of fitted values. MSR_GUI_SYNC.md, tests/msr-generation.test.js.)
  • The parameter table, supported GUI settings, and authoritative MSR text remain consistent through generation, fitting, and synchronization. (Round-trip fixtures for every synchronized construct; shared-code generation in muSRFitlib.js. tests/reference-fixtures.test.js.)
  • Data roots and supported PSI instruments are configurable and documented. (Electron Preferences with persisted data root/startup XML, server-owned MUSRFIT_DATA_ROOT on the web, eight exposed instruments with era/format-validated RUN naming. tests/run-naming.test.js, tests/electron-portable-naming.test.js.)
  • Errors identify missing data, invalid input, missing executables, and failed or non-converged fits without leaving the application busy. (Missing-data outcomes, web health/runtime checks, RRF/t0-bg validation, action lock with Fit/Plot abort, and *** FIT DID NOT CONVERGE *** surfacing in both transports. tests/electron-missing-data.test.js, tests/cgi-integration.test.js.)

The following features may be deferred if they are explicitly documented as unsupported: the model-expression tree (addition/nesting beyond the flat component composition), arbitrary expert-expression reconstruction, session history, manual run files, compound ADDRUN, and multiple fit ranges.

0.9.0-beta.3: correctness baseline

  • Build reference fixtures for Asymmetry, SingleHist, global sharing, FUNCTIONS, bounds, t0/background, FFT, and missing-data workflows.
  • Verify generated MSRs and fitted-parameter feedback against the fixtures.
  • Add integration checks that exercise real musrfit, msr2data, and musrview installations in addition to transport-independent unit tests. (tests/integration-musrfit.test.js runs the hermetic .generated.msr fixtures through real musrfit -e and asserts the converged statistics.)
  • Resolve title handling so editor-owned titles are not replaced unexpectedly by the musrfit -t behavior. (runMSRFileLocal omits -t when the MSR carries an explicit title; tests/electron-msr-title.test.js covers the regression.)
  • Implement the first scan-detection and built-in plot-template tasks from section 2.

0.9.0-beta.4: model and synchronization

Delivered in beta.4 (since beta.3)

  • Add per-run series action chips (reuse / refit / skip) with a bulk "Reuse all / Refit all / Skip all" selector for independent series fits; chips are hidden unless ≥2 data sets with no shared parameters.
  • Allow parameter plotting whenever the parameter table is populated, by building the par.dat inline from the table instead of requiring a generated file.
  • Fail the series plan (rather than silently reusing) when a run set to skip has no on-disk MSR.
  • Fix series-seed and refit edge cases: seed the bootstrap from the first pending run with complete data, and treat a refit of a deleted MSR as a create/fit so the rebuild intent still holds.
  • Move the Electron window and parameter-plot window behind the preload bridge with full process.env and normalized fs.Stats, and route the shared-configuration link through the bridge under hardened settings.
  • Add portable RUN naming via musrfit_startup.xml on desktop and web, ERA/format selection, and scan-kind header guessing (covered below in sections 2 and 3).
  • Complete the 1.0.0-rc.1 hardening gates (preload bridge, CGI input validation/timeouts/cleanup, clean-environment packaging, and web deployment from a clean Apache configuration).

The synchronization layer is implemented and regression-tested; the flat model-component representation remains the supported 1.0 model. The model expression tree is explicitly deferred past 1.0 (see below).

Synchronization (implemented in beta.3 and readiness-tested)

  • Import supported expert-editor changes into one analysis state without guessing the meaning of unsupported expressions.
  • Keep unsupported THEORY/FUNCTIONS content editor-owned and protect it from accidental regeneration.
  • Use one canonical analysis state to render GUI controls, parameter rows, and generated MSR text.
  • Add round-trip tests for every MSR construct declared as synchronized. (parseMSRGUISettings, parseMSRTheoryModel, parseMSRParameters, syncT0BgFromMSR, and buildModelBlocks regeneration are exercised in tests/msr-generation.test.js and tests/reference-fixtures.test.js; design is documented in MSR_GUI_SYNC.md.)

Semantic parameter roles (amplitude, damping, frequency, phase, field) are realized today through the parameter-plot templates and axis labels in muSRFitParameterPlot.js (for example the TF scan template maps Lam/Sgm to damping and Frq/Omega/Nu to frequency). A single authoritative model-role catalogue that drives both generation and plotting is not yet a separate structure; that work rides on the deferred model-expression tree.

Deferred past 1.0: model-expression tree

  • Replace the flat component representation with a model expression tree that can represent addition, multiplication, and nested groups. This requires reworking funcDictionary, buildModelBlocks, and the catalogue-matching importer as a unit; it risks destabilizing the passing generation/sync baseline, so it is intentionally not a 1.0 gate.

0.9.0-beta.5: workflow completeness

  • Complete data-good-bin range and t0/background behavior.
  • Add title and run metadata where they improve analysis context.
  • Initialize a field parameter when the model derives Frq from it via gamma_mu (reverse of the header-field frequency guess).
  • Validate supported filename eras and formats for every exposed instrument.
  • Complete portable startup-XML filename handling from section 3.

Deferred past 1.0 (explicitly documented as out of scope):

  • multiple fit ranges / packings (the current workflow uses one fit interval);
  • manual run files and compound ADDRUN;
  • opt-in web usage collection (config + outcomes as JSONL) to build a corpus for training fitting models — design in docs/WEB_USAGE_COLLECTION.md, not yet implemented.

Optional workflow enhancements after 1.0

These are product improvements to evaluate, not gaps against the old CGI. Header-based initial guesses, scan metadata, and the modeless Run log already exist; extend them only where a concrete analysis workflow needs more.

  • Export plotted spectra as ASCII, optionally compressed, alongside the existing ROOT/JSROOT plots.
  • Evaluate support for archived LEM PPC/ROOT-PPC data if users need it.
  • Expose asymmetric positive MINOS errors in the parameter table.
  • Synchronize fixed Y plot limits between the editor and GUI.
  • Add a durable job manifest with retry/reopen for reproducibility.

1.0.0-rc.1: hardening

  • Move Electron filesystem/process access behind a preload bridge with contextIsolation: true and nodeIntegration: false.
  • Harden CGI input validation, timeouts, job cleanup, and artifact ownership.
  • Test interrupted jobs, repeated actions, malformed files, missing tools, missing data, and stale msr2data output.
  • Build RPM and AppImage packages in a clean environment (fresh machines).
  • Test web deployment from a clean Apache configuration.

1.0.0-rc.2: user validation

  • Run a fixed collection of real analyses through Electron and the web.
  • Compare fitted parameters, statistics, generated MSRs, ROOT plots, and exported parameter data between the two platforms.
  • Collect workflow feedback from experienced users and students.
  • Freeze features and accept only release-blocking fixes and documentation changes.

1.0.0: stable release

  • Freeze and document the supported MSR syntax and workflows.
  • Complete user-facing help, installation, and troubleshooting material.
  • Write the changelog and migration notes.
  • Tag 1.0.0, produce desktop packages, and deploy the matching web build.

2. Scan detection and parameter-plot templates

Goal

When a multi-run fit finishes, determine whether temperature, field, energy, spin rotation, or another supported metadata value is the scan variable. Use that result to initialize a useful fitted-parameter plot rather than requiring the user to construct the same panels for every analysis.

The _par.dat file already contains dataT, dataTErr, dataB, dataE, dataSpinRot, and RUN. These columns should be the primary metadata source for post-fit plotting. dump_header remains useful before fitting or when a parameter-data file does not contain the needed metadata.

Independent-variable detection

  • Represent the result as a scan description containing the selected column, label, unit, confidence, alternatives, and all varying metadata.
  • Include temperature, field, energy, and spin rotation in the candidate registry; use run number as the safe fallback.
  • Ignore changes that are comparable to reported uncertainty or a variable-specific numerical tolerance.
  • Score finite-value coverage, meaningful range, and distinct value groups rather than selecting the first column that is not exactly constant.
  • Select silently only when one variable clearly dominates.
  • Report an ambiguous or mixed scan when two variables vary significantly.
  • Keep a user-selected override when parameter data is refreshed.
  • Later support two-dimensional scans such as X: Energy grouped by temperature, rather than forcing one variable to be discarded.

The compact plot control should communicate both the value and its origin, for example X: Energy (auto). Ambiguous cases should ask once and then remember the selection for the current plot.

Declarative templates

  • Store plot templates as data: identifier, name, compatible X variables, required parameter roles, layout, shared-X behavior, panels, and style.
  • Apply templates by initializing the existing multi-panel plot state; do not create separate plotting functions for each experiment type.
  • Add a built-in three-panel TF scan template containing amplitude, damping rate, and frequency versus the detected X variable.
  • Add a generic parameter template and initial ZF-temperature and field-scan templates after the TF workflow is validated.
  • Add compact Template and X controls to the floating parameter plot.
  • Allow users to modify a template-created plot using the existing panel controls.
  • Interactive color/marker change for fitted-parameter plots: click a plotted marker and adjust color, symbol, size, or opacity for that trace or for the whole data set (branch parameter-plot-interactions).
  • Plot data sets from different *.dat files alongside each other on the same parameter-plot panels ("Add file..." in the plot-options panel; X axis aligned by column name), plus base color palette and per-panel axis-title overrides.

Templates should use semantic roles instead of depending only on exact names. For example, the damping panel can accept Lam or Sgm, while the frequency panel can accept Frq. GUI-generated models should provide these roles directly. Externally opened _par.dat files can use a conservative alias list; ambiguous matches must be shown to the user rather than guessed.

Personal templates

  • Add Save as template only after the built-in format is stable.
  • Persist Electron templates in the application user-data directory.
  • Persist browser templates in local storage until server-side user accounts justify a different design.
  • Support JSON import/export so templates can be shared and reviewed.

Regression cases

  • Temperature-only, field-only, energy-only, and spin-rotation-only scans.
  • Small temperature or field drift that must not override the real scan.
  • Significant variation in two metadata columns, producing an ambiguity.
  • No useful metadata variation, producing a RUN fallback.
  • TF energy data producing the expected three panels and error bars.
  • Missing or multiply matched semantic parameters producing a clear choice instead of an incorrect plot.

3. Portable RUN filenames through musrfit_startup.xml

Problem

muSRFlow currently writes explicit data-root paths into generated RUN and ADDRUN lines. This is deterministic on the machine that generated the MSR but makes the file less portable between /psi.ch/data, /mnt/dmz-data, a web server, and a user's workstation. It also duplicates instrument filename rules that are already represented by musrfit configuration.

The musrfit manual defines:

<data_path>/path/to/instrument/data</data_path>
<run_name_template inst="lem">d%yyyy%/tdc/lem%yy%_his_%rrrr%.root</run_name_template>

%yyyy% and %yy% represent the year, while the number of r characters controls zero-padding of the run number. Multiple paths and templates may exist for one instrument.

  • Add a shared parser for the supported musrfit_startup.xml fields without modifying the user's XML file. (parseMusrfitStartupXML() in muSRFitlib.js reads <data_path> and <run_name_template> only; tested in tests/run-naming.test.js.)
  • Implement template expansion for %yyyy%, %yy%, and arbitrary %rrr...% widths. (expandMusrfitRunTemplate().)
  • Match muSRFlow instrument names to the lowercase XML inst identifiers, including the HAL/HIFI naming difference. (musrfitInstrumentXmlName() maps HAL to hifi.)
  • Try matching templates and formats in configured order and accept a result only when the corresponding data file can be resolved. (portableMusrfitRunCandidates() + the injected resolveMusrfitRunFile platform hook.)
  • Generate portable relative RUN/ADDRUN names when startup-XML resolution is available and validated. (Opt-in via setPortableRunNaming(); wired into runFileNameAuto().)
  • Keep explicit full paths as a documented fallback for manual files, missing/invalid XML, or deliberately self-contained jobs. (The default generation path stays absolute until Portable naming is enabled and a matching template resolves.)
  • Expose the active naming source in Preferences or the status area. (updateStartupNamingSourceIndicator() shows Run naming: musrfit startup XML or Run naming: explicit path in the Electron Preferences dialog; the web equivalent reflects the server startup XML.)

Remaining platform wiring:

  • In Electron, follow musrfit's startup-file discovery order and register a filesystem resolveMusrfitRunFile hook; allow the user to select a specific XML file in Preferences when needed. (muSRFitelec.js installs portableRunResolver, discovers $HOME/.musrfit/musrfit_startup.xml or ./musrfit_startup.xml on top of a Preferences-selected path, and enables portable naming through loadLocalPortableRunNaming(); a selectStartupXmlFile IPC dialog picks the file. Tested in tests/electron-portable-naming.test.js.)
  • In the web runner, register a server-backed resolveMusrfitRunFile hook backed by an administrator-owned startup XML; do not accept arbitrary XML paths from browser requests. (muSRFitweb.js enables portable naming from the health startup_xml block; the CGI resolve_run_name action confirms candidates only under configured <data_path> roots; scripts/deploy-web.sh installs cgi/musrfit_startup.xml to MUSRFIT_STARTUP_XML.)
  • Expose startup-XML readiness in the web health response. (CGI now reports environment.startup_xml.{file,ok,data_paths,templates}.)

The initial implementation should offer two internal modes:

  1. Portable/startup XML: resolve the filename from XML templates and write a relative path that can be combined with <data_path> on another machine.
  2. Explicit path fallback: retain the current generated absolute path when no valid template can be confirmed.

Automatic mode should prefer the portable form only after a resolution check. It must not produce a shorter MSR that cannot be fitted on the current system.

Platform integration

  • In Electron, follow musrfit's startup-file discovery order and allow the user to select a specific XML file in Preferences when needed.
  • In the web runner, use a server-owned startup XML configured by the administrator; do not accept arbitrary XML paths from browser requests.
  • Include startup-XML readiness and the selected naming mode in the web health response.
  • Ensure musrfit, musrview, msr2data, dump_header, and muSRFlow use the same effective configuration for one job.
  • Consider copying a validated server-owned XML into each web job directory because a current-directory startup file has deterministic precedence.

The current musrfit source clearly uses run_name_template through PFindRun for tools such as addRun and dump_header. It is not clear that musrfit's MSR RUN reader expands a run number through these templates directly. Before changing generated MSRs, test the installed executables with a minimal file. If the RUN reader does not expand templates, muSRFlow should expand the template itself and write the resulting relative path; it should not assume that a bare run number will work.

Regression cases

  • XML namespaces, whitespace, multiple <data_path> entries, and multiple templates for one instrument. (Also: XML comments — even prose containing literal tag-like text — are skipped by both the JS and CGI parsers, so the shipped cgi/musrfit_startup.xml header parses cleanly.)
  • Two- and four-digit years plus four- and five-digit run padding.
  • LEM, GPS, LTF, Dolly, VMS, GPD, FLAME, and HAL/HIFI naming rules. (All eight exposed instruments now have direct RUN-line assertions exercised through the shipped default cgi/musrfit_startup.xml, including the era-based template selection.)
  • Historical format transitions and selection among ROOT, BIN, and MDU. (The portable candidate format now follows the resolved path's extension — MUSR-ROOT for .root, HAL PSI-MDU for HAL .bin, PSI-BIN otherwise — instead of the instrument default, fixing an era-transition bug.)
  • Missing XML, malformed XML, unmatched templates, and unresolved files.
  • Equivalent Electron and web RUN lines for the same effective XML, run, year, and instrument.

4. Code consolidation

Consolidate duplicated logic so the CGI and the browser/Electron runtimes do not maintain separate copies. The CGI should only write files and run external executables on the server; computational and text-transformation logic belongs in muSRFitlib.js. Common JS belongs in muSRFitlib.js, Electron-specific code in muSRFitelec.js, and web-specific code in muSRFitweb.js.

Completed

  • Move displayCommand() from muSRFitelec.js to muSRFitlib.js.
  • Move seriesRunDirectiveIdentity() from muSRFitelec.js to muSRFitlib.js.
  • Extract the pure column-header normalization from cleanMusrfitParameterDataHeader() into normalizeParameterDataHeaderLines() in muSRFitlib.js. The Electron function now only wraps the shared transform around file I/O.
  • Load muSRFitlib.js before muSRFitelec.js in tests/parameter-plot.test.js so the moved shared function is available.

Remaining consolidation

  • Extract the shared-configuration encode/decode protocol to muSRFitlib.js, leaving only a platform gzip/gunzip primitive in each adapter. (encodeSharedConfigurationLink() / decodeSharedConfigurationFragment() / parseSharedConfigurationFragment() in muSRFitlib.js; each adapter registers its Node-bridge or CompressionStream primitive.)
  • Extract the copyWebConfigurationLink() orchestration to muSRFitlib.js; each adapter supplies only its clipboard write and URL base. (shareConfigurationLink() in muSRFitlib.js runs the shared copy flow; Electron and web each pass { baseURL, writeClipboard, gzipText }.)
  • Replace amIWeb() branching (including process.cwd() and localStorage persistence) in muSRFitlib.js with platform functions registered by each adapter at load time. (muSRFitelec.js and muSRFitweb.js now publish a window.MusRFlowPlatform contract for the deployment mode, the renderer working directory, and the MSR-file runner; amIWeb() reads the registered mode and only falls back to its heuristic in embedded/test contexts that load no adapter. The renderer no longer reaches for process.cwd() directly.)
  • Replace the amIWeb() dispatch in runMSRFile() with an injected runner registered by each adapter. (Platform.msrFileRunner is set at load time by both adapters; runMSRFile() uses it and keeps the legacy heuristic dispatch only for adapter-free contexts.)
  • Extract the post-restore editor-state synchronization shared by the Electron and web adapters into muSRFitlib.js. (markMSREditorStaleAfterRestore() now runs as the final step of restoreSharedConfiguration(), so a restored edition marks the existing editor documents stale in both deployments; the Electron dialog delegates to the same helper.)
  • Reduce redundant Perl logic in cgi/single_musrfit.cgi where the browser is the sole author of the validated content. Note that series_msr_is_converged, classify_missing_data_output, and most is_valid_generated_global_msr call sites validate server-generated files or command output that the browser never sees in web mode, so they must remain server-side.

5. Post-1.0 security hardening

These defense-in-depth tasks are intentionally deferred until after the stable 1.0.0 workflow release. Pre-1.0 changes should remain focused on reliable fit, plot, editor, synchronization, and result-collection behavior.

  • Review the existing Electron/undici advisories reported by npm audit and test dependency upgrades separately from runtime-installation workflow changes. The new archive extractor is pinned; unrelated upgrades were not applied during the offline installer work.

  • Validate shared-configuration links against a strict schema before they mutate GUI state. Restrict control and parameter identifiers, value types, model names, constraint sizes, and detector-map definitions.

  • Replace parameter-table and custom-parameter innerHTML construction with DOM element creation and property assignment, so restored values are always treated as text rather than markup.

  • Bound compressed and uncompressed shared-configuration payload sizes and reject oversized or excessively nested objects before decompression/restore.

  • Narrow the Electron preload bridge from general filesystem and child-process primitives to the specific file and musrfit operations required by the GUI.

  • Add an Electron/web Content Security Policy and explicit navigation/window controls, with regression tests for shared-link and imported-MSR content.

6. Near-term execution order

  1. Real-command regression baseline, independent-variable detection, the built-in TF scan template, and startup-XML portable naming are established (delivered across beta.3/beta.4).
  2. The 1.0.0-rc.1 hardening gates are complete (preload bridge, CGI hardening, package and clean-web-deploy validation).
  3. Validate the current release candidate with real analyses on Electron and the web, including clean installation and upgrade checks on Linux and Windows/WSLg. Resolve workflow regressions before the stable release.
  4. The model-expression tree is deferred past 1.0 (see beta.4); the current flat-component generation and the synchronization layer are the supported 1.0 model and are protected by fixtures.