Put Windows and Linux setup, upgrades, and troubleshooting in dedicated manual chapters linked directly from the README. Separate user installation from maintainer packaging instructions and refresh generated HTML/PDF manuals. Remove stale checklist references and completed legacy gap descriptions while preserving relevant future workflow improvements and historical release notes.
26 KiB
muSRFlow Roadmap and TODO
Current version: 1.0.0-rc.5. Roadmap reviewed: 2026-09-23; documentation
and obsolete checklist references cleaned up: 2026-10-28.
The 1.0 target is a stable set of documented, regression-tested workflows.
Work is grouped below by release gate and by the larger workflow features
needed for routine analysis. Historical delivery details are in CHANGELOG.md.
1. Version 1.0 release path
Supported 1.0 scope
- Single-run Asymmetry and SingleHist fits are reliable.
(Reference fixtures, real-
musrfitintegration tests, fitted statistic + non-convergence reporting, title handling, t0/background anddata-range fields, and portable RUN naming.tests/reference-fixtures.test.js,tests/integration-musrfit.test.js.) - Multi-run/global fits preserve parameter sharing and run ordering.
(Parameter-table sharing flags, template/global workflow, per-run series
action chips, and independent-series runs with a per-run statistics view.
tests/series-fit-*.test.js,tests/msr-generation.test.js.) - Fit, Plot, FFT, and fitted-parameter plots work in Electron and on the web.
(Native musrview in Electron, ROOT/JSROOT canvases on the web, FFT
toggle/rerun, floating parameter-plot window with scan-variable detection
and declarative templates.
tests/root-plot-clean.test.js,tests/parameter-plot.test.js.) - Template and result/global MSRs can be inspected and edited safely.
(Independent source-of-truth, dirty/lock/stale states, editor ownership
authority flow, and sync import of fitted values.
MSR_GUI_SYNC.md,tests/msr-generation.test.js.) - The parameter table, supported GUI settings, and authoritative MSR text
remain consistent through generation, fitting, and synchronization.
(Round-trip fixtures for every synchronized construct; shared-code
generation in
muSRFitlib.js.tests/reference-fixtures.test.js.) - Data roots and supported PSI instruments are configurable and documented.
(Electron Preferences with persisted data root/startup XML, server-owned
MUSRFIT_DATA_ROOTon the web, eight exposed instruments with era/format-validated RUN naming.tests/run-naming.test.js,tests/electron-portable-naming.test.js.) - Errors identify missing data, invalid input, missing executables, and
failed or non-converged fits without leaving the application busy.
(Missing-data outcomes, web health/runtime checks, RRF/t0-bg validation,
action lock with Fit/Plot abort, and
*** FIT DID NOT CONVERGE ***surfacing in both transports.tests/electron-missing-data.test.js,tests/cgi-integration.test.js.)
The following features may be deferred if they are explicitly documented as
unsupported: the model-expression tree (addition/nesting beyond the flat
component composition), arbitrary expert-expression reconstruction, session
history, manual run files, compound ADDRUN, and multiple fit ranges.
0.9.0-beta.3: correctness baseline
- Build reference fixtures for Asymmetry, SingleHist, global sharing, FUNCTIONS, bounds, t0/background, FFT, and missing-data workflows.
- Verify generated MSRs and fitted-parameter feedback against the fixtures.
- Add integration checks that exercise real
musrfit,msr2data, andmusrviewinstallations in addition to transport-independent unit tests. (tests/integration-musrfit.test.jsruns the hermetic.generated.msrfixtures through realmusrfit -eand asserts the converged statistics.) - Resolve title handling so editor-owned titles are not replaced
unexpectedly by the
musrfit -tbehavior. (runMSRFileLocalomits-twhen the MSR carries an explicit title;tests/electron-msr-title.test.jscovers the regression.) - Implement the first scan-detection and built-in plot-template tasks from section 2.
0.9.0-beta.4: model and synchronization
Delivered in beta.4 (since beta.3)
- Add per-run series action chips (reuse / refit / skip) with a bulk "Reuse all / Refit all / Skip all" selector for independent series fits; chips are hidden unless ≥2 data sets with no shared parameters.
- Allow parameter plotting whenever the parameter table is populated, by
building the
par.datinline from the table instead of requiring a generated file. - Fail the series plan (rather than silently reusing) when a run set to skip has no on-disk MSR.
- Fix series-seed and refit edge cases: seed the bootstrap from the first pending run with complete data, and treat a refit of a deleted MSR as a create/fit so the rebuild intent still holds.
- Move the Electron window and parameter-plot window behind the preload
bridge with full
process.envand normalizedfs.Stats, and route the shared-configuration link through the bridge under hardened settings. - Add portable RUN naming via
musrfit_startup.xmlon desktop and web, ERA/format selection, and scan-kind header guessing (covered below in sections 2 and 3). - Complete the
1.0.0-rc.1hardening gates (preload bridge, CGI input validation/timeouts/cleanup, clean-environment packaging, and web deployment from a clean Apache configuration).
The synchronization layer is implemented and regression-tested; the flat model-component representation remains the supported 1.0 model. The model expression tree is explicitly deferred past 1.0 (see below).
Synchronization (implemented in beta.3 and readiness-tested)
- Import supported expert-editor changes into one analysis state without guessing the meaning of unsupported expressions.
- Keep unsupported THEORY/FUNCTIONS content editor-owned and protect it from accidental regeneration.
- Use one canonical analysis state to render GUI controls, parameter rows, and generated MSR text.
- Add round-trip tests for every MSR construct declared as synchronized.
(
parseMSRGUISettings,parseMSRTheoryModel,parseMSRParameters,syncT0BgFromMSR, andbuildModelBlocksregeneration are exercised intests/msr-generation.test.jsandtests/reference-fixtures.test.js; design is documented inMSR_GUI_SYNC.md.)
Semantic parameter roles (amplitude, damping, frequency, phase, field) are
realized today through the parameter-plot templates and axis labels in
muSRFitParameterPlot.js (for example the TF scan template maps Lam/Sgm
to damping and Frq/Omega/Nu to frequency). A single authoritative
model-role catalogue that drives both generation and plotting is not yet a
separate structure; that work rides on the deferred model-expression tree.
Deferred past 1.0: model-expression tree
- Replace the flat component representation with a model expression tree
that can represent addition, multiplication, and nested groups. This
requires reworking
funcDictionary,buildModelBlocks, and the catalogue-matching importer as a unit; it risks destabilizing the passing generation/sync baseline, so it is intentionally not a 1.0 gate.
0.9.0-beta.5: workflow completeness
- Complete data-good-bin range and t0/background behavior.
- Add title and run metadata where they improve analysis context.
- Initialize a field parameter when the model derives
Frqfrom it viagamma_mu(reverse of the header-field frequency guess). - Validate supported filename eras and formats for every exposed instrument.
- Complete portable startup-XML filename handling from section 3.
Deferred past 1.0 (explicitly documented as out of scope):
- multiple fit ranges / packings (the current workflow uses one fit interval);
- manual run files and compound
ADDRUN; - opt-in web usage collection (config + outcomes as JSONL) to build a corpus
for training fitting models — design in
docs/WEB_USAGE_COLLECTION.md, not yet implemented.
Optional workflow enhancements after 1.0
These are product improvements to evaluate, not gaps against the old CGI. Header-based initial guesses, scan metadata, and the modeless Run log already exist; extend them only where a concrete analysis workflow needs more.
- Export plotted spectra as ASCII, optionally compressed, alongside the existing ROOT/JSROOT plots.
- Evaluate support for archived LEM PPC/ROOT-PPC data if users need it.
- Expose asymmetric positive MINOS errors in the parameter table.
- Synchronize fixed Y plot limits between the editor and GUI.
- Add a durable job manifest with retry/reopen for reproducibility.
1.0.0-rc.1: hardening
- Move Electron filesystem/process access behind a preload bridge with
contextIsolation: trueandnodeIntegration: false. - Harden CGI input validation, timeouts, job cleanup, and artifact ownership.
- Test interrupted jobs, repeated actions, malformed files, missing tools,
missing data, and stale
msr2dataoutput. - Build RPM and AppImage packages in a clean environment (fresh machines).
- Test web deployment from a clean Apache configuration.
1.0.0-rc.2: user validation
- Run a fixed collection of real analyses through Electron and the web.
- Compare fitted parameters, statistics, generated MSRs, ROOT plots, and exported parameter data between the two platforms.
- Collect workflow feedback from experienced users and students.
- Freeze features and accept only release-blocking fixes and documentation changes.
1.0.0: stable release
- Freeze and document the supported MSR syntax and workflows.
- Complete user-facing help, installation, and troubleshooting material.
- Write the changelog and migration notes.
- Tag
1.0.0, produce desktop packages, and deploy the matching web build.
2. Scan detection and parameter-plot templates
Goal
When a multi-run fit finishes, determine whether temperature, field, energy, spin rotation, or another supported metadata value is the scan variable. Use that result to initialize a useful fitted-parameter plot rather than requiring the user to construct the same panels for every analysis.
The _par.dat file already contains dataT, dataTErr, dataB, dataE,
dataSpinRot, and RUN. These columns should be the primary metadata source
for post-fit plotting. dump_header remains useful before fitting or when a
parameter-data file does not contain the needed metadata.
Independent-variable detection
- Represent the result as a scan description containing the selected column, label, unit, confidence, alternatives, and all varying metadata.
- Include temperature, field, energy, and spin rotation in the candidate registry; use run number as the safe fallback.
- Ignore changes that are comparable to reported uncertainty or a variable-specific numerical tolerance.
- Score finite-value coverage, meaningful range, and distinct value groups rather than selecting the first column that is not exactly constant.
- Select silently only when one variable clearly dominates.
- Report an ambiguous or mixed scan when two variables vary significantly.
- Keep a user-selected override when parameter data is refreshed.
- Later support two-dimensional scans such as
X: Energygrouped by temperature, rather than forcing one variable to be discarded.
The compact plot control should communicate both the value and its origin, for
example X: Energy (auto). Ambiguous cases should ask once and then remember
the selection for the current plot.
Declarative templates
- Store plot templates as data: identifier, name, compatible X variables, required parameter roles, layout, shared-X behavior, panels, and style.
- Apply templates by initializing the existing multi-panel plot state; do not create separate plotting functions for each experiment type.
- Add a built-in three-panel TF scan template containing amplitude, damping rate, and frequency versus the detected X variable.
- Add a generic parameter template and initial ZF-temperature and field-scan templates after the TF workflow is validated.
- Add compact
TemplateandXcontrols to the floating parameter plot. - Allow users to modify a template-created plot using the existing panel controls.
- Interactive color/marker change for fitted-parameter plots: click a
plotted marker and adjust color, symbol, size, or opacity for that trace
or for the whole data set (branch
parameter-plot-interactions). - Plot data sets from different
*.datfiles alongside each other on the same parameter-plot panels ("Add file..." in the plot-options panel; X axis aligned by column name), plus base color palette and per-panel axis-title overrides.
Templates should use semantic roles instead of depending only on exact names.
For example, the damping panel can accept Lam or Sgm, while the frequency
panel can accept Frq. GUI-generated models should provide these roles
directly. Externally opened _par.dat files can use a conservative alias list;
ambiguous matches must be shown to the user rather than guessed.
Personal templates
- Add
Save as templateonly after the built-in format is stable. - Persist Electron templates in the application user-data directory.
- Persist browser templates in local storage until server-side user accounts justify a different design.
- Support JSON import/export so templates can be shared and reviewed.
Regression cases
- Temperature-only, field-only, energy-only, and spin-rotation-only scans.
- Small temperature or field drift that must not override the real scan.
- Significant variation in two metadata columns, producing an ambiguity.
- No useful metadata variation, producing a
RUNfallback. - TF energy data producing the expected three panels and error bars.
- Missing or multiply matched semantic parameters producing a clear choice instead of an incorrect plot.
3. Portable RUN filenames through musrfit_startup.xml
Problem
muSRFlow currently writes explicit data-root paths into generated RUN and
ADDRUN lines. This is deterministic on the machine that generated the MSR but
makes the file less portable between /psi.ch/data, /mnt/dmz-data, a web
server, and a user's workstation. It also duplicates instrument filename rules
that are already represented by musrfit configuration.
The musrfit manual defines:
<data_path>/path/to/instrument/data</data_path>
<run_name_template inst="lem">d%yyyy%/tdc/lem%yy%_his_%rrrr%.root</run_name_template>
%yyyy% and %yy% represent the year, while the number of r characters
controls zero-padding of the run number. Multiple paths and templates may exist
for one instrument.
Recommended behavior
- Add a shared parser for the supported
musrfit_startup.xmlfields without modifying the user's XML file. (parseMusrfitStartupXML()inmuSRFitlib.jsreads<data_path>and<run_name_template>only; tested intests/run-naming.test.js.) - Implement template expansion for
%yyyy%,%yy%, and arbitrary%rrr...%widths. (expandMusrfitRunTemplate().) - Match muSRFlow instrument names to the lowercase XML
instidentifiers, including the HAL/HIFI naming difference. (musrfitInstrumentXmlName()mapsHALtohifi.) - Try matching templates and formats in configured order and accept a result
only when the corresponding data file can be resolved.
(
portableMusrfitRunCandidates()+ the injectedresolveMusrfitRunFileplatform hook.) - Generate portable relative
RUN/ADDRUNnames when startup-XML resolution is available and validated. (Opt-in viasetPortableRunNaming(); wired intorunFileNameAuto().) - Keep explicit full paths as a documented fallback for manual files, missing/invalid XML, or deliberately self-contained jobs. (The default generation path stays absolute until Portable naming is enabled and a matching template resolves.)
- Expose the active naming source in Preferences or the status area. (
updateStartupNamingSourceIndicator()showsRun naming: musrfit startup XMLorRun naming: explicit pathin the Electron Preferences dialog; the web equivalent reflects the server startup XML.)
Remaining platform wiring:
- In Electron, follow musrfit's startup-file discovery order and register a
filesystem
resolveMusrfitRunFilehook; allow the user to select a specific XML file in Preferences when needed. (muSRFitelec.jsinstallsportableRunResolver, discovers$HOME/.musrfit/musrfit_startup.xmlor./musrfit_startup.xmlon top of a Preferences-selected path, and enables portable naming throughloadLocalPortableRunNaming(); aselectStartupXmlFileIPC dialog picks the file. Tested intests/electron-portable-naming.test.js.) - In the web runner, register a server-backed
resolveMusrfitRunFilehook backed by an administrator-owned startup XML; do not accept arbitrary XML paths from browser requests. (muSRFitweb.jsenables portable naming from the healthstartup_xmlblock; the CGIresolve_run_nameaction confirms candidates only under configured<data_path>roots;scripts/deploy-web.shinstallscgi/musrfit_startup.xmltoMUSRFIT_STARTUP_XML.) - Expose startup-XML readiness in the web health response. (CGI now reports
environment.startup_xml.{file,ok,data_paths,templates}.)
The initial implementation should offer two internal modes:
- Portable/startup XML: resolve the filename from XML templates and write a
relative path that can be combined with
<data_path>on another machine. - Explicit path fallback: retain the current generated absolute path when no valid template can be confirmed.
Automatic mode should prefer the portable form only after a resolution check. It must not produce a shorter MSR that cannot be fitted on the current system.
Platform integration
- In Electron, follow musrfit's startup-file discovery order and allow the user to select a specific XML file in Preferences when needed.
- In the web runner, use a server-owned startup XML configured by the administrator; do not accept arbitrary XML paths from browser requests.
- Include startup-XML readiness and the selected naming mode in the web health response.
- Ensure
musrfit,musrview,msr2data,dump_header, and muSRFlow use the same effective configuration for one job. - Consider copying a validated server-owned XML into each web job directory because a current-directory startup file has deterministic precedence.
The current musrfit source clearly uses run_name_template through PFindRun
for tools such as addRun and dump_header. It is not clear that musrfit's MSR
RUN reader expands a run number through these templates directly. Before
changing generated MSRs, test the installed executables with a minimal file. If
the RUN reader does not expand templates, muSRFlow should expand the template
itself and write the resulting relative path; it should not assume that a bare
run number will work.
Regression cases
- XML namespaces, whitespace, multiple
<data_path>entries, and multiple templates for one instrument. (Also: XML comments — even prose containing literal tag-like text — are skipped by both the JS and CGI parsers, so the shippedcgi/musrfit_startup.xmlheader parses cleanly.) - Two- and four-digit years plus four- and five-digit run padding.
- LEM, GPS, LTF, Dolly, VMS, GPD, FLAME, and HAL/HIFI naming rules. (All
eight exposed instruments now have direct RUN-line assertions exercised
through the shipped default
cgi/musrfit_startup.xml, including the era-based template selection.) - Historical format transitions and selection among ROOT, BIN, and MDU. (The
portable candidate format now follows the resolved path's extension —
MUSR-ROOT for
.root, HAL PSI-MDU for HAL.bin, PSI-BIN otherwise — instead of the instrument default, fixing an era-transition bug.) - Missing XML, malformed XML, unmatched templates, and unresolved files.
- Equivalent Electron and web RUN lines for the same effective XML, run, year, and instrument.
4. Code consolidation
Consolidate duplicated logic so the CGI and the browser/Electron runtimes do not
maintain separate copies. The CGI should only write files and run external
executables on the server; computational and text-transformation logic belongs in
muSRFitlib.js. Common JS belongs in muSRFitlib.js, Electron-specific code in
muSRFitelec.js, and web-specific code in muSRFitweb.js.
Completed
- Move
displayCommand()frommuSRFitelec.jstomuSRFitlib.js. - Move
seriesRunDirectiveIdentity()frommuSRFitelec.jstomuSRFitlib.js. - Extract the pure column-header normalization from
cleanMusrfitParameterDataHeader()intonormalizeParameterDataHeaderLines()inmuSRFitlib.js. The Electron function now only wraps the shared transform around file I/O. - Load
muSRFitlib.jsbeforemuSRFitelec.jsintests/parameter-plot.test.jsso the moved shared function is available.
Remaining consolidation
- Extract the shared-configuration encode/decode protocol to
muSRFitlib.js, leaving only a platformgzip/gunzipprimitive in each adapter. (encodeSharedConfigurationLink()/decodeSharedConfigurationFragment()/parseSharedConfigurationFragment()inmuSRFitlib.js; each adapter registers its Node-bridge orCompressionStreamprimitive.) - Extract the
copyWebConfigurationLink()orchestration tomuSRFitlib.js; each adapter supplies only its clipboard write and URL base. (shareConfigurationLink()inmuSRFitlib.jsruns the shared copy flow; Electron and web each pass{ baseURL, writeClipboard, gzipText }.) - Replace
amIWeb()branching (includingprocess.cwd()andlocalStoragepersistence) inmuSRFitlib.jswith platform functions registered by each adapter at load time. (muSRFitelec.jsandmuSRFitweb.jsnow publish awindow.MusRFlowPlatformcontract for the deployment mode, the renderer working directory, and the MSR-file runner;amIWeb()reads the registered mode and only falls back to its heuristic in embedded/test contexts that load no adapter. The renderer no longer reaches forprocess.cwd()directly.) - Replace the
amIWeb()dispatch inrunMSRFile()with an injected runner registered by each adapter. (Platform.msrFileRunneris set at load time by both adapters;runMSRFile()uses it and keeps the legacy heuristic dispatch only for adapter-free contexts.) - Extract the post-restore editor-state synchronization shared by the
Electron and web adapters into
muSRFitlib.js. (markMSREditorStaleAfterRestore()now runs as the final step ofrestoreSharedConfiguration(), so a restored edition marks the existing editor documents stale in both deployments; the Electron dialog delegates to the same helper.) - Reduce redundant Perl logic in
cgi/single_musrfit.cgiwhere the browser is the sole author of the validated content. Note thatseries_msr_is_converged,classify_missing_data_output, and mostis_valid_generated_global_msrcall sites validate server-generated files or command output that the browser never sees in web mode, so they must remain server-side.
5. Post-1.0 security hardening
These defense-in-depth tasks are intentionally deferred until after the stable
1.0.0 workflow release. Pre-1.0 changes should remain focused on reliable fit,
plot, editor, synchronization, and result-collection behavior.
-
Review the existing Electron/undici advisories reported by
npm auditand test dependency upgrades separately from runtime-installation workflow changes. The new archive extractor is pinned; unrelated upgrades were not applied during the offline installer work. -
Validate shared-configuration links against a strict schema before they mutate GUI state. Restrict control and parameter identifiers, value types, model names, constraint sizes, and detector-map definitions.
-
Replace parameter-table and custom-parameter
innerHTMLconstruction with DOM element creation and property assignment, so restored values are always treated as text rather than markup. -
Bound compressed and uncompressed shared-configuration payload sizes and reject oversized or excessively nested objects before decompression/restore.
-
Narrow the Electron preload bridge from general filesystem and child-process primitives to the specific file and musrfit operations required by the GUI.
-
Add an Electron/web Content Security Policy and explicit navigation/window controls, with regression tests for shared-link and imported-MSR content.
6. Near-term execution order
- Real-command regression baseline, independent-variable detection, the
built-in TF scan template, and startup-XML portable naming are established
(delivered across
beta.3/beta.4). - The
1.0.0-rc.1hardening gates are complete (preload bridge, CGI hardening, package and clean-web-deploy validation). - Validate the current release candidate with real analyses on Electron and the web, including clean installation and upgrade checks on Linux and Windows/WSLg. Resolve workflow regressions before the stable release.
- The model-expression tree is deferred past 1.0 (see
beta.4); the current flat-component generation and the synchronization layer are the supported1.0model and are protected by fixtures.