diff --git a/CGI_PARITY_CHECKLIST.md b/CGI_PARITY_CHECKLIST.md index 42d9146..400617c 100644 --- a/CGI_PARITY_CHECKLIST.md +++ b/CGI_PARITY_CHECKLIST.md @@ -37,7 +37,7 @@ Status legend: | Run-number expansion (`-`, `:` syntax) | `MSR.pm::ExpandRunNumbers` | `arrayRunNumbers()` | `PARTIAL` | Common ascending single/range/step syntax is supported with stricter validation. Legacy descending `from:to:step` input swaps its endpoints; the current GUI deliberately rejects descending ranges instead. | | Single-run/asymmetry MSR creation | `MSR.pm::CreateMSRUni` | `createMSR()` / `buildMSR()` | `PARTIAL` | Core FITPARAMETER, THEORY/FUNCTIONS, GLOBAL, RUN, COMMANDS, PLOT, and FOURIER blocks are generated. Multiple fit ranges, explicit data-good-bin ranges, and legacy title handling remain gaps. | | Multiple fit ranges / packings | `MSR.pm::CreateMSRUni` (`Tis`, `Tfs`, `BINS` arrays) | Single Fit range row | `MISSING` | Legacy CGI accepted comma-separated range/packing sets and repeated RUN blocks accordingly. Current generation expects one fit interval and one packing value. | -| Global MSR generation | `MSR.pm::CreateMSRGLB`, `msr2data global` | `prepareMSRRunFiles()` + `buildSeriesGlobalMSR()` | `DONE` | Multi-run asymmetry merges the already-fitted per-run MSRs into the combined display MSR in JS (`muSRFitlib.js`) and Perl (`single_musrfit.cgi`). msr2data is no longer asked to build it, because it only writes its `+global` file from a template and re-fits each run in every value-populating mode. Stale output is removed and the merged MSR is validated before use, instead of relying on msr2data's unreliable exit codes. | +| Global MSR generation | `MSR.pm::CreateMSRGLB`, `msr2data global` | `prepareMSRRunFiles()` + `buildSeriesGlobalMSR()` | `DONE` | Electron and the browser use the same JavaScript merger for already-fitted independent runs. In web mode the browser fetches the HTTP-visible fitted MSRs and submits the combined display text; the CGI verifies its artifact names, size, structure, and exact RUN/ADDRUN identity before publishing it. `msr2data` collects the per-run parameter table directly and never re-fits the combined display. | | Parameter table and bounds | `MSR.pm::PrepParamTable` | `cfgParams()`, `formatFitParameterLine()`, fitted-MSR parser | `PARTIAL` | Value/step and two-sided or one-sided bounds are generated correctly. Fitted values and bounds can be imported. The GUI does not separately preserve/display the asymmetric positive MINOS error field from a seven-field fitted parameter line. | | Parameter-data export | `MSR.pm::ExportParams`, post-fit `msr2data` | Web/Electron runners + floating parameter viewer | `DONE` | Both runners generate `_par.dat` after single/global fitting and expose the same artifact to the integrated floating Plotly viewer; the raw table remains available in Results. | @@ -59,7 +59,7 @@ Status legend: | Plot-only and fit+plot execution | Legacy `plotonly` / `fitplot` commands | `muSRFitelec.js`, `muSRFitweb.js`, `cgi/single_musrfit.cgi` | `DONE` | Electron and web have separate Plot/Fit paths. Multi-run jobs assemble the combined display MSR from the per-run fitted files and normalize PLOT run indices before plotting. Fit and Plot share an action lock with visible preparation/execution phases, preventing duplicate or conflicting requests. Missing RUN data from either musrfit or musrview is reported as an amber `Missing data` outcome rather than a failed fit/plot; Fit stops before starting musrview in that case. | | Interactive plot delivery | Legacy `musrview --png` + ImageMagick crop | Native musrview (Electron), `musrview --root` + JSROOT (web) | `DONE` | This intentionally replaces static PNG delivery. Web ROOT files support interactive zoom/resizing and FFT toggling; Electron keeps the native musrview window. | | Spectrum ASCII extraction + gzip | Legacy `musrview --ascii` + `gzip FILENAME.dat` | No equivalent spectrum dump | `MISSING` | Job tarballs and parameter `.dat` export are different artifacts and do not replace the legacy plotted-data ASCII dump. Add only if this download remains useful to users. | -| `msr2data` integration | Legacy `$MSR2DATA` workflow | Local runner + `cgi/single_musrfit.cgi` | `DONE` | msr2data handles the post-fit parameter `.dat` export in both runners. The combined multi-run display MSR is assembled locally from the per-run fitted files instead; the merged MSR is validated explicitly before use. | +| `msr2data` integration | Legacy `$MSR2DATA` workflow | Local runner + `cgi/single_musrfit.cgi` | `DONE` | `msr2data` directly collects the fitted per-run parameter `.dat` table in both runners. Combined display construction is an independent best-effort operation, so an unsupported display model cannot disable Plot parameters. | | Debug/command output | Legacy downloadable `.out` / debug files | `.out` command log + Results details + concise status bar | `DONE` | Current logs record command, combined output, and exit status while keeping routine live status compact. | | Web health/runtime configuration | No direct legacy equivalent | CGI `health` action + `muSRFitweb.js` | `DONE` | Reports runtime/executable/tmp/data-root readiness and refreshes the server-owned data root before actions. | | Dedicated fit statistic display | `musrfit.cgi::PrintOut` / fitted `STATISTIC` block | `parseMSRFitStatistic()` + status bar | `DONE` | After Fit, Electron and web parse the fitted MSR's overall `STATISTIC` line and keep chisq/NDF or maxLH/NDF visible in the status bar; `*** FIT DID NOT CONVERGE ***` is surfaced explicitly. Plot-only actions do not reinterpret an older statistic as a new result. | diff --git a/cgi/single_musrfit.cgi b/cgi/single_musrfit.cgi index ffe54bc..1749475 100755 --- a/cgi/single_musrfit.cgi +++ b/cgi/single_musrfit.cgi @@ -194,7 +194,7 @@ sub rollback_series_receipts { return \@errors; } -sub series_run_directive_signature { +sub series_run_directives { my ($content) = @_; my @directives = (); foreach my $line (split(/\r?\n/, $content || "")) { @@ -209,7 +209,12 @@ sub series_run_directive_signature { $parts[0] = uc($parts[0]); push(@directives, join(" ", @parts[0 .. 4])); } - return sha256_hex(join("\n", @directives)); + return \@directives; +} + +sub series_run_directive_signature { + my ($content) = @_; + return sha256_hex(join("\n", @{series_run_directives($content)})); } sub perform_series_transaction { @@ -855,19 +860,25 @@ sub perform_series_transaction_finalization { }; } -# msr2data represents run-qualified parameters such as Asy_2274 as Asy_ in -# the exported data header. Remove exactly one trailing underscore from header -# fields while preserving embedded underscores and labels such as Asy_PosErr. +# Global msr2data exports represent a run-qualified parameter such as Asy_2274 +# as Asy_, while ordinary per-run exports retain the first run suffix as +# Asy_2274, Asy_2274PosErr, and Asy_2274NegErr. Normalize only suffixes that +# match this collection's run list while preserving embedded underscores. sub clean_parameter_data_header { - my ($path) = @_; + my ($path, $run_numbers) = @_; my $content = read_text_file($path); return 0 if $content eq ""; + my %runs = map { (0 + $_) => 1 } + grep { defined($_) && /^\d+$/ } @{ref($run_numbers) eq "ARRAY" ? $run_numbers : []}; my @lines = split(/\n/, $content, -1); foreach my $line (@lines) { my $probe = $line; $probe =~ s/\r$//; next if $probe =~ /^\s*$/ || $probe =~ /^\s*#/; + $line =~ s{(\S+?)_([0-9]+)((?:_)?(?:PosErr|NegErr))?(?=\s|$)}{ + exists($runs{0 + $2}) ? $1 . (defined($3) ? $3 : "") : $& + }gex; $line =~ s/(\S+)_(?=\s|$)/$1/g; last; } @@ -995,463 +1006,7 @@ sub normalize_plot_runs_line { return $content; } -# Merge the fitted per-run MSR files of an independent series fit into a single -# display-only global MSR. msr2data is deliberately not used for this: it only -# writes its +global file from a template and, in the modes that populate -# per-run values, re-fits each run with musrfit. The merged file keeps each -# run's already-fitted parameter values and lets every RUN block select its own -# parameters through the same mapN/alpha/norm mechanism a genuine msr2data -# global file uses. -my @SERIES_MSR_BLOCK_NAMES = qw(FITPARAMETER THEORY FUNCTIONS GLOBAL RUN COMMANDS PLOT FOURIER STATISTIC); -my %SERIES_MSR_RUN_PARAM_DIRECTIVES = map { $_ => 1 } qw(alpha beta norm backgr.fit); -my $SERIES_MSR_MAX_MAP_SLOTS = 10; - -# Recognize a block header line. The FOURIER block uses a lowercase -# `plot ` directive that would otherwise look like a PLOT header, so a -# genuine PLOT header must carry a numeric plot id ("PLOT 2"); non-numeric -# `plot` lines stay inside FOURIER. -sub series_msr_block_header_name { - my ($trimmed) = @_; - for my $name (@SERIES_MSR_BLOCK_NAMES) { - if ($trimmed =~ /^\Q$name\E(?:\s|$)/i) { - my $candidate = uc($name); - if ($candidate eq "PLOT") { - my $arg = $trimmed; - $arg =~ s/^\Q$name\E//i; - $arg =~ s/^\s+|\s+$//g; - return "" if $arg !~ /^\d/; - } - return $candidate; - } - } - return ""; -} - -sub series_msr_blocks { - my ($content) = @_; - my %blocks; - my $section = ""; - for my $line (split(/\r?\n/, defined($content) ? $content : "")) { - my $trimmed = $line; - $trimmed =~ s/^\s+//; - my $block = series_msr_block_header_name($trimmed); - if ($block ne "") { - my $start = $line; - $start =~ s/^\s+//; - $start =~ s/\s+$//; - $section = $block; - if (!exists($blocks{$block})) { - $blocks{$block} = { header => $start, lines => [] }; - } elsif ($block eq "RUN") { - # RUN is repeatable. Preserve later detector headers inside the - # first collected RUN section so series_msr_run_blocks can - # reconstruct every histogram block. - push(@{$blocks{$block}{lines}}, $start); - } - } elsif ($section ne "") { - push(@{$blocks{$section}{lines}}, $line); - } else { - $blocks{HEADER} = { header => "", lines => [] } - if !exists($blocks{HEADER}); - push(@{$blocks{HEADER}{lines}}, $line); - } - } - return \%blocks; -} - -sub series_msr_parameters { - my ($lines) = @_; - my @params; - for my $line (@{$lines || []}) { - my $trimmed = $line; - $trimmed =~ s/^\s+//; - next if $trimmed eq "" || $trimmed =~ /^#/; - my ($no, $name, $rest) = $trimmed =~ /^(\d+)\s+(\S+)\s*(.*)$/; - next if !defined($no); - $rest = "" if !defined($rest); - $rest =~ s/^\s+|\s+$//g; - $rest =~ s/\s+/ /g; - push(@params, { no => 0 + $no, name => $name, rest => $rest }); - } - return \@params; -} - -sub series_msr_run_blocks { - my ($header, $lines) = @_; - my @blocks; - my $current; - for my $line (($header, @{$lines || []})) { - my $trimmed = $line; - $trimmed =~ s/^\s+//; - if ($trimmed =~ /^RUN\b/i) { - my $start = $line; - $start =~ s/\s+$//; - $current = { header => $start, lines => [] }; - push(@blocks, $current); - } elsif (defined($current)) { - push(@{$current->{lines}}, $line); - } - } - return \@blocks; -} - -sub series_msr_token_shape { - my ($lines) = @_; - return join("\n", map { - join(" ", map { /^\d+$/ ? "#" : $_ } split(/\s+/, $_)) - } @{$lines || []}); -} - -sub format_series_merged_parameter_line { - my ($no, $name, $rest) = @_; - my $line = " $no $name"; - $line .= " $rest" if defined($rest) && $rest ne ""; - return $line; -} - -# The FOURIER block is shared by all RUN blocks in the merged file. musrview -# refuses any plot other than POWER unless the phase line carries one reference -# per run (parR7 par9 ...), which a display file assembled from shared -# parameters cannot express. Mirror msr2data's own global output by forcing the -# plot to POWER; the fitted per-run Fourier phases are deliberately left to the -# individual run files. -sub series_fourier_for_display { - my ($lines) = @_; - my @normalized; - my $saw_plot = 0; - for my $line (@{$lines || []}) { - my $trimmed = $line; - $trimmed =~ s/^\s+//; - if ($trimmed =~ /^plot\b/i) { - $saw_plot = 1; - if ($trimmed !~ /^plot\s+POWER\b/i) { - push(@normalized, "plot POWER"); - } else { - push(@normalized, $line); - } - } else { - push(@normalized, $line); - } - } - push(@normalized, "plot POWER") if !$saw_plot; - return \@normalized; -} - -# Returns [$ok, $reason, $text]. -sub build_series_global_msr { - my ($msr_texts, $run_numbers) = @_; - return [0, "series-merge-invalid-input", ""] - if ref($msr_texts) ne "ARRAY" || ref($run_numbers) ne "ARRAY" - || @{$msr_texts} < 2 || @{$msr_texts} != @{$run_numbers}; - - my @per_run; - for my $index (0 .. $#$msr_texts) { - my $blocks = series_msr_blocks($msr_texts->[$index]); - return [0, "series-merge-source-incomplete", ""] - if !$blocks->{FITPARAMETER} || !$blocks->{THEORY} || !$blocks->{RUN}; - my $params = series_msr_parameters($blocks->{FITPARAMETER}{lines}); - return [0, "series-merge-parameters-incomplete", ""] if !@{$params}; - my $run_blocks = series_msr_run_blocks( - $blocks->{RUN}{header}, $blocks->{RUN}{lines} - ); - return [0, "series-merge-run-blocks-incomplete", ""] if !@{$run_blocks}; - push(@per_run, { - run => "$run_numbers->[$index]", - params => $params, - run_blocks => $run_blocks, - theory => $blocks->{THEORY}{lines} || [], - functions => $blocks->{FUNCTIONS}{lines} || [], - global => $blocks->{GLOBAL}{lines} || [], - commands => $blocks->{COMMANDS}{lines} || [], - plot => $blocks->{PLOT}{lines} || [], - plot_header => $blocks->{PLOT}{header} || "PLOT 0", - fourier => $blocks->{FOURIER}{lines} || [], - statistic => $blocks->{STATISTIC}{lines} || [] - }); - } - - my $num_runs = scalar(@per_run); - my $first = $per_run[0]; - for my $pr (@per_run) { - return [0, "series-merge-parameter-count-mismatch", ""] - if @{$pr->{params}} != @{$first->{params}}; - } - - # A name present in every run is shared; present in one run is run-specific. - my %name_counts; - for my $pr (@per_run) { - $name_counts{$_->{name}}++ for @{$pr->{params}}; - } - for my $name (keys %name_counts) { - return [0, "series-merge-parameter-inconsistency", ""] - if $name_counts{$name} != 1 && $name_counts{$name} != $num_runs; - } - - # All runs must describe the same model: the non-numeric token shape of the - # THEORY and FUNCTIONS blocks is identical, only parameter numbers differ. - for my $pr (@per_run) { - return [0, "series-merge-theory-mismatch", ""] - if series_msr_token_shape($pr->{theory}) ne series_msr_token_shape($first->{theory}); - return [0, "series-merge-functions-mismatch", ""] - if series_msr_token_shape($pr->{functions}) ne series_msr_token_shape($first->{functions}); - } - - # Assign merged numbers: shared parameters first, run-specific grouped per run. - my @merged_params; - my %shared_name_no; - for my $p (@{$first->{params}}) { - if ($name_counts{$p->{name}} == $num_runs - && !exists($shared_name_no{$p->{name}})) { - $shared_name_no{$p->{name}} = scalar(@merged_params) + 1; - push(@merged_params, { name => $p->{name}, rest => $p->{rest} }); - } - } - my @no_map; - for my $run_index (0 .. $num_runs - 1) { - $no_map[$run_index] = {}; - for my $p (@{$per_run[$run_index]{params}}) { - if ($name_counts{$p->{name}} == $num_runs) { - $no_map[$run_index]{$p->{no}} = $shared_name_no{$p->{name}}; - } else { - $no_map[$run_index]{$p->{no}} = scalar(@merged_params) + 1; - push(@merged_params, { name => $p->{name}, rest => $p->{rest} }); - } - } - } - - my %by_number; - for my $pr (@per_run) { - my %map; - $map{$_->{no}} = $_ for @{$pr->{params}}; - $by_number{$pr} = \%map; - } - - # Rewrite THEORY: existing mapN/funN stay, direct integers are renumbered, - # run-specific roles become map slots filled per RUN block below. - my %occupied; - for my $lines ($first->{theory}, $first->{functions}) { - for my $line (@{$lines || []}) { - for my $token (split(/\s+/, $line)) { - $occupied{$1}++ if $token =~ /^map(\d+)$/i; - } - } - } - my $next_slot = 1; - $next_slot++ while $occupied{$next_slot}; - my (%slot_role_name, %slot_position); - my $slot_for = sub { - my ($name, $line_index, $token_index) = @_; - return $slot_role_name{$name} if exists($slot_role_name{$name}); - return 0 if $next_slot > $SERIES_MSR_MAX_MAP_SLOTS; - my $slot = $next_slot++; - $slot_role_name{$name} = $slot; - $slot_position{$slot} = [ $line_index, $token_index ]; - return $slot; - }; - - my @merged_theory; - for my $line_index (0 .. $#{$first->{theory}}) { - my @out; - my $token_index = 0; - for my $token (split(/\s+/, $first->{theory}[$line_index])) { - if ($token =~ /^\d+$/ && $token ne "0") { - my $local_no = 0 + $token; - my $p = $by_number{$first}{$local_no}; - return [0, "series-merge-theory-param-missing", ""] if !$p; - if ($name_counts{$p->{name}} == $num_runs) { - push(@out, $shared_name_no{$p->{name}}); - } else { - my $slot = $slot_for->($p->{name}, $line_index, $token_index); - return [0, "series-merge-map-slots-exhausted", ""] if $slot == 0; - push(@out, "map$slot"); - } - } else { - push(@out, $token); - } - $token_index++; - } - push(@merged_theory, join(" ", @out)); - } - - # FUNCTIONS is shared, so it may only reference parameters that keep their - # number in every RUN block. - my @merged_functions; - for my $line (@{$first->{functions}}) { - my @out; - for my $token (split(/\s+/, $line)) { - if ($token =~ /^par(\d+)$/i) { - my $local_no = 0 + $1; - my $p = $by_number{$first}{$local_no}; - return [0, "series-merge-functions-param-missing", ""] if !$p; - return [0, "series-merge-functions-run-specific", ""] - if $name_counts{$p->{name}} != $num_runs; - push(@out, "par" . $shared_name_no{$p->{name}}); - } else { - push(@out, $token); - } - } - push(@merged_functions, join(" ", @out)); - } - - my $max_slot = 0; - for my $slot (keys %slot_position) { $max_slot = $slot if $slot > $max_slot; } - - my $merged_map_entries = sub { - my ($run_index, $block) = @_; - my @existing_values; - for my $line (@{$block->{lines}}) { - if ($line =~ /^\s*map\b/i) { - my @tokens = split(/\s+/, $line); - @existing_values = @tokens > 1 ? @tokens[1 .. $#tokens] : (); - last; - } - } - my %entries; - my $slot_max = $max_slot; - $slot_max = scalar(@existing_values) if @existing_values > $slot_max; - for my $slot (1 .. $slot_max) { - my $value = 0; - if ($slot <= @existing_values && $existing_values[$slot - 1] =~ /^\d+$/) { - my $local_no = 0 + $existing_values[$slot - 1]; - $value = ($local_no > 0 && defined($no_map[$run_index]{$local_no})) - ? $no_map[$run_index]{$local_no} : $local_no; - } elsif (exists($slot_position{$slot})) { - my ($line_index, $token_index) = @{$slot_position{$slot}}; - my @local_tokens = split(/\s+/, $per_run[$run_index]{theory}[$line_index] || ""); - my $local_no = $local_tokens[$token_index] =~ /^\d+$/ - ? 0 + $local_tokens[$token_index] : 0; - $value = $no_map[$run_index]{$local_no} - if $local_no > 0 && defined($no_map[$run_index]{$local_no}); - } - $entries{$slot} = $value; - } - my @values; - for my $slot (1 .. $slot_max) { push(@values, $entries{$slot}); } - pop @values while @values && $values[$#values] == 0; - return \@values; - }; - - my @output; - push(@output, "# Independent series display MSR for the runs: " . join(" ", @{$run_numbers})); - push(@output, "# Fitted per-run parameters collected without re-fitting; " - . "msr2data handles parameter extraction separately."); - push(@output, "###############################################################"); - push(@output, "FITPARAMETER"); - push(@output, "###############################################################"); - push(@output, "# Common parameters for all runs"); - for my $index (0 .. $#merged_params) { - next if $name_counts{$merged_params[$index]{name}} != $num_runs; - push(@output, format_series_merged_parameter_line( - $index + 1, $merged_params[$index]{name}, $merged_params[$index]{rest} - )); - } - for my $run_index (0 .. $num_runs - 1) { - push(@output, "# Specific parameters for run " . $per_run[$run_index]{run}); - for my $p (@{$per_run[$run_index]{params}}) { - next if $name_counts{$p->{name}} == $num_runs; - push(@output, format_series_merged_parameter_line( - $no_map[$run_index]{$p->{no}}, $p->{name}, $p->{rest} - )); - } - } - push(@output, "###############################################################"); - push(@output, "THEORY"); - push(@output, @merged_theory); - if (@{$first->{functions}}) { - push(@output, "###############################################################"); - push(@output, "FUNCTIONS"); - push(@output, @merged_functions); - } - push(@output, "###############################################################"); - push(@output, "GLOBAL"); - push(@output, @{$first->{global}}); - for my $run_index (0 .. $num_runs - 1) { - push(@output, "###############################################################"); - push(@output, "# RUN blocks for run " . $per_run[$run_index]{run}); - for my $block (@{$per_run[$run_index]{run_blocks}}) { - push(@output, $block->{header}); - my $had_map = 0; - for my $line (@{$block->{lines}}) { - my $trimmed = $line; - $trimmed =~ s/^\s+//; - next if $trimmed eq "" || $trimmed =~ /^#/; - my ($directive) = $trimmed =~ /^(\S+)/; - if ($directive =~ /^map$/i) { $had_map = 1; next; } - if ($SERIES_MSR_RUN_PARAM_DIRECTIVES{lc($directive)}) { - my $rest = $trimmed; - $rest =~ s/^\s*\Q$directive\E\s*//; - my @renumbered; - for my $token (split(/\s+/, $rest)) { - if ($token =~ /^\d+$/) { - my $n = 0 + $token; - push(@renumbered, defined($no_map[$run_index]{$n}) - ? $no_map[$run_index]{$n} : $token); - } else { - push(@renumbered, $token); - } - } - push(@output, $directive . " " . join(" ", @renumbered)); - } else { - push(@output, $line); - } - } - if ($had_map || $max_slot > 0) { - my $values = $merged_map_entries->($run_index, $block); - push(@output, "map " . join(" ", @{$values})) if @{$values}; - } - } - } - push(@output, "###############################################################"); - push(@output, "COMMANDS"); - push(@output, @{$first->{commands}}); - if (@{$first->{fourier}}) { - push(@output, "###############################################################"); - push(@output, "FOURIER"); - push(@output, @{series_fourier_for_display($first->{fourier})}); - } - push(@output, "###############################################################"); - push(@output, $first->{plot_header}); - for my $line (@{$first->{plot}}) { - next if $line =~ /^\s*runs\s+/i; - push(@output, $line); - } - my @plot_runs = (1 .. $num_runs); - push(@output, "runs " . join(" ", @plot_runs)); - - # musrfit always stamps a STATISTIC block into a fitted file, and msr2data's - # global data export keys each run's convergence on that block. The merged - # display MSR is not re-fitted, so carry the per-run statistics forward as - # the ordinary aggregate musrfit would write for a global fit: summed chisq - # and NDF, reduced chisq from those sums. - my ($chisq_sum, $ndf_sum) = (0, 0); - my $statistic_runs = 0; - for my $pr (@per_run) { - for my $line (@{$pr->{statistic}}) { - if ($line =~ /chisq\s*=\s*([^\s,]+).*?NDF\s*=\s*([^\s,]+)/i) { - my ($run_chisq, $run_ndf) = (0 + $1, 0 + $2); - next if $run_ndf <= 0; - $chisq_sum += $run_chisq; - $ndf_sum += $run_ndf; - $statistic_runs++; - } - } - } - if ($statistic_runs) { - my ($sec, $min, $hour, $mday, $mon, $year) = localtime(); - my $timestamp = sprintf("%04d-%02d-%02d %02d:%02d:%02d", - $year + 1900, $mon + 1, $mday, $hour, $min, $sec); - my $reduced = $ndf_sum > 0 ? $chisq_sum / $ndf_sum : 0; - push(@output, "###############################################################"); - push(@output, "STATISTIC --- " . $timestamp); - my $formatted_chisq = sprintf("%.4f", $chisq_sum); - my $formatted_reduced = sprintf("%.6f", $reduced); - $formatted_chisq =~ s/\.?0+$//; - $formatted_reduced =~ s/\.?0+$//; - push(@output, sprintf(" chisq = %s, NDF = %d, chisq/NDF = %s", - $formatted_chisq, $ndf_sum, $formatted_reduced)); - } - return [1, "series-merge-ready", join("\n", @output) . "\n"]; -} +# Display-MSR parsing and merging live in muSRFitlib.js and run in the browser. # Collect a completed independent series without running musrfit again. The # requested non-empty data table and structurally valid combined MSR are the @@ -1473,16 +1028,21 @@ sub perform_series_artifact_collection { }; }; return $failure->("invalid-series-collection", "Invalid series collection request") - if !defined($collection_text) || length($collection_text) > 100_000; + if !defined($collection_text) || length($collection_text) > 5_000_000; my $collection = eval { decode_json($collection_text) }; return $failure->("invalid-series-collection", "Invalid series collection JSON") if $@ || ref($collection) ne "HASH"; my $runs = $collection->{runs}; my $suffix = defined($collection->{suffix}) ? "$collection->{suffix}" : ""; my $fftv = defined($collection->{fftv}) ? "$collection->{fftv}" : "n"; + my $contract_version = defined($collection->{version}) + && !ref($collection->{version}) ? 0 + $collection->{version} : 0; + my $client_combined = exists($collection->{combinedMSR}) + && !ref($collection->{combinedMSR}) ? "$collection->{combinedMSR}" : undef; return $failure->("invalid-series-collection", "Invalid series collection fields") if ref($runs) ne "ARRAY" || @{$runs} < 2 || @{$runs} > 500 - || $suffix !~ /^_[A-Za-z0-9._+\-]+$/ || $fftv !~ /^(?:y|n)$/; + || $suffix !~ /^_[A-Za-z0-9._+\-]+$/ || $fftv !~ /^(?:y|n)$/ + || $contract_version != 1 || !defined($client_combined); my @run_numbers = (); my %seen = (); @@ -1509,15 +1069,27 @@ sub perform_series_artifact_collection { # global-fit results that may already exist in the same HTTP job folder. my $combined_name = $base_run . "+series" . $suffix . ".msr"; my $combined_path = "$work_path/$combined_name"; - # msr2data still needs a base+global.msr staging input. A unique - # internal suffix prevents collision with the normal `_tmpl` global path. - my $template_suffix = "_musrflow_series_" . $$ . "_" . time(); - my $template_name = $base_run . "+global" . $template_suffix . ".msr"; - my $template_path = "$work_path/$template_name"; my $parameter_name = $base_run . "+series" . $suffix . "_par.dat"; my $parameter_path = "$work_path/$parameter_name"; my $out_name = $base_run . "+series" . $suffix . ".out"; my $out_path = "$work_path/$out_name"; + my $artifacts = $collection->{artifacts}; + return $failure->("invalid-series-collection", "Invalid series collection contract") + if ref($artifacts) ne "HASH" + || ($artifacts->{combined} || "") ne $combined_name + || ($artifacts->{parameters} || "") ne $parameter_name + || ($artifacts->{log} || "") ne $out_name + || length($client_combined) > 4_000_000; + + # Collection mutates the job's generated artifacts and starts external + # programs. Serialize it with fitting/promotion operations for this job. + my $lock_path = "$work_path/.musrflow-series.lock"; + return $failure->("series-collection-lock-failed", "Could not open series job lock") + if !sysopen(my $lock_fh, $lock_path, O_RDWR | O_CREAT, 0600); + if (!flock($lock_fh, LOCK_EX | LOCK_NB)) { + close($lock_fh); + return $failure->("series-collection-lock-failed", "Another series operation is active"); + } foreach my $stale_path ($combined_path, $parameter_path) { return $failure->("series-artifact-output-invalid", "A series output path is unsafe") if -l $stale_path || (-e $stale_path && !-f $stale_path); @@ -1534,55 +1106,59 @@ sub perform_series_artifact_collection { my $run_argument = "[" . join(" ", @run_numbers) . "]"; my @source_texts = map { $source_texts{$_} } @run_numbers; - # Build the combined display MSR first. It becomes the canonical - # `+global.msr` file that both musrview and the parameter export - # use, and is assembled directly from the already-fitted per-run files so - # no global re-fit ever takes place. - my $combined_url = ""; - my $merged = build_series_global_msr(\@source_texts, \@run_numbers); - if (!$merged->[0]) { - push(@warnings, "Could not assemble the combined display MSR ($merged->[1])"); - } elsif (!is_valid_generated_global_msr($merged->[2])) { - push(@warnings, "Assembled combined display MSR failed validation"); - } elsif (write_text_file($combined_path, normalize_plot_runs_line( - $merged->[2], join(" ", @run_numbers) - ))) { - $combined_url = "$TMP_URL/$job_name/$combined_name"; + # Collect fitted per-run values directly. This is independent of display + # merging, so models that cannot yet be represented in one display MSR + # still produce the table consumed by Plot parameters. + my $parameter_url = ""; + my $parameter_cmd = "cd " . shell_quote($work_path) + . " && " . shell_quote($MSR2DATA_BIN) + . " " . shell_quote($run_argument) + . " " . shell_quote($suffix) + . " new data -o " . shell_quote($parameter_name); + my $parameter_result = run_logged_command( + $out_path, "msr2data independent series collection", $parameter_cmd + ); + if (-s $parameter_path) { + if (!clean_parameter_data_header($parameter_path, \@run_numbers)) { + push(@warnings, "Could not normalize the parameter-table header"); + } + $parameter_url = "$TMP_URL/$job_name/$parameter_name"; + if ($parameter_result->{raw_status} != 0) { + push(@warnings, "msr2data returned code " . $parameter_result->{exit_code} + . " but produced the parameter table"); + } } else { - push(@warnings, "Could not write the combined display MSR"); + push(@warnings, "msr2data did not produce parameter data"); } - # msr2data is used only for the parameter table, exported with the same - # `new global data` form and `_tmpl` staging as the ordinary multirun - # export, producing the global .dat layout (stripped base names, one row - # per run) that the parameter plot button reads. - my $parameter_url = ""; - my $combined_text = $combined_url ne "" ? read_text_file($combined_path) : ""; - if ($combined_text ne "" && write_text_file($template_path, $combined_text)) { - my $parameter_cmd = "cd " . shell_quote($work_path) - . " && " . shell_quote($MSR2DATA_BIN) - . " " . shell_quote($run_argument) - . " " . shell_quote($template_suffix) - . " new global data -o " . shell_quote($parameter_name); - my $parameter_result = run_logged_command( - $out_path, "msr2data independent series collection", $parameter_cmd - ); - if (-s $parameter_path) { - if (!clean_parameter_data_header($parameter_path)) { - push(@warnings, "Could not normalize the parameter-table header"); - } - # Keep the non-empty file available even when header cleanup fails. - $parameter_url = "$TMP_URL/$job_name/$parameter_name"; - if ($parameter_result->{raw_status} != 0) { - push(@warnings, "msr2data returned code " . $parameter_result->{exit_code} - . " but produced the parameter table"); - } + # The browser carries the display MSR produced by the shared JavaScript + # merger. Verify that it retains exactly the fitted sources' RUN/ADDRUN + # identities before publishing or passing it to musrview. + my $combined_url = ""; + my $combined_text = $client_combined || ""; + if ($combined_text ne "") { + my @expected_directives = map { + @{series_run_directives($_)} + } @source_texts; + my @actual_directives = @{series_run_directives($combined_text)}; + if (!is_valid_generated_global_msr($combined_text)) { + push(@warnings, "Submitted combined display MSR failed validation"); + $combined_text = ""; + } elsif (join("\n", @actual_directives) ne join("\n", @expected_directives)) { + push(@warnings, "Submitted combined display MSR changed RUN identity"); + $combined_text = ""; + } + } + if ($combined_text ne "") { + my $combined_stage = "$work_path/.musrflow-series-display-$$.tmp"; + if (!-e $combined_stage + && write_text_file($combined_stage, $combined_text) + && rename($combined_stage, $combined_path)) { + $combined_url = "$TMP_URL/$job_name/$combined_name"; } else { - push(@warnings, "msr2data did not produce parameter data"); + unlink($combined_stage) if -f $combined_stage && !-l $combined_stage; + push(@warnings, "Could not write the combined display MSR"); } - unlink($template_path); - } elsif ($combined_url ne "") { - push(@warnings, "Could not stage the combined MSR for parameter export"); } my @root_urls = (); @@ -1627,6 +1203,7 @@ sub perform_series_artifact_collection { my $command_log = read_text_file($out_path); my $usable = $parameter_url ne "" || $combined_url ne ""; + close($lock_fh); return { ok => $usable ? JSON::PP::true : JSON::PP::false, code => $usable ? 0 : 1, diff --git a/muSRFitelec.js b/muSRFitelec.js index 5661065..2997030 100644 --- a/muSRFitelec.js +++ b/muSRFitelec.js @@ -723,20 +723,31 @@ function appendFitLog(logFile, text) { } } -// msr2data represents a run-qualified parameter such as Asy_2274 as Asy_ in -// the exported data header. Remove exactly one trailing underscore from header -// fields so downstream plotting sees the original logical parameter name. -// Underscores inside names (for example my_parameter) and error labels such as -// Asy_PosErr are left untouched. -function cleanMusrfitParameterDataHeader(parameterPath) { +// Global msr2data exports represent a run-qualified parameter such as +// Asy_2274 as Asy_, while ordinary per-run exports retain the first run suffix +// as Asy_2274, Asy_2274PosErr, and Asy_2274NegErr. Normalize only suffixes that +// match this collection's run list so downstream plotting sees the logical +// parameter name. Embedded underscores in names remain untouched. +function cleanMusrfitParameterDataHeader(parameterPath, runNumbers = []) { let contents = nodeFs.readFileSync(parameterPath, "utf8"); let newline = contents.includes("\r\n") ? "\r\n" : "\n"; let lines = contents.split(/\r?\n/); + let runs = new Set(runNumbers.map(function(run) { + return String(Number(run)); + }).filter(function(run) { + return /^\d+$/.test(run); + })); for (let i = 0; i < lines.length; i++) { let probe = lines[i].trim(); if (!probe || probe.startsWith("#")) continue; - lines[i] = lines[i].replace(/(\S+)_(?=\s|$)/g, "$1"); + lines[i] = lines[i].replace(/\S+/g, function(column) { + let qualified = /^(.*)_([0-9]+)((?:_)?(?:PosErr|NegErr))?$/.exec(column); + if (qualified && runs.has(String(Number(qualified[2])))) { + return qualified[1] + (qualified[3] || ""); + } + return column.endsWith("_") ? column.slice(0, -1) : column; + }); break; } @@ -975,30 +986,17 @@ window.runSeriesMSRCandidateLocal = runSeriesMSRCandidateLocal; // +global file from a template). A non-empty requested .dat is registered // immediately, and any structurally valid combined MSR is safe for musrview. async function collectIndependentSeriesArtifactsLocal(spec) { - let result = { - ok: false, - reason: "invalid-series-artifact-spec", - parameterDataPath: "", - combinedMSRPath: "", - logPath: "", - plotStarted: false, - warnings: [] - }; - if (!spec || spec.ok !== true || !Array.isArray(spec.runs) - || spec.runs.length < 2 || !Array.isArray(spec.msrPaths) - || spec.msrPaths.length !== spec.runs.length - || !/^_[A-Za-z0-9._+\-]+$/.test(spec.fileNameSuffix || "")) { - return result; - } + let prepared = prepareIndependentSeriesCollection(spec); + let result = normalizeIndependentSeriesCollectionResult({ reason: prepared.reason }); + if (!prepared.ok) return result; - let workPath = nodePath.resolve(spec.workPath); - let suffix = spec.fileNameSuffix; - let baseRun = spec.runs[0]; - for (let index = 0; index < spec.runs.length; index++) { - if (!/^\d+$/.test(spec.runs[index]) - || nodePath.resolve(spec.msrPaths[index]) - !== nodePath.join(workPath, spec.runs[index] + suffix + ".msr") - || !isSafeSeriesRegularFile(spec.msrPaths[index])) { + let workPath = nodePath.resolve(prepared.workPath); + let suffix = prepared.fileNameSuffix; + let baseRun = prepared.baseRun; + for (let index = 0; index < prepared.runs.length; index++) { + if (nodePath.resolve(prepared.msrPaths[index]) + !== nodePath.join(workPath, prepared.runs[index] + suffix + ".msr") + || !isSafeSeriesRegularFile(prepared.msrPaths[index])) { result.reason = "series-artifact-source-invalid"; return result; } @@ -1007,18 +1005,11 @@ async function collectIndependentSeriesArtifactsLocal(spec) { // Independent-series artifacts use their own namespace. A genuine global // fit can legitimately exist beside the same per-run files and must never // be removed or replaced by display-only post-processing. - let combinedName = baseRun + "+series" + suffix + ".msr"; + let combinedName = prepared.combinedName; let combinedPath = nodePath.join(workPath, combinedName); - // msr2data global-data mode requires a `base+global.msr` input. - // Give the staging file a process-unique suffix so an ordinary `_tmpl` - // global-fit template is neither overwritten nor deleted. - let templateSuffix = "_musrflow_series_" + process.pid + "_" - + Date.now().toString(36); - let templateName = baseRun + "+global" + templateSuffix + ".msr"; - let templatePath = nodePath.join(workPath, templateName); - let parameterName = baseRun + "+series" + suffix + "_par.dat"; + let parameterName = prepared.parameterName; let parameterPath = nodePath.join(workPath, parameterName); - let logName = baseRun + "+series" + suffix + ".out"; + let logName = prepared.logName; let logPath = nodePath.join(workPath, logName); result.logPath = logPath; @@ -1035,7 +1026,7 @@ async function collectIndependentSeriesArtifactsLocal(spec) { "# muSRFlow independent series collection\n# Working directory: " + workPath + "\n", { encoding: "utf8", flag: "w" }); - let runsArg = "[" + spec.runs.join(" ") + "]"; + let runsArg = "[" + prepared.runs.join(" ") + "]"; let msr2data = resolveLocalCommand("msr2data"); let env = Object.assign({}, process.env); env.PATH = ["/usr/local/bin", "/usr/bin", "/bin", env.PATH || ""].join(":"); @@ -1063,64 +1054,53 @@ async function collectIndependentSeriesArtifactsLocal(spec) { return code; }; - // Assemble the combined display MSR first. It becomes the canonical - // `+global.msr` file that both musrview and the parameter export - // use, and is built directly from the already-fitted per-run files so no - // global re-fit ever takes place. + // Parameter collection is the primary independent-series result. Use + // msr2data's ordinary per-run data mode—the same operation as the + // established rplcall/msr2data workflow—so an unsupported display merge + // can never prevent Plot parameters from receiving a usable table. + let parameterCode = runMsr2data("parameter msr2data", + [runsArg, suffix, "new", "data", "-o", parameterName]); + if (isSafeSeriesRegularFile(parameterPath) + && nodeFs.statSync(parameterPath).size > 0) { + try { + cleanMusrfitParameterDataHeader(parameterPath, spec.runs); + } catch (error) { + result.warnings.push("Could not normalize the parameter-table header."); + } + result.parameterDataPath = parameterPath; + if (parameterCode !== 0) { + result.warnings.push("msr2data returned code " + parameterCode + + " but produced the parameter table."); + } + } else { + result.warnings.push("msr2data did not produce parameter data."); + } + + // Assemble the combined display MSR independently from the already + // fitted per-run files. No global re-fit takes place. let msrTexts = []; - for (let index = 0; index < spec.msrPaths.length; index++) { - if (!isSafeSeriesRegularFile(spec.msrPaths[index])) { - result.warnings.push("Fitted MSR " + spec.msrPaths[index] + for (let index = 0; index < prepared.msrPaths.length; index++) { + if (!isSafeSeriesRegularFile(prepared.msrPaths[index])) { + result.warnings.push("Fitted MSR " + prepared.msrPaths[index] + " is missing for the combined display."); break; } - msrTexts.push(nodeFs.readFileSync(spec.msrPaths[index], "utf8")); + msrTexts.push(nodeFs.readFileSync(prepared.msrPaths[index], "utf8")); } - if (msrTexts.length === spec.msrPaths.length) { - let merged = buildSeriesGlobalMSR(msrTexts, spec.runs); + if (msrTexts.length === prepared.msrPaths.length) { + let merged = buildSeriesGlobalMSR(msrTexts, prepared.runs); if (merged.ok !== true) { result.warnings.push("Could not assemble the combined display MSR (" + merged.reason + ")."); } else if (!isValidGeneratedGlobalMSR(merged.text)) { result.warnings.push("Assembled combined display MSR failed validation."); } else { - let combinedText = normalizeMSRPlotRuns(merged.text, {}, spec.runs); + let combinedText = normalizeMSRPlotRuns(merged.text, {}, prepared.runs); nodeFs.writeFileSync(combinedPath, combinedText, { encoding: "utf8", flag: "wx" }); result.combinedMSRPath = combinedPath; - - // msr2data is used only for the parameter table, exported with the - // same `new global data` form and `_tmpl` staging as the ordinary - // multirun export, producing the global .dat layout (stripped base - // names, one row per run) that the parameter plot button reads. - nodeFs.copyFileSync(combinedPath, templatePath, nodeFs.constants.COPYFILE_EXCL); - try { - let parameterCode = runMsr2data("parameter msr2data", - [runsArg, templateSuffix, "new", "global", "data", "-o", parameterName]); - if (isSafeSeriesRegularFile(parameterPath) - && nodeFs.statSync(parameterPath).size > 0) { - try { - cleanMusrfitParameterDataHeader(parameterPath); - } catch (error) { - result.warnings.push("Could not normalize the parameter-table header."); - } - // A non-empty table is usable even if header cleanup or the - // external program's status is imperfect. - result.parameterDataPath = parameterPath; - if (parameterCode !== 0) { - result.warnings.push("msr2data returned code " + parameterCode - + " but produced the parameter table."); - } - } else { - result.warnings.push("msr2data did not produce parameter data."); - } - } finally { - if (isSafeSeriesRegularFile(templatePath)) { - nodeFs.unlinkSync(templatePath); - } - } } } else { result.warnings.push("Could not read all fitted per-run MSR files" @@ -1129,7 +1109,7 @@ async function collectIndependentSeriesArtifactsLocal(spec) { if (result.combinedMSRPath) { let musrviewArgs = []; - if (spec.fftv == "y") musrviewArgs.push("-f"); + if (prepared.fftv == "y") musrviewArgs.push("-f"); musrviewArgs.push(combinedName); let viewId = window.MusrfitRunner.nextViewId++; runLocalProcess("musrview series #" + viewId, @@ -1137,15 +1117,12 @@ async function collectIndependentSeriesArtifactsLocal(spec) { { reportStatus: false }); result.plotStarted = true; } - result.ok = Boolean(result.parameterDataPath || result.combinedMSRPath); - result.reason = result.ok - ? "series-artifacts-collected" : "series-artifacts-not-produced"; - return result; + return normalizeIndependentSeriesCollectionResult(result); } catch (error) { result.reason = "series-artifact-collection-failed"; result.warnings.push(error && error.message ? error.message : "Series post-processing failed."); - return result; + return normalizeIndependentSeriesCollectionResult(result); } } diff --git a/muSRFitlib.js b/muSRFitlib.js index 7e2b0e2..7a9d2cf 100644 --- a/muSRFitlib.js +++ b/muSRFitlib.js @@ -4608,14 +4608,18 @@ function buildSeriesGlobalMSR(msrTexts, runs) { let nextSlot = 1; while (occupiedSlots.has(nextSlot)) nextSlot++; let slotRoleName = Object.create(null); - let slotPosition = Object.create(null); // slot -> { lineIndex, tokenIndex } + let slotPosition = Object.create(null); // slot -> { section, lineIndex, tokenIndex } - function slotForParam(name, lineIndex, tokenIndex) { + function slotForParam(name, section, lineIndex, tokenIndex) { if (slotRoleName[name] !== undefined) return slotRoleName[name]; if (nextSlot > SERIES_MSR_MAX_MAP_SLOTS) return 0; let slot = nextSlot++; slotRoleName[name] = slot; - slotPosition[slot] = { lineIndex: lineIndex, tokenIndex: tokenIndex }; + slotPosition[slot] = { + section: section, + lineIndex: lineIndex, + tokenIndex: tokenIndex + }; return slot; } @@ -4631,7 +4635,7 @@ function buildSeriesGlobalMSR(msrTexts, runs) { if (nameCounts[p.name] === numRuns) { out.push(String(sharedNameNo[p.name])); } else { - let slot = slotForParam(p.name, lineIndex, tokenIndex); + let slot = slotForParam(p.name, "theory", lineIndex, tokenIndex); if (slot === 0) return fail("series-merge-map-slots-exhausted"); out.push("map" + slot); } @@ -4646,25 +4650,29 @@ function buildSeriesGlobalMSR(msrTexts, runs) { if (line && typeof line == "object" && line.ok === false) return fail(line.reason); } - // FUNCTIONS is shared by all runs, so it may only reference parameters that - // keep the same number across every RUN block. a function that multiplies - // two run-specific parameters would otherwise force a different curve per - // run block, which the global file cannot express. - let mergedFunctions = first.functions.map(function(line) { + // FUNCTIONS can select run-specific parameters through mapN just like + // THEORY. Allocate a RUN map slot for each such parN reference; genuinely + // shared parameters keep a direct parN reference. + let mergedFunctions = first.functions.map(function(line, lineIndex) { let out = []; + let tokenIndex = 0; for (let token of line.split(/\s+/)) { let parMatch = token.match(/^par(\d+)$/i); if (parMatch) { let localNo = parseInt(parMatch[1], 10); let p = byNumber[0][localNo]; if (!p) return fail("series-merge-functions-param-missing"); - if (nameCounts[p.name] !== numRuns) { - return fail("series-merge-functions-run-specific"); + if (nameCounts[p.name] === numRuns) { + out.push("par" + sharedNameNo[p.name]); + } else { + let slot = slotForParam(p.name, "functions", lineIndex, tokenIndex); + if (slot === 0) return fail("series-merge-map-slots-exhausted"); + out.push("map" + slot); } - out.push("par" + sharedNameNo[p.name]); } else { out.push(token); } + tokenIndex++; } return out.join(" "); }); @@ -4693,17 +4701,24 @@ function buildSeriesGlobalMSR(msrTexts, runs) { let slotMax = Math.max(maxSlot, existing ? existing.length : 0); for (let slot = 1; slot <= slotMax; slot++) { let value = 0; - if (existing && existing[slot - 1] && /^\d+$/.test(existing[slot - 1])) { + if (existing && existing[slot - 1] && /^\d+$/.test(existing[slot - 1]) + && parseInt(existing[slot - 1], 10) > 0) { let localNo = parseInt(existing[slot - 1], 10); value = (localNo > 0 && noMap[runIndex][localNo] !== undefined) ? noMap[runIndex][localNo] : localNo; } else if (slotPosition[slot]) { let position = slotPosition[slot]; - let localTokens = perRun[runIndex].theory[position.lineIndex] - ? perRun[runIndex].theory[position.lineIndex].split(/\s+/) + let sourceLines = position.section === "functions" + ? perRun[runIndex].functions : perRun[runIndex].theory; + let localTokens = sourceLines[position.lineIndex] + ? sourceLines[position.lineIndex].split(/\s+/) : []; let localToken = localTokens[position.tokenIndex]; - let localNo = /^\d+$/.test(localToken || "") ? parseInt(localToken, 10) : 0; + let functionPar = position.section === "functions" + ? String(localToken || "").match(/^par(\d+)$/i) : null; + let localNo = functionPar + ? parseInt(functionPar[1], 10) + : (/^\d+$/.test(localToken || "") ? parseInt(localToken, 10) : 0); if (localNo > 0 && noMap[runIndex][localNo] !== undefined) { value = noMap[runIndex][localNo]; } @@ -4802,26 +4817,32 @@ function buildSeriesGlobalMSR(msrTexts, runs) { for (let i = 1; i <= numRuns; i++) plotRuns.push(String(i)); output.push("runs " + plotRuns.join(" ")); - // musrfit always stamps a STATISTIC block into a fitted file, and msr2data's - // global data export keys each run's convergence on that block. The merged - // display MSR is not re-fitted, so carry the per-run statistics forward as - // the ordinary aggregate musrfit would write for a global fit: summed chisq - // and NDF, reduced chisq from those sums. + // The display MSR is not re-fitted. When every source has the same objective, + // report the sum of its independently fitted statistic and NDF. Never emit a + // partial or mixed-objective aggregate. let statistics = []; + let statisticKind = ""; for (let pr of perRun) { let match = pr.statistic.match( - /chisq\s*=\s*([^\s,]+).*?NDF\s*=\s*([^\s,]+)/i + /(chisq|maxLH)\s*=\s*([^\s,]+).*?NDF\s*=\s*([^\s,]+)/i ); if (!match) continue; - let runChiSq = parseFloat(match[1]); - let runNDF = parseFloat(match[2]); - if (isNaN(runChiSq) || isNaN(runNDF) || runNDF <= 0) continue; - statistics.push({ chiSq: runChiSq, ndf: runNDF }); + let kind = match[1].toLowerCase() === "maxlh" ? "maxLH" : "chisq"; + let runValue = parseFloat(match[2]); + let runNDF = parseFloat(match[3]); + if (isNaN(runValue) || isNaN(runNDF) || runNDF <= 0) continue; + if (statisticKind && statisticKind !== kind) { + statistics = []; + statisticKind = ""; + break; + } + statisticKind = kind; + statistics.push({ value: runValue, ndf: runNDF }); } - if (statistics.length) { - let chiSq = 0; + if (statistics.length === numRuns) { + let total = 0; let ndf = 0; - for (let s of statistics) { chiSq += s.chiSq; ndf += s.ndf; } + for (let s of statistics) { total += s.value; ndf += s.ndf; } let date = new Date(); let timestamp = date.getFullYear() + "-" + String(date.getMonth() + 1).padStart(2, "0") + "-" @@ -4829,13 +4850,13 @@ function buildSeriesGlobalMSR(msrTexts, runs) { + String(date.getHours()).padStart(2, "0") + ":" + String(date.getMinutes()).padStart(2, "0") + ":" + String(date.getSeconds()).padStart(2, "0"); - let formattedChiSq = parseFloat(chiSq.toFixed(4)).toString(); + let formattedTotal = parseFloat(total.toFixed(4)).toString(); let formattedNDF = Math.round(ndf).toString(); - let formattedReduced = parseFloat((chiSq / ndf).toFixed(6)).toString(); + let formattedReduced = parseFloat((total / ndf).toFixed(6)).toString(); output.push("###############################################################"); output.push("STATISTIC --- " + timestamp); - output.push(" chisq = " + formattedChiSq + ", NDF = " + formattedNDF - + ", chisq/NDF = " + formattedReduced); + output.push(" " + statisticKind + " = " + formattedTotal + ", NDF = " + + formattedNDF + ", " + statisticKind + "/NDF = " + formattedReduced); } return { @@ -9469,6 +9490,104 @@ function buildIndependentSeriesArtifactSpec(plan, config = {}) { return result; } +// Normalize the platform-neutral part of a completed-series collection. Local +// filesystem confinement and web URL/job checks remain adapter responsibilities, +// but artifact names and the data exchanged with the CGI must be identical. +function prepareIndependentSeriesCollection(spec) { + let invalid = function(reason) { + return { + ok: false, + reason: reason || "invalid-series-artifact-spec", + workPath: "", + fileNameSuffix: "", + runs: [], + msrPaths: [], + fftv: "n", + baseRun: "", + combinedName: "", + parameterName: "", + logName: "" + }; + }; + if (!spec || spec.ok !== true || !Array.isArray(spec.runs) + || spec.runs.length < 2 || !Array.isArray(spec.msrPaths) + || spec.msrPaths.length !== spec.runs.length + || typeof spec.workPath !== "string" || spec.workPath === "" + || !/^_[A-Za-z0-9._+\-]+$/.test(spec.fileNameSuffix || "")) { + return invalid("invalid-series-artifact-spec"); + } + let runs = spec.runs.map(function(run) { return String(run); }); + if (runs.some(function(run) { return !/^\d+$/.test(run) || Number(run) <= 0; }) + || new Set(runs).size !== runs.length + || spec.msrPaths.some(function(path) { + return typeof path !== "string" || path === ""; + })) { + return invalid("series-artifact-source-invalid"); + } + let baseRun = runs[0]; + let suffix = spec.fileNameSuffix; + return { + ok: true, + reason: "series-collection-request-ready", + workPath: spec.workPath, + fileNameSuffix: suffix, + runs: runs, + msrPaths: spec.msrPaths.slice(), + fftv: spec.fftv == "y" ? "y" : "n", + baseRun: baseRun, + combinedName: baseRun + "+series" + suffix + ".msr", + parameterName: baseRun + "+series" + suffix + "_par.dat", + logName: baseRun + "+series" + suffix + ".out" + }; +} + +function buildIndependentSeriesCollectionRequest(spec, combinedMSR) { + let prepared = prepareIndependentSeriesCollection(spec); + if (!prepared.ok) return prepared; + return { + ok: true, + version: 1, + runs: prepared.runs.slice(), + suffix: prepared.fileNameSuffix, + fftv: prepared.fftv, + artifacts: { + combined: prepared.combinedName, + parameters: prepared.parameterName, + log: prepared.logName + }, + combinedMSR: typeof combinedMSR === "string" ? combinedMSR : "" + }; +} + +function normalizeIndependentSeriesCollectionResult(value) { + value = value && typeof value === "object" ? value : {}; + let result = { + ok: false, + reason: typeof value.reason === "string" && value.reason + ? value.reason : "series-artifacts-not-produced", + parameterDataPath: typeof value.parameterDataPath === "string" + ? value.parameterDataPath : "", + combinedMSRPath: typeof value.combinedMSRPath === "string" + ? value.combinedMSRPath : "", + logPath: typeof value.logPath === "string" ? value.logPath : "", + plotPath: typeof value.plotPath === "string" ? value.plotPath : "", + plotPaths: Array.isArray(value.plotPaths) + ? value.plotPaths.filter(function(path) { + return typeof path === "string" && path !== ""; + }) : [], + plotStarted: value.plotStarted === true, + warnings: Array.isArray(value.warnings) + ? value.warnings.filter(function(warning) { + return typeof warning === "string" && warning !== ""; + }) : [] + }; + result.ok = Boolean(result.parameterDataPath || result.combinedMSRPath); + if (result.ok && (!value.reason || value.reason == "series-artifacts-not-produced")) { + result.reason = "series-artifacts-collected"; + } + return result; +} + async function collectIndependentSeriesArtifacts(plan, config, webMode) { let spec = buildIndependentSeriesArtifactSpec(plan, config); if (!spec.ok) { @@ -9503,7 +9622,7 @@ async function collectIndependentSeriesArtifacts(plan, config, webMode) { warnings: [error && error.message ? error.message : "Series collection failed."] }; } - artifacts = artifacts && typeof artifacts === "object" ? artifacts : {}; + artifacts = normalizeIndependentSeriesCollectionResult(artifacts); let status = {}; if (artifacts.parameterDataPath) status.parameterDataPath = artifacts.parameterDataPath; if (artifacts.combinedMSRPath) status.resultMSR = artifacts.combinedMSRPath; diff --git a/muSRFitweb.js b/muSRFitweb.js index 8adb4a4..332c1b6 100644 --- a/muSRFitweb.js +++ b/muSRFitweb.js @@ -753,40 +753,48 @@ window.runSeriesMSRCandidateWeb = runSeriesMSRCandidateWeb; // never invokes musrfit; it only creates the parameter table, a combined display // MSR, and the ROOT canvas used by the browser viewer. async function collectIndependentSeriesArtifactsWeb(spec) { - let result = { - ok: false, - reason: "invalid-series-artifact-spec", - parameterDataPath: "", - combinedMSRPath: "", - logPath: "", - plotPath: "", - plotStarted: false, - warnings: [] - }; - if (!spec || spec.ok !== true || !Array.isArray(spec.runs) - || spec.runs.length < 2 || !Array.isArray(spec.msrPaths) - || spec.msrPaths.length !== spec.runs.length - || !/^_[A-Za-z0-9._+\-]+$/.test(spec.fileNameSuffix || "")) { - return result; - } - let workUrl = String(spec.workPath || "").replace(/\/+$/, ""); - for (let index = 0; index < spec.runs.length; index++) { - if (!/^\d+$/.test(spec.runs[index]) - || spec.msrPaths[index] - !== workUrl + "/" + spec.runs[index] + spec.fileNameSuffix + ".msr") { + let prepared = prepareIndependentSeriesCollection(spec); + let result = normalizeIndependentSeriesCollectionResult({ reason: prepared.reason }); + if (!prepared.ok) return result; + let workUrl = prepared.workPath.replace(/\/+$/, ""); + for (let index = 0; index < prepared.runs.length; index++) { + if (prepared.msrPaths[index] + !== workUrl + "/" + prepared.runs[index] + prepared.fileNameSuffix + ".msr") { result.reason = "series-artifact-source-invalid"; return result; } } + // Generated job files are HTTP-visible, so the browser can perform the + // pure merge itself using the same implementation as Electron. Failure to + // build a display file does not block the server-side parameter collection. + let clientWarnings = []; + let combinedMSR = ""; + let sourceTexts = await Promise.all(prepared.msrPaths.map(function(path) { + return readAsciiFile(path); + })); + if (sourceTexts.every(function(text) { + return typeof text === "string" && text !== ""; + })) { + let merged = buildSeriesGlobalMSR(sourceTexts, prepared.runs); + if (merged.ok && isValidGeneratedGlobalMSR(merged.text)) { + combinedMSR = normalizeMSRPlotRuns(merged.text, {}, prepared.runs); + } else { + clientWarnings.push("Could not assemble the combined display MSR (" + + (merged.reason || "series-merge-invalid") + ")."); + } + } else { + clientWarnings.push("Could not fetch all fitted per-run MSR files" + + " for the combined display."); + } + let collectionRequest = buildIndependentSeriesCollectionRequest( + prepared, combinedMSR + ); + let request = await postToMusrfitCGI({ action: "series_collect", - fn: spec.msrPaths[0], - collection: JSON.stringify({ - runs: spec.runs, - suffix: spec.fileNameSuffix, - fftv: spec.fftv == "y" ? "y" : "n" - }) + fn: prepared.msrPaths[0], + collection: JSON.stringify(collectionRequest) }); if (!request.ok || !request.payload) { result.reason = "web-series-artifact-request-failed"; @@ -802,8 +810,8 @@ async function collectIndependentSeriesArtifactsWeb(spec) { result.parameterDataPath = response.parameter_url || ""; result.combinedMSRPath = response.msr_url || ""; result.logPath = response.out_url || ""; - result.warnings = Array.isArray(response.warnings) - ? response.warnings.filter(Boolean) : []; + result.warnings = clientWarnings.concat(Array.isArray(response.warnings) + ? response.warnings.filter(Boolean) : []); if (response.command_log && typeof appendJobLog === "function") { appendJobLog(response.command_log.trim()); } @@ -815,13 +823,13 @@ async function collectIndependentSeriesArtifactsWeb(spec) { if (typeof showWebRootPlotWindow === "function") { showWebRootPlotWindow(rootUrls, response.job_name || "", { msrPath: result.combinedMSRPath, - runs: spec.runs.join(" "), + runs: prepared.runs.join(" "), rootItems: Array.isArray(response.root_items) ? response.root_items : [] }); result.plotStarted = true; } } - return result; + return normalizeIndependentSeriesCollectionResult(result); } window.collectIndependentSeriesArtifactsWeb = collectIndependentSeriesArtifactsWeb; diff --git a/package.json b/package.json index efec987..2582193 100644 --- a/package.json +++ b/package.json @@ -10,7 +10,7 @@ "dist:linux": "npm run test:electron && npx --yes electron-builder@26.15.3 --linux rpm AppImage --x64", "dist:rpm": "npm run test:electron && npx --yes electron-builder@26.15.3 --linux rpm --x64", "dist:appimage": "npm run test:electron && npx --yes electron-builder@26.15.3 --linux AppImage --x64", - "test:electron": "node tests/log-highlighting.test.js && node tests/action-state.test.js && node tests/electron-missing-data.test.js && node tests/window-layering.test.js && node tests/msr-generation.test.js && node tests/web-runner.test.js && node tests/root-plot-clean.test.js && node tests/parameter-plot.test.js && node tests/help-window.test.js && node tests/release-packaging.test.js && node tests/addrun.test.js && node tests/fit-statistic.test.js && node tests/series-fit-planner.test.js && node tests/series-fit-result.test.js && node tests/series-fit-inventory.test.js && node tests/series-fit-msr-preparation.test.js && node tests/series-global-merge.test.js && node tests/series-fit-staging.test.js && node tests/series-fit-runner.test.js && node tests/series-runner-adapters.test.js && node tests/series-fit-routing.test.js && node tests/singlehist-parity.test.js && node tests/url-parameters.test.js && node tests/shared-configuration.test.js" + "test:electron": "node tests/log-highlighting.test.js && node tests/action-state.test.js && node tests/electron-missing-data.test.js && node tests/window-layering.test.js && node tests/msr-generation.test.js && node tests/web-runner.test.js && node tests/root-plot-clean.test.js && node tests/parameter-plot.test.js && node tests/help-window.test.js && node tests/release-packaging.test.js && node tests/addrun.test.js && node tests/fit-statistic.test.js && node tests/series-fit-planner.test.js && node tests/series-fit-result.test.js && node tests/series-fit-inventory.test.js && node tests/series-fit-msr-preparation.test.js && node tests/series-artifact-contract.test.js && node tests/series-global-merge.test.js && node tests/series-fit-staging.test.js && node tests/series-fit-runner.test.js && node tests/series-runner-adapters.test.js && node tests/series-fit-routing.test.js && node tests/singlehist-parity.test.js && node tests/url-parameters.test.js && node tests/shared-configuration.test.js" }, "repository": { "type": "git", diff --git a/tests/cgi-series-collection.test.js b/tests/cgi-series-collection.test.js index 792e36f..f7ee64c 100644 --- a/tests/cgi-series-collection.test.js +++ b/tests/cgi-series-collection.test.js @@ -6,11 +6,41 @@ const childProcess = require("child_process"); const fs = require("fs"); const os = require("os"); const path = require("path"); +const vm = require("vm"); const repoRoot = path.resolve(__dirname, ".."); const testRoot = fs.mkdtempSync(path.join(os.tmpdir(), "musrflow-cgi-series-collect-")); const perlLibraryPath = [path.join(repoRoot, "tests", "perl"), process.env.PERL5LIB] .filter(Boolean).join(path.delimiter); +const shared = { console: console, window: { MSREditor: {} } }; +shared.global = shared; +vm.runInNewContext(fs.readFileSync(path.join(repoRoot, "muSRFitlib.js"), "utf8"), shared); + +function mergedDisplayMSR(texts) { + const merged = shared.buildSeriesGlobalMSR(texts, ["514", "515"]); + assert.strictEqual(merged.ok, true); + return shared.normalizeMSRPlotRuns(merged.text, {}, ["514", "515"]); +} + +function seriesCollectionBody(combinedMSR) { + return new URLSearchParams({ + action: "series_collect", + fn: "/tmp/seriesjob/514_asy.msr", + collection: JSON.stringify({ + ok: true, + version: 1, + runs: ["514", "515"], + suffix: "_asy", + fftv: "n", + artifacts: { + combined: "514+series_asy.msr", + parameters: "514+series_asy_par.dat", + log: "514+series_asy.out" + }, + combinedMSR: combinedMSR + }) + }).toString(); +} function fittedMSR(run) { return [ @@ -73,8 +103,10 @@ try { const tmpRoot = path.join(webRoot, "tmp"); const workPath = path.join(tmpRoot, "seriesjob"); fs.mkdirSync(workPath, { recursive: true }); - fs.writeFileSync(path.join(workPath, "514_asy.msr"), fittedMSR(514)); - fs.writeFileSync(path.join(workPath, "515_asy.msr"), fittedMSR(515)); + const first514 = fittedMSR(514); + const first515 = fittedMSR(515); + fs.writeFileSync(path.join(workPath, "514_asy.msr"), first514); + fs.writeFileSync(path.join(workPath, "515_asy.msr"), first515); fs.writeFileSync(path.join(workPath, "514+global_asy.msr"), "genuine global fit\n"); fs.writeFileSync(path.join(workPath, "514+global_tmpl.msr"), "genuine global template\n"); @@ -85,7 +117,7 @@ try { fs.writeFileSync(fakeMsr2data, [ "#!/bin/sh", "printf '%s\\n' \"$*\" >> \"$MUSRFLOW_TEST_M2D_ARGS\"", - "printf 'RUN Asy_ Asy_PosErr\\n514 0.2 0.01\\n515 0.21 0.01\\n' > 514+series_asy_par.dat", + "printf 'dataT Alpha_0514 Alpha_0514PosErr Alpha_0514NegErr my_parameter_0514 CHISQ NDF CHISQred RUN\\n5 0.2 0.01 0.01 1.5 100 95 1.0526 514\\n6 0.21 0.01 0.01 1.6 102 95 1.0737 515\\n' > 514+series_asy_par.dat", "exit 9", "" ].join("\n")); @@ -99,11 +131,7 @@ try { ].join("\n")); fs.chmodSync(fakeMusrview, 0o755); - const body = new URLSearchParams({ - action: "series_collect", - fn: "/tmp/seriesjob/514_asy.msr", - collection: JSON.stringify({ runs: ["514", "515"], suffix: "_asy", fftv: "n" }) - }).toString(); + const body = seriesCollectionBody(mergedDisplayMSR([first514, first515])); const execution = childProcess.spawnSync( "perl", [path.join(repoRoot, "cgi", "single_musrfit.cgi")], { cwd: repoRoot, @@ -137,13 +165,14 @@ try { assert.match(response.warnings.join("\n"), /returned code 9/); const m2dArgs = fs.readFileSync(m2dArgsPath, "utf8").trim().split(/\r?\n/); assert.strictEqual(m2dArgs.length, 1); - assert.match(m2dArgs[0], - /^\[514 515\] _musrflow_series_\d+_\d+ new global data -o 514\+series_asy_par\.dat$/); + assert.strictEqual(m2dArgs[0], + "[514 515] _asy new data -o 514+series_asy_par.dat"); assert.strictEqual(fs.readFileSync(viewArgsPath, "utf8").trim(), "--timeout 300 --root 514+series_asy.msr"); assert.strictEqual(fs.readFileSync( path.join(workPath, "514+series_asy_par.dat"), "utf8" - ).split(/\r?\n/)[0], "RUN Asy Asy_PosErr"); + ).split(/\r?\n/)[0], + "dataT Alpha AlphaPosErr AlphaNegErr my_parameter CHISQ NDF CHISQred RUN"); const combinedText = fs.readFileSync(path.join(workPath, "514+series_asy.msr"), "utf8"); assert.match(combinedText, /^runs\s+1\s+2$/m); assert.match(combinedText, /^# Common parameters for all runs$/m); @@ -174,20 +203,22 @@ try { assert.match(response.command_log, /\[exit code 9\]/); // Repeat the same server path with two detector RUN blocks per physical - // run. The web/Perl merger must preserve all four blocks just like the - // shared JavaScript implementation. - fs.writeFileSync(path.join(workPath, "514_asy.msr"), fittedSingleHistMSR(514)); - fs.writeFileSync(path.join(workPath, "515_asy.msr"), fittedSingleHistMSR(515)); + // run. The CGI publishes the four-block shared-JavaScript result unchanged. + const single514 = fittedSingleHistMSR(514); + const single515 = fittedSingleHistMSR(515); + fs.writeFileSync(path.join(workPath, "514_asy.msr"), single514); + fs.writeFileSync(path.join(workPath, "515_asy.msr"), single515); + const singleHistBody = seriesCollectionBody(mergedDisplayMSR([single514, single515])); const singleHistExecution = childProcess.spawnSync( "perl", [path.join(repoRoot, "cgi", "single_musrfit.cgi")], { cwd: repoRoot, encoding: "utf8", - input: body, + input: singleHistBody, env: Object.assign({}, process.env, { REQUEST_METHOD: "POST", PERL5LIB: perlLibraryPath, CONTENT_TYPE: "application/x-www-form-urlencoded", - CONTENT_LENGTH: String(Buffer.byteLength(body)), + CONTENT_LENGTH: String(Buffer.byteLength(singleHistBody)), MUSRFIT_WEBROOT: webRoot, MUSRFIT_TMPROOT: tmpRoot, MUSRFIT_TMP_URL: "/tmp", @@ -211,6 +242,44 @@ try { assert.strictEqual((singleHistCombined.match(/^forward 21 25$/gm) || []).length, 2); assert.strictEqual((singleHistCombined.match(/^forward 23 27$/gm) || []).length, 2); assert.match(singleHistCombined, /^runs\s+1\s+2\s+3\s+4$/m); + + // Parameter extraction is independent of display merging. A structural + // mismatch prevents the combined MSR, but the per-run msr2data table is + // still returned and can be plotted. + fs.writeFileSync(path.join(workPath, "514_asy.msr"), fittedMSR(514)); + fs.writeFileSync(path.join(workPath, "515_asy.msr"), + fittedMSR(515).replace("asymmetry 1", "asymmetry 1\nsimplExpo 1")); + const dataOnlyBody = seriesCollectionBody(""); + const dataOnlyExecution = childProcess.spawnSync( + "perl", [path.join(repoRoot, "cgi", "single_musrfit.cgi")], { + cwd: repoRoot, + encoding: "utf8", + input: dataOnlyBody, + env: Object.assign({}, process.env, { + REQUEST_METHOD: "POST", + PERL5LIB: perlLibraryPath, + CONTENT_TYPE: "application/x-www-form-urlencoded", + CONTENT_LENGTH: String(Buffer.byteLength(dataOnlyBody)), + MUSRFIT_WEBROOT: webRoot, + MUSRFIT_TMPROOT: tmpRoot, + MUSRFIT_TMP_URL: "/tmp", + MSR2DATA_BIN: fakeMsr2data, + MUSRVIEW_BIN: fakeMusrview, + MUSRFIT_RUNTIME_HOME: tmpRoot, + MUSRFLOW_TEST_M2D_ARGS: m2dArgsPath, + MUSRFLOW_TEST_VIEW_ARGS: viewArgsPath + }) + } + ); + assert.strictEqual(dataOnlyExecution.status, 0, dataOnlyExecution.stderr); + const dataOnlyResponse = JSON.parse( + dataOnlyExecution.stdout.split(/\r?\n\r?\n/).slice(1).join("\n\n") + ); + assert.strictEqual(dataOnlyResponse.ok, true); + assert.strictEqual(dataOnlyResponse.parameter_url, + "/tmp/seriesjob/514+series_asy_par.dat"); + assert.strictEqual(dataOnlyResponse.msr_url, ""); + assert.deepStrictEqual(dataOnlyResponse.root_urls, []); } finally { fs.rmSync(testRoot, { recursive: true, force: true }); } diff --git a/tests/series-artifact-contract.test.js b/tests/series-artifact-contract.test.js new file mode 100644 index 0000000..a99e18c --- /dev/null +++ b/tests/series-artifact-contract.test.js @@ -0,0 +1,54 @@ +// Copyright (C) 2026 Zaher Salman +// SPDX-License-Identifier: GPL-3.0-or-later + +const assert = require("assert"); +const fs = require("fs"); +const path = require("path"); +const vm = require("vm"); + +const repoRoot = path.resolve(__dirname, ".."); +const sandbox = { console: console, window: { MSREditor: {} } }; +sandbox.global = sandbox; +vm.runInNewContext(fs.readFileSync(path.join(repoRoot, "muSRFitlib.js"), "utf8"), sandbox); + +const spec = { + ok: true, + workPath: "/tmp/job", + fileNameSuffix: "_asy", + runs: ["514", "515"], + msrPaths: ["/tmp/job/514_asy.msr", "/tmp/job/515_asy.msr"], + fftv: "y" +}; +const prepared = sandbox.prepareIndependentSeriesCollection(spec); +assert.strictEqual(prepared.ok, true); +assert.strictEqual(prepared.baseRun, "514"); +assert.strictEqual(prepared.combinedName, "514+series_asy.msr"); +assert.strictEqual(prepared.parameterName, "514+series_asy_par.dat"); +assert.strictEqual(prepared.logName, "514+series_asy.out"); + +const request = sandbox.buildIndependentSeriesCollectionRequest(spec, "FITPARAMETER\n..."); +assert.strictEqual(request.ok, true); +assert.strictEqual(request.version, 1); +assert.deepStrictEqual(Array.from(request.runs), ["514", "515"]); +assert.strictEqual(request.artifacts.combined, "514+series_asy.msr"); +assert.strictEqual(request.combinedMSR, "FITPARAMETER\n..."); + +assert.strictEqual(sandbox.prepareIndependentSeriesCollection(Object.assign({}, spec, { + runs: ["514", "514"] +})).reason, "series-artifact-source-invalid"); + +const dataOnly = sandbox.normalizeIndependentSeriesCollectionResult({ + ok: false, + reason: "series-display-merge-failed", + parameterDataPath: "/tmp/job/514+series_asy_par.dat", + warnings: ["display unavailable", "", null] +}); +assert.strictEqual(dataOnly.ok, true); +assert.strictEqual(dataOnly.reason, "series-display-merge-failed"); +assert.deepStrictEqual(Array.from(dataOnly.warnings), ["display unavailable"]); + +const empty = sandbox.normalizeIndependentSeriesCollectionResult({ ok: true }); +assert.strictEqual(empty.ok, false); +assert.strictEqual(empty.reason, "series-artifacts-not-produced"); + +console.log("series artifact contract tests passed"); diff --git a/tests/series-global-merge.test.js b/tests/series-global-merge.test.js index adfeed7..c8e7d4b 100644 --- a/tests/series-global-merge.test.js +++ b/tests/series-global-merge.test.js @@ -293,7 +293,7 @@ function testMergingErrors(lib) { assert.strictEqual(results.source.ok, false); assert.strictEqual(results.source.reason, "series-merge-source-incomplete"); - // A function that multiplies run-specific Asy cannot be shared across runs. + // Run-specific function parameters are represented through RUN map slots. const functionSource = function(run) { const suffix = String(run).padStart(4, "0"); return [ @@ -320,8 +320,12 @@ function testMergingErrors(lib) { results.functionRunSpecific = lib.buildSeriesGlobalMSR( [functionSource(518), functionSource(519)], [518, 519] ); - assert.strictEqual(results.functionRunSpecific.ok, false); - assert.strictEqual(results.functionRunSpecific.reason, "series-merge-functions-run-specific"); + assert.strictEqual(results.functionRunSpecific.ok, true); + sanctuaryCase("run-specific function parameters use map slots", function() { + assert.match(results.functionRunSpecific.text, /^fun1 = map1 \* map2$/m); + assert.match(results.functionRunSpecific.text, /^map\s+1\s+2$/m); + assert.match(results.functionRunSpecific.text, /^map\s+3\s+4$/m); + }); // But a function over a shared parameter stays valid and is renumbered. const sharedFunctionSource = function(run) { @@ -358,6 +362,73 @@ function testMergingErrors(lib) { }); } +function testMaximumLikelihoodStatistic(lib) { + const source = function(run, value) { + return stagedAsymmetryMSR(run, "0.2") + .replace("chisq = 101.2, NDF = 100, chisq/NDF = 1.012000", + "maxLH = " + value + ", NDF = 100, maxLH/NDF = " + + (Number(value) / 100).toFixed(6)); + }; + const result = lib.buildSeriesGlobalMSR( + [source(514, "120.5"), source(515, "121.5")], [514, 515] + ); + assert.strictEqual(result.ok, true); + sanctuaryCase("maximum-likelihood statistics are aggregated", function() { + assert.match(result.text, /^ maxLH = 242, NDF = 200, maxLH\/NDF = 1\.21$/m); + assert.doesNotMatch(result.text, /^ chisq =/m); + }); + + const partial = lib.buildSeriesGlobalMSR( + [source(514, "120.5"), stagedAsymmetryMSR(515, "0.2")], [514, 515] + ); + assert.strictEqual(partial.ok, true); + sanctuaryCase("mixed objectives do not produce a misleading aggregate", function() { + assert.doesNotMatch(partial.text, /^STATISTIC\b/m); + }); +} + +function testFunctionAndDetectorMapsCoexist(lib) { + const source = function(run) { + const suffix = String(run).padStart(4, "0"); + return [ + "FITPARAMETER", + "1 Phase_" + suffix + " 10 0.1 none", + "2 RelL_" + suffix + " 0 0 none", + "3 Field_" + suffix + " 75 0.1 none", + "4 Scale_" + suffix + " 0.0135 0 none", + "5 RelR_" + suffix + " 180 0 none", + "THEORY", + "TFieldCos fun1 fun2", + "FUNCTIONS", + "fun1 = par1 + map1", + "fun2 = par3 * par4", + "GLOBAL", + "fittype 0", + "RUN " + lemToken(run) + " MUE4 PSI MUSR-ROOT", + "map 2 0 0 0 0 0 0 0 0 0", + "forward 21", + "RUN " + lemToken(run) + " MUE4 PSI MUSR-ROOT", + "map 5 0 0 0 0 0 0 0 0 0", + "forward 23", + "COMMANDS", + "MIGRAD", + "PLOT 0", + "runs 1 2", + "" + ].join("\n"); + }; + const result = lib.buildSeriesGlobalMSR([source(514), source(515)], [514, 515]); + assert.strictEqual(result.ok, true); + sanctuaryCase("function maps fill zero slots beside detector maps", function() { + assert.match(result.text, /^fun1 = map2 \+ map1$/m); + assert.match(result.text, /^fun2 = map3 \* map4$/m); + assert.match(result.text, /^map\s+2\s+1\s+3\s+4$/m); + assert.match(result.text, /^map\s+5\s+1\s+3\s+4$/m); + assert.match(result.text, /^map\s+7\s+6\s+8\s+9$/m); + assert.match(result.text, /^map\s+10\s+6\s+8\s+9$/m); + }); +} + (async function() { const lib = loadLibrary(); testMergesTwoAsymmetryRuns(lib); @@ -365,6 +436,8 @@ function testMergingErrors(lib) { testPreservesEverySingleHistRunBlock(lib); testFourierForDisplay(lib); testMergingErrors(lib); + testMaximumLikelihoodStatistic(lib); + testFunctionAndDetectorMapsCoexist(lib); console.log("series global merge tests passed"); })().catch(function(error) { console.error(error); diff --git a/tests/series-runner-adapters.test.js b/tests/series-runner-adapters.test.js index 9f7a837..a2e27ad 100644 --- a/tests/series-runner-adapters.test.js +++ b/tests/series-runner-adapters.test.js @@ -214,8 +214,13 @@ async function testElectronSeriesArtifactCollection() { const fakeChildProcess = { spawnSync: function(command, args, options) { syncCalls.push({ command: command, args: args.slice(), cwd: options.cwd }); - fs.writeFileSync(path.join(options.cwd, "514+series_asy_par.dat"), - "RUN Asy_ Asy_PosErr\n514 0.2 0.01\n515 0.21 0.01\n", "utf8"); + fs.writeFileSync(path.join(options.cwd, "514+series_asy_par.dat"), [ + "dataT dataTErr Alpha_0514 Alpha_0514PosErr Alpha_0514NegErr " + + "my_parameter_0514 my_parameter_0514PosErr CHISQ NDF CHISQred RUN", + "5 0.1 0.2 0.01 0.01 1.5 0.1 100 95 1.0526 514", + "6 0.1 0.21 0.01 0.01 1.6 0.1 102 95 1.0737 515", + "" + ].join("\n"), "utf8"); return { status: 7, stdout: "parameter data written\n", stderr: "" }; }, spawn: function(command, args, options) { @@ -248,13 +253,14 @@ async function testElectronSeriesArtifactCollection() { assert.strictEqual(result.plotStarted, true); assert.match(result.warnings.join("\n"), /returned code 7/); assert.strictEqual(syncCalls.length, 1); - assert.match(syncCalls[0].args[1], /^_musrflow_series_\d+_[a-z0-9]+$/); - assert.deepStrictEqual(Array.from(syncCalls[0].args.slice(0, 1)), ["[514 515]"]); - assert.deepStrictEqual(Array.from(syncCalls[0].args.slice(2)), - ["new", "global", "data", "-o", "514+series_asy_par.dat"]); + assert.deepStrictEqual(Array.from(syncCalls[0].args), + ["[514 515]", "_asy", "new", "data", "-o", "514+series_asy_par.dat"]); assert.deepStrictEqual(Array.from(asyncCalls[0].args), ["514+series_asy.msr"]); - assert.match(fs.readFileSync(result.parameterDataPath, "utf8"), - /^RUN Asy Asy_PosErr/m); + const parameterHeader = fs.readFileSync(result.parameterDataPath, "utf8") + .split(/\r?\n/)[0]; + assert.strictEqual(parameterHeader, + "dataT dataTErr Alpha AlphaPosErr AlphaNegErr " + + "my_parameter my_parameterPosErr CHISQ NDF CHISQred RUN"); const combinedText = fs.readFileSync(result.combinedMSRPath, "utf8"); assert.match(combinedText, /^runs\s+1\s+2$/m); assert.match(combinedText, /^# Common parameters for all runs$/m); @@ -274,9 +280,60 @@ async function testElectronSeriesArtifactCollection() { } } +async function testElectronParameterCollectionSurvivesDisplayMergeFailure() { + const tempDirectory = fs.mkdtempSync(path.join(os.tmpdir(), "musrflow-series-data-only-")); + const syncCalls = []; + const asyncCalls = []; + try { + fs.writeFileSync(path.join(tempDirectory, "514_asy.msr"), + runnableMSR(514, true), "utf8"); + fs.writeFileSync(path.join(tempDirectory, "515_asy.msr"), + runnableMSR(515, true).replace("asymmetry 1", "asymmetry 1\nsimplExpo 1"), + "utf8"); + const sandbox = loadElectronSandbox({ + spawnSync: function(command, args, options) { + syncCalls.push(args.slice()); + fs.writeFileSync(path.join(options.cwd, "514+series_asy_par.dat"), + "RUN Asy_\n514 0.2\n515 0.21\n", "utf8"); + return { status: 0, stdout: "", stderr: "" }; + }, + spawn: function(command, args) { + asyncCalls.push(args.slice()); + const child = new events.EventEmitter(); + child.stdout = new events.EventEmitter(); + child.stderr = new events.EventEmitter(); + return child; + } + }, []); + const result = await sandbox.window.collectIndependentSeriesArtifactsLocal({ + ok: true, + workPath: tempDirectory, + fileNameSuffix: "_asy", + runs: ["514", "515"], + msrPaths: [ + path.join(tempDirectory, "514_asy.msr"), + path.join(tempDirectory, "515_asy.msr") + ], + fftv: "n" + }); + + assert.strictEqual(result.ok, true); + assert.strictEqual(result.parameterDataPath, + path.join(tempDirectory, "514+series_asy_par.dat")); + assert.strictEqual(result.combinedMSRPath, ""); + assert.strictEqual(result.plotStarted, false); + assert.strictEqual(syncCalls.length, 1); + assert.strictEqual(asyncCalls.length, 0); + assert.match(result.warnings.join("\n"), /series-merge-theory-mismatch/); + } finally { + fs.rmSync(tempDirectory, { recursive: true, force: true }); + } +} + Promise.resolve().then(testElectronAdapterFitsOnlyOneMSR) .then(testElectronAdapterRetainsRejectedNewMSR) - .then(testElectronSeriesArtifactCollection).then(function() { + .then(testElectronSeriesArtifactCollection) + .then(testElectronParameterCollectionSurvivesDisplayMergeFailure).then(function() { console.log("series runner adapter tests passed"); }).catch(function(error) { console.error(error); diff --git a/tests/web-runner.test.js b/tests/web-runner.test.js index 25b0a83..cbe7a67 100644 --- a/tests/web-runner.test.js +++ b/tests/web-runner.test.js @@ -272,6 +272,10 @@ async function testWebSeriesCollectionRegistersArtifactsAndPlot() { const logs = []; const lib = loadWebLibrary(async function(url, options) { calls.push({ url: url, options: options }); + if (!options || options.method !== "POST") { + const run = /515_asy\.msr$/.test(url) ? 515 : 514; + return createResponse(convergedWebMSR(run)); + } return createResponse(JSON.stringify({ ok: true, code: 0, @@ -307,13 +311,24 @@ async function testWebSeriesCollectionRegistersArtifactsAndPlot() { assert.strictEqual(result.plotPath, "/tmp/job/514+series_asy_0.root"); assert.strictEqual(result.plotStarted, true); assert.match(result.warnings[0], /returned code 9/); - assert.strictEqual(calls.length, 1); - const body = new URLSearchParams(calls[0].options.body); + assert.strictEqual(calls.length, 3); + assert.deepStrictEqual(calls.slice(0, 2).map(function(call) { return call.url; }), + ["/tmp/job/514_asy.msr", "/tmp/job/515_asy.msr"]); + const body = new URLSearchParams(calls[2].options.body); assert.strictEqual(body.get("action"), "series_collect"); assert.strictEqual(body.get("fn"), "/tmp/job/514_asy.msr"); - assert.deepStrictEqual(JSON.parse(body.get("collection")), { - runs: ["514", "515"], suffix: "_asy", fftv: "n" + const collection = JSON.parse(body.get("collection")); + assert.strictEqual(collection.version, 1); + assert.deepStrictEqual(collection.runs, ["514", "515"]); + assert.strictEqual(collection.suffix, "_asy"); + assert.strictEqual(collection.fftv, "n"); + assert.deepStrictEqual(collection.artifacts, { + combined: "514+series_asy.msr", + parameters: "514+series_asy_par.dat", + log: "514+series_asy.out" }); + assert.match(collection.combinedMSR, /^# Independent series display MSR/m); + assert.match(collection.combinedMSR, /^runs\s+1\s+2$/m); assert.strictEqual(plots.length, 1); assert.deepStrictEqual(Array.from(plots[0].urls), ["/tmp/job/514+series_asy_0.root"]); @@ -321,6 +336,44 @@ async function testWebSeriesCollectionRegistersArtifactsAndPlot() { assert.match(logs[0], /msr2data/); } +async function testWebParameterCollectionSurvivesClientMergeFailure() { + const calls = []; + const lib = loadWebLibrary(async function(url, options) { + calls.push({ url: url, options: options }); + if (!options || options.method !== "POST") { + const run = /515_asy\.msr$/.test(url) ? 515 : 514; + let text = convergedWebMSR(run); + if (run === 515) text = text.replace("asymmetry 1", "asymmetry 1\nsimplExpo 1"); + return createResponse(text); + } + return createResponse(JSON.stringify({ + ok: true, + reason: "series-artifacts-collected", + parameter_url: "/tmp/job/514+series_asy_par.dat", + msr_url: "", + out_url: "/tmp/job/514+series_asy.out", + root_urls: [], + warnings: [] + })); + }); + + const result = await lib.window.collectIndependentSeriesArtifactsWeb({ + ok: true, + workPath: "/tmp/job", + fileNameSuffix: "_asy", + runs: ["514", "515"], + msrPaths: ["/tmp/job/514_asy.msr", "/tmp/job/515_asy.msr"], + fftv: "n" + }); + assert.strictEqual(result.ok, true); + assert.strictEqual(result.parameterDataPath, "/tmp/job/514+series_asy_par.dat"); + assert.strictEqual(result.combinedMSRPath, ""); + assert.strictEqual(result.plotStarted, false); + assert.match(result.warnings.join("\n"), /series-merge-theory-mismatch/); + const body = new URLSearchParams(calls[2].options.body); + assert.strictEqual(JSON.parse(body.get("collection")).combinedMSR, ""); +} + function convergedWebMSR(run) { const suffix = String(run).padStart(4, "0"); return [ @@ -615,6 +668,7 @@ async function runTests() { await testSeriesTransactionRejectsMixedJobDirectories(); await testWebSeriesCandidateUsesDedicatedEndpoint(); await testWebSeriesCollectionRegistersArtifactsAndPlot(); + await testWebParameterCollectionSurvivesClientMergeFailure(); await testWebSeriesInventoryFeedsSharedClassifier(); await testIncompleteWebSeriesInventoryFailsClosed(); await testWebRunLoadsFittedMSRAndArtifacts();