/*! @page pag_run Running PMSCO
\section sec_run Running PMSCO

To run PMSCO you need the PMSCO code and its dependencies (cf. @ref pag_install),
a customized code module that contains the project-specific code,
one or several files containing the scan parameters and experimental data,
and a run-file specifying the calculation parameters.

Please check the <code>projects</code> folder for examples of project modules.


\subsection sec_run_single Single Process

The following instructions assume that PMSCO was installed as a Python site-package according to @ref pag_install.

To run PMSCO from the command prompt:

@code{.sh}
cd work-dir
python -m pmsco -j job-name -r run-file
@endcode

where:

| `work-dir` | Destination directory for output files |
| `run-file` | JSON-formatted configuration file that defines run-time parameters. The format and content of a run file is described in a section @ref pag_runfile. |
| `job-name` | (optional) The job name appears mainly as the prefix of all output files but is also used in the database and other places. The job name can also be declared in the run file. |

In this form, PMSCO is run in one process which handles all calculations sequentially.

Example command line for a single EDAC calculation of the two-atom project:
@code{.sh}
cd work/twoatom
python -m pmsco -j job0001 -r twoatom-hemi.json
@endcode

This command line executes the main pmsco module <code>pmsco.py</code>.
The information which project to load is contained in the <code>twoatom-hemi.json</code> file,
along with all common and specific project arguments.

This example can be run for testing.
All necessary parameters and data files are included in the code repository.


\subsection sec_run_parallel Parallel Processes

PMSCO handles parallelization automatically and transparently.
To start PMSCO in a parallel environment in the login shell,
just prefix the command with <code>mpiexec -np N</code>,
where N is the number of processes.
One process will assume the role of the master, and the remaining will assume the role of slaves.
The slave processes will run the scattering calculations, while the master coordinates the tasks,
and optimizes the model parameters (depending on the operation mode).

For optimum performance, the number of processes should not exceed the number of available processors.
To start an optimization job with multiple processes on a quad-core workstation with hyperthreading:
@code{.sh}
cd work/my_project
mpiexec -np 8 --use-hwthread-cpus python -m pmsco -j my_job002 -r my_project.json
@endcode

The `--use-hwthread` option is necessary on certain hyperthreading architectures.


\subsection sec_run_hpc High-Performance Cluster

PMSCO is ready to run with resource managers on cluster machines.
Code for submitting jobs to Slurm queues is included and can be customized for many machines.
For example, code for the slurm queue of the Ra cluster at PSI is included in the pmsco.schedule module.
Other machines can be supported by sub-classing pmsco.schedule.JobSchedule or pmsco.schedule.SlurmSchedule.

To have PMSCO submit a job, the arguments for the queue are entered in the schedule section of the run file,
cf. @ref pag_runfile.
Then, the same command as for starting a calculation directly will instead submit a job to the queue:

@code{.sh}
python -m pmsco -j job-name -r run-file.json
@endcode

The command creates a separate work directory with copies of the project source, the run-file and the job script.
This job directory will also receive the calculation results.
The full path of the job directory is _output-dir/job-name_.
The directory must not exist when you run the above command to prevent overwriting of previous data.
The job name can be declared in the run file or on the command line.

The command above also loads the project module and scan files.
Many parameter errors are caught this way and can be fixed before the job is submitted to the queue.

The run file offers an option to prepare a script file and not to submit the job immediately
so that you can inspect the job files and submit the job manually.

Be sure to consider the resource allocation policy of the cluster
before you decide on the number of processes.
Requesting less resources will prolong the run time but might increase the scheduling priority.

\subsection sec_run_dirs Directories

Code and data files are typically located in different, possibly machine-specific locations.
This can make it difficult to port a project to another machine and to repeat calculations.
Ideally, a calculation job should be repeatable on different machines
with a minimum of changes to code, input data and parameter files.
Project code (which is under version control)
should never need modifications for porting to another machine.
Run-files (which are considered part of the data) can follow a project-specific or machine-specific directory structure.

PMSCO provides directory resolution at run-time to facilitate writing of portable code.
This is done by a number of directory aliases that can be included as shell-like placeholders, e.g. `${project}`, in file paths.
Some aliases are preset to system-based defaults,
further aliases can be added by the project code or declared in the run file.
Directory aliases can be used in Project.directories
as well as in other Project attributes that hold a file name.

The table below shows the aliases defined and/or required by PMSCO.
The paths are stored in Project.directories.
The aliases are resolved before the actual calculations start (in the Project.validate() method).
The resolved paths are printed to the log at warning level.

| Key | Description | Source | Use |
| --- | --- | --- | --- |
| work | Working directory at program start | PMSCO | |
| home | User's home directory | PMSCO | |
| project | Location of the project module. | PMSCO | Can be used to find auxiliary files that are part of the repository. |
| output | Intermediate and output files. | Must be set by the project or run file | The `output_file` property which serves as the basis of all output files is a concatenation of the `output` directory and `job_name`. |
| report | Directory for graphical output (reports) | Default: `${output}/report` | |
| data (optional) | Location of data (scan) files. | Project or run file | Usage is up to the project. |
| temp | Temporary files | | Reserved. Currently not supported |
| (job tag) | Any job_tags key that maps to a legal directory name can be included in a path | run file | project or run file |
| mode, job_name, project_name | These project attributes can be included in a path if they contain a valid directory name | | |

\subsection sec_run_stop Stopping a PMSCO job

A PMSCO optimization job stops on any one of the following events.

- The model handler is done.
  Depending on the run mode, this happens when the optimization has converged or
  the planned number of iterations or calculations has been reached.
- The number of calculation tasks exceeds the limit configured in `dispatch.MscoMaster.max_calculations`.
  This is meant to prevent excessive and runaway jobs.
  The default value is 1000000. It can be adjusted by the project code if necessary.
- The master process receives a SIGTERM, SIGUSR1 or SIGUSR2 from the operating system.
  The signal can be sent, e.g., by the `kill` command on Linux.
  This doesn´t work on all platforms.
- The time limit configured in `Project.timedelta_limit` is reached.
  This is a soft limit and should be set shorter than the job reservation with the resource manager.
- A file named `finish_pmsco` is present in the output directory.
  This is an easy way for a user to stop a running optimization.
  The file doesn´t need any content.
  It can be created by the `touch` command.

All these stop conditions cause graceful stops.
Running calculation tasks are waited for, but some results on the model level may not be complete.
Final reports of complete models are produced and the output folder is cleaned up.

Stops caused by resource managers such as Slurm are typically not graceful.
The results are in an undefined state, reports are not generated, and temporary files may be left over.
*/
