/*! @page pag_project Setting up a new project
\section sec_project Setting up a new project

This topic guides you through the setup of a new project.
Be sure to check out the examples in the projects folder
and the code documentation as well.

The basic steps are:

1. Create a new package folder under `pmsco/projects`.
   To keep your code and PMSCO separate, you are suggested to start your own pmsco/projects tree
   in a convenient location separate from the PMSCO source code.
2. Add the parent directory of your pmsco/projects tree to the Python path.
3. In the new folder, create a Python module for the project (subsequently called _the project module_).
4. In the project module, define a cluster generator class which inherits from @ref pmsco.cluster.ClusterGenerator.
5. In the project module, define a project class which inherits from @ref pmsco.project.Project.
6. Create one or more run files.

The basic steps listed above are recommended and explained in the following.
In previous versions, other mechanisms of project invocation were available.
They are now obsolete.


\subsection sec_packages Namespace packages

[Python namespace packages](https://realpython.com/python-namespace-package/) provide an easy way
to inject project modules into the PMSCO namespace
while their source files are kept separate from the core PMSCO packages.
This way, PMSCO and the project modules can be under separate version control.

Namespace packages work by extending the Python module search path.
The module loader looks for packages in every entry of the search path
and does not stop at the first match as it would do for a regular package.

The recommended folder structure is:

~~~~~~
pmsco-projects/
+-- pyproject.toml
+-- pmsco/
    +-- projects/
        +-- project1/
            +-- project1.py
            +-- run1.json
            +-- ...
        +-- project2/
        +-- ...
~~~~~~

In place of `pmsco-projects`, `project1`, `project2`, `run1`, you should use distinct names.
The two levels `pmsco` and `projects` should be left as is.
If you now include `pmsco-projects` in the Pyton path,
all of your projects become available within the `pmsco` namespace, i.e.,
you can `import pmsco.projects.project1.project1` in Python.
Furthermore, you can call the module in a run-file without specifying a file path.
You may install multiple project packages if needed.

The recommended way to add `pmsco-projects` to the Python path is by an editable installation.
This will allow you to keep editing your project sources in place.

1. Place your project files in a directory tree similar to `pmsco-projects/pmsco/projects/project1/`.
   The `pmsco/projects` level is mandatory as a part of the path.
   Replace `pmsco-projects` and `project1` by your own choice.
2. Be sure not to create any `__init__.py` files in this directory tree.
3. Copy the `pyproject.toml` file from the PMSCO source into your `pmsco-projects` and adjust its contents.
   At least give the package a distinct name.
4. Select another build backend if necessary.
   The default [uv_build](https://docs.astral.sh/uv/concepts/build-backend/) is recommended for pure Python projects.
5. 'Install' the project locally.
   With uv, call `uv sync --active --inexact`
   while you are in the directory that contains the `pyproject.toml` file.
   In plain pip the corresponding command would be
   `pip install --editable .`.
6. Check that you can `import pmsco.projects.project1.project1` (or whatever your project is called) in a Python shell.

If you encounter problems importing the pmsco modules, check the Python path in a Python shell.
It must contain the `site-packages` directory of your Python environment.
Make sure it does not contain any pmsco or project source directory explicitly.
Also make sure that you don't have any `__init__.py` files in your project tree,
and do not use explicit paths to pmsco or your project anywhere in your source code or shell configuration files.
Be careful not to install packages multiple times in different locations.
In case of trouble, set up a fresh environment.


\subsection sec_project_module Project module

A skeleton of the project module file (with some common imports) may look like this:

~~~~~~{.py}
import logging
import math
import numpy as np
import periodictable as pt
from pathlib import Path

import pmsco.cluster
import pmsco.data
import pmsco.dispatch
import pmsco.elements.bindingenergy
import pmsco.project

logger = logging.getLogger(__name__)


class MyClusterGenerator(pmsco.cluster.ClusterGenerator):
    def create_cluster(self, model, index):
        clu = pmsco.cluster.Cluster()
        # ...
        return clu

    def count_emitters(self, model, index):
        # ...
        return 1


class MyProject(pmsco.project.Project):
    def __init__(self):
        super().__init__()
        # ...
        self.cluster_generator = MyClusterGenerator(self)

    def create_model_space():
        spa = pmsco.project.ModelSpace()
        # ...
        return spa

    def create_params(self, model, index):
        par = pmsco.project.CalculatorParams()
        # ...
        return par
~~~~~~

The main purpose of the `MyProject` class is to bundle the project-specific calculation parameters and code.
The purpose of the `MyClusterGenerator` class is to produce atomic clusters as a function of a number of model parameters.
For the project to be useful, some of the methods in the skeleton above need to be implemented.
The individual methods are discussed in the following.
Further descriptions can be found in the documentation of the code.

\subsection sec_project_cluster Cluster generator

The cluster generator is a project-specific Python object that produces a cluster, i.e., a list of atomic coordinates,
based on a small number of model parameters whenever PMSCO requires it.
The most important method of a cluster generator is `create_cluster`.
At least this method must be implemented for a functional cluster generator.

A generic `count_emitters` method is implemented in the base class.
It needs to be overridden if inequivalent emitters should be calculated in parallel.

\subsubsection sec_project_cluster_create Cluster definition

The `create_cluster` method takes the model parameters (a dictionary)
and the task index (a pmsco.dispatch.CalcID, cf. @ref pag_concepts_tasks) as arguments.
Given these arguments, it creates and fills a @ref pmsco.cluster.Cluster object.
See @ref pmsco.cluster.ClusterGenerator.create_cluster for details on the method contract.

As an example, have a look at the following simplified excerpt from the `twoatom` demo project.

~~~~~~{.py}
class TwoatomCluster(ClusterGenerator):
    # ...

    def create_cluster(self, model, index):
        # access model parameters
        # dAB - distance between atoms in Angstroms
        # th - polar angle in degrees
        # ph - azimuthal angle in degrees
        r = model['dAB']
        th = math.radians(model['th'])
        ph = math.radians(model['ph'])

        # prepare a cluster object
        clu = pmsco.cluster.Cluster()
        # the comment line is optional but can be useful
        clu.comment = "{0} {1}".format(self.__class__, index)
        # set the maximum radius of the cluster (outliers will be ignored)
        clu.set_rmax(r * 2.0)

        # calculate atomic vectors
        dx = r * math.sin(th) * math.cos(ph)
        dy = r * math.sin(th) * math.sin(ph)
        dz = r * math.cos(th)
        a_top = np.array((0.0, 0.0, 0.0))
        a_bot = np.array((-dx, -dy, -dz))

        # add an oxygen atom at a_top position and mark it as emitter
        clu.add_atom('O', a_top, 1)
        # add a copper atom at a_bot position
        clu.add_atom('Cu', a_bot, 0)

        # pass the created cluster to the calculator
        return clu
~~~~~~

In this example, two atoms are added to the cluster.
The @ref pmsco.cluster.Cluster class provides several methods to simplify the task,
such as adding layers or bulk regions, rotation, translation, trim, emitter selection, etc.
Please refer to the documentation of its code for details.
It may also be instructive to have a look at the demo projects.

The main purposes of the cluster object are to store an array of atoms and to read/write cluster files in a variety of formats.
For each atom, the following properties are stored:

- sequential atom index (1-based, maintained by cluster code)
- atom type (chemical element number)
- chemical element symbol from periodic table
- x coordinate of the atom position
- y coordinate of the atom position
- z coordinate of the atom position
- emitter flag (0 = scatterer, 1 = emitter, default 0)
- charge/ionicity (units of elementary charge, default 0)
- scatterer class (default 0)

All of these properties except the scatterer class can be set by the `add_xxxx` methods of the cluster.
The scatterer class is used internally by the atomic scattering factor calculators.
Whether the charge/ionicity is used, depends on the particular calculators, EDAC does not use it, for instance.

\note You do not need to take care how many emitters a calculator allows,
or whether the emitter needs to be at the origin or the first place of the array.
These technical aspects are handled by PMSCO code transparently.

\subsubsection sec_project_cluster_domains Domains

Domains refer to regions of inequivalent structure in the probing region.
This may include regions of different orientation, different lattice constant, or even different structure.
The cluster methods read the requested domain from the `index.domain` argument.
This is an index into the @ref pmsco.project.Project.domains list where each item is a dictionary
that holds additional, invariable structural parameters.

A common case are rotational domains.
In this case, the list of domains may look like `[{"zrot": 0.0}, {"zrot": 60.0}]`, for example,
and the `create_cluster` method would include additional code to rotate the cluster:

~~~~~~{.py}
    def create_cluster(self, model, index):
        # filling atoms here
        # ...

        dom = self.domains[index.domain]
        try:
            z_rot = dom['zrot']
        except KeyError:
            z_rot = 0.0
        if z_rot:
            clu.rotate_z(z_rot)

        # selecting emitters
        # ...

        return clu
~~~~~~

Depending on the complexity of the system, it is advisable to split the code into a separate method for each domain.

The @ref pmsco.project.Project class includes generic code to add intensities of domains incoherently
(cf. @ref pmsco.project.Project.combine_domains).
In this case, the model space should contain parameters 'wdom0', 'wdom1', etc.,
that define the weights of domain 0, 1, etc.

To avoid correlations between parameters, one domain must have a fixed weight:
Typically, 'wdom0' is left undefined and defaults to 1.

\subsubsection sec_project_cluster_emitters Emitter configurations

If a project uses a large cluster and/or many emitters,
it may be more efficient to generate emitter-specific cluster configurations,
for instance to leverage process parallelization,
or to produce small, local clusters around the emitter site.
This concept is called _emitter configurations_ and explained in detail in @ref pag_concepts_emitter.

To implement emitter configurations, override the `count_emitters` method to return the number of emitter configurations.
In the simplest case, this is the number of inequivalent emitters:

~~~~~~{.py}
    def count_emitters(self, model, index):
        index = index._replace(emit=-1)
        clu = self.create_cluster(model, index)
        return clu.get_emitter_count()
~~~~~~

Next, modify the `create_cluster` method to check the emitter index (`index.emit`).
If it is -1, the method must return the full cluster with all inequivalent emitters marked.
If it is positive, only the corresponding emitter configuration must be marked.
For example, if each emitting atom represents a separate emitter configuration:

~~~~~~{.py}
    def create_cluster(self, model, index):
        # filling atoms here
        # ...

        # select all possible emitters (atoms of a specific element) in a cylindrical volume
        # idx_emit is an array of atom numbers (0-based atom index)
        idx_emit = clu.find_index_cylinder(origin, r_xy, r_z, self.project.scans[index.scan].emitter)

        # if PMSCO asks for a specific emitter, restrict the array index:
        if index.emit >= 0:
            idx_emit = idx_emit[index.emit]

        # mark the selected emitters
        clu.data['e'][idx_emit] = 1

        return clu
~~~~~~

Now, the individual emitter configurations are calculated in separate tasks
which can run in parallel in a multi-process environment.
Note that the processing time of EDAC scales linearly with the number of emitters.


\subsection sec_project_project Project class

Most commonly, a project class overrides the `__init__`, `create_model_space` and `create_params` methods.
Most other inherited methods can be overridden optionally,
for instance `validate`, `setup`, `calc_modulation`, `rfactor`,
as well as the combine methods `combine_rfactors`, `combine_domains`, `combine_emitters`, etc.
This introduction shall focus on the three most important methods.


\subsubsection sec_project_project_init Initialization and defaults

The `__init__` method defines and initializes project properties with default values.
It may also redefine properties of the base class.
The following code is just an example to give some ideas.

~~~~~~{.py}
class MyProject(pmsco.project.Project):
    def __init__(self):
        # call the inherited method first
        super().__init__()
        # re-define an inherited property
        self.directories["data"] = Path("/home/pmsco/data")
        # define a scan dictionary
        self.scan_dict = {}
        # fill the scan dictionary
        self.build_scan_dict()
        # create the cluster generator
        self.cluster_generator = MyClusterGenerator(self)
        # declare the list of domains (at least one is required)
        self.domains = [{"zrot": 0.}]

    def build_scan_dict(self):
        self.scan_dict["empty"] = {"filename": "${pmsco}/projects/common/empty-hemiscan.etpi",
                                   "emitter": "Si", "initial_state": "2p3/2"}
        self.scan_dict["Si2p"]  = {"filename": "${data}/xpd-Si2p.etpis",
                                   "emitter": "Si", "initial_state": "2p3/2"}
~~~~~~

A scan dictionary is one way to specify locations and metadata of experimental files centrally in the project code.
The scan can then be selected by the dictionary key rather than copying file locations.

Note that all public attributes can be assigned from a run file.
This happens after the `__init__` method.
The values set by `__init__` serve as default values.

\subsubsection sec_project_project_space Model space

The model space defines the keys and value ranges of the model parameters.
There are three ways to declare the model space in order of priority:

1. Declare the model space in the run-file.
2. Assign a ModelSpace to the self.model_space property directly in the `__init__` method.
3. Implement the `create_model_space` method.

The third way may look like this:

~~~~~~{.py}
class MyProject(pmsco.project.Project):
    def create_model_space(self):
        # create an empty model space
        spa = pmsco.project.ModelSpace()

        # add parameters
        spa.add_param('dAB',    2.05, width=0.25, step=0.05)
        spa.add_param('th',    15.00, 0.00, 30.00, 1.00)
        spa.add_param('ph',    90.00)
        spa.add_param('V0',    21.96, width=10.0, step=1.0)
        spa.add_param('Zsurf',  1.50)
        spa.add_param('wdom1',  0.5,  0.10, 10.00, 0.10)

        # return the model space
        return spa
~~~~~~

This code declares six model parameters: `dAB`, `th`, `ph`, `V0`, `Zsurf` and `wdom1`.
Three of them are structural parameters (used by the cluster generator above),
two are used by the `create_params` method (see below),
and `wdom1` is used in @ref pmsco.project.Project.combine_domains
while summing up contributions from different domains.

The values in the arguments list correspond to the start value (initial guess),
the lower and upper boundaries of the value range,
and the step size for optimizers that require it.
If just one value is given the parameter is held constant during the optimization.
The range can, alternatively, be specified by the `width` argument.

A similar declaration in a run-file could look like as follows (some parameters omitted for brevity).
Parameter values can be numeric constants,
or simple Python math expressions in double quotes.

~~~~~~{.py}
{
  "project": {
    // ...
    "model_space": {
      "dAB": {
        "start": "2.0 / math.cos(math.radians(15.0))",
        "width": 0.25,
        "step": 0.05
      },
      "th": {
        "start": 15.0,
        "min": 0.0,
        "max": 30.0,
        "step": 1.0
      },
      "Zsurf": {
        "start": 1.50
      }
      // ...
    }
  }
}
~~~~~~

\subsubsection sec_project_project_params Calculation parameters

Non-structural parameters that are needed for the input files of the calculators are passed
in a @ref pmsco.project.CalculatorParams object.
This object is created and filled in the `create_params` method of the project class.

The following example is from the `twoatoms` demo project:

~~~~~~{.py}
class MyProject(pmsco.project.Project):
    def create_params(self, model, index):
        params = pmsco.project.CalculatorParams()

        # meta data
        params.title = "two-atom demo"
        params.comment = "{0} {1}".format(self.__class__, index)

        # initial state and binding energy
        initial_state = self.scans[index.scan].initial_state
        params.initial_state = initial_state
        emitter = self.scans[index.scan].emitter
        params.binding_energy = pt.elements.symbol(emitter).binding_energy[initial_state]

        # experimental setup
        params.polarization = "H"
        params.polar_incidence_angle = 60.0
        params.azimuthal_incidence_angle = 0.0
        params.experiment_temperature = 300.0

        # material parameters
        params.z_surface = model['Zsurf']
        params.work_function = 4.5
        params.inner_potential = model['V0']
        params.debye_temperature = 356.0

        # multiple-scattering parameters (EDAC)
        params.emitters = []
        params.lmax = 15
        params.dmax = 5.0
        params.orders = [25]

        return params
~~~~~~

Most of the code is generic and can be copied to other projects.
Only the experimental and material parameters need to be adjusted.
Other properties can be changed as needed, see @ref pmsco.project.CalculatorParams.


\subsection sec_project_args Passing run-time parameters

The recommended way of passing calculation parameters is via @ref pag_runfile.
Run-files allow for a complete separation of code and data in a generic and flexible way.
Program code can be managed by a version control system,
and run-files can be stored along with the results.
This simplifies the reproduction of previous calculations and documentation of the workflow.

For testing and simple projects, it is possible to hard-code all parameters in the project class.


\subsubsection sec_project_args_runfile Setting up a run-file

The usage and format of run-files is described in detail under @ref pag_runfile.


\subsubsection sec_project_args_code Hard-coded arguments

Though it's normally recommended to declare all parameters in the run-file,
parameter values can also be hard-coded in the constructor and/or the validate method of the project class.
Which method to use depends on the processing stage.

The constructor can set default values for rarely changing parameters.
The declarations in the run-file override the defaults from the constructor.
If some parameters need adjusting _after_ the run-file has been loaded,
this can be done in the validate` method.

The call sequence of the methods is as follows.

1. `Project.__init__`:
   The constructor is usually overridden by the project.
   The constructor must call the superclass before applying its values.
2. `Project.set_properties`:
   Sets the parameters from the run-file and resolves class names.
   This method can be overridden if additional classes need resolving after loading the run-file.
   It must call the superclass.
3. `Project.validate`: Parameters are validated, i.e., checked and made consistent.
   Handler classes are resolved.
   The `validate` method or its sub-methods can be overridden by the project.
   The inherited method should be called.

*/
