Update README.md
This commit is contained in:
ozerov_d
110
README.md
110
README.md
@ -18,7 +18,7 @@ Runs on files produced by [sf-daq](https://github.com/paulscherrerinstitute/sf_d
|
||||
* [After beamtime](#usage3)
|
||||
|
||||
## Description
|
||||
Automatic Processing tool checks for the new files/runs produced by sf-daq and runs automatically workload (currently - indexing (from crystfel)) and fills logbook (google spreadsheet) with information with some daq parameters from the sf-daq and processing.
|
||||
Automatic Processing tool checks for the new files/runs produced by sf-daq and runs automatically workload (currently - indexing (by crystfel)) and fills logbook (google spreadsheet) with information with some daq parameters from the sf-daq and processing.
|
||||
|
||||
## Installation<a name="installation"></a>
|
||||
|
||||
@ -41,7 +41,39 @@ Automatic Processing tool can also be installed from scratch with:
|
||||
In case of installation from source, so different location of the code and conda environment - change correspondingly line in scripts/env_setup.sh
|
||||
|
||||
## Configuration files<a name="config"></a>
|
||||
Here description of configuration files will be, together with format and hints of usage
|
||||
|
||||
### BEAM_ENERGY.txt
|
||||
This file should contain a beam energy values (in eV). There must be one line with the default values and (optionally) it's possible to define beam energy values, different from defaults for specific runs(scans). Example:
|
||||
> $ cat BEAM_ENERGY.txt
|
||||
>
|
||||
> DEFAULT 11330.0
|
||||
>
|
||||
> run9876 11001.2
|
||||
>
|
||||
> run9870 12015.1
|
||||
>
|
||||
(for the runs 9876 and 9870 - 11001.2 and 12015.1 photon beam energy will be used, while for any other - 11330.0 value)
|
||||
|
||||
### DETECTOR_DISTANCE.txt
|
||||
This file should contain a detector distance (from sample to detector) in meter. Format is similar to BEAM_ENERGY.txt file, so for example:
|
||||
> $ cat DETECTOR_DISTANCE.txt
|
||||
>
|
||||
> DEFAULT 0.09369
|
||||
>
|
||||
> run9988 0.09212
|
||||
>
|
||||
> run9977 0.09413
|
||||
>
|
||||
(for runs 9988 and 9977 - 9.212cm and 9.413cm will be used as detector distance, for all other runs a default value of 9.369cm will be used)
|
||||
|
||||
### env_setup.sh
|
||||
During preparation [step](#usage1) this file should be filled with the proper values for the beamline name(alvra or bernina or..), pgroup name (p12345), DETECTOR_NAME (JF17T16V01) used in experiment, THRESHOLD_INDEXING (can be changed, adapted, in run_index.sh file, see latter) and LOGBOOK (url to google spreadsheet which will be used for automatic filling)
|
||||
|
||||
### run_index.sh
|
||||
|
||||
this file contains indexing parameters used by crystfel.
|
||||
|
||||
**HINT** - in case several proteins are used during expertiment, it's possible to define different indexing parameters for each of them: in case run_index.<cell_name>.sh file is present - indexing parameters from that file will be used to process <cell_name> protein sample, if not present(default) - run_index.sh parameters are used
|
||||
|
||||
## Google Authentication (optional)<a name="google-api"></a>
|
||||
|
||||
@ -53,52 +85,60 @@ Here description of configuration files will be, together with format and hints
|
||||
|
||||
### Before beamtime<a name="usage1"></a>
|
||||
|
||||
* make directory in res/ space of corresponding pgroup and fill it with configuration files (name **ap** is used as directory name in examples below, but any name can be choosen):
|
||||
* make directory in res/ space of corresponding pgroup and fill it with configuration files (name **ap** is used as directory name in examples below, but any name can be choosen):
|
||||
|
||||
> $ mkdir p12345/res/ap
|
||||
>
|
||||
> $ cd p12345/res/ap
|
||||
>
|
||||
> $ /sf/jungfrau/applications/ap/scripts/prepare.sh
|
||||
> $ mkdir p12345/res/ap
|
||||
>
|
||||
> $ cd p12345/res/ap
|
||||
>
|
||||
> $ /sf/jungfrau/applications/ap/scripts/prepare.sh
|
||||
|
||||
* make corresponding changes in the configuration files (see section [Configuration files]):
|
||||
* make corresponding changes in the configuration files (see section [Configuration files]):
|
||||
|
||||
* BEAM_ENERGY.txt
|
||||
* BEAM_ENERGY.txt
|
||||
|
||||
* DETECTOR_DISTANCE.txt
|
||||
* DETECTOR_DISTANCE.txt
|
||||
|
||||
* env_setup.sh
|
||||
* env_setup.sh
|
||||
|
||||
* run_index.sh
|
||||
* run_index.sh
|
||||
|
||||
* create file <DETECTOR_NAME>.geom (DETECTOR_NAME is variable defined by you in env_setup.sh file) with the crystfel geometry file for corresponding detector (example : JF17T16V01.geom file for CrystallinaMX instrument)
|
||||
* create file <DETECTOR_NAME>.geom (DETECTOR_NAME is variable defined by you in env_setup.sh file) with the crystfel geometry file for corresponding detector (example : JF17T16V01.geom file for CrystallinaMX instrument)
|
||||
|
||||
* put in ap/CELL directory cell files of the protein which will be exposed during beamtime (format of the files should be readable by crystfel). Name of the cell files needs to be <cell_name>.cell.
|
||||
* put in ap/CELL directory cell files of the protein which will be exposed during beamtime (format of the files should be readable by crystfel). Name of the cell files needs to be <cell_name>.cell.
|
||||
|
||||
> $ ls res/ap/CELL
|
||||
>
|
||||
> lyso.cell hewl.cell
|
||||
> $ ls res/ap/CELL
|
||||
>
|
||||
> lyso.cell hewl.cell
|
||||
|
||||
* (optional, if filling is requested)
|
||||
**HINT** - in case there are several space group at which protein can be indexed, it's possible to run automatically indexing in the *alternative* space group. To do this - provide an alternative space group settings in the file <cell_name>.cell_alternative. Example:
|
||||
|
||||
* create (an empty) google spreadsheet
|
||||
> $ ls res/ap/CELL
|
||||
>
|
||||
> lyso.cell chim.cell chim.cell_alternative
|
||||
|
||||
* create (several distinct) credentials files (see section [google authentication](#google-api) how to create service accounts and keys if not done before) and store them with the names in the config directory (it's important to have file with name credentials.json and have few(3 is enough) with names credentials-1.json, credentials-2.json...):
|
||||
> $ ls res/ap/credentials*json
|
||||
>
|
||||
>credentials.json credentials-1.json credentials-2.json credentials-3.json
|
||||
runs with the <cell_name>=lyso will be indexed using lyso.cell file, while for the <cell_name>=chim - indexing will be done twice, using chim.cell and chim.cell_alternative files (and results of both indexing will be filled in logbook)
|
||||
|
||||
***RECOMMENDATION*** - use/generate new credentials files for each beamtime to not expose experiment information
|
||||
* (optional, if filling is requested)
|
||||
|
||||
* give write access to the google spreadsheet to the service-accounts (recommended) or give full editor access to all who know url of the logbook(quicker, but not recommended action). To find e-mails of the service accounts:
|
||||
> $ grep client_email credentials*json
|
||||
* create (an empty) google spreadsheet
|
||||
|
||||
* edit env_setup.sh file to fill URL_TO_GOOGLE_SPREADSHEET(https://...) to the LOGBOOK variable
|
||||
* create (several distinct) credentials files (see section [google authentication](#google-api) how to create service accounts and keys if not done before) and store them with the names in the config directory (it's important to have file with name credentials.json and have few(3 is enough) with names credentials-1.json, credentials-2.json...):
|
||||
> $ ls res/ap/credentials*json
|
||||
>
|
||||
>credentials.json credentials-1.json credentials-2.json credentials-3.json
|
||||
|
||||
* setup/prepare spreadsheet for automatic filling:
|
||||
> $ . ./env_setup.sh
|
||||
>
|
||||
> $ python /sf/jungrau/applications/ap/ap/update-spreadsheet.py --setup --log ${LOGBOOK}
|
||||
***RECOMMENDATION*** - use/generate new credentials files for each beamtime to not expose experiment information
|
||||
|
||||
* give write access to the google spreadsheet to the service-accounts (recommended) or give full editor access to all who know url of the logbook(quicker, but not recommended action). To find e-mails of the service accounts:
|
||||
> $ grep client_email credentials*json
|
||||
|
||||
* edit env_setup.sh file to fill URL_TO_GOOGLE_SPREADSHEET(https://...) to the LOGBOOK variable
|
||||
|
||||
* setup/prepare spreadsheet for automatic filling:
|
||||
> $ . ./env_setup.sh
|
||||
>
|
||||
> $ python /sf/jungrau/applications/ap/ap/update-spreadsheet.py --setup --log ${LOGBOOK}
|
||||
|
||||
|
||||
|
||||
@ -141,12 +181,14 @@ in case of need to re-run indexing (new config parameters, new geometry file etc
|
||||
|
||||
#### pausing indexing<a name="usage2_pause"></a>
|
||||
in case of unknown processing parameters (detector distance, geometry file(beam center), not yet known cell file...), it's possible to pause (not start indexing jobs) putting semaphore file NO_INDEXING in config directory
|
||||
|
||||
> $ touch res/ap/NO_INDEXING
|
||||
once this file is removed - all not indexed runs will be processed
|
||||
|
||||
once this file is removed - all not indexed runs will be processed by the tool
|
||||
|
||||
### After Beamtime<a name="usage3"></a>
|
||||
|
||||
* stop automatic processing executable (Ctrl-c ap.py process) once all runs are processed for this beamtime
|
||||
* stop automatic processing executable (Ctrl-c ap.py process) once all runs are processed for this beamtime (no active jobs and filling of the logbook is finished)
|
||||
|
||||
* remove credentials*json files and revoke api-keys in [Google Developer Console](https://console.developers.google.com/) (->"Service Accounts", for each account -> click on "Actions: ..." and choose "Manage Keys", then remove key)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user