Add code for 1D detector #69

Merged
usov_i merged 118 commits from det1d into master 2020-10-22 12:07:50 +02:00
usov_i commented 2020-09-15 14:27:17 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

*Created by: ivan-usov*
usov_i commented 2020-09-15 14:30:29 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

@JakHolzer , I created a pull request will all changes grouped on the same branch. This way it will be easier to introduce changes, that might affect all files at the same time. You should be able to push to that branch (det1d) too.

*Created by: ivan-usov* @JakHolzer , I created a pull request will all changes grouped on the same branch. This way it will be easier to introduce changes, that might affect all files at the same time. You should be able to push to that branch (`det1d`) too.
usov_i commented 2020-09-15 16:21:59 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

@JakHolzer , if you want, you can try to simplify reading of ccl files in a similar way to fe1d8044a3

*Created by: ivan-usov* @JakHolzer , if you want, you can try to simplify reading of ccl files in a similar way to https://github.com/paulscherrerinstitute/pyzebra/pull/11/commits/fe1d8044a3f2f6e24927a4f8baf4aa0894a56140
usov_i commented 2020-09-15 16:27:39 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

I will try 👍

*Created by: JakHolzer* I will try 👍
usov_i commented 2020-09-15 16:46:31 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

Btw, that commit also renames "omega" into "om", and "counts" into "Counts", that is consistent with the column names in dat files. I hope it doesn't break you other code, otherwise let me know what names you prefer, it will be easy to fix, e.g. with

# row_names = next(infile).split()
row_names = ("NP", "omega", "counts", ...)
*Created by: ivan-usov* Btw, that commit also renames "omega" into "om", and "counts" into "Counts", that is consistent with the column names in dat files. I hope it doesn't break you other code, otherwise let me know what names you prefer, it will be easy to fix, e.g. with ``` # row_names = next(infile).split() row_names = ("NP", "omega", "counts", ...) ```
usov_i commented 2020-09-15 16:50:03 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

I think it should be fine. The ccl files does not have the headers and
there is only one data column, rest is numbers or is calculated, not read.
So I think its alright, important think is that you know what is what :)

út 15. 9. 2020 v 16:46 odesílatel Ivan Usov notifications@github.com
napsal:

@ivan-usov commented on this pull request.

In pyzebra/load_1D.py
https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r488727539
:

  •            variable = variable.strip()
    
  •            if variable in META_VARS_FLOAT:
    
  •                det_variables["meta"][variable] = float(value)
    
  •            elif variable in META_VARS_STR:
    
  •                det_variables["meta"][variable] = str(value)[:-1].strip()
    
  •            elif variable in META_UB_MATRIX:
    
  •                det_variables["meta"][variable] = re.findall(r"[-+]?\d*\.\d+|\d+", str(value))
    
  •        if "#data" in line:
    
  •            # this is the end of metadata and the start of data section
    
  •            break
    
  •    # read data
    
  •    if det_variables["file_type"] == "ccl":
    
  •        decimal = list()
    
  •        data = infile.readlines()
    

Btw, that commit also renames "omega" into "om", and "counts" into
"Counts", that is consistent with the column names in dat files. I hope it
doesn't break you other code, otherwise let me know what names you prefer,
it will be easy to fix, e.g. with

row_names = next(infile).split()

row_names = ["NP", "omega", "counts", ...]


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r488727539,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2BBRVPO2FLEFKK74RXULLSF545RANCNFSM4RNBW7BQ
.

*Created by: JakHolzer* I think it should be fine. The ccl files does not have the headers and there is only one data column, rest is numbers or is calculated, not read. So I think its alright, important think is that you know what is what :) út 15. 9. 2020 v 16:46 odesílatel Ivan Usov <notifications@github.com> napsal: > *@ivan-usov* commented on this pull request. > ------------------------------ > > In pyzebra/load_1D.py > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r488727539> > : > > > + variable = variable.strip() > + if variable in META_VARS_FLOAT: > + det_variables["meta"][variable] = float(value) > + elif variable in META_VARS_STR: > + det_variables["meta"][variable] = str(value)[:-1].strip() > + elif variable in META_UB_MATRIX: > + det_variables["meta"][variable] = re.findall(r"[-+]?\d*\.\d+|\d+", str(value)) > + > + if "#data" in line: > + # this is the end of metadata and the start of data section > + break > + > + # read data > + if det_variables["file_type"] == "ccl": > + decimal = list() > + data = infile.readlines() > > Btw, that commit also renames "omega" into "om", and "counts" into > "Counts", that is consistent with the column names in dat files. I hope it > doesn't break you other code, otherwise let me know what names you prefer, > it will be easy to fix, e.g. with > > # row_names = next(infile).split() > row_names = ["NP", "omega", "counts", ...] > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r488727539>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AM2BBRVPO2FLEFKK74RXULLSF545RANCNFSM4RNBW7BQ> > . >
usov_i commented 2020-09-16 11:27:34 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

These paddings look artificial to me. Do you think it was just tabs between numbers, and the different padding sizes are because of different lengths of those written numbers?

*Created by: ivan-usov* These paddings look artificial to me. Do you think it was just tabs between numbers, and the different padding sizes are because of different lengths of those written numbers?
usov_i commented 2020-09-16 12:56:31 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

Well, the structure of the files seems very uniform and aligned, so I
wanted to keep that. I have never worked with the software that uses it, so
I'm not sure how it reads the data. My suggestion is to leave it like this
and we can ask Oksana to try load one file with padding and second with
tabs to see if it works.

st 16. 9. 2020 v 11:27 odesílatel Ivan Usov notifications@github.com
napsal:

@ivan-usov commented on this pull request.

In pyzebra/comm_export.py
https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r489296559
:

  •        corr_value = np.abs(value * np.sin(ang1) * np.cos(ang2))
    
  •        return corr_value
    

+def export_comm(data, path, lorentz=False):

  • """exports data in the *.comm format
  • :param lorentz: perform Lorentz correction
  • :param path: path to file + name
  • :arg data - data to export, is dict after peak fitting
  • """
  • align = ">"
  • if data["meta"]["indices"] == "hkl":
  •    extension = ".comm"
    
  •    padding = [6, 4, 10, 8]
    

These paddings look artificial to me. Do you think it was just tabs
between numbers, and the different padding sizes are because of different
lengths of those written numbers?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/paulscherrerinstitute/pyzebra/pull/11#pullrequestreview-489441678,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2BBRU7NNDFJCK25MZNFQTSGCAJNANCNFSM4RNBW7BQ
.

*Created by: JakHolzer* Well, the structure of the files seems very uniform and aligned, so I wanted to keep that. I have never worked with the software that uses it, so I'm not sure how it reads the data. My suggestion is to leave it like this and we can ask Oksana to try load one file with padding and second with tabs to see if it works. st 16. 9. 2020 v 11:27 odesílatel Ivan Usov <notifications@github.com> napsal: > *@ivan-usov* commented on this pull request. > ------------------------------ > > In pyzebra/comm_export.py > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r489296559> > : > > > + corr_value = np.abs(value * np.sin(ang1) * np.cos(ang2)) > + return corr_value > + > + > +def export_comm(data, path, lorentz=False): > + """exports data in the *.comm format > + :param lorentz: perform Lorentz correction > + :param path: path to file + name > + :arg data - data to export, is dict after peak fitting > + > + """ > + > + align = ">" > + if data["meta"]["indices"] == "hkl": > + extension = ".comm" > + padding = [6, 4, 10, 8] > > These paddings look artificial to me. Do you think it was just tabs > between numbers, and the different padding sizes are because of different > lengths of those written numbers? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#pullrequestreview-489441678>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AM2BBRU7NNDFJCK25MZNFQTSGCAJNANCNFSM4RNBW7BQ> > . >
usov_i commented 2020-09-21 11:14:30 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

This parameter is (will be) used by export comm to decide if the fitted area or numerically integrated area is to be exported. User has to be able to choose, non the less numerically integrated area is default.

*Created by: JakHolzer* This parameter is (will be) used by export comm to decide if the fitted area or numerically integrated area is to be exported. User has to be able to choose, non the less numerically integrated area is default.
usov_i commented 2020-09-21 11:24:45 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

I'm sorry, I think that the answer I send from email got stuck somewhere. The files look very organized, so I think its really read line by line in this and first 6 chars correspond to number and so on. Oksana would like to keep it this way.

*Created by: JakHolzer* I'm sorry, I think that the answer I send from email got stuck somewhere. The files look very organized, so I think its really read line by line in this and first 6 chars correspond to number and so on. Oksana would like to keep it this way.
usov_i commented 2020-10-01 18:42:53 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

the detectorDistance is a number in ccl files, but in dat files its date, which breaks the code. I asked Oksana if it can be fixed in the software that produces the *.dat files. If it for whatever reason cant be fixed, I suggest to move it to META_VARS_STR.

*Created by: JakHolzer* the detectorDistance is a number in ccl files, but in dat files its date, which breaks the code. I asked Oksana if it can be fixed in the software that produces the *.dat files. If it for whatever reason cant be fixed, I suggest to move it to META_VARS_STR.
usov_i commented 2020-10-08 11:46:50 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

Does it need to be individual for each measurement? I would expect a user wanting to export all measurements in a similar way, thus having this parameter global.

*Created by: ivan-usov* Does it need to be individual for each measurement? I would expect a user wanting to export all measurements in a similar way, thus having this parameter global.
usov_i commented 2020-10-08 12:23:24 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

Generally, user wants to export all fitted or integrated, but option to select individually in case of bad fit should be kept. However, I dont expect it to be used very often.

*Created by: JakHolzer* Generally, user wants to export all fitted or integrated, but option to select individually in case of bad fit should be kept. However, I dont expect it to be used very often.
usov_i commented 2020-10-13 12:14:42 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

Review: Approved

*Created by: JakHolzer* **Review:** Approved
usov_i commented 2020-10-14 15:51:50 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

@JakHolzer , I would suggest not to extend this PR into parametric study, and merge it rather sooner than later. Do you have changes that are not related to parametric study still to be added here?

*Created by: ivan-usov* @JakHolzer , I would suggest not to extend this PR into parametric study, and merge it rather sooner than later. Do you have changes that are not related to parametric study still to be added here?
usov_i commented 2020-10-14 16:48:53 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

None that I can think of at the moment. I think that parametric study is
now complete (hopefully), maybe some minor additions such as saving the
data if we agree how to do it. Nonetheless Romain would like to fit more
peaks, which was not what the ccl code was written for, so there might come
some changes into fit2 again if I find a way how to make it generic.

st 14. 10. 2020 v 15:52 odesílatel Ivan Usov notifications@github.com
napsal:

@JakHolzer https://github.com/JakHolzer , I would suggest not to extend
this PR into parametric study, and merge it rather sooner than later. Do
you have changes that are not related to parametric study still to be added
here?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/paulscherrerinstitute/pyzebra/pull/11#issuecomment-708417862,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2BBRQJ3Y5KD7PXEU2R4JTSKWUIRANCNFSM4RNBW7BQ
.

*Created by: JakHolzer* None that I can think of at the moment. I think that parametric study is now complete (hopefully), maybe some minor additions such as saving the data if we agree how to do it. Nonetheless Romain would like to fit more peaks, which was not what the ccl code was written for, so there might come some changes into fit2 again if I find a way how to make it generic. st 14. 10. 2020 v 15:52 odesílatel Ivan Usov <notifications@github.com> napsal: > @JakHolzer <https://github.com/JakHolzer> , I would suggest not to extend > this PR into parametric study, and merge it rather sooner than later. Do > you have changes that are not related to parametric study still to be added > here? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#issuecomment-708417862>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AM2BBRQJ3Y5KD7PXEU2R4JTSKWUIRANCNFSM4RNBW7BQ> > . >
usov_i commented 2020-10-19 15:19:51 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

The key should be just an integer 1, e.g. measurements[1] = dict(data_cols)

*Created by: ivan-usov* The key should be just an integer 1, e.g. `measurements[1] = dict(data_cols)`
usov_i commented 2020-10-19 15:45:13 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

@JakHolzer , could you confirm that this is correct, especially when the number of measurements is odd

*Created by: ivan-usov* @JakHolzer , could you confirm that this is correct, especially when the number of measurements is odd
usov_i commented 2020-10-19 15:45:53 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

Oh I see, I apologise. Thank you for so throughout correction :)

po 19. 10. 2020 v 15:44 odesílatel Ivan Usov notifications@github.com
napsal:

@ivan-usov commented on this pull request.

In pyzebra/load_1D.py
https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507738757
:

@@ -165,7 +165,15 @@ def parse_1D(fileobj, data_type):
for name, val in zip(col_names, line.split()):
data_cols[name].append(float(val))

  •    measurements = dict(data_cols)
    
  •    data_cols['h_index'] = float(metadata['title'].split()[-3])
    
  •    data_cols['k_index'] = float(metadata['title'].split()[-2])
    
  •    data_cols['l_index'] = float(metadata['title'].split()[-1])
    
  •    data_cols['temperature'] = metadata['temp']
    
  •    data_cols['mag_field'] = metadata['mf']
    
  •    data_cols['omega_angle'] = metadata['omega']
    
  •    data_cols['number_of_measurements'] = len(data_cols['om'])
    
  •    data_cols['monitor'] = data_cols['Monitor1'][0]
    
  •    measurements["1"] = dict(data_cols)
    

The key should be just an integer 1, e.g. measurements[1] =
dict(data_cols)


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/paulscherrerinstitute/pyzebra/pull/11#pullrequestreview-511751197,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2BBRUQGCFAYYI3DQJYJ2DSLQ7CDANCNFSM4RNBW7BQ
.

*Created by: JakHolzer* Oh I see, I apologise. Thank you for so throughout correction :) po 19. 10. 2020 v 15:44 odesílatel Ivan Usov <notifications@github.com> napsal: > *@ivan-usov* commented on this pull request. > ------------------------------ > > In pyzebra/load_1D.py > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507738757> > : > > > @@ -165,7 +165,15 @@ def parse_1D(fileobj, data_type): > for name, val in zip(col_names, line.split()): > data_cols[name].append(float(val)) > > - measurements = dict(data_cols) > + data_cols['h_index'] = float(metadata['title'].split()[-3]) > + data_cols['k_index'] = float(metadata['title'].split()[-2]) > + data_cols['l_index'] = float(metadata['title'].split()[-1]) > + data_cols['temperature'] = metadata['temp'] > + data_cols['mag_field'] = metadata['mf'] > + data_cols['omega_angle'] = metadata['omega'] > + data_cols['number_of_measurements'] = len(data_cols['om']) > + data_cols['monitor'] = data_cols['Monitor1'][0] > + measurements["1"] = dict(data_cols) > > The key should be just an integer 1, e.g. measurements[1] = > dict(data_cols) > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#pullrequestreview-511751197>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AM2BBRUQGCFAYYI3DQJYJ2DSLQ7CDANCNFSM4RNBW7BQ> > . >
usov_i commented 2020-10-19 15:58:53 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

I believe so, I've tested that now on some model data and it seems everything is in order. The omega angle in the first line (not the array) is centre of the the scan and doesent have to be in the omega array. Ive tried it on odd and even numbers and mean of omega array is equal to omega value.
for pints in 20, 21:
omega = 31.4
step = 0.1

om = np.linspace(float(omega) - (int(pints) / 2) * float(step),
    float(omega) + (int(pints) / 2) * float(step),
    int(pints))

print(om)
print(np.mean(om))

gives:
[30.4 30.50526316 30.61052632 30.71578947 30.82105263 30.92631579
31.03157895 31.13684211 31.24210526 31.34736842 31.45263158 31.55789474
31.66315789 31.76842105 31.87368421 31.97894737 32.08421053 32.18947368
32.29473684 32.4 ]
31.4
[30.35 30.455 30.56 30.665 30.77 30.875 30.98 31.085 31.19 31.295
31.4 31.505 31.61 31.715 31.82 31.925 32.03 32.135 32.24 32.345
32.45 ]
31.4

In odd, the omega is middle value, in even case the omega itself is not there. Or am I missing something?

*Created by: JakHolzer* I believe so, I've tested that now on some model data and it seems everything is in order. The omega angle in the first line (not the array) is centre of the the scan and doesent have to be in the omega array. Ive tried it on odd and even numbers and mean of omega array is equal to omega value. for pints in 20, 21: omega = 31.4 step = 0.1 om = np.linspace(float(omega) - (int(pints) / 2) * float(step), float(omega) + (int(pints) / 2) * float(step), int(pints)) print(om) print(np.mean(om)) gives: [30.4 30.50526316 30.61052632 30.71578947 30.82105263 30.92631579 31.03157895 31.13684211 31.24210526 31.34736842 31.45263158 31.55789474 31.66315789 31.76842105 31.87368421 31.97894737 32.08421053 32.18947368 32.29473684 32.4 ] 31.4 [30.35 30.455 30.56 30.665 30.77 30.875 30.98 31.085 31.19 31.295 31.4 31.505 31.61 31.715 31.82 31.925 32.03 32.135 32.24 32.345 32.45 ] 31.4 In odd, the omega is middle value, in even case the omega itself is not there. Or am I missing something?
usov_i commented 2020-10-19 16:12:09 +02:00 (Migrated from gitlab.psi.ch)

Created by: ivan-usov

OK, I didn't know how this "om" array should be constructed at all. This is clearly dependent on how daq works, so if you know that this array is calculated this way, then it's all good.

*Created by: ivan-usov* OK, I didn't know how this "om" array should be constructed at all. This is clearly dependent on how daq works, so if you know that this array is calculated this way, then it's all good.
usov_i commented 2020-10-19 16:18:28 +02:00 (Migrated from gitlab.psi.ch)

Created by: JakHolzer

Well, its not necessary for dat files which actually carry the om array, in
ccl its simplified to omega centre, step and number of points, so this is
only way of getting it.

po 19. 10. 2020 v 16:12 odesílatel Ivan Usov notifications@github.com
napsal:

@ivan-usov commented on this pull request.

In pyzebra/load_1D.py
https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507781946
:

  •        # first line
    
  •        measurement_number, *params = line.split()
    
  •        for param, (param_name, param_type) in zip(params, ccl_first_line):
    
  •            d[param_name] = param_type(param)
    
  •        decimal.append(bool(Decimal(d["h_index"]) % 1 == 0))
    
  •        decimal.append(bool(Decimal(d["k_index"]) % 1 == 0))
    
  •        decimal.append(bool(Decimal(d["l_index"]) % 1 == 0))
    
  •        # second line
    
  •        next_line = next(fileobj)
    
  •        params = next_line.split()
    
  •        for param, (param_name, param_type) in zip(params, ccl_second_line):
    
  •            d[param_name] = param_type(param)
    
  •        d["om"] = np.linspace(
    

OK, I didn't know how this "om" array should be constructed at all. This
is clearly dependent on how daq works, so if you know that this array is
calculated this way, then it's all good.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507781946,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2BBRTHNOPYSCPDKIF76KDSLRCMZANCNFSM4RNBW7BQ
.

*Created by: JakHolzer* Well, its not necessary for dat files which actually carry the om array, in ccl its simplified to omega centre, step and number of points, so this is only way of getting it. po 19. 10. 2020 v 16:12 odesílatel Ivan Usov <notifications@github.com> napsal: > *@ivan-usov* commented on this pull request. > ------------------------------ > > In pyzebra/load_1D.py > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507781946> > : > > > + # first line > + measurement_number, *params = line.split() > + for param, (param_name, param_type) in zip(params, ccl_first_line): > + d[param_name] = param_type(param) > + > + decimal.append(bool(Decimal(d["h_index"]) % 1 == 0)) > + decimal.append(bool(Decimal(d["k_index"]) % 1 == 0)) > + decimal.append(bool(Decimal(d["l_index"]) % 1 == 0)) > + > + # second line > + next_line = next(fileobj) > + params = next_line.split() > + for param, (param_name, param_type) in zip(params, ccl_second_line): > + d[param_name] = param_type(param) > + > + d["om"] = np.linspace( > > OK, I didn't know how this "om" array should be constructed at all. This > is clearly dependent on how daq works, so if you know that this array is > calculated this way, then it's all good. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507781946>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AM2BBRTHNOPYSCPDKIF76KDSLRCMZANCNFSM4RNBW7BQ> > . >
usov_i commented 2020-10-22 12:07:50 +02:00 (Migrated from gitlab.psi.ch)

Merged by: ivan-usov at 2020-10-22 10:07:50 UTC

*Merged by: ivan-usov at 2020-10-22 10:07:50 UTC*
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: zebra/pyzebra#69
No description provided.