@JakHolzer , I created a pull request will all changes grouped on the same branch. This way it will be easier to introduce changes, that might affect all files at the same time. You should be able to push to that branch (det1d) too.
*Created by: ivan-usov*
@JakHolzer , I created a pull request will all changes grouped on the same branch. This way it will be easier to introduce changes, that might affect all files at the same time. You should be able to push to that branch (`det1d`) too.
*Created by: ivan-usov*
@JakHolzer , if you want, you can try to simplify reading of ccl files in a similar way to https://github.com/paulscherrerinstitute/pyzebra/pull/11/commits/fe1d8044a3f2f6e24927a4f8baf4aa0894a56140
Btw, that commit also renames "omega" into "om", and "counts" into "Counts", that is consistent with the column names in dat files. I hope it doesn't break you other code, otherwise let me know what names you prefer, it will be easy to fix, e.g. with
*Created by: ivan-usov*
Btw, that commit also renames "omega" into "om", and "counts" into "Counts", that is consistent with the column names in dat files. I hope it doesn't break you other code, otherwise let me know what names you prefer, it will be easy to fix, e.g. with
```
# row_names = next(infile).split()
row_names = ("NP", "omega", "counts", ...)
```
I think it should be fine. The ccl files does not have the headers and
there is only one data column, rest is numbers or is calculated, not read.
So I think its alright, important think is that you know what is what :)
# this is the end of metadata and the start of data section
break
# read data
if det_variables["file_type"] == "ccl":
decimal = list()
data = infile.readlines()
Btw, that commit also renames "omega" into "om", and "counts" into
"Counts", that is consistent with the column names in dat files. I hope it
doesn't break you other code, otherwise let me know what names you prefer,
it will be easy to fix, e.g. with
*Created by: JakHolzer*
I think it should be fine. The ccl files does not have the headers and
there is only one data column, rest is numbers or is calculated, not read.
So I think its alright, important think is that you know what is what :)
út 15. 9. 2020 v 16:46 odesílatel Ivan Usov <notifications@github.com>
napsal:
> *@ivan-usov* commented on this pull request.
> ------------------------------
>
> In pyzebra/load_1D.py
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r488727539>
> :
>
> > + variable = variable.strip()
> + if variable in META_VARS_FLOAT:
> + det_variables["meta"][variable] = float(value)
> + elif variable in META_VARS_STR:
> + det_variables["meta"][variable] = str(value)[:-1].strip()
> + elif variable in META_UB_MATRIX:
> + det_variables["meta"][variable] = re.findall(r"[-+]?\d*\.\d+|\d+", str(value))
> +
> + if "#data" in line:
> + # this is the end of metadata and the start of data section
> + break
> +
> + # read data
> + if det_variables["file_type"] == "ccl":
> + decimal = list()
> + data = infile.readlines()
>
> Btw, that commit also renames "omega" into "om", and "counts" into
> "Counts", that is consistent with the column names in dat files. I hope it
> doesn't break you other code, otherwise let me know what names you prefer,
> it will be easy to fix, e.g. with
>
> # row_names = next(infile).split()
> row_names = ["NP", "omega", "counts", ...]
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r488727539>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM2BBRVPO2FLEFKK74RXULLSF545RANCNFSM4RNBW7BQ>
> .
>
These paddings look artificial to me. Do you think it was just tabs between numbers, and the different padding sizes are because of different lengths of those written numbers?
*Created by: ivan-usov*
These paddings look artificial to me. Do you think it was just tabs between numbers, and the different padding sizes are because of different lengths of those written numbers?
Well, the structure of the files seems very uniform and aligned, so I
wanted to keep that. I have never worked with the software that uses it, so
I'm not sure how it reads the data. My suggestion is to leave it like this
and we can ask Oksana to try load one file with padding and second with
tabs to see if it works.
:arg data - data to export, is dict after peak fitting
"""
align = ">"
if data["meta"]["indices"] == "hkl":
extension = ".comm"
padding = [6, 4, 10, 8]
These paddings look artificial to me. Do you think it was just tabs
between numbers, and the different padding sizes are because of different
lengths of those written numbers?
*Created by: JakHolzer*
Well, the structure of the files seems very uniform and aligned, so I
wanted to keep that. I have never worked with the software that uses it, so
I'm not sure how it reads the data. My suggestion is to leave it like this
and we can ask Oksana to try load one file with padding and second with
tabs to see if it works.
st 16. 9. 2020 v 11:27 odesílatel Ivan Usov <notifications@github.com>
napsal:
> *@ivan-usov* commented on this pull request.
> ------------------------------
>
> In pyzebra/comm_export.py
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r489296559>
> :
>
> > + corr_value = np.abs(value * np.sin(ang1) * np.cos(ang2))
> + return corr_value
> +
> +
> +def export_comm(data, path, lorentz=False):
> + """exports data in the *.comm format
> + :param lorentz: perform Lorentz correction
> + :param path: path to file + name
> + :arg data - data to export, is dict after peak fitting
> +
> + """
> +
> + align = ">"
> + if data["meta"]["indices"] == "hkl":
> + extension = ".comm"
> + padding = [6, 4, 10, 8]
>
> These paddings look artificial to me. Do you think it was just tabs
> between numbers, and the different padding sizes are because of different
> lengths of those written numbers?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#pullrequestreview-489441678>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM2BBRU7NNDFJCK25MZNFQTSGCAJNANCNFSM4RNBW7BQ>
> .
>
This parameter is (will be) used by export comm to decide if the fitted area or numerically integrated area is to be exported. User has to be able to choose, non the less numerically integrated area is default.
*Created by: JakHolzer*
This parameter is (will be) used by export comm to decide if the fitted area or numerically integrated area is to be exported. User has to be able to choose, non the less numerically integrated area is default.
I'm sorry, I think that the answer I send from email got stuck somewhere. The files look very organized, so I think its really read line by line in this and first 6 chars correspond to number and so on. Oksana would like to keep it this way.
*Created by: JakHolzer*
I'm sorry, I think that the answer I send from email got stuck somewhere. The files look very organized, so I think its really read line by line in this and first 6 chars correspond to number and so on. Oksana would like to keep it this way.
the detectorDistance is a number in ccl files, but in dat files its date, which breaks the code. I asked Oksana if it can be fixed in the software that produces the *.dat files. If it for whatever reason cant be fixed, I suggest to move it to META_VARS_STR.
*Created by: JakHolzer*
the detectorDistance is a number in ccl files, but in dat files its date, which breaks the code. I asked Oksana if it can be fixed in the software that produces the *.dat files. If it for whatever reason cant be fixed, I suggest to move it to META_VARS_STR.
Does it need to be individual for each measurement? I would expect a user wanting to export all measurements in a similar way, thus having this parameter global.
*Created by: ivan-usov*
Does it need to be individual for each measurement? I would expect a user wanting to export all measurements in a similar way, thus having this parameter global.
Generally, user wants to export all fitted or integrated, but option to select individually in case of bad fit should be kept. However, I dont expect it to be used very often.
*Created by: JakHolzer*
Generally, user wants to export all fitted or integrated, but option to select individually in case of bad fit should be kept. However, I dont expect it to be used very often.
@JakHolzer , I would suggest not to extend this PR into parametric study, and merge it rather sooner than later. Do you have changes that are not related to parametric study still to be added here?
*Created by: ivan-usov*
@JakHolzer , I would suggest not to extend this PR into parametric study, and merge it rather sooner than later. Do you have changes that are not related to parametric study still to be added here?
None that I can think of at the moment. I think that parametric study is
now complete (hopefully), maybe some minor additions such as saving the
data if we agree how to do it. Nonetheless Romain would like to fit more
peaks, which was not what the ccl code was written for, so there might come
some changes into fit2 again if I find a way how to make it generic.
@JakHolzer https://github.com/JakHolzer , I would suggest not to extend
this PR into parametric study, and merge it rather sooner than later. Do
you have changes that are not related to parametric study still to be added
here?
*Created by: JakHolzer*
None that I can think of at the moment. I think that parametric study is
now complete (hopefully), maybe some minor additions such as saving the
data if we agree how to do it. Nonetheless Romain would like to fit more
peaks, which was not what the ccl code was written for, so there might come
some changes into fit2 again if I find a way how to make it generic.
st 14. 10. 2020 v 15:52 odesílatel Ivan Usov <notifications@github.com>
napsal:
> @JakHolzer <https://github.com/JakHolzer> , I would suggest not to extend
> this PR into parametric study, and merge it rather sooner than later. Do
> you have changes that are not related to parametric study still to be added
> here?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#issuecomment-708417862>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM2BBRQJ3Y5KD7PXEU2R4JTSKWUIRANCNFSM4RNBW7BQ>
> .
>
*Created by: JakHolzer*
Oh I see, I apologise. Thank you for so throughout correction :)
po 19. 10. 2020 v 15:44 odesílatel Ivan Usov <notifications@github.com>
napsal:
> *@ivan-usov* commented on this pull request.
> ------------------------------
>
> In pyzebra/load_1D.py
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507738757>
> :
>
> > @@ -165,7 +165,15 @@ def parse_1D(fileobj, data_type):
> for name, val in zip(col_names, line.split()):
> data_cols[name].append(float(val))
>
> - measurements = dict(data_cols)
> + data_cols['h_index'] = float(metadata['title'].split()[-3])
> + data_cols['k_index'] = float(metadata['title'].split()[-2])
> + data_cols['l_index'] = float(metadata['title'].split()[-1])
> + data_cols['temperature'] = metadata['temp']
> + data_cols['mag_field'] = metadata['mf']
> + data_cols['omega_angle'] = metadata['omega']
> + data_cols['number_of_measurements'] = len(data_cols['om'])
> + data_cols['monitor'] = data_cols['Monitor1'][0]
> + measurements["1"] = dict(data_cols)
>
> The key should be just an integer 1, e.g. measurements[1] =
> dict(data_cols)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#pullrequestreview-511751197>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM2BBRUQGCFAYYI3DQJYJ2DSLQ7CDANCNFSM4RNBW7BQ>
> .
>
I believe so, I've tested that now on some model data and it seems everything is in order. The omega angle in the first line (not the array) is centre of the the scan and doesent have to be in the omega array. Ive tried it on odd and even numbers and mean of omega array is equal to omega value.
for pints in 20, 21:
omega = 31.4
step = 0.1
In odd, the omega is middle value, in even case the omega itself is not there. Or am I missing something?
*Created by: JakHolzer*
I believe so, I've tested that now on some model data and it seems everything is in order. The omega angle in the first line (not the array) is centre of the the scan and doesent have to be in the omega array. Ive tried it on odd and even numbers and mean of omega array is equal to omega value.
for pints in 20, 21:
omega = 31.4
step = 0.1
om = np.linspace(float(omega) - (int(pints) / 2) * float(step),
float(omega) + (int(pints) / 2) * float(step),
int(pints))
print(om)
print(np.mean(om))
gives:
[30.4 30.50526316 30.61052632 30.71578947 30.82105263 30.92631579
31.03157895 31.13684211 31.24210526 31.34736842 31.45263158 31.55789474
31.66315789 31.76842105 31.87368421 31.97894737 32.08421053 32.18947368
32.29473684 32.4 ]
31.4
[30.35 30.455 30.56 30.665 30.77 30.875 30.98 31.085 31.19 31.295
31.4 31.505 31.61 31.715 31.82 31.925 32.03 32.135 32.24 32.345
32.45 ]
31.4
In odd, the omega is middle value, in even case the omega itself is not there. Or am I missing something?
OK, I didn't know how this "om" array should be constructed at all. This is clearly dependent on how daq works, so if you know that this array is calculated this way, then it's all good.
*Created by: ivan-usov*
OK, I didn't know how this "om" array should be constructed at all. This is clearly dependent on how daq works, so if you know that this array is calculated this way, then it's all good.
Well, its not necessary for dat files which actually carry the om array, in
ccl its simplified to omega centre, step and number of points, so this is
only way of getting it.
for param, (param_name, param_type) in zip(params, ccl_second_line):
d[param_name] = param_type(param)
d["om"] = np.linspace(
OK, I didn't know how this "om" array should be constructed at all. This
is clearly dependent on how daq works, so if you know that this array is
calculated this way, then it's all good.
*Created by: JakHolzer*
Well, its not necessary for dat files which actually carry the om array, in
ccl its simplified to omega centre, step and number of points, so this is
only way of getting it.
po 19. 10. 2020 v 16:12 odesílatel Ivan Usov <notifications@github.com>
napsal:
> *@ivan-usov* commented on this pull request.
> ------------------------------
>
> In pyzebra/load_1D.py
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507781946>
> :
>
> > + # first line
> + measurement_number, *params = line.split()
> + for param, (param_name, param_type) in zip(params, ccl_first_line):
> + d[param_name] = param_type(param)
> +
> + decimal.append(bool(Decimal(d["h_index"]) % 1 == 0))
> + decimal.append(bool(Decimal(d["k_index"]) % 1 == 0))
> + decimal.append(bool(Decimal(d["l_index"]) % 1 == 0))
> +
> + # second line
> + next_line = next(fileobj)
> + params = next_line.split()
> + for param, (param_name, param_type) in zip(params, ccl_second_line):
> + d[param_name] = param_type(param)
> +
> + d["om"] = np.linspace(
>
> OK, I didn't know how this "om" array should be constructed at all. This
> is clearly dependent on how daq works, so if you know that this array is
> calculated this way, then it's all good.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/paulscherrerinstitute/pyzebra/pull/11#discussion_r507781946>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM2BBRTHNOPYSCPDKIF76KDSLRCMZANCNFSM4RNBW7BQ>
> .
>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Created by: ivan-usov
Created by: ivan-usov
@JakHolzer , I created a pull request will all changes grouped on the same branch. This way it will be easier to introduce changes, that might affect all files at the same time. You should be able to push to that branch (
det1d) too.Created by: ivan-usov
@JakHolzer , if you want, you can try to simplify reading of ccl files in a similar way to https://github.com/paulscherrerinstitute/pyzebra/pull/11/commits/fe1d8044a3f2f6e24927a4f8baf4aa0894a56140
Created by: JakHolzer
I will try 👍
Created by: ivan-usov
Btw, that commit also renames "omega" into "om", and "counts" into "Counts", that is consistent with the column names in dat files. I hope it doesn't break you other code, otherwise let me know what names you prefer, it will be easy to fix, e.g. with
Created by: JakHolzer
I think it should be fine. The ccl files does not have the headers and
there is only one data column, rest is numbers or is calculated, not read.
So I think its alright, important think is that you know what is what :)
út 15. 9. 2020 v 16:46 odesílatel Ivan Usov notifications@github.com
napsal:
Created by: ivan-usov
These paddings look artificial to me. Do you think it was just tabs between numbers, and the different padding sizes are because of different lengths of those written numbers?
Created by: JakHolzer
Well, the structure of the files seems very uniform and aligned, so I
wanted to keep that. I have never worked with the software that uses it, so
I'm not sure how it reads the data. My suggestion is to leave it like this
and we can ask Oksana to try load one file with padding and second with
tabs to see if it works.
st 16. 9. 2020 v 11:27 odesílatel Ivan Usov notifications@github.com
napsal:
Created by: JakHolzer
This parameter is (will be) used by export comm to decide if the fitted area or numerically integrated area is to be exported. User has to be able to choose, non the less numerically integrated area is default.
Created by: JakHolzer
I'm sorry, I think that the answer I send from email got stuck somewhere. The files look very organized, so I think its really read line by line in this and first 6 chars correspond to number and so on. Oksana would like to keep it this way.
Created by: JakHolzer
the detectorDistance is a number in ccl files, but in dat files its date, which breaks the code. I asked Oksana if it can be fixed in the software that produces the *.dat files. If it for whatever reason cant be fixed, I suggest to move it to META_VARS_STR.
Created by: ivan-usov
Does it need to be individual for each measurement? I would expect a user wanting to export all measurements in a similar way, thus having this parameter global.
Created by: JakHolzer
Generally, user wants to export all fitted or integrated, but option to select individually in case of bad fit should be kept. However, I dont expect it to be used very often.
Created by: JakHolzer
Review: Approved
Created by: ivan-usov
@JakHolzer , I would suggest not to extend this PR into parametric study, and merge it rather sooner than later. Do you have changes that are not related to parametric study still to be added here?
Created by: JakHolzer
None that I can think of at the moment. I think that parametric study is
now complete (hopefully), maybe some minor additions such as saving the
data if we agree how to do it. Nonetheless Romain would like to fit more
peaks, which was not what the ccl code was written for, so there might come
some changes into fit2 again if I find a way how to make it generic.
st 14. 10. 2020 v 15:52 odesílatel Ivan Usov notifications@github.com
napsal:
Created by: ivan-usov
The key should be just an integer 1, e.g.
measurements[1] = dict(data_cols)Created by: ivan-usov
@JakHolzer , could you confirm that this is correct, especially when the number of measurements is odd
Created by: JakHolzer
Oh I see, I apologise. Thank you for so throughout correction :)
po 19. 10. 2020 v 15:44 odesílatel Ivan Usov notifications@github.com
napsal:
Created by: JakHolzer
I believe so, I've tested that now on some model data and it seems everything is in order. The omega angle in the first line (not the array) is centre of the the scan and doesent have to be in the omega array. Ive tried it on odd and even numbers and mean of omega array is equal to omega value.
for pints in 20, 21:
omega = 31.4
step = 0.1
gives:
[30.4 30.50526316 30.61052632 30.71578947 30.82105263 30.92631579
31.03157895 31.13684211 31.24210526 31.34736842 31.45263158 31.55789474
31.66315789 31.76842105 31.87368421 31.97894737 32.08421053 32.18947368
32.29473684 32.4 ]
31.4
[30.35 30.455 30.56 30.665 30.77 30.875 30.98 31.085 31.19 31.295
31.4 31.505 31.61 31.715 31.82 31.925 32.03 32.135 32.24 32.345
32.45 ]
31.4
In odd, the omega is middle value, in even case the omega itself is not there. Or am I missing something?
Created by: ivan-usov
OK, I didn't know how this "om" array should be constructed at all. This is clearly dependent on how daq works, so if you know that this array is calculated this way, then it's all good.
Created by: JakHolzer
Well, its not necessary for dat files which actually carry the om array, in
ccl its simplified to omega centre, step and number of points, so this is
only way of getting it.
po 19. 10. 2020 v 16:12 odesílatel Ivan Usov notifications@github.com
napsal:
Merged by: ivan-usov at 2020-10-22 10:07:50 UTC