zebra/pyzebra
3
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

error bars after merging in ccl integrate #46

New Issue
Closed
opened 2021-11-22 11:43:41 +01:00 by usov_i · 8 comments
usov_i commented 2021-11-22 11:43:41 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: rsibille-psi

I merge the following data (experiment 20211529):
files 8662 and 8674

Just by looking visually, the error bars are much larger after merging, while obviously they should get smaller when increasing the statistics...

*Created by: rsibille-psi* I merge the following data (experiment 20211529): files 8662 and 8674 Just by looking visually, the error bars are much larger after merging, while obviously they should get smaller when increasing the statistics...
usov_i commented 2021-11-22 13:38:48 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: ivan-usov

@rsibille-psi , true, there is an error in the code (a missing divider), however, thinking a bit more about how the calculation of average in this case should be done properly, it might not be as trivial. Probably, there is a simple way to calculate statistics for neutron counts I'm just not aware of.

After the fix of that issue we would use an average for the number of counts (counts1 + counts2 + ...) / N and sqrt(sigma1^2 + sigma2^2 + ...) / N for the error of that average (currently, dividing by N is missing for sigma). However, for the situation with different sigma values for the same measurement, the direct averaging might not be the most right approach, see for example https://physics.stackexchange.com/a/452809

*Created by: ivan-usov* @rsibille-psi , true, there is an error in the code (a missing divider), however, thinking a bit more about how the calculation of average in this case should be done properly, it might not be as trivial. Probably, there is a simple way to calculate statistics for neutron counts I'm just not aware of. After the fix of that issue we would use an average for the number of counts `(counts1 + counts2 + ...) / N` and `sqrt(sigma1^2 + sigma2^2 + ...) / N` for the error of that average (currently, dividing by `N` is missing for `sigma`). However, for the situation with different `sigma` values for the same measurement, the direct averaging might not be the most right approach, see for example https://physics.stackexchange.com/a/452809
usov_i commented 2021-11-22 14:54:24 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: rsibille-psi

thanks!
In our case, we assume (and it's correct unless there is a problem with the instrument motors or temperature etc) that the error is purely statistical on counting, not related to uncertainties.
Therefore we should simply have
counts = (counts1 + counts2 + ...)/N
error = sqrt((counts1 + counts2 + ...)/N)

*Created by: rsibille-psi* thanks! In our case, we assume (and it's correct unless there is a problem with the instrument motors or temperature etc) that the error is purely statistical on counting, not related to uncertainties. Therefore we should simply have counts = (counts1 + counts2 + ...)/N error = sqrt((counts1 + counts2 + ...)/N)
usov_i commented 2021-11-22 15:19:20 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: ivan-usov

OK, but if I understood it correctly, this way will not lead to a statistic improvement. For example, assume we have 2 measurements at the same motor position, both with 100 counts. Each individual measurement will have an error = sqrt(100) = 10, at the same time, based on your formula, their average error = sqrt((100+100)/2) = 10. And this will be true for any number of averaged measurements. Does it make sense to implement it this way?

*Created by: ivan-usov* OK, but if I understood it correctly, this way will not lead to a statistic improvement. For example, assume we have 2 measurements at the same motor position, both with 100 counts. Each individual measurement will have an `error = sqrt(100) = 10`, at the same time, based on your formula, their `average error = sqrt((100+100)/2) = 10`. And this will be true for any number of averaged measurements. Does it make sense to implement it this way?
usov_i commented 2021-11-22 15:24:58 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: rsibille-psi

sorry.... I positioned the parenthesis wrongly.

it should be:

counts = (counts1 + counts2 + ...)/N
error = sqrt((counts1 + counts2 + ...))/N

*Created by: rsibille-psi* sorry.... I positioned the parenthesis wrongly. it should be: counts = (counts1 + counts2 + ...)/N error = sqrt((counts1 + counts2 + ...))/N
usov_i commented 2021-11-22 15:28:43 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: ivan-usov

pyzebra test server is restarted with the fix

*Created by: ivan-usov* pyzebra test server is restarted with the fix
usov_i commented 2021-11-22 15:30:39 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: rsibille-psi

thank you Ivan !

*Created by: rsibille-psi* thank you Ivan !
usov_i commented 2021-11-22 15:38:30 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: rsibille-psi

I just checked merging exactly the same scans with our old tool "fit", and the results are now exactly the same.
So I think this fix can also go on the main server.
Thank you!

*Created by: rsibille-psi* I just checked merging exactly the same scans with our old tool "fit", and the results are now exactly the same. So I think this fix can also go on the main server. Thank you!
usov_i commented 2021-11-22 16:05:46 +01:00 (Migrated from gitlab.psi.ch)
Copy Link
Copy Source

Created by: ivan-usov

The main server is updated and restarted

*Created by: ivan-usov* The main server is updated and restarted
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
sibille_r (sibille_r) usov_i (usov_i) zaharko (zaharko)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: zebra/pyzebra#46