Issue merging 2 datasets #15
Loading…
x
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Created by: ivan-usov
@JakHolzer I pushed an experimental code to the test server, in which I'm trying to merge 2 datasets:
b28fe39bbb/pyzebra/app/panel_ccl_integrate.py (L123-L125)
Running it with test data:
zebra2020n000039.ccl
followed by appendingzebra2020n000035.dat
errors with the following stack trace:Can you reproduce that? Any ideas what's wrong?
Created by: JakHolzer
Hello Ivan,
I'm away from the computer at the moment, but I think it could be caused by
not assigning the monitor value in dat file perhaps. I will check when
I get home from supermarket.
Best wishes,
Jakub
On Mon, 26 Oct 2020, 16:55 Ivan Usov, notifications@github.com wrote:
Created by: JakHolzer
Hello Ivan,
I reproduced the error. There is some problem with the scan function,
basically its happening because it found two matching scans, 3 and 157 and
157 and 3, but they are listed as two separate.
[33.0, 16.5, 138.8, -23.3] [(3, 157)]
[32.8, 16.4, 138.8, -23.0] [(157, 3)]
therefore, when the merge runs, it merges 157 into 3 and than it tries to
run again and merge 3 with 157, but 157 was deleted when it was merged with
3. I will try to fix it by tomorrow.
Jakub
po 26. 10. 2020 v 16:55 odesílatel Ivan Usov notifications@github.com
napsal:
Created by: JakHolzer
I was trying to find some way to fi it permanently, but could not really do
it simply. The fix I did kind of works, but it will suffer from the same
problem, in the future. It will probably have to be rewritten and I'm not
yet sure how. When we worked with hkl, i created the dictionary where the
keys are the hkl and values are the numbers of the scan and it worked well,
because the hkl was precise every time, i.e. it was always 1 0 1, not 0.98
0.001 1.01 Now this is the case and root for the problem here, it the bug
you reported, the scans with angles:
[32.97, 16.48, 138.81, -23.26]
[32.84, 16.42, 138.82, -22.97]
gets matched, which is correct, but are saved with rounded second digit.
i.e:
[33.0, 16.5, 138.8, -23.3] [(3, 157)]
[32.8, 16.4, 138.8, -23.0] [(157, 3)]
so it thinks that there are two scans. the correct way should be:
[33.0, 16.0, 139.0, -23.0] [(3, 157), (157, 3)], which it does now, but it
will fail happily when one of the angles in first scan will be 1.4 and in
second 1.6. I will try to come up with a better solution, perhaps filter
the reverse duplicates from the beginning. Any hints welcomed 😓
I will try to come up with something quickly.
Best,
Jakub
po 26. 10. 2020 v 19:08 odesílatel Jakub Holzer holzer.jakub@gmail.com
napsal:
Created by: ivan-usov
I didn't thoroughly check the code in
param_study_moduls.py
, but to my understanding, merging should happen in-place like in the following pseudo code:Maybe d1_meta needs some updates too, I don't know. But I also don't understand what you're trying to achieve by splitting this single operation into all these functions.
Created by: JakHolzer
Well, the idea at the beginning was to have some sort of output for the
user to check what is in the ccls and decide. The list of matching scans
was also meant be used for other operations, such as point by point
subtraction, but it obviously doesent work. I will write something with
your idea. Thanks!
út 27. 10. 2020 v 10:34 odesílatel Ivan Usov notifications@github.com
napsal:
Created by: ivan-usov
Let's keep it as simple as possible. I don't expect that, practically, users will need to look at what they have in both files and then decide to merge or not.
Here we handle just the first step of merging one datasets (.ccl) with an additional scan (.dat) or another dataset (.ccl). Point by point subtraction, as far as I understood, happens between datasets after all fitting etc. So, let's not consider it yet, because it will be a different operation.
Created by: JakHolzer
Alright, I would go now with the yesterdays fix and I will supply new
function soon.
út 27. 10. 2020 v 11:07 odesílatel Ivan Usov notifications@github.com
napsal:
Created by: JakHolzer
Hello Ivan,
I apologise for the delay. I was trying to rewrite the function in the
manner you suggested, but still running into problems such as runtime
errors cause by dictionary change or keyerrors caused by deleted scans.
This always leads me to similar solutions where the process is split into
two function, where one scans and gives output to the second, that merges.
Generally, it is necessary to scan the ccl files, because sometimes there
can be repeated scans even before merging with other datasets. I will keep
trying, but progress has not been great so far.
Best,
Jakub
út 27. 10. 2020 v 11:18 odesílatel Jakub Holzer holzer.jakub@gmail.com
napsal:
Created by: ivan-usov
@JakHolzer , there is now two files with operations for dictionary scanning and merging. I assume that the old file
ccl_dict_operation.py
is not supported anymore and can be removed?Created by: JakHolzer
Yes, there are some extra functions there zvut they rely on the scan_dict
function, which we are not be using anymore, once we have a suitable
function.
On Fri, 30 Oct 2020, 14:39 Ivan Usov, notifications@github.com wrote:
Created by: ivan-usov
The merging procedure was refactored, so the issue is not actual anymore