APOG_public/acsmnode
0
0
Fork 0
mirror of https://gitea.psi.ch/APOG/acsmnode.git synced 2025-06-24 21:21:08 +02:00
Code Issues Packages Projects Releases Wiki Activity

Update TODO.md. Added some todos and deleted one that is done now.

Browse Source
This commit is contained in:
simon_l
2025-05-28 11:48:26 +02:00
parent 196e237838
commit 3a23db1a12
1 changed files with 47 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

92
TODO.md
View File

@ -1,45 +1,47 @@
# TODO # TODO
* [from Leïla] Correct error when flags are loaded in the flagging app * [from Leïla] Data Flagging App. Correct error when flags are loaded in the app.
* [from Leïla] End the file at end of year (or filter only current year) * [from Leïla] Before Step 5. The file should contain only data of the current year.
* [from Leïla] Change "9999.999" to "9999999.9999" in header * [from Leïla] Ebas converter. Change "9999.999" to "9999999.9999" in header
* [from Leïla] Update Detection limit values in L2 header: take the ones (1h) from limits_of_detection.yaml * [from Leïla] Ebas converter. Update Detection limit values in L2 header: take the ones (1h) from limits_of_detection.yaml
* [from Leïla] For PAY, calculate error as 50% of concentration * [from Leïla] Ebas converter. Correct errors (uncertainties) that they can't be lower than 0.0001
* [from Leïla] Correct errors (uncertainties) that they can't be lower than 0.0001 * [from Leïla] Flag using validity threshold. Change flow rate values to 10% of flow rate ref.
* [from Leïla] Change flow rate values to 10% of flow rate ref * [from Leïla] In Step 1. Creation of a new collection should be an option and not automatic.
* [New] Create data flows to validate and replace existing data chain params. data/campaignData/param/ -> pipelines/params and campaignDescriptor.yaml -> acsm data converter. * [from Leïla] The data chain (except Step 5) should also include Org_mx and Ord_err. To discuss together.
* [New] DIMA. Add dictionaries to explain variables at different levels. * [from Leïla] Step 4.1. Add a step to verify ACSM data with external instruments (MPSS, eBC). To discuss together.
* [New] DIMA. Modify data integration pipeline to not add current datetime in filename when not specified. * [New] Create data flows to validate and replace existing data chain params. data/campaignData/param/ -> pipelines/params and campaignDescriptor.yaml -> acsm data converter.
* [New] DIMA. Set data/ folder as default and only possible output folder * [New] DIMA. Add dictionaries to explain variables at different levels.
* [New] DIMA. Review DataManager File Update. Generated CSV files are not transfering properly. * [New] DIMA. Modify data integration pipeline to not add current datetime in filename when not specified.
* [New] DIMA. Ensure code snippets that open and close HDF5 files do so securly and do not leave the file locked * [New] DIMA. Set data/ folder as default and only possible output folder
* [New] DIMA. Revise default csv file reader and enhance it to infer datetime column and format. * [New] DIMA. Review DataManager File Update. Generated CSV files are not transfering properly.
* [New] Learn how to run docker-compose.yaml * [New] DIMA. Ensure code snippets that open and close HDF5 files do so securly and do not leave the file locked
* [New] EBAS_SUBMISSION_STEP. Extend time filter to ranges, create a merge data frame function, and construct file name and output dir dynammically. It is currently hardcoded. * [New] DIMA. Revise default csv file reader and enhance it to infer datetime column and format.
* [New] Finish command line interface for visualize_datatable_vars and add modes, --flags, --dataset, and save to figures folder in repo * [New] Learn how to run docker-compose.yaml
* Implement flagging-app specific data operations such as: * [New] EBAS_SUBMISSION_STEP. Extend time filter to ranges, create a merge data frame function, and construct file name and output dir dynammically. It is currently hardcoded.
1. [New item] When verify flags from checklist is active, enable delete-flag button to delete flag associated with active cell on table. * [New] Finish command line interface for visualize_datatable_vars and add modes, --flags, --dataset, and save to figures folder in repo
2. [New item] When verify and ready to trasnfer items on checklist are active, enable record-flags button to record verified flags into the HDF5 file. * Implement flagging-app specific data operations such as:
3. [New item] When all checklist items active, enable apply button to apply flags to the time series data and save it to the HDF5 file. 1. [New item] When verify flags from checklist is active, enable delete-flag button to delete flag associated with active cell on table.
1. ~~Define data manager obj with apply flags behavior.~~ 2. [New item] When verify and ready to trasnfer items on checklist are active, enable record-flags button to record verified flags into the HDF5 file.
2. Define metadata answering who did the flagging and quality assurance tests? 3. [New item] When all checklist items active, enable apply button to apply flags to the time series data and save it to the HDF5 file.
3. Update intruments/dictionaries/ACSM_TOFWARE_flags.yaml and instruments/readers/flag_reader.py to describe metadata elements based on dictionary. 1. ~~Define data manager obj with apply flags behavior.~~
4. ~~Update DIMA data integration pipeline to allowed user-defined file naming template~~ 2. Define metadata answering who did the flagging and quality assurance tests?
5. ~~Design and implement flag visualization feature: click flag on table and display on figure shaded region when feature is enabled~~ 3. Update intruments/dictionaries/ACSM_TOFWARE_flags.yaml and instruments/readers/flag_reader.py to describe metadata elements based on dictionary.
6. Implement schema validator on yaml/json representation of hdf5 metadata 4. ~~Update DIMA data integration pipeline to allowed user-defined file naming template~~
7. Implement updates to 'actris level' and 'processing_script' after operation applied to data/file? 5. ~~Design and implement flag visualization feature: click flag on table and display on figure shaded region when feature is enabled~~
6. Implement schema validator on yaml/json representation of hdf5 metadata
* ~~When `Create Flag` is clicked, modify the title to indicate that we are in flagging mode and ROIs can be drawn by dragging.~~ 7. Implement updates to 'actris level' and 'processing_script' after operation applied to data/file?
* ~~Update `Commit Flag` logic:~~ * ~~When `Create Flag` is clicked, modify the title to indicate that we are in flagging mode and ROIs can be drawn by dragging.~~
~~3. Update recorded flags directory, and add provenance information to each flag (which instrument and channel belongs to).~~
* ~~Update `Commit Flag` logic:~~
* Record collected flag information initially in a YAML or JSON file. Is this faster than writing directly to the HDF5 file? ~~3. Update recorded flags directory, and add provenance information to each flag (which instrument and channel belongs to).~~
* Should we actively transfer collected flags by clicking a button? after commit button is pressed, each flag is now stored in an independent json file * Record collected flag information initially in a YAML or JSON file. Is this faster than writing directly to the HDF5 file?
* Enable some form of chunk storage and visualization from the HDF5 file. Iterate over chunks for faster display versus access time. * Should we actively transfer collected flags by clicking a button? after commit button is pressed, each flag is now stored in an independent json file
1. Do I need to modify DIMA?
2. What is a good chunk size? * Enable some form of chunk storage and visualization from the HDF5 file. Iterate over chunks for faster display versus access time.
3. What Dash component can we use to iterate over the chunks? 1. Do I need to modify DIMA?
2. What is a good chunk size?
![Screenshot](figures/flagging_app_screenshot.JPG) 3. What Dash component can we use to iterate over the chunks?
![Screenshot](figures/flagging_app_screenshot.JPG)