APOG_public/acsmnode
0
0
Fork 0
mirror of https://gitea.psi.ch/APOG/acsmnode.git synced 2025-06-24 13:11:08 +02:00
Code Issues Packages Projects Releases Wiki Activity

Update TODO.md. Added some todos and deleted one that is done now.

Browse Source
This commit is contained in:
simon_l
2025-05-28 11:48:26 +02:00
parent 196e237838
commit 3a23db1a12
1 changed files with 47 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

92
TODO.md
View File

@ -1,45 +1,47 @@
# TODO
* [from Leïla] Correct error when flags are loaded in the flagging app
* [from Leïla] End the file at end of year (or filter only current year)
* [from Leïla] Change "9999.999" to "9999999.9999" in header
* [from Leïla] Update Detection limit values in L2 header: take the ones (1h) from limits_of_detection.yaml
* [from Leïla] For PAY, calculate error as 50% of concentration
* [from Leïla] Correct errors (uncertainties) that they can't be lower than 0.0001
* [from Leïla] Change flow rate values to 10% of flow rate ref
* [New] Create data flows to validate and replace existing data chain params. data/campaignData/param/ -> pipelines/params and campaignDescriptor.yaml -> acsm data converter.
* [New] DIMA. Add dictionaries to explain variables at different levels.
* [New] DIMA. Modify data integration pipeline to not add current datetime in filename when not specified.
* [New] DIMA. Set data/ folder as default and only possible output folder
* [New] DIMA. Review DataManager File Update. Generated CSV files are not transfering properly.
* [New] DIMA. Ensure code snippets that open and close HDF5 files do so securly and do not leave the file locked
* [New] DIMA. Revise default csv file reader and enhance it to infer datetime column and format.
* [New] Learn how to run docker-compose.yaml
* [New] EBAS_SUBMISSION_STEP. Extend time filter to ranges, create a merge data frame function, and construct file name and output dir dynammically. It is currently hardcoded.
* [New] Finish command line interface for visualize_datatable_vars and add modes, --flags, --dataset, and save to figures folder in repo
* Implement flagging-app specific data operations such as:
1. [New item] When verify flags from checklist is active, enable delete-flag button to delete flag associated with active cell on table.
2. [New item] When verify and ready to trasnfer items on checklist are active, enable record-flags button to record verified flags into the HDF5 file.
3. [New item] When all checklist items active, enable apply button to apply flags to the time series data and save it to the HDF5 file.
1. ~~Define data manager obj with apply flags behavior.~~
2. Define metadata answering who did the flagging and quality assurance tests?
3. Update intruments/dictionaries/ACSM_TOFWARE_flags.yaml and instruments/readers/flag_reader.py to describe metadata elements based on dictionary.
4. ~~Update DIMA data integration pipeline to allowed user-defined file naming template~~
5. ~~Design and implement flag visualization feature: click flag on table and display on figure shaded region when feature is enabled~~
6. Implement schema validator on yaml/json representation of hdf5 metadata
7. Implement updates to 'actris level' and 'processing_script' after operation applied to data/file?
* ~~When `Create Flag` is clicked, modify the title to indicate that we are in flagging mode and ROIs can be drawn by dragging.~~
* ~~Update `Commit Flag` logic:~~
~~3. Update recorded flags directory, and add provenance information to each flag (which instrument and channel belongs to).~~
* Record collected flag information initially in a YAML or JSON file. Is this faster than writing directly to the HDF5 file?
* Should we actively transfer collected flags by clicking a button? after commit button is pressed, each flag is now stored in an independent json file
* Enable some form of chunk storage and visualization from the HDF5 file. Iterate over chunks for faster display versus access time.
1. Do I need to modify DIMA?
2. What is a good chunk size?
3. What Dash component can we use to iterate over the chunks?
![Screenshot](figures/flagging_app_screenshot.JPG)
# TODO
* [from Leïla] Data Flagging App. Correct error when flags are loaded in the app.
* [from Leïla] Before Step 5. The file should contain only data of the current year.
* [from Leïla] Ebas converter. Change "9999.999" to "9999999.9999" in header
* [from Leïla] Ebas converter. Update Detection limit values in L2 header: take the ones (1h) from limits_of_detection.yaml
* [from Leïla] Ebas converter. Correct errors (uncertainties) that they can't be lower than 0.0001
* [from Leïla] Flag using validity threshold. Change flow rate values to 10% of flow rate ref.
* [from Leïla] In Step 1. Creation of a new collection should be an option and not automatic.
* [from Leïla] The data chain (except Step 5) should also include Org_mx and Ord_err. To discuss together.
* [from Leïla] Step 4.1. Add a step to verify ACSM data with external instruments (MPSS, eBC). To discuss together.
* [New] Create data flows to validate and replace existing data chain params. data/campaignData/param/ -> pipelines/params and campaignDescriptor.yaml -> acsm data converter.
* [New] DIMA. Add dictionaries to explain variables at different levels.
* [New] DIMA. Modify data integration pipeline to not add current datetime in filename when not specified.
* [New] DIMA. Set data/ folder as default and only possible output folder
* [New] DIMA. Review DataManager File Update. Generated CSV files are not transfering properly.
* [New] DIMA. Ensure code snippets that open and close HDF5 files do so securly and do not leave the file locked
* [New] DIMA. Revise default csv file reader and enhance it to infer datetime column and format.
* [New] Learn how to run docker-compose.yaml
* [New] EBAS_SUBMISSION_STEP. Extend time filter to ranges, create a merge data frame function, and construct file name and output dir dynammically. It is currently hardcoded.
* [New] Finish command line interface for visualize_datatable_vars and add modes, --flags, --dataset, and save to figures folder in repo
* Implement flagging-app specific data operations such as:
1. [New item] When verify flags from checklist is active, enable delete-flag button to delete flag associated with active cell on table.
2. [New item] When verify and ready to trasnfer items on checklist are active, enable record-flags button to record verified flags into the HDF5 file.
3. [New item] When all checklist items active, enable apply button to apply flags to the time series data and save it to the HDF5 file.
1. ~~Define data manager obj with apply flags behavior.~~
2. Define metadata answering who did the flagging and quality assurance tests?
3. Update intruments/dictionaries/ACSM_TOFWARE_flags.yaml and instruments/readers/flag_reader.py to describe metadata elements based on dictionary.
4. ~~Update DIMA data integration pipeline to allowed user-defined file naming template~~
5. ~~Design and implement flag visualization feature: click flag on table and display on figure shaded region when feature is enabled~~
6. Implement schema validator on yaml/json representation of hdf5 metadata
7. Implement updates to 'actris level' and 'processing_script' after operation applied to data/file?
* ~~When `Create Flag` is clicked, modify the title to indicate that we are in flagging mode and ROIs can be drawn by dragging.~~
* ~~Update `Commit Flag` logic:~~
~~3. Update recorded flags directory, and add provenance information to each flag (which instrument and channel belongs to).~~
* Record collected flag information initially in a YAML or JSON file. Is this faster than writing directly to the HDF5 file?
* Should we actively transfer collected flags by clicking a button? after commit button is pressed, each flag is now stored in an independent json file
* Enable some form of chunk storage and visualization from the HDF5 file. Iterate over chunks for faster display versus access time.
1. Do I need to modify DIMA?
2. What is a good chunk size?
3. What Dash component can we use to iterate over the chunks?
![Screenshot](figures/flagging_app_screenshot.JPG)