Commit Graph

Select branches

Hide Pull Requests

changes_leila

main

more_changes

renku/main

#1

Mono Color

• d290daaa39 Update pipelines/steps/visualize_datatable_vars.py. Specify alternative path to flags in case they are not in the HDF5 file. We assume there is an append directory where they may be. Otherwise it throws and error. Florez Ospina Juan Felipe 2025-04-04 16:06:11 +02:00
• b3b5516aed Fix typo in README.md Florez Ospina Juan Felipe 2025-04-04 09:46:36 +02:00
• 17f497f1c5 Update README.md with instructions to set up network drive access credentials and update TODO.md Florez Ospina Juan Felipe 2025-04-02 14:40:55 +02:00
• 9a23bf283a Fix output directory in campaignDescripor.yaml Florez Ospina Juan Felipe 2025-04-02 14:17:45 +02:00
• 73c0409207 Update README.md with instruction to build, run, and stop docker container on which the data chain will run Florez Ospina Juan Felipe 2025-04-02 14:10:52 +02:00
• 73648ebe31 Update Dockerfile and docker-compose.yaml. By default, container can use cifs to map network drives so no need to install, and added bash command to docker compose file to run both jupyterlab and dash board simultanously Florez Ospina Juan Felipe 2025-04-02 14:09:44 +02:00
• c6b010702c Remove run_container.sh. This is now done by running the docker-compose.yaml file using docker Florez Ospina Juan Felipe 2025-04-02 11:27:42 +02:00
• 643a305782 Update ACSM data chain workflow with markdown descriptions Florez Ospina Juan Felipe 2025-04-02 11:26:03 +02:00
• f0da41e914 Add docker-compose.yaml to replace run_container.sh Florez Ospina Juan Felipe 2025-04-02 11:17:20 +02:00
• 1e7f92d6e8 Update ignore files Florez Ospina Juan Felipe 2025-04-02 11:15:36 +02:00
• 14cb24fb4b Add config files .sh .yaml and .md to dockerignore Florez Ospina Juan Felipe 2025-03-18 09:21:26 +01:00
• aac6015986 Update README with Docker container workflow Florez Ospina Juan Felipe 2025-03-18 08:57:51 +01:00
• b7cbf7dd14 Add bash script to run container Florez Ospina Juan Felipe 2025-03-18 08:57:06 +01:00
• 41506519d3 WIP: Update Dockerfile with requirements.txt that downgrade versions so that ebas converter can run. We need to rethink the Dockerfile container workflow using docker compose Florez Ospina Juan Felipe 2025-03-18 08:55:52 +01:00
• 862a2c46e6 Clean up notebooks Florez Ospina Juan Felipe 2025-03-18 08:53:51 +01:00
• b0d68747e9 Update code to rearrange table columns based on column_order from acsm_to_ebas.yaml. This can be removed in the future. Florez Ospina Juan Felipe 2025-03-18 07:23:54 +01:00
• 261c8b8154 Add column order attributes, storing the order in which variables need to be displayed to adhere to Nora's initial implementation. Florez Ospina Juan Felipe 2025-03-18 07:22:06 +01:00
• f3010bff47 Fix import problem. The app didn't run from a notebooks/<>.ipynb. This however may not be needed any longer. Is it good practice to run dashboards from a jupyter notebook? Florez Ospina Juan Felipe 2025-03-18 07:14:44 +01:00
• 395b734cd6 Fix bug reported in TODO.md. FLAGGING APP didnt record the parent channel correctly, that is the channel that originated a particular flag. An additional problem was detected and corrected where _metadata.json was confused as flag. Florez Ospina Juan Felipe 2025-03-17 14:21:26 +01:00
• 895a44cc8c Update notebooks/demo_acsm_pipeline.ipynb by introducing global append and input file variables. Florez Ospina Juan Felipe 2025-03-14 15:26:37 +01:00
• d5a89b2fbd Add more items to TODO.md Florez Ospina Juan Felipe 2025-03-14 15:25:47 +01:00
• f90708b043 Update todo.md Florez Ospina Juan Felipe 2025-03-14 15:17:31 +01:00
• 9504576fc0 Refactor steps using main functions from each step instead of subprocess with command line interface Florez Ospina Juan Felipe 2025-03-14 13:44:03 +01:00
• b93cb7517b Add comment to indicate where params are defined in original code. Florez Ospina Juan Felipe 2025-03-14 13:40:22 +01:00
• 7d277e3e3b Clean up print statements. Florez Ospina Juan Felipe 2025-03-14 13:39:12 +01:00
• 39a9ab07a2 Refactor command line interface to run a main function on the parsed arguments. Florez Ospina Juan Felipe 2025-03-14 13:36:52 +01:00
• 8cdd8a0771 Completed command line interface for pipelines/steps/prepare_ebas_submission.py. This finishes WIP associated with commit 2eb88e4. Florez Ospina Juan Felipe 2025-03-14 13:09:09 +01:00
• bfc9f0ab82 Cross out item. Requirements of data input to converter are defined in pipelines/dictionaries/acsm_to_ebas.yaml Florez Ospina Juan Felipe 2025-03-14 13:05:29 +01:00
• f3612267c1 Update dima to latest version. Florez Ospina Juan Felipe 2025-03-14 10:18:30 +01:00
• b643eb2d26 Refactored command line interface, generate_species_flags, and created functions to improve performance and code clarity. Florez Ospina Juan Felipe 2025-03-13 14:17:46 +01:00
• 97f253074b Add in try-except block a finally statement to close hdf5 file for secure file closing. This was done before in the try case which is not good practice Florez Ospina Juan Felipe 2025-03-13 14:06:01 +01:00
• 0fe2b564b7 WIP: Change to copy command as COPY . /acsmdc. Florez Ospina Juan Felipe 2025-03-12 16:22:39 +01:00
• 05cf34032f WIP: Add dockerignore and Dockerfile. I have built the Docker image and seems to be working fine but haven't tested all aspects of the applicaton Florez Ospina Juan Felipe 2025-03-12 16:15:49 +01:00
• d6fa51effc Fix typo and missing key. 1) Update allowed yaml files in load_project_yaml_files by adding pipelines/params/calibration_factors.yaml. 2) Correct filename access after os.path.split operation. Florez Ospina Juan Felipe 2025-03-10 13:02:38 +01:00
• 4ca99f4f04 Rewrote load_calib_file() in terms pipelines.steps.utils.load_project_yaml_files(). In the future we will replace it completely so we can validate and manage all yaml files in the chain in a more centralized manner. Florez Ospina Juan Felipe 2025-03-10 11:46:24 +01:00
• ecdf3687ba Remove/replace def load_parameters(flag_type) with steps.utils.load_project_yaml_files(.). This is to have a more centralized treatment of yaml files inside the data chain. Florez Ospina Juan Felipe 2025-03-10 11:38:51 +01:00
• cf073126b9 Updated acsm raw to ebas terminology mapping. It is now complete! Florez Ospina Juan Felipe 2025-03-07 16:53:22 +01:00
• 2eb88e444c WIP: finished last commit, and tested the whole step. Now we need to implement the command line interface. Florez Ospina Juan Felipe 2025-03-07 16:51:06 +01:00
• eb8a36dc8e Update pipelines/steps/apply_calibration_factors.py to generate error table and cleaned up command line interface. Florez Ospina Juan Felipe 2025-03-07 16:49:32 +01:00
• cbad39355b Reformat LOD variable names for simplicity Florez Ospina Juan Felipe 2025-03-07 16:46:48 +01:00
• 78340464aa Add functions: generate_error_dataframe() with missing values, metadata_dict_to_dataframe(), and load_project_yaml_files() to easily access data from yaml files in the project. Florez Ospina Juan Felipe 2025-03-07 16:45:33 +01:00
• 93f49f7fd1 Updated third party submodule to latest version Florez Ospina Juan Felipe 2025-03-07 13:45:54 +01:00
• 35e08d387f Add pipelines/params/station_params.yaml, where station dependent temperature and preassure are to be defined. Crossed out associated TODO.md item. Florez Ospina Juan Felipe 2025-03-07 11:23:48 +01:00
• 5f9d72e9e1 Replace submodule:third_party/actris_acsm_converter with third_party/acsmProcessingSoftware at Nora/main branch. The idea is to add Nora's modification to original repo to maintain a clear provenance. Florez Ospina Juan Felipe 2025-03-07 09:41:23 +01:00
• f035afbf3b Fix duplicate time stamp problems and added code to check for NaTs before ebas submission files.) Florez Ospina Juan Felipe 2025-03-04 09:02:37 +01:00
• 9a23744f63 Add acsm to ebas dictionary Florez Ospina Juan Felipe 2025-03-04 09:01:09 +01:00
• 6eccbb5018 Moved get_metadata() from pipelines/steps/prepare_ebas_submission.py to utils.py Florez Ospina Juan Felipe 2025-03-03 18:55:46 +01:00
• def67a0562 Added correct calibration parameters to calibration factors table Florez Ospina Juan Felipe 2025-03-03 14:23:45 +01:00
• d3a491ea85 Renamed pipeline step join_tables.py to prepare_ebas_submission.py for accuracy. This step performs more than joining tables, it also does renaming using dictionaries and generates the ebas data submission package Florez Ospina Juan Felipe 2025-03-03 10:48:03 +01:00
• 877c0178d7 WIP: Added first version of join_tables and tested txt generation. We should probaly make this step encompasse name renaming for EBAS and rename step to be more generic. Change join_tables.py to generate_ebas_converter_input_files Florez Ospina Juan Felipe 2025-02-28 17:23:00 +01:00
• a9533ae3e8 Refine function. Abstract out load parameters as function to simplify code. Florez Ospina Juan Felipe 2025-02-28 17:18:13 +01:00
• 6890377134 Remove dataset_name parameter from command line interface. Instead i now infer it assuming there is a consistent input filename in the hdf5 file that can be found using hard coded keywords. Florez Ospina Juan Felipe 2025-02-28 17:17:33 +01:00
• 499ba3d1c7 Update TODO.md Florez Ospina Juan Felipe 2025-02-27 09:14:19 +01:00
• 33f29a8080 WIP: Completed flag generation for species, command line interface is now clear but still does not consider potentially existing flags. Review TODO.md, for efficiency we need to simplify flag assignment and reconciliation for the species flag type. Florez Ospina Juan Felipe 2025-02-26 15:11:12 +01:00
• 600c60be36 Append variable specification for explicit definition of diagnostics and species to pipelines/params/calibration_params.yaml. This is to support flagging operations to particular variables Florez Ospina Juan Felipe 2025-02-26 15:07:58 +01:00
• 6d5d74e616 Append flag_ranking to app/flags/ebas_dict.yaml. This is to disambiguate/overide conflicting flags Florez Ospina Juan Felipe 2025-02-26 15:05:06 +01:00
• e6868da3d4 Update dima to latest version Florez Ospina Juan Felipe 2025-02-25 13:34:52 +01:00
• 94cd536204 Updated dima to latest version Florez Ospina Juan Felipe 2025-02-22 18:08:14 +01:00
• 4edd6809e9 WIP: Redesigned command line functionality. options --species is half baked :). It runs without error but second part needs validation. Juan Felipe Florez Ospina 2025-02-22 16:00:27 +01:00
• e24d451542 Updated thirdparty submodule to latest version Juan Felipe Florez Ospina 2025-02-22 15:02:17 +01:00
• 456f4c297f Adjust code due to changes in flag definition location. Florez Ospina Juan Felipe 2025-02-20 17:28:43 +01:00
• 60e1c35745 Move flag definitions from ...utils.py to flags/ebas_flags.yaml Florez Ospina Juan Felipe 2025-02-20 17:26:50 +01:00
• dc34e07f62 Add actris_acsm_converter as a third_party submodule Florez Ospina Juan Felipe 2025-02-20 16:07:47 +01:00
• ae15333207 Update layout figure Juan Felipe Florez Ospina 2025-02-17 15:46:40 +01:00
• de33977e71 Start restructuring of app to make it more modular, and improve layout a bit Juan Felipe Florez Ospina 2025-02-17 15:44:51 +01:00
• 0454b54b8c Fix bug: When flag folder empty, it now returns [] (as opposed to None) Juan Felipe Florez Ospina 2025-02-17 15:42:57 +01:00
• 0911260f26 Save changes. Juan Felipe Florez Ospina 2025-02-15 18:20:58 +01:00
• 25f3ee12a4 Fix bug. Saved flags from apps were not associated with right parent variable. Juan Felipe Florez Ospina 2025-02-15 18:10:29 +01:00
• 46767996b8 Modify experimental campaign descriptor to point to yearly acsm_tofware data. Juan Felipe Florez Ospina 2025-02-15 18:08:14 +01:00
• a516479036 Change data lineage format. Now, data lineage information is accessed using name of the file as key. Juan Felipe Florez Ospina 2025-02-15 17:57:50 +01:00
• 3123fec9da Add new version of dash board graphical user interphase Florez Ospina Juan Felipe 2025-02-13 19:59:20 +01:00
• 10fdd1b8c7 Fix bug when saving flags to flagsFolder and subsequent visualization of existing flags. Florez Ospina Juan Felipe 2025-02-13 19:58:30 +01:00
• e53d8dec8c Simplify load_flags() function by removing redundant h5 verification step Florez Ospina Juan Felipe 2025-02-13 19:56:44 +01:00
• 11ecdebf1b Move flagging app to app folder with initialized callbacks/ and components/ modules Florez Ospina Juan Felipe 2025-02-13 18:13:45 +01:00
• 479c11d0d4 Move app scripts to app folder. This is to improve modularity of the app Florez Ospina Juan Felipe 2025-02-13 18:03:41 +01:00
• 1a86dc2065 Implement instrument, file, and variable selection dropdown menus. Florez Ospina Juan Felipe 2025-02-13 17:56:08 +01:00
• 79f9badb1c Update dima submodule to latest version. Florez Ospina Juan Felipe 2025-02-11 13:16:14 +01:00
• f2d46101b1 Add demo jupyter notebook to visualize variables and their associated flags Florez Ospina Juan Felipe 2025-02-11 13:15:19 +01:00
• ca08044569 Update keys of data_lineage_metadata.yaml. keys are now the filenames in the folder. Florez Ospina Juan Felipe 2025-02-11 13:14:14 +01:00
• 80fbe3af91 Update params/validity_thresholds.yaml with values from 2024 JFJ ACSM data Florez Ospina Juan Felipe 2025-02-11 13:12:00 +01:00
• 0623c2462b Adjust command line interface of steps/apply_... and steps/compute_aut... Florez Ospina Juan Felipe 2025-02-11 13:07:29 +01:00
• f2854d4ca0 Implement pipelines/steps/visualize_datatable_vars.py Florez Ospina Juan Felipe 2025-02-11 13:03:46 +01:00
• e7ad4d2a36 Add new demo for acsm pipeline and clearout the other notebooks Florez Ospina Juan Felipe 2025-02-10 16:15:56 +01:00
• 3c482c1d21 Rename and move jupyter notebook demos to notebooks folder Florez Ospina Juan Felipe 2025-02-10 16:14:49 +01:00
• b966151d9c Update with template for compute_cpc_flags(). This still needs to be completed. Florez Ospina Juan Felipe 2025-02-10 10:54:18 +01:00
• ca2b8b6787 Rename apply_diagnostic_flags.py to compute_automated_flags.py because the function also computes both cpc and diagnostic based flags Florez Ospina Juan Felipe 2025-02-10 10:45:55 +01:00
• e2958b608f Update dima to latest version Florez Ospina Juan Felipe 2025-02-08 14:52:27 +01:00
• d93003f5c4 Add import statement and adjust dependencies. Juan Felipe Florez Ospina 2025-02-06 16:57:46 +01:00
• ad95d11a0d Implement utils module for reusable functions across steps in the pipelines Juan Felipe Florez Ospina 2025-02-06 16:54:36 +01:00
• 6ed9fe1eb1 Rename pipeline step to apply_diagnositc_flags, it perform a check based on specified thresholds and then assigns I, V, or M labels producing a labeling of a variable. Juan Felipe Florez Ospina 2025-02-06 16:25:40 +01:00
• f8b900422b Rename function Juan Felipe Florez Ospina 2025-02-06 16:15:19 +01:00
• 5e979b4e5e Update dima submodule to latest version Juan Felipe Florez Ospina 2025-02-06 16:08:35 +01:00
• 2c2b154528 Update to account for yaml file attribute renamings. Juan Felipe Florez Ospina 2025-02-05 18:15:43 +01:00
• e2b5aa9d69 Update file structure to depend on calibration_parameters.yaml and update thresholds for JFJ/ACSM_TOFWARE/2024 data. Juan Felipe Florez Ospina 2025-02-05 17:05:07 +01:00
• afd6ef4d1b Fix typo in denominators. Juan Felipe Florez Ospina 2025-02-05 16:49:36 +01:00
• bc779bd583 Update calibration parameters for JFJ/ACSM_TOFWARE/2024 data. Juan Felipe Florez Ospina 2025-02-05 16:48:14 +01:00
• 48e2127fef Reimplement apply_calibration_factors() in terms of load_calibration_file() and compute_calibration_factors() to reduce complexity and improve modularity. Juan Felipe Florez Ospina 2025-02-04 18:12:42 +01:00
• d4681b32ca Update dima to latest version Juan Felipe Florez Ospina 2025-02-04 16:50:37 +01:00
• a5a9fa9c3f Add anchor to default params from tofware and use alias to complete interval specific params coming from time period specific calibrations Juan Felipe Florez Ospina 2025-02-04 14:31:42 +01:00
• 484eb1ae2c Change path to file to be relative with respect to project directory path. Juan Felipe Florez Ospina 2025-02-03 16:13:19 +01:00