APOG_public/acsm-fairifier
0
0
Fork 0
mirror of https://gitea.psi.ch/APOG/acsm-fairifier.git synced 2026-08-31 11:10:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • e7863078aa Add data chain step /update_datachain_params.py. This uses params specified in input data folder to update current data chain params florez_j 2025-04-11 11:07:52 +02:00
  • cea83396d2 Add function to estimate project directory path florez_j 2025-04-11 11:05:41 +02:00
  • 68a7d5dc2b Fix out of bound error in pipelines/steps/visualize_datatable_vars.py florez_j 2025-04-11 11:04:00 +02:00
  • c9e25c2483 Add new placeholder for submitter name and remove metadata element 'station' conflicting with third party converter florez_j 2025-04-11 11:02:38 +02:00
  • 13bcd88a93 Replace station attribute with station_abbr, so it does not conflict with ebas database naming florez_j 2025-04-11 10:41:05 +02:00
  • 6c8cb61df2 I added some todos simon_l 2025-04-10 16:51:46 +02:00
  • b0b3ccef66 Update campaignDescriptor.yaml with originator and submitter info florez_j 2025-04-08 21:27:51 +02:00
  • 4725d85e61 Update data chain notebook florez_j 2025-04-08 21:26:36 +02:00
  • 881209bf82 Update steps to allow filenames for PAYERN station florez_j 2025-04-08 21:26:01 +02:00
  • 3dfed2c5f3 Update pipelines/steps/prepare_ebas_submission.py. Rmoved hard coded paths and build output name using metadata from campaign descriptor. Also, we can now specify month ranges florez_j 2025-04-08 18:44:46 +02:00
  • 5dd280e88c Update pipelines/steps/utils.py. Now we can load the campaign descriptor using load_project_yaml_files() florez_j 2025-04-08 18:43:01 +02:00
  • bfaaeb4416 Add header templates for JFJ and PAY florez_j 2025-04-08 18:35:44 +02:00
  • ca0366338a Add pipeline step to update actris_header with submitter and originator florez_j 2025-04-08 18:34:51 +02:00
  • 0e625b558a Update third party submodule florez_j 2025-04-08 15:46:18 +02:00
  • 00a85505c8 Fix bug when infering name of flags file from dataset name florez_j 2025-04-04 16:53:41 +02:00
  • dcf6b8ec41 Clean up and document steps in notebooks/demo_visualize_diagnostic_flags_from_hdf5_file.ipynb florez_j 2025-04-04 16:07:07 +02:00
  • d290daaa39 Update pipelines/steps/visualize_datatable_vars.py. Specify alternative path to flags in case they are not in the HDF5 file. We assume there is an append directory where they may be. Otherwise it throws and error. florez_j 2025-04-04 16:06:11 +02:00
  • b3b5516aed Fix typo in README.md florez_j 2025-04-04 09:46:36 +02:00
  • 17f497f1c5 Update README.md with instructions to set up network drive access credentials and update TODO.md florez_j 2025-04-02 14:40:55 +02:00
  • 9a23bf283a Fix output directory in campaignDescripor.yaml florez_j 2025-04-02 14:17:45 +02:00
  • 73c0409207 Update README.md with instruction to build, run, and stop docker container on which the data chain will run florez_j 2025-04-02 14:10:52 +02:00
  • 73648ebe31 Update Dockerfile and docker-compose.yaml. By default, container can use cifs to map network drives so no need to install, and added bash command to docker compose file to run both jupyterlab and dash board simultanously florez_j 2025-04-02 14:09:44 +02:00
  • c6b010702c Remove run_container.sh. This is now done by running the docker-compose.yaml file using docker florez_j 2025-04-02 11:27:42 +02:00
  • 643a305782 Update ACSM data chain workflow with markdown descriptions florez_j 2025-04-02 11:26:03 +02:00
  • f0da41e914 Add docker-compose.yaml to replace run_container.sh florez_j 2025-04-02 11:17:20 +02:00
  • 1e7f92d6e8 Update ignore files florez_j 2025-04-02 11:15:36 +02:00
  • 14cb24fb4b Add config files .sh .yaml and .md to dockerignore florez_j 2025-03-18 09:21:26 +01:00
  • aac6015986 Update README with Docker container workflow florez_j 2025-03-18 08:57:51 +01:00
  • b7cbf7dd14 Add bash script to run container florez_j 2025-03-18 08:57:06 +01:00
  • 41506519d3 WIP: Update Dockerfile with requirements.txt that downgrade versions so that ebas converter can run. We need to rethink the Dockerfile container workflow using docker compose florez_j 2025-03-18 08:55:52 +01:00
  • 862a2c46e6 Clean up notebooks florez_j 2025-03-18 08:53:51 +01:00
  • b0d68747e9 Update code to rearrange table columns based on column_order from acsm_to_ebas.yaml. This can be removed in the future. florez_j 2025-03-18 07:23:54 +01:00
  • 261c8b8154 Add column order attributes, storing the order in which variables need to be displayed to adhere to Nora's initial implementation. florez_j 2025-03-18 07:22:06 +01:00
  • f3010bff47 Fix import problem. The app didn't run from a notebooks/<>.ipynb. This however may not be needed any longer. Is it good practice to run dashboards from a jupyter notebook? florez_j 2025-03-18 07:14:44 +01:00
  • 395b734cd6 Fix bug reported in TODO.md. FLAGGING APP didnt record the parent channel correctly, that is the channel that originated a particular flag. An additional problem was detected and corrected where _metadata.json was confused as flag. florez_j 2025-03-17 14:21:26 +01:00
  • 895a44cc8c Update notebooks/demo_acsm_pipeline.ipynb by introducing global append and input file variables. florez_j 2025-03-14 15:26:37 +01:00
  • d5a89b2fbd Add more items to TODO.md florez_j 2025-03-14 15:25:47 +01:00
  • f90708b043 Update todo.md florez_j 2025-03-14 15:17:31 +01:00
  • 9504576fc0 Refactor steps using main functions from each step instead of subprocess with command line interface florez_j 2025-03-14 13:44:03 +01:00
  • b93cb7517b Add comment to indicate where params are defined in original code. florez_j 2025-03-14 13:40:22 +01:00
  • 7d277e3e3b Clean up print statements. florez_j 2025-03-14 13:39:12 +01:00
  • 39a9ab07a2 Refactor command line interface to run a main function on the parsed arguments. florez_j 2025-03-14 13:36:52 +01:00
  • 8cdd8a0771 Completed command line interface for pipelines/steps/prepare_ebas_submission.py. This finishes WIP associated with commit 2eb88e4. florez_j 2025-03-14 13:09:09 +01:00
  • bfc9f0ab82 Cross out item. Requirements of data input to converter are defined in pipelines/dictionaries/acsm_to_ebas.yaml florez_j 2025-03-14 13:05:29 +01:00
  • f3612267c1 Update dima to latest version. florez_j 2025-03-14 10:18:30 +01:00
  • b643eb2d26 Refactored command line interface, generate_species_flags, and created functions to improve performance and code clarity. florez_j 2025-03-13 14:17:46 +01:00
  • 97f253074b Add in try-except block a finally statement to close hdf5 file for secure file closing. This was done before in the try case which is not good practice florez_j 2025-03-13 14:06:01 +01:00
  • 0fe2b564b7 WIP: Change to copy command as COPY . /acsmdc. florez_j 2025-03-12 16:22:39 +01:00
  • 05cf34032f WIP: Add dockerignore and Dockerfile. I have built the Docker image and seems to be working fine but haven't tested all aspects of the applicaton florez_j 2025-03-12 16:15:49 +01:00
  • d6fa51effc Fix typo and missing key. 1) Update allowed yaml files in load_project_yaml_files by adding pipelines/params/calibration_factors.yaml. 2) Correct filename access after os.path.split operation. florez_j 2025-03-10 13:02:38 +01:00
  • 4ca99f4f04 Rewrote load_calib_file() in terms pipelines.steps.utils.load_project_yaml_files(). In the future we will replace it completely so we can validate and manage all yaml files in the chain in a more centralized manner. florez_j 2025-03-10 11:46:24 +01:00
  • ecdf3687ba Remove/replace def load_parameters(flag_type) with steps.utils.load_project_yaml_files(.). This is to have a more centralized treatment of yaml files inside the data chain. florez_j 2025-03-10 11:38:51 +01:00
  • cf073126b9 Updated acsm raw to ebas terminology mapping. It is now complete! florez_j 2025-03-07 16:53:22 +01:00
  • 2eb88e444c WIP: finished last commit, and tested the whole step. Now we need to implement the command line interface. florez_j 2025-03-07 16:51:06 +01:00
  • eb8a36dc8e Update pipelines/steps/apply_calibration_factors.py to generate error table and cleaned up command line interface. florez_j 2025-03-07 16:49:32 +01:00
  • cbad39355b Reformat LOD variable names for simplicity florez_j 2025-03-07 16:46:48 +01:00
  • 78340464aa Add functions: generate_error_dataframe() with missing values, metadata_dict_to_dataframe(), and load_project_yaml_files() to easily access data from yaml files in the project. florez_j 2025-03-07 16:45:33 +01:00
  • 93f49f7fd1 Updated third party submodule to latest version florez_j 2025-03-07 13:45:54 +01:00
  • 35e08d387f Add pipelines/params/station_params.yaml, where station dependent temperature and preassure are to be defined. Crossed out associated TODO.md item. florez_j 2025-03-07 11:23:48 +01:00
  • 5f9d72e9e1 Replace submodule:third_party/actris_acsm_converter with third_party/acsmProcessingSoftware at Nora/main branch. The idea is to add Nora's modification to original repo to maintain a clear provenance. florez_j 2025-03-07 09:41:23 +01:00
  • f035afbf3b Fix duplicate time stamp problems and added code to check for NaTs before ebas submission files.) florez_j 2025-03-04 09:02:37 +01:00
  • 9a23744f63 Add acsm to ebas dictionary florez_j 2025-03-04 09:01:09 +01:00
  • 6eccbb5018 Moved get_metadata() from pipelines/steps/prepare_ebas_submission.py to utils.py florez_j 2025-03-03 18:55:46 +01:00
  • def67a0562 Added correct calibration parameters to calibration factors table florez_j 2025-03-03 14:23:45 +01:00
  • d3a491ea85 Renamed pipeline step join_tables.py to prepare_ebas_submission.py for accuracy. This step performs more than joining tables, it also does renaming using dictionaries and generates the ebas data submission package florez_j 2025-03-03 10:48:03 +01:00
  • 877c0178d7 WIP: Added first version of join_tables and tested txt generation. We should probaly make this step encompasse name renaming for EBAS and rename step to be more generic. Change join_tables.py to generate_ebas_converter_input_files florez_j 2025-02-28 17:23:00 +01:00
  • a9533ae3e8 Refine function. Abstract out load parameters as function to simplify code. florez_j 2025-02-28 17:18:13 +01:00
  • 6890377134 Remove dataset_name parameter from command line interface. Instead i now infer it assuming there is a consistent input filename in the hdf5 file that can be found using hard coded keywords. florez_j 2025-02-28 17:17:33 +01:00
  • 499ba3d1c7 Update TODO.md florez_j 2025-02-27 09:14:19 +01:00
  • 33f29a8080 WIP: Completed flag generation for species, command line interface is now clear but still does not consider potentially existing flags. Review TODO.md, for efficiency we need to simplify flag assignment and reconciliation for the species flag type. florez_j 2025-02-26 15:11:12 +01:00
  • 600c60be36 Append variable specification for explicit definition of diagnostics and species to pipelines/params/calibration_params.yaml. This is to support flagging operations to particular variables florez_j 2025-02-26 15:07:58 +01:00
  • 6d5d74e616 Append flag_ranking to app/flags/ebas_dict.yaml. This is to disambiguate/overide conflicting flags florez_j 2025-02-26 15:05:06 +01:00
  • e6868da3d4 Update dima to latest version florez_j 2025-02-25 13:34:52 +01:00
  • 94cd536204 Updated dima to latest version florez_j 2025-02-22 18:08:14 +01:00
  • 4edd6809e9 WIP: Redesigned command line functionality. options --species is half baked :). It runs without error but second part needs validation. florez_j 2025-02-22 16:00:27 +01:00
  • e24d451542 Updated thirdparty submodule to latest version florez_j 2025-02-22 15:02:17 +01:00
  • 456f4c297f Adjust code due to changes in flag definition location. florez_j 2025-02-20 17:28:43 +01:00
  • 60e1c35745 Move flag definitions from ...utils.py to flags/ebas_flags.yaml florez_j 2025-02-20 17:26:50 +01:00
  • dc34e07f62 Add actris_acsm_converter as a third_party submodule florez_j 2025-02-20 16:07:47 +01:00
  • ae15333207 Update layout figure florez_j 2025-02-17 15:46:40 +01:00
  • de33977e71 Start restructuring of app to make it more modular, and improve layout a bit florez_j 2025-02-17 15:44:51 +01:00
  • 0454b54b8c Fix bug: When flag folder empty, it now returns [] (as opposed to None) florez_j 2025-02-17 15:42:57 +01:00
  • 0911260f26 Save changes. florez_j 2025-02-15 18:20:58 +01:00
  • 25f3ee12a4 Fix bug. Saved flags from apps were not associated with right parent variable. florez_j 2025-02-15 18:10:29 +01:00
  • 46767996b8 Modify experimental campaign descriptor to point to yearly acsm_tofware data. florez_j 2025-02-15 18:08:14 +01:00
  • a516479036 Change data lineage format. Now, data lineage information is accessed using name of the file as key. florez_j 2025-02-15 17:57:50 +01:00
  • 3123fec9da Add new version of dash board graphical user interphase florez_j 2025-02-13 19:59:20 +01:00
  • 10fdd1b8c7 Fix bug when saving flags to flagsFolder and subsequent visualization of existing flags. florez_j 2025-02-13 19:58:30 +01:00
  • e53d8dec8c Simplify load_flags() function by removing redundant h5 verification step florez_j 2025-02-13 19:56:44 +01:00
  • 11ecdebf1b Move flagging app to app folder with initialized callbacks/ and components/ modules florez_j 2025-02-13 18:13:45 +01:00
  • 479c11d0d4 Move app scripts to app folder. This is to improve modularity of the app florez_j 2025-02-13 18:03:41 +01:00
  • 1a86dc2065 Implement instrument, file, and variable selection dropdown menus. florez_j 2025-02-13 17:56:08 +01:00
  • 79f9badb1c Update dima submodule to latest version. florez_j 2025-02-11 13:16:14 +01:00
  • f2d46101b1 Add demo jupyter notebook to visualize variables and their associated flags florez_j 2025-02-11 13:15:19 +01:00
  • ca08044569 Update keys of data_lineage_metadata.yaml. keys are now the filenames in the folder. florez_j 2025-02-11 13:14:14 +01:00
  • 80fbe3af91 Update params/validity_thresholds.yaml with values from 2024 JFJ ACSM data florez_j 2025-02-11 13:12:00 +01:00
  • 0623c2462b Adjust command line interface of steps/apply_... and steps/compute_aut... florez_j 2025-02-11 13:07:29 +01:00
  • f2854d4ca0 Implement pipelines/steps/visualize_datatable_vars.py florez_j 2025-02-11 13:03:46 +01:00
  • e7ad4d2a36 Add new demo for acsm pipeline and clearout the other notebooks florez_j 2025-02-10 16:15:56 +01:00
  • 3c482c1d21 Rename and move jupyter notebook demos to notebooks folder florez_j 2025-02-10 16:14:49 +01:00