 florez_j
|
14cb24fb4b
|
Add config files .sh .yaml and .md to dockerignore
|
2025-03-18 09:21:26 +01:00 |
|
|
florez_j
|
aac6015986
|
Update README with Docker container workflow
|
2025-03-18 08:57:51 +01:00 |
|
|
florez_j
|
b7cbf7dd14
|
Add bash script to run container
|
2025-03-18 08:57:06 +01:00 |
|
|
florez_j
|
41506519d3
|
WIP: Update Dockerfile with requirements.txt that downgrade versions so that ebas converter can run. We need to rethink the Dockerfile container workflow using docker compose
|
2025-03-18 08:55:52 +01:00 |
|
|
florez_j
|
862a2c46e6
|
Clean up notebooks
|
2025-03-18 08:53:51 +01:00 |
|
|
florez_j
|
b0d68747e9
|
Update code to rearrange table columns based on column_order from acsm_to_ebas.yaml. This can be removed in the future.
|
2025-03-18 07:23:54 +01:00 |
|
|
florez_j
|
261c8b8154
|
Add column order attributes, storing the order in which variables need to be displayed to adhere to Nora's initial implementation.
|
2025-03-18 07:22:06 +01:00 |
|
|
florez_j
|
f3010bff47
|
Fix import problem. The app didn't run from a notebooks/<>.ipynb. This however may not be needed any longer. Is it good practice to run dashboards from a jupyter notebook?
|
2025-03-18 07:14:44 +01:00 |
|
|
florez_j
|
395b734cd6
|
Fix bug reported in TODO.md. FLAGGING APP didnt record the parent channel correctly, that is the channel that originated a particular flag. An additional problem was detected and corrected where _metadata.json was confused as flag.
|
2025-03-17 14:21:26 +01:00 |
|
|
florez_j
|
895a44cc8c
|
Update notebooks/demo_acsm_pipeline.ipynb by introducing global append and input file variables.
|
2025-03-14 15:26:37 +01:00 |
|
|
florez_j
|
d5a89b2fbd
|
Add more items to TODO.md
|
2025-03-14 15:25:47 +01:00 |
|
|
florez_j
|
f90708b043
|
Update todo.md
|
2025-03-14 15:17:31 +01:00 |
|
|
florez_j
|
9504576fc0
|
Refactor steps using main functions from each step instead of subprocess with command line interface
|
2025-03-14 13:44:03 +01:00 |
|
|
florez_j
|
b93cb7517b
|
Add comment to indicate where params are defined in original code.
|
2025-03-14 13:40:22 +01:00 |
|
|
florez_j
|
7d277e3e3b
|
Clean up print statements.
|
2025-03-14 13:39:12 +01:00 |
|
|
florez_j
|
39a9ab07a2
|
Refactor command line interface to run a main function on the parsed arguments.
|
2025-03-14 13:36:52 +01:00 |
|
|
florez_j
|
8cdd8a0771
|
Completed command line interface for pipelines/steps/prepare_ebas_submission.py. This finishes WIP associated with commit 2eb88e4.
|
2025-03-14 13:09:09 +01:00 |
|
|
florez_j
|
bfc9f0ab82
|
Cross out item. Requirements of data input to converter are defined in pipelines/dictionaries/acsm_to_ebas.yaml
|
2025-03-14 13:05:29 +01:00 |
|
|
florez_j
|
f3612267c1
|
Update dima to latest version.
|
2025-03-14 10:18:30 +01:00 |
|
|
florez_j
|
b643eb2d26
|
Refactored command line interface, generate_species_flags, and created functions to improve performance and code clarity.
|
2025-03-13 14:17:46 +01:00 |
|
|
florez_j
|
97f253074b
|
Add in try-except block a finally statement to close hdf5 file for secure file closing. This was done before in the try case which is not good practice
|
2025-03-13 14:06:01 +01:00 |
|
|
florez_j
|
0fe2b564b7
|
WIP: Change to copy command as COPY . /acsmdc.
|
2025-03-12 16:22:39 +01:00 |
|
|
florez_j
|
05cf34032f
|
WIP: Add dockerignore and Dockerfile. I have built the Docker image and seems to be working fine but haven't tested all aspects of the applicaton
|
2025-03-12 16:15:49 +01:00 |
|
|
florez_j
|
d6fa51effc
|
Fix typo and missing key. 1) Update allowed yaml files in load_project_yaml_files by adding pipelines/params/calibration_factors.yaml. 2) Correct filename access after os.path.split operation.
|
2025-03-10 13:02:38 +01:00 |
|
|
florez_j
|
4ca99f4f04
|
Rewrote load_calib_file() in terms pipelines.steps.utils.load_project_yaml_files(). In the future we will replace it completely so we can validate and manage all yaml files in the chain in a more centralized manner.
|
2025-03-10 11:46:24 +01:00 |
|
|
florez_j
|
ecdf3687ba
|
Remove/replace def load_parameters(flag_type) with steps.utils.load_project_yaml_files(.). This is to have a more centralized treatment of yaml files inside the data chain.
|
2025-03-10 11:38:51 +01:00 |
|
|
florez_j
|
cf073126b9
|
Updated acsm raw to ebas terminology mapping. It is now complete!
|
2025-03-07 16:53:22 +01:00 |
|
|
florez_j
|
2eb88e444c
|
WIP: finished last commit, and tested the whole step. Now we need to implement the command line interface.
|
2025-03-07 16:51:06 +01:00 |
|
|
florez_j
|
eb8a36dc8e
|
Update pipelines/steps/apply_calibration_factors.py to generate error table and cleaned up command line interface.
|
2025-03-07 16:49:32 +01:00 |
|
|
florez_j
|
cbad39355b
|
Reformat LOD variable names for simplicity
|
2025-03-07 16:46:48 +01:00 |
|
|
florez_j
|
78340464aa
|
Add functions: generate_error_dataframe() with missing values, metadata_dict_to_dataframe(), and load_project_yaml_files() to easily access data from yaml files in the project.
|
2025-03-07 16:45:33 +01:00 |
|
|
florez_j
|
93f49f7fd1
|
Updated third party submodule to latest version
|
2025-03-07 13:45:54 +01:00 |
|
|
florez_j
|
35e08d387f
|
Add pipelines/params/station_params.yaml, where station dependent temperature and preassure are to be defined. Crossed out associated TODO.md item.
|
2025-03-07 11:23:48 +01:00 |
|
|
florez_j
|
5f9d72e9e1
|
Replace submodule:third_party/actris_acsm_converter with third_party/acsmProcessingSoftware at Nora/main branch. The idea is to add Nora's modification to original repo to maintain a clear provenance.
|
2025-03-07 09:41:23 +01:00 |
|
|
florez_j
|
f035afbf3b
|
Fix duplicate time stamp problems and added code to check for NaTs before ebas submission files.)
|
2025-03-04 09:02:37 +01:00 |
|
|
florez_j
|
9a23744f63
|
Add acsm to ebas dictionary
|
2025-03-04 09:01:09 +01:00 |
|
|
florez_j
|
6eccbb5018
|
Moved get_metadata() from pipelines/steps/prepare_ebas_submission.py to utils.py
|
2025-03-03 18:55:46 +01:00 |
|
|
florez_j
|
def67a0562
|
Added correct calibration parameters to calibration factors table
|
2025-03-03 14:23:45 +01:00 |
|
|
florez_j
|
d3a491ea85
|
Renamed pipeline step join_tables.py to prepare_ebas_submission.py for accuracy. This step performs more than joining tables, it also does renaming using dictionaries and generates the ebas data submission package
|
2025-03-03 10:48:03 +01:00 |
|
|
florez_j
|
877c0178d7
|
WIP: Added first version of join_tables and tested txt generation. We should probaly make this step encompasse name renaming for EBAS and rename step to be more generic. Change join_tables.py to generate_ebas_converter_input_files
|
2025-02-28 17:23:00 +01:00 |
|
|
florez_j
|
a9533ae3e8
|
Refine function. Abstract out load parameters as function to simplify code.
|
2025-02-28 17:18:13 +01:00 |
|
|
florez_j
|
6890377134
|
Remove dataset_name parameter from command line interface. Instead i now infer it assuming there is a consistent input filename in the hdf5 file that can be found using hard coded keywords.
|
2025-02-28 17:17:33 +01:00 |
|
|
florez_j
|
499ba3d1c7
|
Update TODO.md
|
2025-02-27 09:14:19 +01:00 |
|
|
florez_j
|
33f29a8080
|
WIP: Completed flag generation for species, command line interface is now clear but still does not consider potentially existing flags. Review TODO.md, for efficiency we need to simplify flag assignment and reconciliation for the species flag type.
|
2025-02-26 15:11:12 +01:00 |
|
|
florez_j
|
600c60be36
|
Append variable specification for explicit definition of diagnostics and species to pipelines/params/calibration_params.yaml. This is to support flagging operations to particular variables
|
2025-02-26 15:07:58 +01:00 |
|
|
florez_j
|
6d5d74e616
|
Append flag_ranking to app/flags/ebas_dict.yaml. This is to disambiguate/overide conflicting flags
|
2025-02-26 15:05:06 +01:00 |
|
|
florez_j
|
e6868da3d4
|
Update dima to latest version
|
2025-02-25 13:34:52 +01:00 |
|
|
florez_j
|
94cd536204
|
Updated dima to latest version
|
2025-02-22 18:08:14 +01:00 |
|
|
florez_j
|
4edd6809e9
|
WIP: Redesigned command line functionality. options --species is half baked :). It runs without error but second part needs validation.
|
2025-02-22 16:00:27 +01:00 |
|
|
florez_j
|
e24d451542
|
Updated thirdparty submodule to latest version
|
2025-02-22 15:02:17 +01:00 |
|