 florez_j
|
00a85505c8
|
Fix bug when infering name of flags file from dataset name
|
2025-04-04 16:53:41 +02:00 |
|
|
florez_j
|
dcf6b8ec41
|
Clean up and document steps in notebooks/demo_visualize_diagnostic_flags_from_hdf5_file.ipynb
|
2025-04-04 16:07:07 +02:00 |
|
|
florez_j
|
d290daaa39
|
Update pipelines/steps/visualize_datatable_vars.py. Specify alternative path to flags in case they are not in the HDF5 file. We assume there is an append directory where they may be. Otherwise it throws and error.
|
2025-04-04 16:06:11 +02:00 |
|
|
florez_j
|
b3b5516aed
|
Fix typo in README.md
|
2025-04-04 09:46:36 +02:00 |
|
|
florez_j
|
17f497f1c5
|
Update README.md with instructions to set up network drive access credentials and update TODO.md
|
2025-04-02 14:40:55 +02:00 |
|
|
florez_j
|
9a23bf283a
|
Fix output directory in campaignDescripor.yaml
|
2025-04-02 14:17:45 +02:00 |
|
|
florez_j
|
73c0409207
|
Update README.md with instruction to build, run, and stop docker container on which the data chain will run
|
2025-04-02 14:10:52 +02:00 |
|
|
florez_j
|
73648ebe31
|
Update Dockerfile and docker-compose.yaml. By default, container can use cifs to map network drives so no need to install, and added bash command to docker compose file to run both jupyterlab and dash board simultanously
|
2025-04-02 14:09:44 +02:00 |
|
|
florez_j
|
c6b010702c
|
Remove run_container.sh. This is now done by running the docker-compose.yaml file using docker
|
2025-04-02 11:27:42 +02:00 |
|
|
florez_j
|
643a305782
|
Update ACSM data chain workflow with markdown descriptions
|
2025-04-02 11:26:03 +02:00 |
|
|
florez_j
|
f0da41e914
|
Add docker-compose.yaml to replace run_container.sh
|
2025-04-02 11:17:20 +02:00 |
|
|
florez_j
|
1e7f92d6e8
|
Update ignore files
|
2025-04-02 11:15:36 +02:00 |
|
|
florez_j
|
14cb24fb4b
|
Add config files .sh .yaml and .md to dockerignore
|
2025-03-18 09:21:26 +01:00 |
|
|
florez_j
|
aac6015986
|
Update README with Docker container workflow
|
2025-03-18 08:57:51 +01:00 |
|
|
florez_j
|
b7cbf7dd14
|
Add bash script to run container
|
2025-03-18 08:57:06 +01:00 |
|
|
florez_j
|
41506519d3
|
WIP: Update Dockerfile with requirements.txt that downgrade versions so that ebas converter can run. We need to rethink the Dockerfile container workflow using docker compose
|
2025-03-18 08:55:52 +01:00 |
|
|
florez_j
|
862a2c46e6
|
Clean up notebooks
|
2025-03-18 08:53:51 +01:00 |
|
|
florez_j
|
b0d68747e9
|
Update code to rearrange table columns based on column_order from acsm_to_ebas.yaml. This can be removed in the future.
|
2025-03-18 07:23:54 +01:00 |
|
|
florez_j
|
261c8b8154
|
Add column order attributes, storing the order in which variables need to be displayed to adhere to Nora's initial implementation.
|
2025-03-18 07:22:06 +01:00 |
|
|
florez_j
|
f3010bff47
|
Fix import problem. The app didn't run from a notebooks/<>.ipynb. This however may not be needed any longer. Is it good practice to run dashboards from a jupyter notebook?
|
2025-03-18 07:14:44 +01:00 |
|
|
florez_j
|
395b734cd6
|
Fix bug reported in TODO.md. FLAGGING APP didnt record the parent channel correctly, that is the channel that originated a particular flag. An additional problem was detected and corrected where _metadata.json was confused as flag.
|
2025-03-17 14:21:26 +01:00 |
|
|
florez_j
|
895a44cc8c
|
Update notebooks/demo_acsm_pipeline.ipynb by introducing global append and input file variables.
|
2025-03-14 15:26:37 +01:00 |
|
|
florez_j
|
d5a89b2fbd
|
Add more items to TODO.md
|
2025-03-14 15:25:47 +01:00 |
|
|
florez_j
|
f90708b043
|
Update todo.md
|
2025-03-14 15:17:31 +01:00 |
|
|
florez_j
|
9504576fc0
|
Refactor steps using main functions from each step instead of subprocess with command line interface
|
2025-03-14 13:44:03 +01:00 |
|
|
florez_j
|
b93cb7517b
|
Add comment to indicate where params are defined in original code.
|
2025-03-14 13:40:22 +01:00 |
|
|
florez_j
|
7d277e3e3b
|
Clean up print statements.
|
2025-03-14 13:39:12 +01:00 |
|
|
florez_j
|
39a9ab07a2
|
Refactor command line interface to run a main function on the parsed arguments.
|
2025-03-14 13:36:52 +01:00 |
|
|
florez_j
|
8cdd8a0771
|
Completed command line interface for pipelines/steps/prepare_ebas_submission.py. This finishes WIP associated with commit 2eb88e4.
|
2025-03-14 13:09:09 +01:00 |
|
|
florez_j
|
bfc9f0ab82
|
Cross out item. Requirements of data input to converter are defined in pipelines/dictionaries/acsm_to_ebas.yaml
|
2025-03-14 13:05:29 +01:00 |
|
|
florez_j
|
f3612267c1
|
Update dima to latest version.
|
2025-03-14 10:18:30 +01:00 |
|
|
florez_j
|
b643eb2d26
|
Refactored command line interface, generate_species_flags, and created functions to improve performance and code clarity.
|
2025-03-13 14:17:46 +01:00 |
|
|
florez_j
|
97f253074b
|
Add in try-except block a finally statement to close hdf5 file for secure file closing. This was done before in the try case which is not good practice
|
2025-03-13 14:06:01 +01:00 |
|
|
florez_j
|
0fe2b564b7
|
WIP: Change to copy command as COPY . /acsmdc.
|
2025-03-12 16:22:39 +01:00 |
|
|
florez_j
|
05cf34032f
|
WIP: Add dockerignore and Dockerfile. I have built the Docker image and seems to be working fine but haven't tested all aspects of the applicaton
|
2025-03-12 16:15:49 +01:00 |
|
|
florez_j
|
d6fa51effc
|
Fix typo and missing key. 1) Update allowed yaml files in load_project_yaml_files by adding pipelines/params/calibration_factors.yaml. 2) Correct filename access after os.path.split operation.
|
2025-03-10 13:02:38 +01:00 |
|
|
florez_j
|
4ca99f4f04
|
Rewrote load_calib_file() in terms pipelines.steps.utils.load_project_yaml_files(). In the future we will replace it completely so we can validate and manage all yaml files in the chain in a more centralized manner.
|
2025-03-10 11:46:24 +01:00 |
|
|
florez_j
|
ecdf3687ba
|
Remove/replace def load_parameters(flag_type) with steps.utils.load_project_yaml_files(.). This is to have a more centralized treatment of yaml files inside the data chain.
|
2025-03-10 11:38:51 +01:00 |
|
|
florez_j
|
cf073126b9
|
Updated acsm raw to ebas terminology mapping. It is now complete!
|
2025-03-07 16:53:22 +01:00 |
|
|
florez_j
|
2eb88e444c
|
WIP: finished last commit, and tested the whole step. Now we need to implement the command line interface.
|
2025-03-07 16:51:06 +01:00 |
|
|
florez_j
|
eb8a36dc8e
|
Update pipelines/steps/apply_calibration_factors.py to generate error table and cleaned up command line interface.
|
2025-03-07 16:49:32 +01:00 |
|
|
florez_j
|
cbad39355b
|
Reformat LOD variable names for simplicity
|
2025-03-07 16:46:48 +01:00 |
|
|
florez_j
|
78340464aa
|
Add functions: generate_error_dataframe() with missing values, metadata_dict_to_dataframe(), and load_project_yaml_files() to easily access data from yaml files in the project.
|
2025-03-07 16:45:33 +01:00 |
|
|
florez_j
|
93f49f7fd1
|
Updated third party submodule to latest version
|
2025-03-07 13:45:54 +01:00 |
|
|
florez_j
|
35e08d387f
|
Add pipelines/params/station_params.yaml, where station dependent temperature and preassure are to be defined. Crossed out associated TODO.md item.
|
2025-03-07 11:23:48 +01:00 |
|
|
florez_j
|
5f9d72e9e1
|
Replace submodule:third_party/actris_acsm_converter with third_party/acsmProcessingSoftware at Nora/main branch. The idea is to add Nora's modification to original repo to maintain a clear provenance.
|
2025-03-07 09:41:23 +01:00 |
|
|
florez_j
|
f035afbf3b
|
Fix duplicate time stamp problems and added code to check for NaTs before ebas submission files.)
|
2025-03-04 09:02:37 +01:00 |
|
|
florez_j
|
9a23744f63
|
Add acsm to ebas dictionary
|
2025-03-04 09:01:09 +01:00 |
|
|
florez_j
|
6eccbb5018
|
Moved get_metadata() from pipelines/steps/prepare_ebas_submission.py to utils.py
|
2025-03-03 18:55:46 +01:00 |
|
|
florez_j
|
def67a0562
|
Added correct calibration parameters to calibration factors table
|
2025-03-03 14:23:45 +01:00 |
|