|
|
989e99078a
|
Merge branch 'main' of https://gitlab.psi.ch/apog/acsmnode
This updates todo.md
|
2025-05-26 13:12:52 +02:00 |
|
|
|
52fdb7a8ad
|
Fix number of variables by subtracting one. Start_time is now excluded from count.
|
2025-05-26 13:12:40 +02:00 |
|
|
|
f7e1c31550
|
added len(interpolated_cpc)
|
2025-05-21 16:35:50 +02:00 |
|
|
|
a7faf69c96
|
Changed missing value code generator, any number greater than the largest possible one that can be represented in terms of 9s is capped
|
2025-05-21 15:17:25 +02:00 |
|
|
|
e39c23bbad
|
Update generate_flags.py to skip checking for cpc flags when not available
|
2025-05-21 09:48:19 +02:00 |
|
|
|
01769f10e0
|
Implement new step to remove column from already created nas file.
|
2025-05-21 09:43:36 +02:00 |
|
|
|
a9461cb1aa
|
Added limits on y axis for flags visualization
|
2025-05-20 16:27:11 +02:00 |
|
|
|
cc1b445d38
|
Implement binary flags for visualization purposes, and fix bug with cpc flags.
|
2025-05-16 14:22:17 +02:00 |
|
|
|
58d7b425aa
|
Add generate cpc flags pipelines/steps/generate_flags.py
|
2025-05-15 15:01:11 +02:00 |
|
|
|
37671a3685
|
update pipelines/steps/visualize_datatable_vars.py with interactive plots and refactor dependent jupyter nb accordingly
|
2025-05-13 11:32:46 +02:00 |
|
|
|
4408514439
|
Rename attribute station --> station_abbr
|
2025-04-11 11:20:48 +02:00 |
|
|
|
e7863078aa
|
Add data chain step /update_datachain_params.py. This uses params specified in input data folder to update current data chain params
|
2025-04-11 11:07:52 +02:00 |
|
|
|
cea83396d2
|
Add function to estimate project directory path
|
2025-04-11 11:05:41 +02:00 |
|
|
|
68a7d5dc2b
|
Fix out of bound error in pipelines/steps/visualize_datatable_vars.py
|
2025-04-11 11:04:00 +02:00 |
|
|
|
c9e25c2483
|
Add new placeholder for submitter name and remove metadata element 'station' conflicting with third party converter
|
2025-04-11 11:02:38 +02:00 |
|
|
|
881209bf82
|
Update steps to allow filenames for PAYERN station
|
2025-04-08 21:26:01 +02:00 |
|
|
|
3dfed2c5f3
|
Update pipelines/steps/prepare_ebas_submission.py. Rmoved hard coded paths and build output name using metadata from campaign descriptor. Also, we can now specify month ranges
|
2025-04-08 18:44:46 +02:00 |
|
|
|
5dd280e88c
|
Update pipelines/steps/utils.py. Now we can load the campaign descriptor using load_project_yaml_files()
|
2025-04-08 18:43:01 +02:00 |
|
|
|
ca0366338a
|
Add pipeline step to update actris_header with submitter and originator
|
2025-04-08 18:34:51 +02:00 |
|
|
|
00a85505c8
|
Fix bug when infering name of flags file from dataset name
|
2025-04-04 16:53:41 +02:00 |
|
|
|
d290daaa39
|
Update pipelines/steps/visualize_datatable_vars.py. Specify alternative path to flags in case they are not in the HDF5 file. We assume there is an append directory where they may be. Otherwise it throws and error.
|
2025-04-04 16:06:11 +02:00 |
|
|
|
b0d68747e9
|
Update code to rearrange table columns based on column_order from acsm_to_ebas.yaml. This can be removed in the future.
|
2025-03-18 07:23:54 +01:00 |
|
|
|
7d277e3e3b
|
Clean up print statements.
|
2025-03-14 13:39:12 +01:00 |
|
|
|
39a9ab07a2
|
Refactor command line interface to run a main function on the parsed arguments.
|
2025-03-14 13:36:52 +01:00 |
|
|
|
8cdd8a0771
|
Completed command line interface for pipelines/steps/prepare_ebas_submission.py. This finishes WIP associated with commit 2eb88e4.
|
2025-03-14 13:09:09 +01:00 |
|
|
|
b643eb2d26
|
Refactored command line interface, generate_species_flags, and created functions to improve performance and code clarity.
|
2025-03-13 14:17:46 +01:00 |
|
|
|
97f253074b
|
Add in try-except block a finally statement to close hdf5 file for secure file closing. This was done before in the try case which is not good practice
|
2025-03-13 14:06:01 +01:00 |
|
|
|
d6fa51effc
|
Fix typo and missing key. 1) Update allowed yaml files in load_project_yaml_files by adding pipelines/params/calibration_factors.yaml. 2) Correct filename access after os.path.split operation.
|
2025-03-10 13:02:38 +01:00 |
|
|
|
4ca99f4f04
|
Rewrote load_calib_file() in terms pipelines.steps.utils.load_project_yaml_files(). In the future we will replace it completely so we can validate and manage all yaml files in the chain in a more centralized manner.
|
2025-03-10 11:46:24 +01:00 |
|
|
|
ecdf3687ba
|
Remove/replace def load_parameters(flag_type) with steps.utils.load_project_yaml_files(.). This is to have a more centralized treatment of yaml files inside the data chain.
|
2025-03-10 11:38:51 +01:00 |
|
|
|
2eb88e444c
|
WIP: finished last commit, and tested the whole step. Now we need to implement the command line interface.
|
2025-03-07 16:51:06 +01:00 |
|
|
|
eb8a36dc8e
|
Update pipelines/steps/apply_calibration_factors.py to generate error table and cleaned up command line interface.
|
2025-03-07 16:49:32 +01:00 |
|
|
|
78340464aa
|
Add functions: generate_error_dataframe() with missing values, metadata_dict_to_dataframe(), and load_project_yaml_files() to easily access data from yaml files in the project.
|
2025-03-07 16:45:33 +01:00 |
|
|
|
f035afbf3b
|
Fix duplicate time stamp problems and added code to check for NaTs before ebas submission files.)
|
2025-03-04 09:02:37 +01:00 |
|
|
|
6eccbb5018
|
Moved get_metadata() from pipelines/steps/prepare_ebas_submission.py to utils.py
|
2025-03-03 18:55:46 +01:00 |
|
|
|
def67a0562
|
Added correct calibration parameters to calibration factors table
|
2025-03-03 14:23:45 +01:00 |
|
|
|
d3a491ea85
|
Renamed pipeline step join_tables.py to prepare_ebas_submission.py for accuracy. This step performs more than joining tables, it also does renaming using dictionaries and generates the ebas data submission package
|
2025-03-03 10:48:03 +01:00 |
|
|
|
877c0178d7
|
WIP: Added first version of join_tables and tested txt generation. We should probaly make this step encompasse name renaming for EBAS and rename step to be more generic. Change join_tables.py to generate_ebas_converter_input_files
|
2025-02-28 17:23:00 +01:00 |
|
|
|
a9533ae3e8
|
Refine function. Abstract out load parameters as function to simplify code.
|
2025-02-28 17:18:13 +01:00 |
|
|
|
6890377134
|
Remove dataset_name parameter from command line interface. Instead i now infer it assuming there is a consistent input filename in the hdf5 file that can be found using hard coded keywords.
|
2025-02-28 17:17:33 +01:00 |
|
|
|
33f29a8080
|
WIP: Completed flag generation for species, command line interface is now clear but still does not consider potentially existing flags. Review TODO.md, for efficiency we need to simplify flag assignment and reconciliation for the species flag type.
|
2025-02-26 15:11:12 +01:00 |
|
|
|
4edd6809e9
|
WIP: Redesigned command line functionality. options --species is half baked :). It runs without error but second part needs validation.
|
2025-02-22 16:00:27 +01:00 |
|
|
|
e24d451542
|
Updated thirdparty submodule to latest version
|
2025-02-22 15:02:17 +01:00 |
|
|
|
0911260f26
|
Save changes.
|
2025-02-15 18:20:58 +01:00 |
|
|
|
a516479036
|
Change data lineage format. Now, data lineage information is accessed using name of the file as key.
|
2025-02-15 17:57:50 +01:00 |
|
|
|
ca08044569
|
Update keys of data_lineage_metadata.yaml. keys are now the filenames in the folder.
|
2025-02-11 13:14:14 +01:00 |
|
|
|
0623c2462b
|
Adjust command line interface of steps/apply_... and steps/compute_aut...
|
2025-02-11 13:07:29 +01:00 |
|
|
|
f2854d4ca0
|
Implement pipelines/steps/visualize_datatable_vars.py
|
2025-02-11 13:03:46 +01:00 |
|
|
|
b966151d9c
|
Update with template for compute_cpc_flags(). This still needs to be completed.
|
2025-02-10 10:54:18 +01:00 |
|
|
|
ca2b8b6787
|
Rename apply_diagnostic_flags.py to compute_automated_flags.py because the function also computes both cpc and diagnostic based flags
|
2025-02-10 10:45:55 +01:00 |
|
