|
|
2ea9269f75
|
Replaced applymap to .apply because the former is being depricated
|
2024-06-17 13:47:54 +02:00 |
|
|
|
86a811e6aa
|
Created function to save dataframes with annotations in hdf5 format
|
2024-06-17 13:36:05 +02:00 |
|
|
|
652f311c8d
|
Added metadata printer method and rewrote slightly a few class terms.
|
2024-06-17 08:44:44 +02:00 |
|
|
|
9f6533e53b
|
Incorporated dataframe_to_np_structured_array(df: pd.DataFrame) from another module.
|
2024-06-16 18:39:30 +02:00 |
|
|
|
bda5e87cc8
|
Incorporated dataframe_to_np_structured_array(df: pd.DataFrame) from another module.
|
2024-06-16 18:26:12 +02:00 |
|
|
|
d2e53dca3f
|
Moved dataframe_to_np_structured_array(df: pd.DataFrame) to src/g5505_utils.py. This is a more generic function that can be used more broadly accross modules.
|
2024-06-16 18:25:08 +02:00 |
|
|
|
f7f91aa105
|
Removed lenthy example.
|
2024-06-13 16:03:04 +02:00 |
|
|
|
1c937222fd
|
Replaced add_data_level_info to add_dataset.
|
2024-06-13 16:01:27 +02:00 |
|
|
|
2a28d45b13
|
Developed a metadata harvesting object to facilitate metadata collection throught the code.
|
2024-06-13 15:47:02 +02:00 |
|
|
|
71e1fffd1a
|
Modified a few variable values in yaml files so that they are within expected values.
|
2024-06-13 15:45:39 +02:00 |
|
|
|
622661d4d3
|
Abstracted a code snippet from def create_hdf5_file_from_filesystem_path(..) as transfer_file_dict_to_hdf5() so that it can be reusable.
|
2024-06-13 15:44:01 +02:00 |
|
|
|
9b70493fbf
|
Modified hardcoded paths to adapt with respect to the parent directory
|
2024-06-11 17:30:58 +02:00 |
|
|
|
cf82678f9e
|
Implemented a data extraction module to access data from an hdf5 file in the form of dataframes.
|
2024-06-11 10:38:04 +02:00 |
|
|
|
ba5b8cb407
|
Removed data table split into categorical and numerical variables and numering is only introduce to disambiguate repeated columns.
|
2024-06-10 16:18:51 +02:00 |
|
|
|
ed33e77380
|
Removed additional numbering from some intrument specifications. These are now only added if the column names are ambigous.
|
2024-06-10 16:14:13 +02:00 |
|
|
|
b47ed2b3f4
|
Fixed bug in the case where data_integration_mode = 'collection'.
|
2024-06-07 16:45:00 +02:00 |
|
|
|
9d28c3e1d6
|
Updated instrument names from ICAD/HONO and ICAD/NO2 to HONO and NO2.
|
2024-06-07 16:41:41 +02:00 |
|
|
|
1d241f663c
|
Updated file reader and data integration with datastart and dataend properties.
|
2024-06-04 13:37:20 +02:00 |
|
|
|
05e580527e
|
renamed folder src/instrument_descriptions/ to src/intruments/ and moved text_data_sources.yaml in there.
|
2024-06-04 10:54:09 +02:00 |
|
|
|
f6154a6777
|
Added .strip to column names to remove unwanted characters (\r|\t|\n) and included units description to timestamps.
|
2024-06-04 09:57:37 +02:00 |
|
|
|
e3dcb1110a
|
Simplified and documented parse_attribute function.
|
2024-06-04 09:51:12 +02:00 |
|
|
|
1f7bf98c96
|
Modified temperature units from °C to Celcius for simpler string encoding. It seems ascii codec cannot encode such a character
|
2024-06-04 09:44:09 +02:00 |
|
|
|
0cc4a1f215
|
Updated treemap visualization to select only root metadata, which is of string type.
|
2024-06-03 14:17:42 +02:00 |
|
|
|
236693c66c
|
Tracking metadata file: output_files/smog_chamber_study_2022-07-26_NatashaG.yaml
|
2024-06-03 07:52:47 +02:00 |
|
|
|
52d8399bdd
|
Tracking metadata file: output_files/smog_chamber_study_2022-07-26_NatashaG.yaml
|
2024-06-03 07:44:53 +02:00 |
|
|
|
afeb2241fc
|
Tracking metadata file: output_files/smog_chamber_study_2022-07-26_NatashaG.yaml
|
2024-06-03 07:30:11 +02:00 |
|
|
|
156027a934
|
Tracking metadata file: output_files/smog_chamber_study_2022-07-26_NatashaG.yaml
|
2024-06-02 18:28:34 +02:00 |
|
|
|
f344a45c94
|
Tracking metadata file: output_files/smog_chamber_study_2022-07-26_NatashaG.yaml
|
2024-06-02 18:25:39 +02:00 |
|
|
|
866d4aa4d9
|
Updated root metadata display in treemaps
|
2024-06-02 16:43:54 +02:00 |
|
|
|
25daf66b19
|
Updated instrument attributes with datetime_format and desired_format.
|
2024-06-02 16:14:30 +02:00 |
|
|
|
2b2874cfdc
|
Modified annotate_root_dir function.
|
2024-06-02 16:02:48 +02:00 |
|
|
|
85f0e69c2c
|
Updated reader to standardize timestamps to a desired format when possible. The desired format is set in text_data_sources.yaml.
|
2024-06-02 15:59:01 +02:00 |
|
|
|
d0395fff5b
|
Implemented functions for data extraction from hdf5 files.
|
2024-05-31 12:39:10 +02:00 |
|
|
|
16dda1e834
|
Incorporated jupyter notebook of simple example metadata annotation workflow.
|
2024-05-30 12:24:12 +02:00 |
|
|
|
d15c8924b5
|
Updated readme file
|
2024-05-30 12:21:17 +02:00 |
|
|
|
4f462578ef
|
Updated notebook documentation and included an example metadata annotation notebook.
|
2024-05-30 12:20:34 +02:00 |
|
|
|
8fa587ef19
|
Removed html file no longer useful.
|
2024-05-30 12:18:28 +02:00 |
|
|
|
894936f107
|
Updated YAML config file parsing logic to account for changes in config file description.
|
2024-05-30 12:16:54 +02:00 |
|
|
|
3e21ecde7b
|
Decomposed experiment_data into experiment_startdate and experiment_enddate.
|
2024-05-30 12:15:49 +02:00 |
|
|
|
b2e807788f
|
Made def third_update_hdf5_file_with_review more modular by separating data update and git operations, resulting new functions that can be reused in less restrictive matadata annotation contexts.
|
2024-05-29 15:26:48 +02:00 |
|
|
|
1987d1610f
|
Implemented a git operations module for automated git ops, based on subprocess.
|
2024-05-29 15:17:09 +02:00 |
|
|
|
c1e5bc9ddd
|
Updated readme file.
|
2024-05-29 11:24:46 +02:00 |
|
|
|
8226f616dd
|
Updated readme file
|
2024-05-29 11:23:33 +02:00 |
|
|
|
fb1c627104
|
Updated project name in configuration file
|
2024-05-28 15:06:25 +02:00 |
|
|
|
4fb5ed58b1
|
Changed ordering of data integration config files so that they align with our experimental campaign hierarchy.
|
2024-05-28 14:43:32 +02:00 |
|
|
|
82754e26b0
|
Refactored due to updates in the file reader function.
|
2024-05-28 14:41:34 +02:00 |
|
|
|
0f505df45c
|
added the feature to activate or deactivate data copying before reading the input file. This is to avoid redundant copying when we are already working on file copies.
|
2024-05-28 14:40:14 +02:00 |
|
|
|
e0d84d7822
|
Updated some of the raname_as metadata for all instruments so that it is much machine readable and perhpas be used as an alternative to the original name in future releases.
|
2024-05-28 14:37:43 +02:00 |
|
|
|
54e4301e93
|
Modified function to return list of paths when config_file.yaml integration mode = experimental step.
|
2024-05-28 11:29:32 +02:00 |
|
|
|
dfd14fd029
|
Improved parsing from HDF5 attr dict to yaml compatible dict. Now we can parse HDF5 compound attributes (structured np arrays).
|
2024-05-28 11:27:44 +02:00 |
|
