|
|
5c6fcabf91
|
Updated the yaml instrument descriptions.
|
2024-07-10 13:29:14 +02:00 |
|
|
|
73beb83278
|
Moved parse_attribute() from ..review_lib.py into ...utils.py and backpropagate (refactored) changes to respective modules.
|
2024-07-10 11:32:00 +02:00 |
|
|
|
2ce925735d
|
Modified return datetime output to a format without colons, which could be problematic for filenaming.
|
2024-07-10 09:47:56 +02:00 |
|
|
|
0a0b4ac41d
|
Moved a few functions from ...reader.py and hdf5_lib.py into ..utils.py, and refactored accordingly.
|
2024-07-10 09:19:30 +02:00 |
|
|
|
0c74c52e09
|
Removed smogchamber reader because its funtionality is now integrated into g5505_file_reader.py.
|
2024-07-09 16:13:01 +02:00 |
|
|
|
afc6c93823
|
Removed non utilized code.
|
2024-07-08 15:29:13 +02:00 |
|
|
|
cb7d914908
|
Cleaned code and modified def create_hdf5_file_from_dataframe to create group hierichy implicitly from path rather than recursively.
|
2024-07-08 15:24:48 +02:00 |
|
|
|
92eca4d79e
|
Moved remaining git operations in metadata_review_lib.py to git_ops.py and refactored accoringly
|
2024-07-05 15:46:20 +02:00 |
|
|
|
77386432f8
|
Merge branch 'main' of https://gitlab.psi.ch/5505/dima
|
2024-07-02 16:50:08 +02:00 |
|
|
|
177a5aa2a1
|
Updated documentation.
|
2024-07-02 16:49:48 +02:00 |
|
|
|
ca8570b3b0
|
Merge branch 'main' of https://gitlab.psi.ch/5505/dima
|
2024-07-01 16:20:06 +02:00 |
|
|
|
ba6d89d8e1
|
Modified created at function to output date time and time zone
|
2024-07-01 16:19:28 +02:00 |
|
|
|
c074e45892
|
Renamed script_name to processing_file.
|
2024-07-01 16:17:25 +02:00 |
|
|
|
b816e62f3b
|
Made a few edits.
|
2024-06-21 15:55:44 +02:00 |
|
|
|
db199f81e0
|
Merge branch 'main' of https://gitlab.psi.ch/5505/dima
|
2024-06-21 15:42:46 +02:00 |
|
|
|
6d6caf96db
|
Cleared out outputs.
|
2024-06-21 15:42:23 +02:00 |
|
|
|
5ab775ecac
|
Added a few root level metadata names and definitions
|
2024-06-21 15:40:38 +02:00 |
|
|
|
cedfe614e7
|
Implemented input argument to enable append information to exisintg attributes, which must take the values of either strings or lists.
|
2024-06-20 15:32:33 +02:00 |
|
|
|
106795ae59
|
Added a few lines to detect the existence of the file and change the file mode from 'w' to 'a' based on that information.
|
2024-06-20 09:03:47 +02:00 |
|
|
|
498a51cbc6
|
Updated function to add project level metadata at the root group of the hdf5 file.
|
2024-06-19 18:31:11 +02:00 |
|
|
|
06c5c6d84b
|
Incorporated method to MetadataHarvester class to collect project level metadata.
|
2024-06-19 18:30:02 +02:00 |
|
|
|
04558e7785
|
Added code to parse dict attributes.
|
2024-06-18 14:42:51 +02:00 |
|
|
|
a6868d985d
|
Fixed bug regarding datetime to str column conversion in dataframe by using .map(srt) (element wise operation) as opposed to .apply(str)
|
2024-06-18 09:21:46 +02:00 |
|
|
|
b66dc11a62
|
Replaced applymap to .apply because the former is being depricated
|
2024-06-17 13:47:54 +02:00 |
|
|
|
ed1641af55
|
Created function to save dataframes with annotations in hdf5 format
|
2024-06-17 13:36:05 +02:00 |
|
|
|
0eba80db41
|
Added metadata printer method and rewrote slightly a few class terms.
|
2024-06-17 08:44:44 +02:00 |
|
|
|
c68e800967
|
Incorporated dataframe_to_np_structured_array(df: pd.DataFrame) from another module.
|
2024-06-16 18:39:30 +02:00 |
|
|
|
e4de4edf28
|
Incorporated dataframe_to_np_structured_array(df: pd.DataFrame) from another module.
|
2024-06-16 18:26:12 +02:00 |
|
|
|
2d4ecec806
|
Moved dataframe_to_np_structured_array(df: pd.DataFrame) to src/g5505_utils.py. This is a more generic function that can be used more broadly accross modules.
|
2024-06-16 18:25:08 +02:00 |
|
|
|
6f5c49dc64
|
Removed lenthy example.
|
2024-06-13 16:03:04 +02:00 |
|
|
|
a301d42ad5
|
Replaced add_data_level_info to add_dataset.
|
2024-06-13 16:01:27 +02:00 |
|
|
|
0fb14b7c6c
|
Developed a metadata harvesting object to facilitate metadata collection throught the code.
|
2024-06-13 15:47:02 +02:00 |
|
|
|
f43d86e729
|
Modified a few variable values in yaml files so that they are within expected values.
|
2024-06-13 15:45:39 +02:00 |
|
|
|
9ab9aa49c4
|
Abstracted a code snippet from def create_hdf5_file_from_filesystem_path(..) as transfer_file_dict_to_hdf5() so that it can be reusable.
|
2024-06-13 15:44:01 +02:00 |
|
|
|
cbca94390f
|
Modified hardcoded paths to adapt with respect to the parent directory
|
2024-06-11 17:30:58 +02:00 |
|
|
|
e7ed6145f0
|
Implemented a data extraction module to access data from an hdf5 file in the form of dataframes.
|
2024-06-11 10:38:04 +02:00 |
|
|
|
a410bde23e
|
Removed data table split into categorical and numerical variables and numering is only introduce to disambiguate repeated columns.
|
2024-06-10 16:18:51 +02:00 |
|
|
|
1ec7ad76ff
|
Removed additional numbering from some intrument specifications. These are now only added if the column names are ambigous.
|
2024-06-10 16:14:13 +02:00 |
|
|
|
726e9b3503
|
Fixed bug in the case where data_integration_mode = 'collection'.
|
2024-06-07 16:45:00 +02:00 |
|
|
|
dba5bc9ea7
|
Updated instrument names from ICAD/HONO and ICAD/NO2 to HONO and NO2.
|
2024-06-07 16:41:41 +02:00 |
|
|
|
197ad0288a
|
Updated file reader and data integration with datastart and dataend properties.
|
2024-06-04 13:37:20 +02:00 |
|
|
|
9dcc757acc
|
renamed folder src/instrument_descriptions/ to src/intruments/ and moved text_data_sources.yaml in there.
|
2024-06-04 10:54:09 +02:00 |
|
|
|
a6ddb24eeb
|
Added .strip to column names to remove unwanted characters (\r|\t|\n) and included units description to timestamps.
|
2024-06-04 09:57:37 +02:00 |
|
|
|
fa2990527e
|
Simplified and documented parse_attribute function.
|
2024-06-04 09:51:12 +02:00 |
|
|
|
014bd14fcd
|
Modified temperature units from °C to Celcius for simpler string encoding. It seems ascii codec cannot encode such a character
|
2024-06-04 09:44:09 +02:00 |
|
|
|
385267a98f
|
Updated treemap visualization to select only root metadata, which is of string type.
|
2024-06-03 14:17:42 +02:00 |
|
|
|
560481610c
|
Updated root metadata display in treemaps
|
2024-06-02 16:43:54 +02:00 |
|
|
|
c74b6c1a91
|
Updated instrument attributes with datetime_format and desired_format.
|
2024-06-02 16:14:30 +02:00 |
|
|
|
1054367f12
|
Modified annotate_root_dir function.
|
2024-06-02 16:02:48 +02:00 |
|
|
|
d335836a7d
|
Updated reader to standardize timestamps to a desired format when possible. The desired format is set in text_data_sources.yaml.
|
2024-06-02 15:59:01 +02:00 |
|
