f37ba4705aIncluded .h5 files for now, but they should be enable later on through git LFS.Florez Ospina Juan Felipe2024-03-26 16:18:48 +01:00
a727e38db4Implemented hdf5_vis.py, which is a hdf5 visualization library to obtain treemap and yaml representations of hdf5 files.Florez Ospina Juan Felipe2024-03-26 16:14:40 +01:00
1bf1f60bebAdded lines to treat string attributes as fixed-length strings, which are represented as bytes that need to be decoded with utf-8. There are a few advantages, and hdf5 reader provide more precise behavior than variable length stringsFlorez Ospina Juan Felipe2024-03-22 17:28:47 +01:00
13cb6395aaRestructured the way table_preamble attribute is represented. Now it is a list of strings as opposed to a multilinear string with special characters like \n. This is to avoid parsing problems in the yalm files.Florez Ospina Juan Felipe2024-03-22 17:26:30 +01:00
fff935f551Included optional argument in make_copy function and commented out a few lines that increase dataset storage complexity.Florez Ospina Juan Felipe2024-03-21 17:16:14 +01:00
4244e39232Implemented hdf5_vis.py to gather functions that display or represent properties of an hdf5 file in a human readable file format like yalm or html files that enble interative visualizations on the browser.Florez Ospina Juan Felipe2024-03-21 16:30:27 +01:00
e389ffbefeRelocated def display_group_hierarchy_on_a_treemap(filename: str) to hdf5_vis.pyFlorez Ospina Juan Felipe2024-03-21 16:27:54 +01:00
8004a891aaIncluded lines to work on copies of files, and removed .strip() to create the table preamble because it destroyed txt structure.Florez Ospina Juan Felipe2024-03-19 14:55:49 +01:00
63e7fb28d0Removed the 'backup_' name from the copied file so that the orignal name is preserved in the hdf5 file. The file copy storage location is enough to distinguise it from the original file.Florez Ospina Juan Felipe2024-03-19 14:07:37 +01:00
2271be6eccMoved make_file_copy internal function to g5505_utils.py module because it can be reused accross file readers.Florez Ospina Juan Felipe2024-03-19 11:58:08 +01:00
7fe254755fReplaced attributes, previously extracted from the table_preamble in .txt and .dat files with a single dataset attribute called table_preamble that contains the whole table preamble.Florez Ospina Juan Felipe2024-03-19 11:40:35 +01:00
b886066133Simplified code and corrected buggy if statement. Included input verification steps and OS path normalization.Florez Ospina Juan Felipe2024-03-19 11:11:05 +01:00
1a4294e0c2Modified to received unified dictionary structure and transform it into equivalent group datasets and attribute structure.Florez Ospina Juan Felipe2024-02-16 16:52:21 +01:00
a2818975b5Modified to include all relevant inputs to hdf5 file creation for different groupsFlorez Ospina Juan Felipe2024-02-16 16:50:13 +01:00
a6400b9ac4Inclided function to copy an entired h5 file in a group. Useful when integrating h5 files into new ones.Florez Ospina Juan Felipe2024-02-15 16:01:15 +01:00
13416ddeecAdded a configuration file to select available group dependent file reader functions. This is to enable modularity and scalability.Florez Ospina Juan Felipe2024-02-15 15:57:45 +01:00
e449021a6dIntegrated data and time columns in smps txt files as a single column called TimestampFlorez Ospina Juan Felipe2024-02-14 09:31:59 +01:00
e082e066b1Implemented a file reader module for smog chamber file formats, smog_chamber_group_reader.py. its use is equivalent to that of g5505_file_reader.py, to read file formats of the multiphase chemistry group.Florez Ospina Juan Felipe2024-02-13 16:44:32 +01:00
03dcc62f9aRenamed the main functions for the sake of clarity and transfered a few functions to the g5505_utils.py module.Florez Ospina Juan Felipe2024-02-13 16:40:52 +01:00
d62327ba25Created g5505_utils.py module with very specific helper functions of the 5505 group. This is to remove clutter from the main module hdf5_lib.pyFlorez Ospina Juan Felipe2024-02-13 16:39:05 +01:00
8f757ca68cImplemented filtering mechanishm in function create_hdf5_file_from_filesystem_path, which allows only directory paths containing a key word in include list to be stored in the hdf5 file group structure.Florez Ospina Juan Felipe2024-02-02 16:30:29 +01:00
be43367bc0Updated function create_hdf5_file_from_filesystem_path(ofilename,input_file_system_path), which now can integrate xps ibw files as datasets in the resulting file.Florez Ospina Juan Felipe2024-02-01 15:29:03 +01:00
118191d93aImplemented minor change to output dictionary. keys meant to be dataset's attributes are now access to the key attributes_key, that take the value of a dictionary.Florez Ospina Juan Felipe2024-02-01 15:26:05 +01:00
9a88fbf3aeImplementated module to read file of common types arising in multiphase chemistry group experiments. Status: in progres.Florez Ospina Juan Felipe2024-01-31 16:47:28 +01:00
755d0cbf2eImplemented partially the func create_hdf5_file_from_filesystem_path, which reads a file system path and copies if directory structure in a hdf5 file. Next steps require aligning the structure in the context of our FAIR data file hierarchy and adding data to it. Last commit of the year :). Happy holidays!Florez Ospina Juan Felipe2023-12-22 15:05:28 +01:00
ce0e7e08b1Refined the construct_mask code so that it works with datetime defined intervals.Florez Ospina Juan Felipe2023-11-09 16:26:54 +01:00
bf4d03f369Added output_files' folder, environment.yml file, and jupyter notebook demo on how to create and visualize hdf5 files.Florez Ospina Juan Felipe2023-11-02 15:49:35 +01:00
25c0f07cc3Implemented 1) function to add metadata to root folder in existing hdf5 file, 2) piece of code to display root folder's metadata on treemap's hoover.Florez Ospina Juan Felipe2023-11-02 15:46:14 +01:00
86be738216Rerun jupyter notebook and updated FileList.h5 and napp_plotlib.py due to changes on code dependencies.Florez Ospina Juan Felipe2023-10-31 14:33:25 +01:00
cc52fafc44Implemented a few new features such as create_group_hierarchy of any depth, get_parent_child_relationships, and display_group_hierarchy_on_a_treemap. Additionally, unified read_hdf5_as_dataframe's and documented it better.Florez Ospina Juan Felipe2023-10-31 14:30:34 +01:00
87b256d93eUpdated create_hdf5_file.py with input, enabling either botton-up and top-down construction of group hierarchies. So, far, implementation works for top-down construction where group hierarchies can be of any depth and defined by categorical columns.Florez Ospina Juan Felipe2023-10-26 14:44:53 +02:00
d95a08caaaImplemented partially validation of group hierarchy input param. It needs to be extended to deeper group hierarchies and perhaps output nonconforming lowerlevel groups.Florez Ospina Juan Felipe2023-10-20 16:56:27 +02:00
7eb1686d1fModified creat_hdf5_file to admit group_by_list of functions that will be used to create group hierarchy.Florez Ospina Juan Felipe2023-10-19 16:57:33 +02:00
7ff1d8d09bImplemented package to interface openbis with python and upload hdf5 file data from Thorsten data into openbis.Florez Ospina Juan Felipe2023-10-18 17:00:01 +02:00
0248ca3c6eAdded a new function read_hdf5_as_dataframe_v2 which supports hdf5 files without datasets and replaced the dataframe construction procedure to a column wise basis.Florez Ospina Juan Felipe2023-10-10 09:00:41 +02:00
3e9a26f147Initial commit. hdf5_lib.py contains functions to read hdf5 file as dataframe. and napp_plotlib contains functions to plot image and spectra. jupyter notebook included to illustrate functions.Florez Ospina Juan Felipe2023-08-22 14:28:27 +02:00
Florez Ospina Juan Felipe