14ae29bf3cCorrected parsing problem from hdf5 to yaml attribute. Single element arrays are now represented as a scalar as opposed to a list with a single element.Florez Ospina Juan Felipe2024-04-26 12:54:41 +02:00
c64cad6779Removed this workflow because it is redundant. Replaced this with active creation of review branch in gitlabFlorez Ospina Juan Felipe2024-04-24 17:05:25 +02:00
074d2e3954Removed config_file output file naming and instead user now inputs desired output filename. Also added input argument to introduce root level metadata.Florez Ospina Juan Felipe2024-04-18 19:14:06 +02:00
a1c88fdb5aAdded lines to flatten (shorten) original directory paths in the resulting hdf5 file.Florez Ospina Juan Felipe2024-04-17 15:20:26 +02:00
8005b60579Included a boolean input argument hdf5_upload to deactivate hdf5 upload for testing.Florez Ospina Juan Felipe2024-04-07 17:09:01 +02:00
edd1bbf5beAdded an import and treemap to png statemets, but for some reason didnot work, and took forever to run. So, I left the lines but for now commented them out.Florez Ospina Juan Felipe2024-04-07 16:55:37 +02:00
5e70d9158bDeleted function third_complete_metadata_review() because forth_complete_metadata_review() is the same. Also, modified a substring of their name from complete to submit and submit to save for clarity. Usually submission is the last step of a review process.Florez Ospina Juan Felipe2024-04-05 17:10:34 +02:00
d68dc98070Implemented some safeguards that enable only commits of untracked metadata review filesFlorez Ospina Juan Felipe2024-04-04 14:20:13 +02:00
719e9d6672Repurposed the role of the config_file.py. Now it only provides functions to select the file_readers based on group id and produce a created_at timestamp.Florez Ospina Juan Felipe2024-04-03 13:55:54 +02:00
f9b31c06fdReimplemented file filtering, first file extension contraints are imposed and then file keyword contraints.Florez Ospina Juan Felipe2024-04-03 13:49:16 +02:00
9cde013be0Modified node values as the number of children of each group. When nodes are datasets, their value is 1.Florez Ospina Juan Felipe2024-04-02 18:48:50 +02:00
9071120e50Refactored code to read .dat and .txt files in binary mode first rb, then the prespecified encoding is used to decode the lines. This is to have more control over the decoding process and be able to better spot possible encoding errors.Florez Ospina Juan Felipe2024-04-02 18:35:04 +02:00
39cae66936Implemented a two important changes. 1. filename of output file is not passed as input but it is automatically computed based on an input config_param dict. 2) input filenames in file system path are now filtered on an initial walk through the directory tree. This is to use stored path filenames for prunning directory tree, later on.Florez Ospina Juan Felipe2024-04-02 17:33:58 +02:00
2b568ff05aImplemented jupyter notebook to run data integration workflow. Tested all usecases defined in config. So far so good.Florez Ospina Juan Felipe2024-03-28 18:22:40 +01:00
6fb5253d21Corrected a few bugs; deletion of useless buggy line and configuration of text reader with latin-1 encoding for a few cases.Florez Ospina Juan Felipe2024-03-28 18:20:57 +01:00
bbff419313Removed strange bug when reading .TXT smps files. Specified latin-1 encoding and relaxed error detection to ignore.Florez Ospina Juan Felipe2024-03-28 17:43:26 +01:00
06429e6defGeneralized workflow functions to consider reviewer attributes such as initials and type e.g., data-owner and metadata-reviewer.Florez Ospina Juan Felipe2024-03-28 16:11:01 +01:00
37fd603943Completed first version of metadata_review_lib.py. Still need to test and correct possible bugs.Florez Ospina Juan Felipe2024-03-28 13:59:47 +01:00
54e30ef9ecImplemented git add and commit for second metadata review step, and create it function to checkout branches.Florez Ospina Juan Felipe2024-03-27 14:23:16 +01:00
6aa98b71b3Implemented git add and commit for second metadata review step, and create it function to checkout branches.Florez Ospina Juan Felipe2024-03-27 14:22:25 +01:00
Florez Ospina Juan Felipe