|
|
9aa7c0ece8
|
Clean import statements
|
2025-06-19 20:49:36 +02:00 |
|
|
|
33d1f20d38
|
Update src/hdf5_writer.py to consider data lineage metadata in data ingestion process
|
2025-06-07 15:31:13 +02:00 |
|
|
|
6b43c95a8d
|
Implemented hdf5_file_reader.py and updated register.yaml and hdf5_writer.py. This replaces previous function __copy_file_in_group().
|
2025-02-25 12:25:15 +01:00 |
|
|
|
14b738818c
|
Merge branch 'main' into 'feature/DB_for_FileReader_Repo'
# Conflicts:
# instruments/filereader_registry.py
# pipelines/data_integration.py
# src/hdf5_writer.py
|
2025-02-25 10:41:02 +01:00 |
|
|
|
68344964ac
|
Implemented create_hdf5_from_filesystem_new() using new instrument readers cml interface and subprocesses. This facilitates extension of file reading capabilities by collaborators without requiring changes to file_registry.py. Only additions in folders and registry.yaml.
|
2025-02-24 18:48:03 +01:00 |
|
|
|
2cdd6925af
|
Merge branch 'main' of https://gitlab.psi.ch/5505-public/dima
|
2025-02-22 18:02:45 +01:00 |
|
|
|
bc1d65d469
|
Fix import for filereader_registry.py after moving it from intruments/readers/ one level above.
|
2025-02-22 17:59:00 +01:00 |
|
|
|
81be6b54c8
|
Change import statements with try except to enable explicit import of submodules from import to avoid conflicts with parent project.
|
2025-02-22 17:10:53 +01:00 |
|
|
|
df0aca97df
|
Implement data_lineage_metadata.json detection and then use it to annotate associated file.
|
2025-02-10 15:56:34 +01:00 |
|
|
|
46ca26a983
|
Enable instrumentFolder of form <instFolder>/<category>/ to be trasfered without flatenning
|
2025-02-07 16:24:21 +01:00 |
|
|
|
36780d1a63
|
Add try except block to trigger errors for invalid group names.
|
2025-02-06 16:07:45 +01:00 |
|
|
|
58386ca10b
|
Add property to extracted dataset as dataframe. Now time column is of datetime type to facilitate downstream procesing.
|
2025-02-04 17:23:32 +01:00 |
|
|
|
d89aebd861
|
Implement method in hdf5 manager to infer datetime column in dataset
|
2025-02-04 17:13:01 +01:00 |
|
|
|
e358d4ab64
|
Synch with remote repo
|
2025-02-03 10:31:48 +01:00 |
|
|
|
1b2184d8e1
|
Update unload operation to remove reference and fix logic error to dataset metadata extraction.
|
2025-01-24 10:28:43 +01:00 |
|
|
|
fc139e0ae5
|
Relocated to visualization module
|
2024-12-02 15:39:41 +01:00 |
|
|
|
d79877cc9b
|
Moved hdf5_lib.py to visualization folder
|
2024-12-02 15:34:44 +01:00 |
|
|
|
22e201d063
|
Removed bacause some of the functionalities have been outsourced to other modules src/hdf5_ops.py and src/hdf5_writer.py
|
2024-11-26 11:55:06 +01:00 |
|
|
|
fed9aa0a1b
|
Update to DIMA package path resolution from file.
|
2024-11-24 19:45:18 +01:00 |
|
|
|
6dc89d34fe
|
Commented out metadata info about group members for a given group. This is to simplify yaml or json representation of the metadata.
|
2024-11-24 15:57:54 +01:00 |
|
|
|
3db2eeb041
|
Moved func create_hdf5_file_from_dataframe() from hdf5_lib_part2 into hdf5_write.py
|
2024-11-24 11:30:08 +01:00 |
|
|
|
2740330709
|
Moved read_mtable_as_dataframe(filename) to src/hdf5_ops.py
|
2024-11-24 11:03:44 +01:00 |
|
|
|
53fae27472
|
Check whether h5 file being written exists. If so, we do not overwrite it because it may be underdoing refinement, changes or updates, for archiving, sharing, or publishing.
|
2024-11-24 10:38:13 +01:00 |
|
|
|
2988d816af
|
Added read_mtable_as_dataframe(filename) back so that jupyter notebook can use it to demonstrate some functionality
|
2024-11-23 16:31:29 +01:00 |
|
|
|
870d60a789
|
Improved progress description stdout
|
2024-11-10 18:21:00 +01:00 |
|
|
|
29c6d86583
|
Included cli commands update and serialize to simplify running metadata revision pipeline.
|
2024-10-29 07:56:43 +01:00 |
|
|
|
c0ec7e9b26
|
Moved git related operations from pipelines/ to src/git_ops.py
|
2024-10-28 16:30:34 +01:00 |
|
|
|
a8ceffba8b
|
Corrected import statements due to dependency name changes
|
2024-10-17 16:52:42 +02:00 |
|
|
|
8129949db9
|
Renamed module: src/hdf5_lib.py -> src/hdf5_writer.py
|
2024-10-17 10:53:51 +02:00 |
|
|
|
d49e7ae9d5
|
Replaced read_dataset_from_hdf5file(hdf5_file_path, dataset_path) with HDF5DataOpsManager.extract_dataset_as_dataframe(self,dataset_name)
|
2024-10-17 10:46:19 +02:00 |
|
|
|
1e5581769b
|
Robustified metadata and dataset extraction methods by requiring explicit load of file obj before their use. Renamed a few functions and fixed types in print statements.
|
2024-10-10 11:28:23 +02:00 |
|
|
|
800a5aca49
|
Renamed open_file() --> load_file_obj() and close_file() --> unload_file_obj() to focus more on the management operations on the files that actual file handling operations.
|
2024-10-10 10:47:44 +02:00 |
|
|
|
47f095bdc8
|
Robustifed metadata revision methods with error detection conditions and try-except statements. Metadata revision methods now do not have the capability of opening the file.
|
2024-10-10 10:39:10 +02:00 |
|
|
|
8ea111a0c7
|
Fixed bug in HDF5DataOpsManager.append_dataset() and added 'creation_date' metadata attribute when instrument (groups) are created.
|
2024-10-09 16:06:44 +02:00 |
|
|
|
bd4ced00ba
|
Fixed bug, causing input_path normalization operation to damage Windows network drive paths. Basically, os.path.normpath(path_to_input_directory).strip(os.sep) replaced by os.path.normpath(path_to_input_directory).rstrip(os.sep)
|
2024-10-07 16:16:12 +02:00 |
|
|
|
c103268102
|
Fixed bugs in update_file() method and create_hdf5_file_from_filesystem_path()
|
2024-10-03 09:32:25 +02:00 |
|
|
|
d49f511dbd
|
Added .update_file() method, which enables complementary data structure updates to existing file with same name as append_dir's head.
|
2024-10-02 14:38:35 +02:00 |
|
|
|
ea898ca3c5
|
Added file openning mode as input parameter. Now, mode can only take values in ['w','r+']
|
2024-10-02 13:54:59 +02:00 |
|
|
|
4f0361c6c5
|
Removed construct_attributes_dict(attrs_obj) and replaced by {key: utils.to_serializable_dtype(val) for key, val in obj.attrs.items()}
|
2024-10-01 10:42:20 +02:00 |
|
|
|
1b0c666132
|
Made two helper functions private by adding the prefix __
|
2024-10-01 09:31:41 +02:00 |
|
|
|
14a1d032b9
|
Deleted annotate_root_dir(filename,annotation_dict: dict), and outsourced functionality to HDF5DataOpsManager.append_metadata() or .update_metadata() at obj_name = '/'
|
2024-10-01 09:19:14 +02:00 |
|
|
|
96500063fb
|
Implemented metadata append, rename, delete, and update operations on the hdf5 manager object and refactored metadata update script based on yaml file to use said operations.
|
2024-09-30 16:32:39 +02:00 |
|
|
|
c992662a1f
|
Renamed to_yaml() as serialize_metadata() and introduce input parameter output_format, which allows yaml or json.
|
2024-09-26 16:23:09 +02:00 |
|
|
|
02a7c4d834
|
Performed a few function relocations and deletions from src/hdf5_lib.py into src/hdf5_ops.py and made a copy of previous version as src/hdf5_lib_part2.py
|
2024-09-26 15:13:31 +02:00 |
|
|
|
8f9e2fc594
|
Moved is_structured_array() and to_serializable_dtype() to utils, ranamed a few functions and propagated changes to dependent modules.
|
2024-09-26 14:03:11 +02:00 |
|
|
|
7ab615019a
|
Renamed take_yml_snapshot_of_hdf5_file func as to_yaml func
|
2024-09-25 16:49:44 +02:00 |
|
|
|
57d49a8db0
|
Moved take_yml_snapshot_of_hdf5_file func and associted helper functions from hdf5_vis.py into hdf5_ops.py
|
2024-09-25 16:42:44 +02:00 |
|
|
|
7304655ba5
|
Moved take_yml_snapshot_of_hdf5_file func and associted helper functions from hdf5_vis.py into hdf5_ops.py
|
2024-09-25 16:40:16 +02:00 |
|
|
|
32ba2a13cd
|
Renamed make_dtype_yaml_compatible func as to_serializable_dtype func
|
2024-09-25 16:36:50 +02:00 |
|
|
|
dd8fc1a906
|
Robustified definition of path_to_input_dir arg or parameter by ensuring is always defined using forward slashes and then is normalized to the os specification. Improved dry run = True of copy directory func.
|
2024-09-25 15:12:19 +02:00 |
|
