f4f27b7084
Add draft of dima documentation
florez_j2024-11-26 13:40:43 +01:00
11ca454b94
Removed bacause some of the functionalities have been outsourced to other modules src/hdf5_ops.py and src/hdf5_writer.py
florez_j2024-11-26 11:55:06 +01:00
31a129694c
Updated instrument dictionaries with variable descriptor names aligned with CF metadata conventions.
florez_j2024-11-25 15:49:49 +01:00
5c61e2391a
Update to DIMA package path resolution from file.
florez_j2024-11-24 19:45:18 +01:00
1b2b319295
Attempt to dynamically resolve path to dima package, when excecuted from command line.
florez_j2024-11-24 17:37:38 +01:00
1174ffc8b8
Commented out metadata info about group members for a given group. This is to simplify yaml or json representation of the metadata.
florez_j2024-11-24 15:57:54 +01:00
d7f6e52068
Reimplemented code snippet using hdf5_Writer module
florez_j2024-11-24 11:32:26 +01:00
967be876d1
Moved func create_hdf5_file_from_dataframe() from hdf5_lib_part2 into hdf5_write.py
florez_j2024-11-24 11:30:08 +01:00
0330773f08
Moved read_mtable_as_dataframe(filename) to src/hdf5_ops.py
florez_j2024-11-24 11:03:44 +01:00
3122c4482f
Removed logs/ folder. This is usually created locally to trace certain file copying, transfer and conversion processes.
florez_j2024-11-24 10:51:05 +01:00
c257ab6072
Rerun jupyternotebooks to check their functionality after relocating them to notebooks. OpenBis related python scripts still need to be tested.
florez_j2024-11-24 10:45:40 +01:00
02ded9c11a
Add hidden.py to the list. This file may contain sensitive information is only to be accessed locally or from a secure location.
florez_j2024-11-24 10:41:32 +01:00
b24d33ab15
Check whether h5 file being written exists. If so, we do not overwrite it because it may be underdoing refinement, changes or updates, for archiving, sharing, or publishing.
florez_j2024-11-24 10:38:13 +01:00
ca314f971d
Add utility functions add_project_path_to_sys_path() to set up path to DIMA's modules dynamically.
florez_j2024-11-24 10:08:19 +01:00
a928c4ef4c
Moved demos .py to notebooks. Note: Maybe turn them to jupyternotebooks for consistency
florez_j2024-11-24 07:49:50 +01:00
6701bc06ad
Added read_mtable_as_dataframe(filename) back so that jupyter notebook can use it to demonstrate some functionality
florez_j2024-11-23 16:31:29 +01:00
fd92bce802
Implemented sanitize dataframe function to deal with 'O' which may have numbers or strings detected as string types. Then we use it prior to convert dataframe into structured numpy array.
florez_j2024-11-23 16:28:49 +01:00
44073e3816
Replaced read_dataset_from_hdf5file(hdf5_file_path, dataset_path) with HDF5DataOpsManager.extract_dataset_as_dataframe(self,dataset_name)
florez_j2024-10-17 10:46:19 +02:00
f1b2c64f66
Fixed bug when file reader not available. File reader registry now returns a reade that maps input to None.
florez_j2024-10-14 16:03:03 +02:00
2a330fcf92
Added 'filename_format' attribute to YAML schema. It takes as value a string of comma separated keys from available attributes in YAML file.
florez_j2024-10-14 16:01:24 +02:00
1954542031
Fixed bug introduce in logger due to invalid date naming replace : with -
florez_j2024-10-10 14:29:36 +02:00
ea82af2cd5
Cleaned up import statements and comment out path append operations
florez_j2024-10-10 14:27:50 +02:00
7d94ce29dd
Attempt to initialize dima/utils as a module
florez_j2024-10-10 11:53:27 +02:00
1c2588d85f
Attemp to initialize dima as a module
florez_j2024-10-10 11:43:02 +02:00
2a9d69c757
Robustified metadata and dataset extraction methods by requiring explicit load of file obj before their use. Renamed a few functions and fixed types in print statements.
florez_j2024-10-10 11:28:23 +02:00
7653e982a4
Updated function dependencies to reflect changes made to hdf5_ops.py
florez_j2024-10-10 11:02:05 +02:00
6be3b31247
Renamed open_file() --> load_file_obj() and close_file() --> unload_file_obj() to focus more on the management operations on the files that actual file handling operations.
florez_j2024-10-10 10:47:44 +02:00
568f747a69
Robustifed metadata revision methods with error detection conditions and try-except statements. Metadata revision methods now do not have the capability of opening the file.
florez_j2024-10-10 10:39:10 +02:00
31c9db98ca
Changed datetime format output of created_at() function as '%Y-%m-%d %H:%M:%S.%f'
florez_j2024-10-09 16:07:40 +02:00
fe96134383
Fixed bug in HDF5DataOpsManager.append_dataset() and added 'creation_date' metadata attribute when instrument (groups) are created.
florez_j2024-10-09 16:06:44 +02:00
9a3bf77f37
Created file reader for acsm tofware files, updated registry and updated yaml file with instrument specific terms and reader config params.
florez_j2024-10-07 16:18:14 +02:00
c321a17943
Fixed bug, causing input_path normalization operation to damage Windows network drive paths. Basically, os.path.normpath(path_to_input_directory).strip(os.sep) replaced by os.path.normpath(path_to_input_directory).rstrip(os.sep)
florez_j2024-10-07 16:16:12 +02:00
dc7f156367
Updated README.md with guide for intrument dependent file reader extensions and updated TODO.md with pending tasks.
florez_j2024-10-03 11:31:51 +02:00
89e9dd9ab1
Fixed bugs in update_file() method and create_hdf5_file_from_filesystem_path()
florez_j2024-10-03 09:32:25 +02:00
098a79531c
Added new instrument (flagging app) file reading capabilities. It includes two files a flag_reader.py that takes flag.json files produced by the app into a standard intermidiate representation, and a yaml file with instrument dependent description terms. Last, we modified the filereader_registry.py to find the new instrument file reader.
florez_j2024-10-03 09:07:06 +02:00
9b5d777a5b
Added .update_file() method, which enables complementary data structure updates to existing file with same name as append_dir's head.
florez_j2024-10-02 14:38:35 +02:00
aad0a7c3fb
Added file openning mode as input parameter. Now, mode can only take values in ['w','r+']
florez_j2024-10-02 13:54:59 +02:00
4420f81642
Removed construct_attributes_dict(attrs_obj) and replaced by {key: utils.to_serializable_dtype(val) for key, val in obj.attrs.items()}
florez_j2024-10-01 10:42:20 +02:00
4d48e84e50
Made two helper functions private by adding the prefix __
florez_j2024-10-01 09:31:41 +02:00
8cd4b7d925
Deleted annotate_root_dir(filename,annotation_dict: dict), and outsourced functionality to HDF5DataOpsManager.append_metadata() or .update_metadata() at obj_name = '/'
florez_j2024-10-01 09:19:14 +02:00
6f5d4adcee
Implemented metadata append, rename, delete, and update operations on the hdf5 manager object and refactored metadata update script based on yaml file to use said operations.
florez_j2024-09-30 16:32:39 +02:00
afe31288a0
Refactored a few function calls due to ranming changes in utils module
florez_j2024-09-27 08:58:35 +02:00
96dad0bfb1
Renamed to_yaml() as serialize_metadata() and introduce input parameter output_format, which allows yaml or json.
florez_j2024-09-26 16:23:09 +02:00
85b0e5ab74
Performed a few function relocations and deletions from src/hdf5_lib.py into src/hdf5_ops.py and made a copy of previous version as src/hdf5_lib_part2.py
florez_j2024-09-26 15:13:31 +02:00
a92660049f
Moved is_structured_array() and to_serializable_dtype() to utils, ranamed a few functions and propagated changes to dependent modules.
florez_j2024-09-26 14:03:11 +02:00
a57e46d89c
Renamed take_yml_snapshot_of_hdf5_file func as to_yaml func
florez_j2024-09-25 16:49:44 +02:00
7b221599d8
Moved take_yml_snapshot_of_hdf5_file func and associted helper functions from hdf5_vis.py into hdf5_ops.py
florez_j2024-09-25 16:42:44 +02:00
1e93a2c552
Moved take_yml_snapshot_of_hdf5_file func and associted helper functions from hdf5_vis.py into hdf5_ops.py
florez_j2024-09-25 16:40:16 +02:00
10554fc41e
Renamed make_dtype_yaml_compatible func as to_serializable_dtype func
florez_j2024-09-25 16:36:50 +02:00
df2f7b3de6
Abstracted reusable steps in integration_sources as dima_pipeline.py and added functionality to make a collection of hdf5 files, where each represents an single experiment of campaign.
florez_j2024-09-25 15:23:23 +02:00
1e1499c28a
Robustified definition of path_to_input_dir arg or parameter by ensuring is always defined using forward slashes and then is normalized to the os specification. Improved dry run = True of copy directory func.
florez_j2024-09-25 15:12:19 +02:00
0dbec94374
Fixed instrument_dir estimation to be bottom up, ie, based on path to file. Otherwise, it does not work when dima used as submodule
florez_j2024-09-19 15:47:11 +02:00
9eeb9d6380
Moved src/metadata_review_lib.py pipelines/metadata_revision.py
florez_j2024-09-17 16:55:22 +02:00
07401c895f
Moved src/data_integration_lib.py -> pipelines/data_integration.py
florez_j2024-09-17 15:32:23 +02:00
2dd033bcb3
Refactored code into functions to parse and validate yaml condif file and to perform specified data integration task using a pipeline like software structure.
florez_j2024-09-17 15:28:11 +02:00
d63f522588
Major update. Remove file filtering option and outputname input arg. The output name is now the same as the path_to_input_dir + .h5. By default, the hdf5 writer preserves second level subdirectories and the rest are flattend. dir filtering is outsource to copy_dir_with_constraints from utils-
florez_j2024-09-16 16:35:09 +02:00
9c641c0dae
Restructured a bit to include the default case of copying an imnput directory without any constraints. Also, added dry_run input argument that returns a path to files dict representation of output directory without making an actual copy. Useful when input directory is already safe to work with directly
florez_j2024-09-16 15:38:30 +02:00
7a9f7a8c59
Renamed parameter 'input_file_system_path' to 'path_to_input_directory' for clarity.
florez_j2024-09-16 14:24:55 +02:00
e8e2473ebe
Added new method to retreive metadata from h5file at a given obj path
florez_j2024-09-13 14:52:07 +02:00
96a2e96b6a
Fixed import statement after module's relocation
florez_j2024-08-23 16:23:57 +02:00
e4b04b4484
Modified to use filereader_registry.py.
florez_j2024-08-23 16:10:23 +02:00
9789d312f9
Removed and splitted into instruments/readers/filereader_registry.py instruments/readers/g5505_text_reader.py instruments/readers/xps_ibw_reader.py
florez_j2024-08-23 16:09:04 +02:00
d866c8f9f9
Split instruments/readers/g5505_file_reader.py into a fileregistry.py and independent file readers. This is to improve instrument modularity and additions
florez_j2024-08-23 16:06:44 +02:00
aad51fb1b8
Renamed to reflect better the functionality of the file
florez_j2024-08-23 15:50:14 +02:00
d985115125
Integrated copy h5 file into group functionality, imported from g5505_file_reader
florez_j2024-08-23 15:47:04 +02:00
a4b7c6a8b0
Moved copy_file_in_group() into hdf5_lib.py because it is not really doing the same role of all filereaders
florez_j2024-08-23 15:45:32 +02:00
b5c200d588
Moved all yaml files with dictionary terms for each instrument to dictionaries folder
florez_j2024-08-23 14:32:23 +02:00
18165eca1a
Modified import statements to account for reader module's relocation.
florez_j2024-08-23 13:27:26 +02:00
15b76f7704
Fixed a few import dependencies after relocating this file.
florez_j2024-08-23 10:57:13 +02:00
a0f44a1f4b
Moved src/g5505_file_reader.py -> instruments/readers/g5505_file_reader.py to increase modularity with respect to new intrument additions.
florez_j2024-08-23 10:11:29 +02:00
1112a214e9
Moved src/g5505_utils.py to utils/g5505_utils.py
florez_j2024-08-23 07:27:39 +02:00
d7fc38abd9
Moved get_parent_relationships func into hdf5_vis.py and cleaned up unused import statements
florez_j2024-08-22 09:50:26 +02:00
05d1133e32
Moved get_parent_child_relationships() funct from hdf5_lib.py tinto hdf5_vis.py to avoid circular dependency between the lower level and higher level module. Thus removed also src.hdf5_lib.py import statement.
florez_j2024-08-22 09:47:57 +02:00
d7c7808400
Implemented method for appending new attributes to an specific object.
florez_j2024-08-16 09:32:58 +02:00
bb250e9940
Implemented method to reformat a given column in a datatable holding datetime info into a desired datetime format. During data integration this will serve to normalize datatime formats across data tables
florez_j2024-08-16 08:08:28 +02:00
florez_j