|
|
02a7c4d834
|
Performed a few function relocations and deletions from src/hdf5_lib.py into src/hdf5_ops.py and made a copy of previous version as src/hdf5_lib_part2.py
|
2024-09-26 15:13:31 +02:00 |
|
|
|
8f9e2fc594
|
Moved is_structured_array() and to_serializable_dtype() to utils, ranamed a few functions and propagated changes to dependent modules.
|
2024-09-26 14:03:11 +02:00 |
|
|
|
7ab615019a
|
Renamed take_yml_snapshot_of_hdf5_file func as to_yaml func
|
2024-09-25 16:49:44 +02:00 |
|
|
|
57d49a8db0
|
Moved take_yml_snapshot_of_hdf5_file func and associted helper functions from hdf5_vis.py into hdf5_ops.py
|
2024-09-25 16:42:44 +02:00 |
|
|
|
7304655ba5
|
Moved take_yml_snapshot_of_hdf5_file func and associted helper functions from hdf5_vis.py into hdf5_ops.py
|
2024-09-25 16:40:16 +02:00 |
|
|
|
32ba2a13cd
|
Renamed make_dtype_yaml_compatible func as to_serializable_dtype func
|
2024-09-25 16:36:50 +02:00 |
|
|
|
3e143fb9c7
|
Abstracted reusable steps in integration_sources as dima_pipeline.py and added functionality to make a collection of hdf5 files, where each represents an single experiment of campaign.
|
2024-09-25 15:23:23 +02:00 |
|
|
|
dd8fc1a906
|
Robustified definition of path_to_input_dir arg or parameter by ensuring is always defined using forward slashes and then is normalized to the os specification. Improved dry run = True of copy directory func.
|
2024-09-25 15:12:19 +02:00 |
|
|
|
90d43a46f8
|
Fixed instrument_dir estimation to be bottom up, ie, based on path to file. Otherwise, it does not work when dima used as submodule
|
2024-09-19 15:47:11 +02:00 |
|
|
|
0e354a0f14
|
Moved src/metadata_review_lib.py pipelines/metadata_revision.py
|
2024-09-17 16:55:22 +02:00 |
|
|
|
de859102ab
|
Moved src/data_integration_lib.py -> pipelines/data_integration.py
|
2024-09-17 15:32:23 +02:00 |
|
|
|
59861c3aa8
|
Refactored code into functions to parse and validate yaml condif file and to perform specified data integration task using a pipeline like software structure.
|
2024-09-17 15:28:11 +02:00 |
|
|
|
7b3b453db1
|
Major update. Remove file filtering option and outputname input arg. The output name is now the same as the path_to_input_dir + .h5. By default, the hdf5 writer preserves second level subdirectories and the rest are flattend. dir filtering is outsource to copy_dir_with_constraints from utils-
|
2024-09-16 16:35:09 +02:00 |
|
|
|
eec38f61d7
|
Restructured a bit to include the default case of copying an imnput directory without any constraints. Also, added dry_run input argument that returns a path to files dict representation of output directory without making an actual copy. Useful when input directory is already safe to work with directly
|
2024-09-16 15:38:30 +02:00 |
|
|
|
6d91c043f8
|
Renamed parameter 'input_file_system_path' to 'path_to_input_directory' for clarity.
|
2024-09-16 14:24:55 +02:00 |
|
|
|
85b4909713
|
Fixed import statement
|
2024-09-13 15:11:25 +02:00 |
|
|
|
0f913e5002
|
move def get_parent_child_relationships(file: h5py.File) from ..._vis.py to ..._ops.py
|
2024-09-13 14:59:11 +02:00 |
|
|
|
4813359a4f
|
src/hdf5_data_extraction.py -> src/hdf5_ops.py
|
2024-09-13 14:55:12 +02:00 |
|
|
|
4525c1ba04
|
Added new method to retreive metadata from h5file at a given obj path
|
2024-09-13 14:52:07 +02:00 |
|
|
|
3e1a46ebc7
|
Fixed import statement after module's relocation
|
2024-08-23 16:23:57 +02:00 |
|
|
|
926dc9208a
|
Modified to use filereader_registry.py.
|
2024-08-23 16:10:23 +02:00 |
|
|
|
1e0da55abc
|
Removed and splitted into instruments/readers/filereader_registry.py instruments/readers/g5505_text_reader.py instruments/readers/xps_ibw_reader.py
|
2024-08-23 16:09:04 +02:00 |
|
|
|
0a58e86bcb
|
Split instruments/readers/g5505_file_reader.py into a fileregistry.py and independent file readers. This is to improve instrument modularity and additions
|
2024-08-23 16:06:44 +02:00 |
|
|
|
cfae414b0e
|
Renamed to reflect better the functionality of the file
|
2024-08-23 15:50:14 +02:00 |
|
|
|
b499ef2845
|
Integrated copy h5 file into group functionality, imported from g5505_file_reader
|
2024-08-23 15:47:04 +02:00 |
|
|
|
7ad4e686a7
|
Moved copy_file_in_group() into hdf5_lib.py because it is not really doing the same role of all filereaders
|
2024-08-23 15:45:32 +02:00 |
|
|
|
33ad9acdd4
|
Moved all yaml files with dictionary terms for each instrument to dictionaries folder
|
2024-08-23 14:32:23 +02:00 |
|
|
|
f20e02d62f
|
Added ACSM_TOFWARE metadata descriptions
|
2024-08-23 14:23:32 +02:00 |
|
|
|
17dd1f1864
|
Modified import statements to account for reader module's relocation.
|
2024-08-23 13:27:26 +02:00 |
|
|
|
a33e2b681f
|
Fixed a few import dependencies after relocating this file.
|
2024-08-23 10:57:13 +02:00 |
|
|
|
e76ed79f1e
|
Moved src/g5505_file_reader.py -> instruments/readers/g5505_file_reader.py to increase modularity with respect to new intrument additions.
|
2024-08-23 10:11:29 +02:00 |
|
|
|
b22b0e94e4
|
Moved src/g5505_utils.py to utils/g5505_utils.py
|
2024-08-23 07:27:39 +02:00 |
|
|
|
9d917226af
|
Moved get_parent_relationships func into hdf5_vis.py and cleaned up unused import statements
|
2024-08-22 09:50:26 +02:00 |
|
|
|
da6cca1632
|
Moved get_parent_child_relationships() funct from hdf5_lib.py tinto hdf5_vis.py to avoid circular dependency between the lower level and higher level module. Thus removed also src.hdf5_lib.py import statement.
|
2024-08-22 09:47:57 +02:00 |
|
|
|
d6dce9a392
|
Implemented method for appending new attributes to an specific object.
|
2024-08-16 09:32:58 +02:00 |
|
|
|
ea13f2b71b
|
Implemented method to reformat a given column in a datatable holding datetime info into a desired datetime format. During data integration this will serve to normalize datatime formats across data tables
|
2024-08-16 08:08:28 +02:00 |
|
|
|
3fc96a89d2
|
Added method to reformat columns containing datetime byte strings into a desired datetime formated object
|
2024-08-14 16:22:28 +02:00 |
|
|
|
6a0ae327ae
|
Changed link to descriptions according to new instrument folder location.
|
2024-08-12 13:40:43 +02:00 |
|
|
|
9d77f4815c
|
Modified code to point to new instrument folders location. Also, upgrated code to accept either a user specified location or the default location
|
2024-08-12 13:40:01 +02:00 |
|
|
|
ac23822b0e
|
Moved instruments folder outside src/.
|
2024-08-12 10:09:21 +02:00 |
|
|
|
291a5cc1b6
|
Implemented dataset append method in HDF5DatOpsAPI
|
2024-08-09 15:25:09 +02:00 |
|
|
|
52a2303054
|
Developed a class to manage data operations on a given hdf5 file
|
2024-08-09 13:23:54 +02:00 |
|
|
|
fbc8c5ebc3
|
Removed time stamp configuration attributes from ACSM_TOFWARE, because it can be messy for a configuration file.
|
2024-08-08 11:24:41 +02:00 |
|
|
|
780b2302b3
|
Updated file with new instrument configuration ACSM.
|
2024-08-07 16:38:52 +02:00 |
|
|
|
99fb2de6d8
|
Moved ext_to_reader_dict to g5505_file_reader.py and replaced redear selection based on g5505_reader.select_file_reader(hdf5_file_path).
|
2024-08-07 16:30:36 +02:00 |
|
|
|
381d330ee6
|
Moved hdf5_file_path to file reader mapping and extension definitions to g5505_file_reader_module.py. Created functions to compute file_reader key from path to file in the hdf5 file and select the reader based on the key. This should enable more modular file reader selection.
|
2024-08-07 16:21:22 +02:00 |
|
|
|
1acbd2f758
|
Modified reader to output table_preamble as a dataset as opposed to attributes of the file. I believe this is better for readability of the metadata given that those preambles can sometimes contain large ammounts of text.
|
2024-08-02 14:37:06 +02:00 |
|
|
|
d7f7223d31
|
Modified .yaml config files to satisty metadata naming expectations.
|
2024-07-17 08:50:24 +02:00 |
|
|
|
79a593cbbb
|
Changed names of expected root level metadata attributes.
|
2024-07-17 08:48:47 +02:00 |
|
|
|
6c50625002
|
Added attribution insertion order tracking at the root level and reorganized a few import statements.
|
2024-07-17 08:41:40 +02:00 |
|
