|
|
c40d138563
|
Create new version v1.1.0. Data integration pipeline now does disk space check and skips data transfer if destination files already exist.
|
2025-06-28 16:21:26 +02:00 |
|
|
|
f3ff32e049
|
Update to pipelines/data_integration.py. Added feature to use environment variable MOUNT_DRIVE defined in .env file.
|
2025-06-22 12:11:48 +02:00 |
|
|
|
a70b012da5
|
Simplify output dir and file naming
|
2025-06-20 11:47:19 +02:00 |
|
|
|
773b6a6fbe
|
Fix import statement in pipelines.data_integration.py
|
2025-03-14 10:12:57 +01:00 |
|
|
|
064b8b3a62
|
Update import statements in pipelines/data_integration.py. from instruments.readers import ... -> from instruments import ...
|
2025-02-25 09:21:52 +01:00 |
|
|
|
821d314cb6
|
Change import statements with try except to enable explicit import of submodules from import to avoid conflicts with parent project.
|
2025-02-22 17:10:53 +01:00 |
|
|
|
32bba4239a
|
Synch with remote repo
|
2025-02-03 10:31:48 +01:00 |
|
|
|
1ae607f73b
|
Add validation step to yaml file validation to ensure list type and a minimun length for the 'instrument_datafolder' keyword.
|
2025-01-22 15:55:21 +01:00 |
|
|
|
1b2b319295
|
Attempt to dynamically resolve path to dima package, when excecuted from command line.
|
2024-11-24 17:37:38 +01:00 |
|
|
|
d13e10e44f
|
Modified logger setup to create monthly logs
|
2024-11-21 11:46:11 +01:00 |
|
|
|
ca2c98eebc
|
Fixed command line interface bug
|
2024-11-10 18:19:59 +01:00 |
|
|
|
8d17bf267c
|
Major code refactoring and simplifications to enhance modularity. Included a command line interface.
|
2024-11-01 09:52:41 +01:00 |
|
|
|
69b73c26b0
|
Corrected import statements due to dependency name changes
|
2024-10-17 16:52:42 +02:00 |
|
|
|
2a330fcf92
|
Added 'filename_format' attribute to YAML schema. It takes as value a string of comma separated keys from available attributes in YAML file.
|
2024-10-14 16:01:24 +02:00 |
|
|
|
1954542031
|
Fixed bug introduce in logger due to invalid date naming replace : with -
|
2024-10-10 14:29:36 +02:00 |
|
|
|
df2f7b3de6
|
Abstracted reusable steps in integration_sources as dima_pipeline.py and added functionality to make a collection of hdf5 files, where each represents an single experiment of campaign.
|
2024-09-25 15:23:23 +02:00 |
|
|
|
07401c895f
|
Moved src/data_integration_lib.py -> pipelines/data_integration.py
|
2024-09-17 15:32:23 +02:00 |
|
