|
|
bea19f5ed8
|
Reorganize data for testing
|
2025-09-30 00:51:21 +02:00 |
|
|
|
cde421edda
|
cleanup config organization
|
2025-09-30 00:23:12 +02:00 |
|
|
|
203bd9d740
|
feat: add to run_config.yaml number of processes for the dask cluster and option to select start and end dates for processing
|
2025-09-29 10:09:04 +02:00 |
|
|
|
872d2c5ac4
|
fix: Add retry logic and consistent partitioning for distributed processing
|
2025-09-12 10:29:15 +02:00 |
|
|
|
d3a7448883
|
Improve conversion from original csv or zip files to parquet with more robust schema definition
|
2025-09-11 16:18:21 +02:00 |
|
|
|
641871a567
|
test: add test for path extraction from file directory
|
2025-09-09 19:14:30 +02:00 |
|
|
|
f437b1c5fe
|
feat: add the possibility to decide the saving partition schema between date or date/hour
|
2025-09-09 17:09:38 +02:00 |
|
|
|
29e2351341
|
Feat: user can now decide frequency of repartition of dask dataframes after being loaded (both hk and pbp)
|
2025-09-09 16:03:42 +02:00 |
|
|
|
0e932e9a70
|
test: add parquet files to use for testing
|
2025-09-09 14:25:13 +02:00 |
|
|
|
21e14ae2f1
|
chore: update run config
|
2025-08-22 18:20:26 +02:00 |
|
|
|
f42a308474
|
Feature: processing is now divided in time chunks to reduce size of dask graph
|
2025-08-22 11:37:58 +02:00 |
|
|
|
063b01e73f
|
Test: PbP and HK parquet files added for testing
|
2025-08-20 19:08:28 +02:00 |
|
|
|
a2cc520ff2
|
feat: possibility to choose between running locally and via slurm cluster
|
2025-08-14 11:48:52 +02:00 |
|
|
|
18b8635147
|
chore: moved config files part 2
|
2025-08-14 10:42:44 +02:00 |
|
|
|
d470f2e811
|
chore: moved config files
|
2025-08-14 10:41:44 +02:00 |
|
|
|
053d0d5b75
|
Refactor: separte the workflow for the option to calculate distributions for BC (numb and amss), scattering and time delay
|
2025-08-13 16:26:39 +02:00 |
|
|
|
5f3d25817c
|
Refactor: moved calibration of PbP data to new file and clear separation between run_config and instr_config
|
2025-08-13 15:28:57 +02:00 |
|
|
|
6f87b4cc79
|
Refactor: implemented input run_config.yaml file
|
2025-08-13 14:09:58 +02:00 |
|
|
|
74c5a77b48
|
test: manually added case-specific instrument_config parameters
|
2025-08-04 13:03:24 +02:00 |
|
|
|
ad897be5db
|
feat: pre-populate YAML export with calibration and threshold placeholders
|
2025-08-04 12:58:17 +02:00 |
|
|
|
f5c5420209
|
feat: add CLI utility to convert SP2-XR .ini files to editable YAML configs
|
2025-08-04 12:33:09 +02:00 |
|
|
|
1d8f12ee35
|
feat: add ini→yaml conversion and tests for SP2-XR instrument parameters
|
2025-08-04 12:26:03 +02:00 |
|
|
|
5690fcdcaf
|
refactor: split read_csv_files_with_dask into modular load, enrich, and save functions
|
2025-07-25 20:28:24 +02:00 |
|
|
|
399df18e81
|
test: add real-data tests for PbP and HK ZIP input files
|
2025-07-25 15:33:33 +02:00 |
|
|
|
b443aa5db8
|
style: Ruff auto‑fixes in SP2XR_toolkit
|
2025-07-25 12:12:38 +02:00 |
|
