mirror of
https://gitea.psi.ch/APOG/acsm-fairifier.git
synced 2026-01-19 13:25:23 +01:00
269 lines
9.0 KiB
Plaintext
269 lines
9.0 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# ACSM Data Chain Workflow\n",
|
|
"\n",
|
|
"In this notebook, we will go through our **ACSM Data Chain**. This involves the following steps:\n",
|
|
"\n",
|
|
"1. Run the data integration pipeline to retrieve ACSM input data and prepare it for processing. \n",
|
|
"2. Perform QC/QA analysis. \n",
|
|
"3. (Optional) Conduct visual analysis for flag validation. \n",
|
|
"4. Prepare input data and QC/QA analysis results for submission to the EBAS database. \n",
|
|
"\n",
|
|
"## Import Libraries and Data Chain Steps\n",
|
|
"\n",
|
|
"* Execute (or Run) the cell below."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"import os\n",
|
|
"# Set up project root directory\n",
|
|
"\n",
|
|
"\n",
|
|
"notebook_dir = os.getcwd() # Current working directory (assumes running from notebooks/)\n",
|
|
"project_path = os.path.normpath(os.path.join(notebook_dir, \"..\")) # Move up to project root\n",
|
|
"dima_path = os.path.normpath(os.path.join(project_path, \"dima\")) # Move up to project root\n",
|
|
"\n",
|
|
"if project_path not in sys.path: # Avoid duplicate entries\n",
|
|
" sys.path.append(project_path)\n",
|
|
"if dima_path not in sys.path:\n",
|
|
" sys.path.insert(0,dima_path)\n",
|
|
"#sys.path.append(os.path.join(root_dir,'dima','instruments'))\n",
|
|
"#sys.path.append(os.path.join(root_dir,'dima','src'))\n",
|
|
"#sys.path.append(os.path.join(root_dir,'dima','utils'))\n",
|
|
"\n",
|
|
"#import dima.visualization.hdf5_vis as hdf5_vis\n",
|
|
"#import dima.pipelines.data_integration as data_integration\n",
|
|
"import subprocess\n",
|
|
"\n",
|
|
"\n",
|
|
"for item in sys.path:\n",
|
|
" print(item)\n",
|
|
"\n",
|
|
"from dima.pipelines.data_integration import run_pipeline as get_campaign_data\n",
|
|
"from pipelines.steps.apply_calibration_factors import main as apply_calibration_factors\n",
|
|
"from pipelines.steps.generate_flags import main as generate_flags\n",
|
|
"from pipelines.steps.prepare_ebas_submission import main as prepare_ebas_submission \n",
|
|
"from pipelines.steps.update_actris_header import main as update_actris_header\n",
|
|
"from pipelines.steps.utils import load_project_yaml_files\n",
|
|
"\n",
|
|
"campaign_descriptor = load_project_yaml_files(project_path, \"campaignDescriptor.yaml\")\n",
|
|
"YEAR = campaign_descriptor['year']\n",
|
|
"STATION = campaign_descriptor['station']"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 1: Retrieve Input Data from a Network Drive\n",
|
|
"\n",
|
|
"* Create a configuration file (i.e., a `.yaml` file) following the example provided in the input folder.\n",
|
|
"* Set up the input and output directory paths.\n",
|
|
"* Execute the cell."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"path_to_config_file = '../campaignDescriptor.yaml'\n",
|
|
"paths_to_hdf5_files = get_campaign_data(path_to_config_file)\n",
|
|
"# Select campaign data file and append directory\n",
|
|
"CAMPAIGN_DATA_FILE = paths_to_hdf5_files[0]\n",
|
|
"APPEND_DATA_DIR = os.path.splitext(CAMPAIGN_DATA_FILE)[0]\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 2: Calibrate Input Campaign Data and Save Data Products\n",
|
|
"\n",
|
|
"* Set up the input and output directory paths.\n",
|
|
"* Execute the cell."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"path_to_data_file = CAMPAIGN_DATA_FILE\n",
|
|
"path_to_calibration_file = '../pipelines/params/calibration_factors.yaml'\n",
|
|
"#dataset_name = 'ACSM_TOFWARE/2024/ACSM_JFJ_2024_timeseries.txt/data_table'\n",
|
|
"#command = ['python', 'pipelines/steps/apply_calibration_factors.py', path_to_data_file, dataset_name, path_to_calibration_file]\n",
|
|
"#status = subprocess.run(command, capture_output=True, check=True)\n",
|
|
"#print(status.stdout.decode())\n",
|
|
"\n",
|
|
"apply_calibration_factors(path_to_data_file,path_to_calibration_file)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 3: Perform QC/QA Analysis\n",
|
|
"\n",
|
|
"* Generate automated flags based on validity thresholds for diagnostic channels.\n",
|
|
"* (Optional) Generate manual flags using the **Data Flagging App**, accessible at: \n",
|
|
" [http://localhost:8050/](http://localhost:8050/)\n",
|
|
"* Execute the cell."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dataset_name = f'ACSM_TOFWARE/{YEAR}/ACSM_{STATION}_{YEAR}_meta.txt/data_table'\n",
|
|
"path_to_config_file = 'pipelines/params/validity_thresholds.yaml'\n",
|
|
"#command = ['python', 'pipelines/steps/compute_automated_flags.py', path_to_data_file, dataset_name, path_to_config_file]\n",
|
|
"#status = subprocess.run(command, capture_output=True, check=True)\n",
|
|
"#print(status.stdout.decode())\n",
|
|
"generate_flags(path_to_data_file, 'diagnostics')\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## (Optional) Step 3.1: Inspect Previously Generated Flags for Correctness\n",
|
|
"\n",
|
|
"* Perform flag validation using the Jupyter Notebook workflow available at: \n",
|
|
" [../notebooks/demo_visualize_diagnostic_flags_from_hdf5_file.ipynb](demo_visualize_diagnostic_flags_from_hdf5_file.ipynb)\n",
|
|
"* Follow the notebook steps to visually inspect previously generated flags."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 4: Apply Diagnostic and Manual Flags to Variables of Interest\n",
|
|
"\n",
|
|
"* Generate flags for species based on previously collected QC/QA flags.\n",
|
|
"* Execute the cell."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#CAMPAIGN_DATA_FILE = '../data/collection_JFJ_2024_2025-04-08_2025-04-08.h5'\n",
|
|
"path_to_data_file = CAMPAIGN_DATA_FILE\n",
|
|
"dataset_name = f'ACSM_TOFWARE/{YEAR}/ACSM_{STATION}_{YEAR}_meta.txt/data_table'\n",
|
|
"path_to_config_file = 'pipelines/params/validity_thresholds.yaml'\n",
|
|
"#command = ['python', 'pipelines/steps/compute_automated_flags.py', path_to_data_file, dataset_name, path_to_config_file]\n",
|
|
"#status = subprocess.run(command, capture_output=True, check=True)\n",
|
|
"#print(status.stdout.decode())\n",
|
|
"generate_flags(path_to_data_file, 'species')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 5: Generate Campaign Data in EBAS Format\n",
|
|
"\n",
|
|
"* Gather and set paths to the required data products produced in the previous steps.\n",
|
|
"* Execute the cell."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"DATA_DIR = f\"{APPEND_DATA_DIR}/ACSM_TOFWARE_processed/{YEAR}\"\n",
|
|
"FLAGS_DIR = f\"{APPEND_DATA_DIR}/ACSM_TOFWARE_flags/{YEAR}\"\n",
|
|
"\n",
|
|
"PATH1 = f\"{DATA_DIR}/ACSM_{STATION}_{YEAR}_timeseries_calibrated.csv\"\n",
|
|
"PATH2 = f\"{DATA_DIR}/ACSM_{STATION}_{YEAR}_timeseries_calibrated_err.csv\"\n",
|
|
"PATH3 = f\"{DATA_DIR}/ACSM_{STATION}_{YEAR}_timeseries_calibration_factors.csv\"\n",
|
|
"PATH4 = f\"{FLAGS_DIR}/ACSM_{STATION}_{YEAR}_timeseries_flags.csv\"\n",
|
|
"\n",
|
|
"month = \"2-3\"\n",
|
|
"update_actris_header('../campaignDescriptor.yaml')\n",
|
|
"prepare_ebas_submission([PATH1, PATH2, PATH3], PATH4, month)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 6: Save Data Products to an HDF5 File\n",
|
|
"\n",
|
|
"* Gather and set paths to the required data products produced in the previous steps.\n",
|
|
"* Execute the cell.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import dima.src.hdf5_ops as dataOps \n",
|
|
"#print(os.curdir)\n",
|
|
"\n",
|
|
"\n",
|
|
"dataManager = dataOps.HDF5DataOpsManager(CAMPAIGN_DATA_FILE)\n",
|
|
"print(dataManager.file_path)\n",
|
|
"print(APPEND_DATA_DIR)\n",
|
|
"dataManager.update_file(APPEND_DATA_DIR)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dataManager = dataOps.HDF5DataOpsManager(path_to_data_file)\n",
|
|
"dataManager.load_file_obj()\n",
|
|
"dataManager.extract_and_load_dataset_metadata()\n",
|
|
"df = dataManager.dataset_metadata_df\n",
|
|
"print(df.head(10))\n",
|
|
"dataManager.unload_file_obj()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "dash_multi_chem_env",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|