121 lines
3.2 KiB
Plaintext
121 lines
3.2 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Data integration workflow of experimental campaign\n",
|
|
"\n",
|
|
"In this notebook, we will go through a our data integration workflow. This involves the following steps:\n",
|
|
"\n",
|
|
"1. Specify data integration file through YAML configuration file.\n",
|
|
"2. Create an integrated HDF5 file of experimental campaign from configuration file.\n",
|
|
"3. Display the created HDF5 file using a treemap\n",
|
|
"\n",
|
|
"## Import libraries and modules\n",
|
|
"\n",
|
|
"* Excecute (or Run) the Cell below"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"import os\n",
|
|
"# Set up project root directory\n",
|
|
"root_dir = os.path.abspath(os.curdir)\n",
|
|
"sys.path.append(root_dir)\n",
|
|
"\n",
|
|
"import src.hdf5_vis as hdf5_vis\n",
|
|
"import src.data_integration_lib as dilib\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 1: Specify data integration task through YAML configuration file\n",
|
|
"\n",
|
|
"* Create your configuration file (i.e., *.yaml file) adhering to the example yaml file in the input folder.\n",
|
|
"* Set up input directory and output directory paths and Excecute Cell.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#output_filename_path = 'output_files/unified_file_smog_chamber_2024-04-07_UTC-OFST_+0200_NG.h5'\n",
|
|
"yaml_config_file_path = 'input_files/data_integr_config_file_LI.yaml'\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 2: Create an integrated HDF5 file of experimental campaign.\n",
|
|
"\n",
|
|
"* Excecute Cell. Here we run the function `integrate_data_sources` with input argument as the previously specified YAML config file."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"hdf5_file_path = dilib.integrate_data_sources(yaml_config_file_path)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Display integrated HDF5 file using a treemap\n",
|
|
"\n",
|
|
"* Excecute Cell. A visual representation in html format of the integrated file should be displayed and stored in the output directory folder"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"if isinstance(hdf5_file_path ,list):\n",
|
|
" for path_item in hdf5_file_path :\n",
|
|
" hdf5_vis.display_group_hierarchy_on_a_treemap(path_item)\n",
|
|
"else:\n",
|
|
" hdf5_vis.display_group_hierarchy_on_a_treemap(hdf5_file_path)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "multiphase_chemistry_env",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|