SP2XR/docs/api-reference.md

API Reference

Core Modules

sp2xr.io

Input/output operations for data conversion and processing.

Functions:

• csv_to_parquet(source_dir, target_dir, config_file, filter_pattern) - Convert CSV/ZIP files to Parquet format
• process_sp2xr_file(file_path, config_file, output_dir) - Process individual SP2XR data files
• read_sp2xr_csv(file_path, schema, **kwargs) - Read SP2XR CSV/ZIP files with schema validation
• load_matching_hk_file(pbp_file, hk_dir, hk_schema) - Load housekeeping file matching a PbP file
• enrich_sp2xr_dataframe(df, filename) - Add time-derived and metadata columns (date, hour, filename)
• save_sp2xr_parquet(df, output_dir, partition_cols) - Save DataFrame to partitioned Parquet format

---

sp2xr.calibration

Complete calibration workflow with flags and processing.

Calibration Functions:

• calibrate_single_particle(ddf, instr_config, run_config) - Complete calibration workflow with quality flags
• calibrate_particle_data(df, config) - Apply scattering and incandescence calibrations to particle data
• apply_calibration(df, config) - Legacy wrapper for backward compatibility
• apply_inc_calibration(df, calib_params) - Apply incandescence calibration (BC mass calculation)
• apply_scatt_calibration(df, calib_params) - Apply scattering calibration (optical diameter calculation)

Calibration Curve Functions:

• polynomial(x, coeffs) - Generic polynomial curve: a0 + a1*x + a2*x^2 + ...
• powerlaw(x, a, b) - Generic power-law calibration: a * x^b

Mass/Diameter Conversion:

• BC_mass_to_diam(mass_fg, material='fullerene') - Convert BC mass (fg) to mass-equivalent diameter (nm)
• BC_diam_to_mass(diam_nm, material='fullerene') - Convert BC diameter (nm) to mass (fg)
• mass2meqDiam(mass, rho_eff) - Calculate mass-equivalent diameter from mass and effective density

Distribution Conversion:

• dNdlogDp_to_dMdlogDp(dNdlogDp, dp_nm, rho) - Convert number distribution to mass distribution

---

sp2xr.calibration_constants

Spline coefficient functions for BC calibration materials.

Core Functions:

• SP2_calibCurveSpline(x, spline_coeffs) - Spline interpolation for density calculations
• SP2_calibCurveSplineCheck(spline_coeffs) - Validate spline coefficient dimensions

Aquadag Calibration Functions:

• Aquadag_RhoVsLogMspline_Var25() - Aquadag density vs. log(mass) spline coefficients (Var25)
• Aquadag_RhoVsLogMspline_Var8() - Aquadag density vs. log(mass) spline coefficients (Var8)
• Aquadag_RhoVsLogDspline_Var25() - Aquadag density vs. log(diameter) spline (Var25)
• Aquadag_RhoVsLogDspline_Var8() - Aquadag density vs. log(diameter) spline (Var8)
• Aquadag_RhoVsLogDspline_Var8_old() - Legacy Aquadag diameter-to-density spline

Fullerene Calibration Functions:

• Fullerene_RhoVsLogMspline_Var2() - Fullerene density vs. log(mass) spline (Var2)
• Fullerene_RhoVsLogMspline_Var5() - Fullerene density vs. log(mass) spline (Var5)
• Fullerene_RhoVsLogMspline_Var8() - Fullerene density vs. log(mass) spline (Var8)
• Fullerene_RhoVsLogMspline_Var8_old() - Legacy Fullerene mass-to-density spline
• Fullerene_RhoVsLogDspline_Var2() - Fullerene diameter-to-density spline (Var2)
• Fullerene_RhoVsLogDspline_Var5() - Fullerene diameter-to-density spline (Var5)
• Fullerene_RhoVsLogDspline_Var8() - Fullerene diameter-to-density spline (Var8)
• Fullerene_RhoVsLogDspline_Var8_old() - Legacy Fullerene diameter-to-density spline

Glassy Carbon Calibration Functions:

• GlassyCarbonAlpha_Mass2Diam() - Glassy carbon mass-to-diameter conversion coefficients
• GlassyCarbonAlpha_Diam2Mass() - Glassy carbon diameter-to-mass conversion coefficients

---

sp2xr.flag_single_particle_data

Quality control flagging based on instrument parameters.

Functions:

• define_flags(df, flag_config) - Define quality control flags (transit time, FWHM, peak intensity, etc.)
• add_thin_thick_flags(df, lag_threshold) - Add thin/thick coating classification flags based on lag time

Constants:

• FLAG_COLS - List of flag column names used in quality filtering

---

sp2xr.distribution

Size/mass distribution calculations.

Main Functions:

• process_histograms(ddf, config, inc_bins, inc_ctrs, scatt_bins, scatt_ctrs, timelag_bins, timelag_ctrs, chunk_start, client) - Calculate size/mass distributions (main workflow function)
• process_hist_and_dist_partition(partition, dt_s, inc_mass_bin_lims, scatt_bin_lims, timelag_bins_lims, calc_dist, flow_dt) - Process histogram and distribution for a single partition

Bin Utilities:

• make_bin_arrays(min_val, max_val, n_bins, log_scale=True) - Create bin arrays for histograms
• bin_lims_to_ctrs(bin_lims) - Convert bin limits to bin centers
• bin_ctrs_to_lims(bin_ctrs) - Convert bin centers to bin limits
• get_dlogp(bin_lims) - Calculate log bin widths (for dN/dlogDp calculations)

Distribution Calculations:

• calculate_histogram(series, bins, dt_s, flow_sccm) - Calculate histogram from series with time/flow normalization
• counts2numConc(counts, dlogDp, dt_s, flow_sccm) - Convert counts to number concentration (dN/dlogDp)
• dNdlogDp_to_dMdlogDp(dNdlogDp, dp_nm, rho) - Convert number distribution to mass distribution

Metadata:

• make_hist_meta(bin_ctrs, dt_index) - Create metadata DataFrame for histogram output

---

sp2xr.resample_pbp_hk

Time resampling functions for data aggregation.

Functions:

• build_dt_summary(pdf, dt_s) - Resample single-particle data to time bins (aggregate particle counts)
• resample_hk_partition(pdf, dt) - Partition-wise resampling of housekeeping data to specified time resolution
• join_pbp_with_flow(ddf_pbp, flow_series, config) - Join particle data with flow measurements
• aggregate_dt(ddf_pbp_dt, ddf_hk_dt, config) - Aggregate PbP and HK data at specified time resolution

---

sp2xr.concentrations

Concentration calculations for different particle types.

Functions:

• add_concentrations(df, dt) - Add BC mass concentration, scattering mass concentration, and number concentration columns

---

sp2xr.helpers

Utility functions for file handling, argument parsing, cluster initialization, and configuration management.

Configuration Management

• load_and_resolve_config(args) - Load and merge YAML configuration with command-line arguments
• load_yaml_cfg(path) - Load YAML configuration file
• parse_args() - Parse command-line arguments for pipeline scripts
• apply_sets(config, set_args) - Apply command-line overrides (--set) to configuration
• get(config, key_path, default=None) - Get nested configuration value using dot notation
• choose(cli_val, config, key_path, default) - Choose between CLI argument and config value
• validate_config_compatibility(config) - Validate configuration compatibility and consistency

Cluster/Dask Management

• initialize_cluster(config) - Initialize Dask cluster (local or SLURM)
• make_slurm_cluster(config) - Create SLURM cluster with specified resources
• make_local_cluster(config) - Create local Dask cluster with auto-detected resources

File Operations

• find_files(directory, pattern) - Recursively find files matching pattern
• find_matching_hk_file(pbp_file, hk_dir) - Find housekeeping file matching a PbP file
• extract_base_filename(file_path) - Extract base filename from SP2XR file path
• extract_sp2xr_filename_parts(filename) - Extract filename components (timestamp, instrument ID, etc.)

Time/Partition Management

• extract_partitioned_datetimes(parquet_path) - Extract timestamps from partitioned Parquet paths
• get_time_chunks_from_range(start, end, freq) - Generate time chunk tuples for processing
• delete_partition_if_exists(output_path, partition_values) - Delete specific partition directory
• floor_index_to_dt(df, dt_s) - Replace DatetimeIndex with lower-second floored values
• calculate_delta_sec(time1, time2) - Calculate time difference in seconds
• extract_datetime(filename) - Extract datetime from SP2XR filename

INI/YAML Conversion

• read_xr_ini_file(ini_path) - Read SP2-XR .ini calibration file
• find_and_validate_ini_files(directory) - Find and validate .ini files (ensure consistency)
• export_xr_ini_to_yaml(ini_path, yaml_path) - Convert .ini file to YAML format
• export_xr_ini_to_yaml_with_source(ini_path, yaml_path) - Convert .ini to YAML with source metadata

Utilities

• chunks(lst, n) - Yield successive n-sized chunks from list
• partition_rowcount(ddf) - Count total rows in Dask DataFrame

---

sp2xr.schema

Data schema definitions and type enforcement for SP2XR data streams.

Constants:

• CANONICAL_DTYPES - Dictionary mapping column names to canonical data types
• DEFAULT_FLOAT - Default float dtype for numeric columns

Functions:

• enforce_schema(df) - Cast DataFrame columns to canonical data types
• cast_and_arrow(df) - Cast to canonical dtypes and convert to PyArrow backend