diff --git a/docs/.~lock.ClusterFinderCUDA_optimizations.pptx# b/docs/.~lock.ClusterFinderCUDA_optimizations.pptx# new file mode 100644 index 00000000..5b76f65c --- /dev/null +++ b/docs/.~lock.ClusterFinderCUDA_optimizations.pptx# @@ -0,0 +1 @@ +,ferjao_k,pc-moench-04.psi.ch,11.08.2026 11:46,file:///home/ferjao_k/.config/libreoffice/4; \ No newline at end of file diff --git a/docs/ClusterFinderCUDA_optimizations.pptx b/docs/ClusterFinderCUDA_optimizations.pptx index 4dbdb834..f09a07cf 100644 Binary files a/docs/ClusterFinderCUDA_optimizations.pptx and b/docs/ClusterFinderCUDA_optimizations.pptx differ diff --git a/docs/benchmark_opt1_opt6_results.md b/docs/benchmark_opt1_opt6_results.md new file mode 100644 index 00000000..5ad28fd3 --- /dev/null +++ b/docs/benchmark_opt1_opt6_results.md @@ -0,0 +1,467 @@ +# ClusterFinderCUDA — Benchmark Results, opt1 → opt6 + +Consolidated, verified performance numbers for the CUDA cluster-finder optimization +ladder. Every number below is traced to the code, notebook cell, or profiler report +that produced it, and each is tagged **quotable** or **not quotable** with the reason. + +Companion documents: +- `docs/pedestal_precision_f32_cancellation.md` — why the naive f32 pedestal failed and how B1 fixes it +- `docs/cuda_optimization_recap.md`, `docs/optimization_summary.md` — earlier narrative/roadmap notes +- `docs/ClusterFinderCUDA_optimizations.pptx` — the deck these numbers feed + +--- + +## 1. Environment + +| item | value | +|---|---| +| GPU | NVIDIA GeForce RTX 4090 (Ada, sm_89), 24 GB, driver 595.71.05 | +| GPU clocks | idle 210 MHz → boost 3120 MHz; **persistence mode disabled** | +| FP64 rate | 1/64 of FP32 on this part (relevant to opt6) | +| CPU | AMD Ryzen 9 7950X, 16 cores / 32 threads | +| RAM | 125 GiB, **no swap** | +| CUDA | 12.4 (nvcc V12.4.131) | +| Profiler | Nsight Systems 2024.5.1 (`/opt/nvidia/nsight-systems/2024.5.1`) | +| Host | `pc-moench-04` | +| Branch | `bench/opt2-pipeline` (off `feature/cuda_clusterfinder` @ `ce256dd`) | + +**Dataset** — MOENCH, MAX IV beamtime, Cu fluorescence: + +``` +/mnt/sls_det_storage/moench_data/2603_MaxIVBeamtime/2026032408/process/xrf/ + Cu_factor_10_data_master_0.json (100 000 frames, 400×400 uint16) + Cu_factor_10_pedestal_master_0.json (1 000 frames used for pedestal training) +``` + +Frame size 400×400×2 B = 320 000 B (312.5 KiB). All finders trained on the +**same 1 000 pedestal frames**; +`n_sigma = 5` throughout. Data pre-loaded into RAM with `read_n()` so file I/O is +outside every timing loop. + +--- + +## 2. The optimization ladder + +| step | what changed | how it is measured | +|---|---|---| +| **baseline** | `ClusterFinderMT`, 48 threads | `ClusterFinderMT(..., n_threads=48)` + `ClusterCollector` | +| **opt1** | first CUDA port: 1 stream, one launch per frame, no batching | `ClusterFinderCUDAOpt2(..., n_streams=1)` + `find_clusters()` per frame | +| **opt2** | multi-stream scaffolding + host-side batching (bulk memcpy) | `ClusterFinderCUDAOpt2(..., n_streams=4)` + `find_clusters_batched()`, batch 2000 | +| **opt3** | pipeline rework: remove per-round sync barriers, fixed-size D2H | `ClusterFinderCUDA(..., n_streams=4)` + `find_clusters_batched()`, **no pinning** | +| **opt4** | DMA-speed transfers via pinned host input | opt3 + `register_input_buffer(data)` | +| **opt5** | CUDA Graphs (pre-recorded H2D→kernel→D2H per stream) | `ClusterFinderCUDAGraph(...)` + pinned input | +| **opt6** | **first kernel optimization**: f32 device pedestal + variance rewrite (B1) | rebuild with `DEVICE_PED_TYPE = float` | + +opt1–opt5 are **pipeline/host-side**; opt6 is the first change to the **kernel itself**. + +### Code behind each step + +| step | primary source | +|---|---| +| opt1, opt2 | [`include/aare/ClusterFinderCUDAOpt2.hpp`](../include/aare/ClusterFinderCUDAOpt2.hpp), [`include/aare/clusterfinder_kernel_opt2.cuh`](../include/aare/clusterfinder_kernel_opt2.cuh) — snapshot of commit `88e0e8d` (pre-refactor pipeline), namespace `aare::device_opt2` | +| opt3, opt4, opt6 | [`include/aare/ClusterFinderCUDA.hpp`](../include/aare/ClusterFinderCUDA.hpp), [`include/aare/clusterfinder_kernel.cuh`](../include/aare/clusterfinder_kernel.cuh) | +| opt5 | [`include/aare/ClusterFinderCUDA_graph.hpp`](../include/aare/ClusterFinderCUDA_graph.hpp) | +| bindings | [`python/src/bind_ClusterFinderCUDAOpt2.hpp`](../python/src/bind_ClusterFinderCUDAOpt2.hpp), [`python/src/cuda_bindings.cu`](../python/src/cuda_bindings.cu) | +| factories | [`python/aare/ClusterFinder.py`](../python/aare/ClusterFinder.py) | + +Relevant history: `3ed773e` (multi-stream+batched) → `ac96d1f` (mixed precision) → +`88e0e8d` (**opt1/opt2 snapshot**) → `6a12e3d` (pipeline refactor = opt3) → +`4c66802` (FP32 pedestal, introduced the tail) → `5922c73` (async API) → +`1bf317f` (local-max fix) → `a42d71c` (graphs = opt5). + +### Precision configuration + +Two type aliases in [`clusterfinder_kernel.cuh:16-17`](../include/aare/clusterfinder_kernel.cuh#L16-L17): + +```cpp +using COMPUTE_TYPE = float; // stencil arithmetic — float in ALL builds below +using DEVICE_PED_TYPE = double; // device pedestal — the opt6 knob (double → float) +``` + +- **"f64 build"** in this document = `COMPUTE_TYPE=float`, `DEVICE_PED_TYPE=double` (mixed precision). +- **"f32 build"** (opt6) = `COMPUTE_TYPE=float`, `DEVICE_PED_TYPE=float` (100 % f32). + +> ⚠️ `ClusterFinderCUDAOpt2` is templated on `PEDESTAL_TYPE`, which its binding pins +> to `double` ([`bind_ClusterFinderCUDAOpt2.hpp:14`](../python/src/bind_ClusterFinderCUDAOpt2.hpp#L14)). +> **opt1 and opt2 are therefore unaffected by the opt6 flip** — their numbers are +> identical in both campaigns by construction. Only opt3/opt4/opt5 respond to opt6. + +--- + +## 3. Methodology — what is measurable and what is not + +Three measurement artifacts were identified and controlled. **All three matter for +how numbers may be quoted on a slide.** + +### 3.1 First-run soft page faults (dominant, ~1.5–4 s per cell) + +Each timed cell materializes ~233 M clusters ≈ **10 GB of host result heap**. On the +first execution in a process, every 4 kB page must be faulted in and zeroed by the OS +(millions of minor faults). Measured cost on this machine: **0.7 µs/fault**. + +Instrumentation added to all seven timed notebook cells: + +```python +import resource +def _faults(): + r = resource.getrusage(resource.RUSAGE_SELF) + return r.ru_minflt, r.ru_majflt +``` +bracketing the timed region, printing `minor faults: N (~N×0.7µs est.)`. + +**Protocol: quote the run where the fault counter has plateaued** (< ~200 k). Cold +numbers are inflated by 25–45 %. Verified by direct correlation (§9). + +### 3.2 CUDA-event `kernel_ms` inflates under multi-stream saturation + +`avg_kernel_time_ms()` uses a CUDA event pair on the kernel's own stream. It measures +**elapsed time on that stream's timeline**, which includes queue-wait when other +streams are competing for SMs. Consequences: + +- In transfer-paced regimes: `event ≈ true kernel + ~5–7 µs` launch gap → usable. +- In kernel-saturated regimes: **inflated up to 3.5×** → **not quotable**. +- Symptom of saturation: the derived `PCIe + overhead = wall/N − kernel_ms` **goes + negative** (kernels overlap, so wall/frame < kernel/frame). + +Ground truth requires Nsight Systems (§8). + +### 3.3 Profiler distorts wall clock + +Under `nsys`, wall time per frame is ~4× the unprofiled value (API tracing overhead). +**Take per-operation GPU times from nsys; take wall times from unprofiled runs.** The +standalone probe script also pays the full first-touch fault tax inside its single +timed call (fresh process, no warm pass), so **its wall times are not throughput +numbers** either. + +### 3.4 Other controls + +- GPU clock ramp (210 MHz → 3.1 GHz) is real but < 0.1 % of a multi-second run; the + invariance of `kernel_ms` across loaded/idle runs confirms it is not a factor. +- The `cf_cuda_v1` notebook cell builds a histogram **inside** its timed loop + (~4.5 s over 100 k frames). It is a reference row, **not part of the arc**. +- `ClusterFinderMT` cannot restart after `stop()`; the CPU baseline is therefore + always a first-pass number and carries its own ~1.8 s of allocator faults. + +--- + +## 4. Campaign A — 3×3 pipeline arc (f64 pedestal) ★ headline + +**Config**: `cluster_size=(3,3)`, `N=100 000`, `n_frames_pd=1000`, `n_sigma=5`, +`BATCH_SIZE=2000`, `n_streams=4`, `max_clusters_per_frame=3000`. +**Source**: [`python/tests/ClusterFinderCUDA_perf.ipynb`](../python/tests/ClusterFinderCUDA_perf.ipynb), +warm pass (steady state, faults plateaued). + +| step | variant | wall [s] | FPS | µs/frame | kernel [µs/fr] | host ovhd [µs/fr] | **vs CPU** | step gain | minor faults | +|---|---|--:|--:|--:|--:|--:|--:|--:|--:| +| baseline | CPU MT (48 threads) | 21.00 | 4 761 | 210.0 | — | — | 1.00× | — | 2.63 M | +| **opt1** | 1 stream, per-frame | 6.68 | 14 968 | 66.8 | 23 | 44 | **3.14×** | 3.14× | 1 | +| **opt2** | 4 streams + batching | 4.32 | 23 134 | 43.2 | 22 | 21 | **4.86×** | 1.55× | 64 931 | +| **opt3** | pipeline rework, no pin | 3.76 | 26 588 | 37.6 | 24 | 14 | **5.58×** | 1.15× | 59 814 | +| **opt4** | + pinned input (DMA) | 2.72 | 36 810 | 27.2 | 24 | 3 | **7.73×** | 1.38× | 1 027 | +| **opt5** | + CUDA Graph | 2.53 | 39 472 | 25.3 | 25 * | — | **8.29×** | 1.07× | 1 032 | + +\* graph reports kernel+PCIe+overhead combined; the kernel is not timed separately. + +Reference row (not in the arc): current finder, per-frame, histogram inside the timed +loop — 11.35 s / 8 813 FPS / 1.85×. + +**Status: quotable.** All CUDA rows at steady state; CPU baseline is first-pass by +necessity (see §3.4) — if a fault-corrected CPU (≈19.2 s) is preferred, all speedups +shrink by ~9 % (e.g. opt5 8.29× → 7.57×). Be consistent and state the choice. + +### The story in one row — host overhead + +``` +opt1 → opt2 → opt3 → opt4 host overhead per frame + 44 → 21 → 14 → 3 µs (kernel fixed at ~23 µs) +``` + +opt4's 3 µs means H2D/kernel/D2H are essentially fully overlapped; opt5 then trims +residual launch overhead. **After opt4 the 3×3 pipeline is host/PCIe-bound, not +kernel-bound** — which is exactly why opt6 does nothing here (§5). + +--- + +## 5. Campaign B — opt6 at 3×3 (100 % f32) + +Same config as Campaign A; rebuilt with `DEVICE_PED_TYPE = float`. +**Source**: same notebook, warm pass. + +| step | wall f64 → f32 [s] | FPS f64 → f32 | kernel f64 → f32 [µs] | Δ wall | +|---|--:|--:|--:|--:| +| CPU baseline | 21.00 → 21.45 | 4 761 → 4 662 | — | (run-to-run) | +| v1 per-frame (serialized) | 11.35 → 10.57 | 8 813 → 9 465 | **24 → 13** | **−7 %** ✓ | +| opt3 (no pin) | 3.76 → 3.88 | 26 588 → 25 798 | **24 → 14** | +3 % (noise) | +| opt4 (pinned) | 2.72 → 2.83 | 36 810 → 35 372 | **24 → 16** | +4 % (noise) | +| opt5 (graph) | 2.53 → 2.53 | 39 472 → 39 503 | — | **0 %** | +| opt1 / opt2 | 6.68 → 6.73 / 4.32 → 4.88 | — | 23 / 22 unchanged | n/a — f64-pinned class | + +**Result: at 3×3, opt6 halves the kernel and buys nothing end-to-end.** The kernel +(13–16 µs) sits *below* the ~25–27 µs/frame transfer+host floor, so it is entirely +hidden by stream overlap. The control is the serialized `v1` path, where the kernel +*cannot* hide: there the saving does appear (−7 % wall ≈ the 8–10 µs/frame kernel gain). + +The f32 opt2 row (4.88 s) still carried 822 k faults and is **not quotable**; it is +also irrelevant, being the f64-pedestal class. + +**Status: quotable** (opt3/opt4/opt5/v1 rows). + +--- + +## 6. Campaign C — opt6 at 9×9 (where the kernel *is* the bottleneck) ★ + +**Config change**: `cluster_size=(9,9)`, `N=20 000`, `n_streams=8`, +`max_clusters_per_frame=1500`, everything else unchanged. +Cap 1500 chosen so the CUDA finders record **all** clusters the CPU finds. +**Source**: same notebook, warm runs (faults ≤ 102). + +| build | path | wall [s] | FPS | µs/frame | kernel [µs/fr] | derived ovhd | vs CPU | clusters/frame | +|---|---|--:|--:|--:|--:|--:|--:|--:| +| **f64 ped** | opt4 batched+pin | 1.540 | 12 988 | 77.0 | 171 † | **−94** † | 6.01× | 1422.13 | +| **f64 ped** | opt5 graph | 1.482 | 13 491 | 74.1 | 74 * | — | 6.24× | 1422.13 | +| **f32 ped** | opt4 batched+pin | 1.423 | 14 057 | 71.2 | 33 | +38 | 6.61× | 1422.33 | +| **f32 ped** | opt5 graph | 1.372 | 14 580 | 68.6 | 69 * | — | 6.86× | 1422.33 | + +† **Not quotable** — event-timer inflation under 8-stream saturation; the negative +overhead is the tell-tale. True kernel time is 43 µs (§8). \* combined metric. + +CPU baselines (derived from the printed speedup ratios): f64 run ≈ 9.25 s (2 162 FPS), +f32 run ≈ 9.41 s (2 126 FPS). + +**Result: at 9×9, opt6 is worth ~8 % end-to-end** (1.540 → 1.423 s batched; +1.482 → 1.372 s graph) **and flips the regime**: the f64 build is kernel-bound +(negative derived overhead), the f32 build is transfer-bound (+38 µs). + +**Status: wall/FPS/counts quotable; kernel column must come from §8.** + +--- + +## 7. The rule that unifies Campaigns B and C + +Per-frame GPU operation profile at 9×9 (nsys, §8): + +``` +f64: kernel 43 µs | D2H 19 µs | H2D 13 µs → kernel is the tallest bar +f32: kernel 26 µs | D2H 20 µs | H2D 13 µs → near-balanced +3×3: kernel 13–24 µs, below the ~27 µs host/PCIe floor → always hidden +``` + +> **f32 buys exactly the distance between the kernel bar and the next-tallest bar.** +> Nothing at 3×3 (kernel already hidden); ~8 % at 9×9/f64 (kernel on the critical +> path); more for larger windows, faster links, or fewer streams. + +--- + +## 8. Nsight Systems ground truth (9×9, cap 1500) + +**Script**: [`python/tests/nsys_kernel_probe.py`](../python/tests/nsys_kernel_probe.py) +(also in the session scratchpad). Trains 1 000 pedestal frames, runs one batched pass +over 2 000 frames, prints wall + event `kernel_ms`. + +```bash +nsys profile --trace=cuda --sample=none --cpuctxsw=none -o probe_s1 \ + python nsys_kernel_probe.py 1 2000 # 1 stream → kernels serialized +nsys profile --trace=cuda --sample=none --cpuctxsw=none -o probe_s8 \ + python nsys_kernel_probe.py 8 2000 # 8 streams → deck config +nsys stats --report cuda_gpu_kern_sum --report cuda_gpu_mem_time_sum probe_s1.nsys-rep +``` + +### Kernel — `aare::device::find_clusters_in_single_frame>` + +| build | streams | instances | **avg** | median | min | max | σ | +|---|--:|--:|--:|--:|--:|--:|--:| +| f64 ped | 1 | 2 000 | **43 011 ns** | 42 945 | 40 929 | 45 632 | 580 | +| f64 ped | 8 | 2 000 | 46 714 ns | 46 880 | 41 632 | 61 441 | 1 988 | +| f32 ped | 1 | 2 000 | **25 602 ns** | 25 569 | 24 737 | 27 073 | 411 | +| f32 ped | 8 | 2 000 | 25 840 ns | 25 728 | 24 864 | 30 624 | 578 | + +> ### **opt6 kernel result: 43.0 µs → 25.6 µs = −40 %** (9×9, exclusive time) + +### Memory operations (1-stream, uncontended) + +| op | f64 | f32 | payload | effective BW (decimal) | +|---|--:|--:|---|--:| +| D2H (clusters) | 19 424 ns | 19 845 ns | 1500 × 328 B = 492 000 B (480.5 KiB) | **25.3 GB/s** (23.6 GiB/s) | +| H2D (frame) | 13 210 ns | 13 451 ns | 400×400×2 B = 320 000 B (312.5 KiB) | **24.2 GB/s** (22.6 GiB/s) | +| memset | 359 ns | 365 ns | — | — | + +The H2D payload is **one frame**; the D2H payload is one frame's cluster buffer +(`max_clusters_per_frame` × `sizeof(Cluster)`, transferred at +fixed size regardless of how many clusters were actually found). + +PCIe 4.0 ×16 theoretical is 31.5 GB/s (16 GT/s × 16 lanes × 128b/130b), so H2D +reaches **77% of theoretical** — the signature of a real DMA path. Pageable +transfers, which the driver stages through a hidden pinned buffer, run ~15 GB/s. + +Transfers are identical between builds and at full DMA speed — confirming pinning +(opt4) is doing its job and that only the kernel changed. + +### Cross-validation of the event timer + +| build | streams | nsys avg | event `kernel_ms` | offset | +|---|--:|--:|--:|--:| +| f64 | 1 | 43.0 µs | 0.050 ms | +7.0 µs (launch gap) | +| f64 | 8 | 46.7 µs | 0.149 ms | **+102 µs (queue-wait)** | +| f32 | 1 | 25.6 µs | 0.033 ms | +7.4 µs | +| f32 | 8 | 25.8 µs | 0.033 ms | +7.2 µs | + +Independently reproduced unprofiled by the user (f32 build, `N=20000`): event +`kernel_ms` = **0.030–0.032** warm (= 25.6 µs + ~5–6 µs gap); 0.047 and 0.184 on +cold-clock first invocations. + +Note the 8-stream instance time stretches only +9 % (f64) / +1 % (f32): **one 9×9 +kernel nearly fills the GPU, so streams queue rather than co-execute.** Multi-streaming +at 9×9 buys transfer overlap, not kernel concurrency. + +**Reports retained** (openable in `nsys-ui` for timeline figures): +`probe_s1.nsys-rep`, `probe_s8.nsys-rep` (f64), `probe_f32_s1.nsys-rep`, +`probe_f32_s8.nsys-rep` (f32). + +--- + +## 9. Supporting study — the page-fault artifact + +### Synthetic isolation + +Allocating ~8 GB in ClusterVector-sized chunks (90 000 × 93 kB), touching every page, +freeing, repeating in one process: + +| run | wall | minor faults | +|---|--:|--:| +| 1 (cold heap) | 2.05 s | 2 046 594 | +| 2 (warm heap) | 0.07 s | 2 232 | +| 3 (warm heap) | 0.07 s | 2 016 | + +**30× faster, 1000× fewer faults**, same allocations — glibc retains the arenas. + +### In situ (opt2 cell, three consecutive executions) + +| run | wall [s] | FPS | minor faults | +|---|--:|--:|--:| +| 1 | 6.110 | 16 366 | 2 625 948 | +| 2 | 4.872 | 20 526 | 729 866 | +| 3 | 4.452 | 22 459 | 185 885 | + +Correlation Δwall vs Δfaults: + +| interval | Δwall | Δfaults | implied cost | +|---|--:|--:|--:| +| 1 → 2 | 1.238 s | 1 895 667 | **0.65 µs/fault** | +| 2 → 3 | 0.420 s | 543 981 | **0.77 µs/fault** | +| 1 → 3 | 1.658 s | 2 439 648 | **0.68 µs/fault** | + +Reconstruction of run 1 from run 3: +`4.452 s + 2 439 648 × 0.68 µs = 6.111 s` vs **measured 6.110 s** (1 ms error over 6 s). + +Kernel time was constant (0.022 ms) across all three — the GPU is not involved. +**Conclusion: the entire first-run penalty is OS page population of the host result +heap.** Persistence mode / clock ramp are not responsible. + +--- + +## 10. Correctness (held constant across the whole arc) + +### 3×3, N = 100 000 + +| finder | clusters | /frame | diff vs CPU | +|---|--:|--:|--:| +| CPU MT | 233 085 343 | 2330.85 | — | +| opt1 | 233 094 770 | 2330.95 | 0.0040 % | +| opt2 | 233 093 553 | 2330.94 | 0.0035 % | +| opt3 / opt4 / opt5 (f64) | 233 093 484 | 2330.93 | 0.0035 % | +| opt3 / opt4 / opt5 (**f32**) | 233 093 554 – 233 094 465 | 2330.94 | 0.0039 % | + +The residual ~0.004 % is the known per-frame vs per-pixel pedestal-update difference +(analysed in `ClusterFinderFrozen_vs_CUDA.ipynb`), **not** a precision effect. + +**f64 vs f32 builds differ by ~70 clusters out of 233 M (3 × 10⁻⁷).** + +### 9×9, N = 20 000 + +| build | clusters | /frame | +|---|--:|--:| +| f64 | 28 442 582 | 1422.13 | +| f32 | 28 446 667 | 1422.33 | + +### Two fixes that made this possible + +**(a) B1 — per-pixel offset accumulation** (`docs/pedestal_precision_f32_cancellation.md`). +Before: naive f32 pedestal produced **+28.06 % clusters and an unphysical high-energy +tail** — catastrophic cancellation in `var = sum2/n − mean²` (both terms ≈ 2.17 × 10⁷, +variance ≈ 2025) drove quiet pixels to `rms=0`, so they fired every frame. Fix: freeze +a per-pixel baseline `X0 ≈ round(mean)` at t=0 and accumulate centered `Y = X − X0`. +After: full-f32 matches f64 to **3 × 10⁻⁷**. **opt6 is only shippable because of B1.** + +**(b) Test3 local-max gate backported** into +[`clusterfinder_kernel_opt2.cuh`](../include/aare/clusterfinder_kernel_opt2.cuh) +so opt1/opt2 stop over-counting extended charge-shared events: + +| | before | after | +|---|--:|--:| +| opt1 | 233 940 268 (2339.40/fr) | 233 094 770 (2330.95/fr) | +| opt2 | 233 931 015 (2339.31/fr) | 233 093 553 (2330.94/fr) | + +−0.36 %, now matching CPU. Kernel time unchanged (the gate is an early `return`). + +--- + +## 11. Numbers that must NOT be used + +| number | why | +|---|---| +| 9×9 **cap = 1000** runs (f64: 1.277 s / 15 657 FPS / 7.37×; f32: 1.087 s / 18 395 FPS / 8.53×) | cluster cap saturated → **exactly** 1000.00/frame, ~30 % of clusters truncated; CPU found ~1422/frame. Timing story survives but counts and speedups are dishonest. Superseded by cap 1500 (§6). | +| `kernel_ms` under multi-stream saturation (f64 9×9: 0.171 / 0.192; 8-stream first-launch 0.184) | event-timer queue-wait inflation up to 3.5× (§3.2). Use nsys §8. | +| Negative "PCIe + overhead" (−0.094, −0.128, −0.007) | arithmetic artifact of `wall/N − inflated kernel`. Useful only as a *kernel-bound indicator*, never as a transfer cost. | +| Any wall time from `nsys_kernel_probe.py` (5.3–6.6 s for 20 k frames) | fresh process → full first-touch fault tax inside the timed call; single mega-batch; profiler overhead when traced. | +| Cold-pass notebook numbers (opt2 6.18 s, opt3 5.29 s, batched 4.45 s, graph 4.02 s) | 2.1–2.6 M unresolved page faults each (§9). | +| `cf_cuda_v1` cell as an arc data point | builds a histogram inside the timed loop (~4.5 s / 100 k frames). Valid only as the *serialized-path* control in §5. | + +--- + +## 12. Reproduction index + +| artifact | path | role | +|---|---|---| +| main benchmark notebook | `python/tests/ClusterFinderCUDA_perf.ipynb` | all wall/FPS/count numbers; 7 instrumented timed cells | +| nsys probe | `python/tests/nsys_kernel_probe.py` | exclusive kernel + memcpy times | +| correctness notebook | `python/tests/ClusterFinderFrozen_vs_CUDA.ipynb` | CPU↔CUDA agreement analysis | +| precision study | `docs/pedestal_precision_f32_cancellation.md` | B1 derivation | +| opt1/opt2 class | `include/aare/ClusterFinderCUDAOpt2.hpp` | pre-refactor pipeline snapshot | +| opt1/opt2 kernel | `include/aare/clusterfinder_kernel_opt2.cuh` | `88e0e8d` kernel + backported Test3 gate | +| opt3/opt4/opt6 | `include/aare/ClusterFinderCUDA.hpp`, `include/aare/clusterfinder_kernel.cuh` | current pipeline; precision knob at lines 16–17 | +| opt5 | `include/aare/ClusterFinderCUDA_graph.hpp` | graph-based finder | +| deck | `docs/ClusterFinderCUDA_optimizations.pptx` | target | + +### Protocol to reproduce a clean number + +1. Idle machine. Optionally `sudo nvidia-smi -pm 1` (kills ~1–3 s of per-process + driver init; does **not** affect the fault artifact). +2. Restart the Jupyter kernel; run the notebook top to bottom (cold pass). +3. **Re-run each timed CUDA cell** until its `minor faults` line plateaus + (< ~200 k; usually 2–3 executions). Quote that run. +4. The CPU baseline cannot be re-run (`ClusterFinderMT.stop()` is terminal) — + re-run the *Build finders* + pedestal cells first if a warm CPU number is needed. +5. For kernel times: use `nsys` at `n_streams=1`, never the event timer under load. + +--- + +## 13. Slide-ready takeaways + +1. **The pipeline arc (opt1 → opt5) is monotonic**: 3.14× → 4.86× → 5.58× → 7.73× → + **8.29×** over a 48-thread CPU baseline, at identical correctness (0.004 %). +2. **The optimization story is host overhead collapsing**: 44 → 21 → 14 → **3 µs/frame** + against a fixed ~23 µs kernel. By opt4 the GPU is fed almost perfectly. +3. **opt6 (f32 pedestal) is a kernel win, not always a throughput win**: kernel + **43.0 → 25.6 µs (−40 %, nsys-verified)**, but end-to-end **0 % at 3×3** and + **−8 % wall at 9×9**. The bottleneck is workload-dependent. +4. **The unifying rule**: f32 buys the distance between the kernel bar and the + next-tallest bar in the per-frame GPU profile. +5. **opt6 is only correct because of the B1 variance rewrite** — the naive f32 pedestal + gave +28 % clusters and an unphysical tail; the centered accumulation restores + agreement with f64 to 3 × 10⁻⁷. +6. **Measurement discipline was necessary to get here**: soft page faults (0.7 µs each, + up to 4 s per run) and CUDA-event queue-wait inflation (up to 3.5×) both had to be + identified and controlled before any number was trustworthy. +7. **Where the next win is**: after opt4 the 3×3 pipeline is PCIe/host-bound. Further + kernel work has no payoff at that window size; the next target is transfer volume + (on-GPU reduction, keeping results on device) rather than kernel speed. diff --git a/docs/deck/build_deck.py b/docs/deck/build_deck.py new file mode 100644 index 00000000..0c3b10da --- /dev/null +++ b/docs/deck/build_deck.py @@ -0,0 +1,744 @@ +"""Rebuild docs/ClusterFinderCUDA_optimizations.pptx — opt1..opt6, in the deck's +own design language (extracted from the original file).""" +from pptx import Presentation +from pptx.util import Inches as In, Pt, Emu +from pptx.dml.color import RGBColor +from pptx.enum.text import PP_ALIGN, MSO_ANCHOR +from pptx.enum.shapes import MSO_SHAPE +from lxml import etree +from pathlib import Path +from PIL import Image + +FIGS = Path(__file__).parent / "figs" +OUT = Path("/home/ferjao_k/aare/docs/ClusterFinderCUDA_optimizations.pptx") + +# ---------------------------------------------------------------- design tokens +BG = RGBColor(0x0B, 0x10, 0x18) +PANEL = RGBColor(0x12, 0x1A, 0x28) +CODEBG = RGBColor(0x0E, 0x14, 0x20) +RULE = RGBColor(0x1E, 0x28, 0x36) +ACCENT = RGBColor(0x1E, 0x90, 0xC2) +AMBER = RGBColor(0xE8, 0xB2, 0x5C) +PALE = RGBColor(0xE7, 0xED, 0xF4) +TEXT2 = RGBColor(0xA5, 0xB2, 0xC4) +MUTED = RGBColor(0x6B, 0x7A, 0x90) + +UI, MONO = "Segoe UI", "Consolas" +W, H = 13.333, 7.5 +M = 0.7 # left margin +COL = 7.9 # left column width +RAIL_X, RAIL_W = 9.2, 3.5 # right rail + +prs = Presentation() +prs.slide_width, prs.slide_height = In(W), In(H) +BLANK = prs.slide_layouts[6] +N_SLIDES = 19 + + +# ------------------------------------------------------------------- helpers +def new_slide(): + s = prs.slides.add_slide(BLANK) + bg = etree.SubElement(s._element, "{http://schemas.openxmlformats.org/presentationml/2006/main}bg") + pr = etree.SubElement(bg, "{http://schemas.openxmlformats.org/presentationml/2006/main}bgPr") + fill = etree.SubElement(pr, "{http://schemas.openxmlformats.org/drawingml/2006/main}solidFill") + clr = etree.SubElement(fill, "{http://schemas.openxmlformats.org/drawingml/2006/main}srgbClr") + clr.set("val", "0B1018") + etree.SubElement(pr, "{http://schemas.openxmlformats.org/drawingml/2006/main}effectLst") + s._element.insert(0, bg) + return s + + +def rect(s, x, y, w, h, color, shape=MSO_SHAPE.RECTANGLE): + sh = s.shapes.add_shape(shape, In(x), In(y), In(w), In(h)) + sh.fill.solid(); sh.fill.fore_color.rgb = color + sh.line.fill.background(); sh.shadow.inherit = False + return sh + + +def tb(s, x, y, w, h, anchor=MSO_ANCHOR.TOP): + box = s.shapes.add_textbox(In(x), In(y), In(w), In(h)) + tf = box.text_frame + tf.word_wrap = True + tf.margin_left = tf.margin_right = tf.margin_top = tf.margin_bottom = 0 + tf.vertical_anchor = anchor + return tf + + +def para(tf, first=False, space_after=0, space_before=0, line=None, align=None): + p = tf.paragraphs[0] if first else tf.add_paragraph() + p.space_after = Pt(space_after); p.space_before = Pt(space_before) + if line: p.line_spacing = line + if align: p.alignment = align + return p + + +def run(p, text, size=11, color=TEXT2, font=UI, bold=False, italic=False, spc=None): + r = p.add_run(); r.text = text + f = r.font + f.name, f.size, f.bold, f.italic = font, Pt(size), bold, italic + f.color.rgb = color + if spc is not None: + r.font._rPr.set("spc", str(int(spc * 100))) + return r + + +# ------------------------------------------------------------------ chrome +def chrome(s, idx, eyebrow, title, title_size=27): + rect(s, M, 0.60, 0.35, 0.035, ACCENT) + tf = tb(s, 1.17, 0.50, 10.33, 0.32) + run(para(tf, True), eyebrow.upper(), 9, MUTED, bold=True, spc=1.6) + + tf = tb(s, M, 0.86, 11.9, 1.0) + run(para(tf, True, line=1.05), title, title_size, PALE, bold=True) + + # progress bar + span, n = 11.0, N_SLIDES + pitch = span / n; wseg = pitch * 0.90 + for i in range(n): + rect(s, M + i * pitch, 7.28, wseg, 0.045, ACCENT if i <= idx - 1 else RULE) + tf = tb(s, 12.0, 7.14, 0.9, 0.3) + run(para(tf, True, align=PP_ALIGN.RIGHT), f"{idx} / {n}", 8.5, MUTED) + + +def bullets(s, x, y, w, items, size=11, gap=7): + tf = tb(s, x, y, w, 0.3) + for i, it in enumerate(items): + color, txt = (it if isinstance(it, tuple) else (TEXT2, it)) + p = para(tf, i == 0, space_after=gap, line=1.25) + run(p, "• ", size, MUTED) + # inline emphasis with **...** + for j, part in enumerate(txt.split("**")): + if part: + run(p, part, size, PALE if j % 2 else color, bold=bool(j % 2)) + return tf + + +def code(s, x, y, w, lines, size=8.5, title=None): + lh = 0.148 + h = 0.24 + len(lines) * lh + (0.22 if title else 0) + rect(s, x, y, w, h, CODEBG, MSO_SHAPE.ROUNDED_RECTANGLE) + ty = y + 0.12 + if title: + tf = tb(s, x + 0.18, ty, w - 0.36, 0.2) + run(para(tf, True), title, 7.5, MUTED, bold=True, spc=1.2) + ty += 0.22 + tf = tb(s, x + 0.18, ty, w - 0.36, h - 0.24) + for i, ln in enumerate(lines): + p = para(tf, i == 0, line=1.12) + if ln.strip().startswith(("//", "#")): + run(p, ln, size, MUTED, MONO) + continue + for j, part in enumerate(ln.split("«")): + for k, seg in enumerate(part.split("»")): + if not seg: continue + hi = (j > 0 and k == 0) + run(p, seg, size, ACCENT if hi else TEXT2, MONO, bold=hi) + return h + + +def callout(s, x, y, w, text, h=0.78, color=ACCENT, size=10.5): + rect(s, x + 0.045, y, w - 0.045, h, PANEL) + rect(s, x, y, 0.045, h, color) + tf = tb(s, x + 0.28, y + 0.10, w - 0.5, h - 0.2, MSO_ANCHOR.MIDDLE) + p = para(tf, True, line=1.2) + for j, part in enumerate(text.split("**")): + if part: + run(p, part, size, PALE if j % 2 else TEXT2, bold=bool(j % 2)) + + +def rail(s, items, y0=2.0, divider=True): + if divider: + rect(s, 8.95, 2.0, 0.012, 4.55, RULE) + y = y0 + for it in items: + kind = it[0] + if kind == "label": + tf = tb(s, RAIL_X, y, RAIL_W, 0.26) + run(para(tf, True), it[1].upper(), 8.5, MUTED, bold=True, spc=1.4) + y += 0.28 + elif kind == "stat": + _, lab, val, col = it + tf = tb(s, RAIL_X, y, RAIL_W, 0.24) + run(para(tf, True), lab.upper(), 8.5, MUTED, spc=1.2) + tf = tb(s, RAIL_X, y + 0.24, RAIL_W, 0.6) + run(para(tf, True), val, 26, col, bold=True) + y += 0.98 + elif kind == "row": + _, lab, val, col = it + tf = tb(s, RAIL_X, y, RAIL_W, 0.24) + run(para(tf, True), lab.upper(), 8.5, MUTED, spc=1.2) + tf = tb(s, RAIL_X, y + 0.22, RAIL_W, 0.3) + run(para(tf, True), val, 13, col, bold=True) + y += 0.66 + elif kind == "note": + tf = tb(s, RAIL_X, y, RAIL_W, 0.9) + run(para(tf, True, line=1.25), it[1], 9, TEXT2) + y += 0.30 + 0.17 * (len(it[1]) // 42 + 1) + elif kind == "gap": + y += it[1] + return y + + +def figure(s, name, x, y, w): + p = FIGS / f"{name}.png" + iw, ih = Image.open(p).size + h = w * ih / iw + s.shapes.add_picture(str(p), In(x), In(y), In(w), In(h)) + return h + + +def caption(s, x, y, w, text, size=9): + tf = tb(s, x, y, w, 0.3) + run(para(tf, True, line=1.25), text, size, MUTED) + + +# =========================================================== 1 · TITLE +s = new_slide() +rect(s, 0, 0, 0.16, H, ACCENT) +tf = tb(s, M + 0.3, 0.85, 11, 0.3) +run(para(tf, True), "AARE · PSI HYBRID PIXEL DETECTORS · CUDA CLUSTERFINDER", + 9.5, MUTED, bold=True, spc=1.8) + +tf = tb(s, M + 0.3, 1.35, 11.4, 1.7) +run(para(tf, True, line=1.02), "Feeding the GPU", 46, PALE, bold=True) +tf = tb(s, M + 0.3, 2.30, 11.4, 0.8) +run(para(tf, True, line=1.05), "Six optimization steps of the CUDA ClusterFinder", + 22, ACCENT) + +tf = tb(s, M + 0.3, 3.25, 9.6, 0.8) +run(para(tf, True, line=1.3), + "Five of the six steps never touch the arithmetic. They are about keeping " + "a 24 µs kernel supplied with data — and about learning to measure honestly.", + 12.5, TEXT2) + +stats = [("×8.29", "VS 48-THREAD CPU", ACCENT), ("39,472", "FRAMES / SECOND", PALE), + ("25.3 µs", "PER FRAME, END TO END", PALE), ("0.004%", "CLUSTER-COUNT DRIFT", AMBER)] +for i, (v, l, c) in enumerate(stats): + x = M + 0.3 + i * 2.85 + rect(s, x, 4.35, 0.035, 0.95, c) + tf = tb(s, x + 0.22, 4.35, 2.5, 0.55) + run(para(tf, True), v, 30, c, bold=True) + tf = tb(s, x + 0.22, 4.98, 2.5, 0.3) + run(para(tf, True), l, 8.5, MUTED, spc=1.2) + +rect(s, M + 0.3, 6.05, 11.0, 0.012, RULE) +tf = tb(s, M + 0.3, 6.25, 11.4, 0.6) +run(para(tf, True, line=1.35), + "RTX 4090 (Ada, sm_89) · PCIe 4.0 ×16 · Mönch 400×400 uint16 · 3×3 clusters · " + "100 000 frames · Cu fluorescence, MAX IV", 10, MUTED) +tf = tb(s, M + 0.3, 6.62, 11.4, 0.3) +run(para(tf, True), "Khalil Ferjaoui · Paul Scherrer Institut", 10, TEXT2) + +# =========================================================== 2 · PROBLEM +s = new_slide() +chrome(s, 2, "The problem & the baseline", "What has to happen to every frame") +bullets(s, M, 1.95, COL, [ + "Per pixel: subtract a **running pedestal** (mean ± rms), keep pixels above " + "**nσ · rms**, cut a 3×3 cluster around each local maximum.", + "400×400 = 160 k pixels, **312.5 kB per frame**; Cu data yields ~2 330 clusters " + "per frame at 3×3.", + "The pedestal is **updated by every non-photon pixel**, every frame — so the " + "arithmetic and the data movement are coupled.", +]) +code(s, M, 3.55, COL, [ + "// the whole algorithm, per pixel", + "v = frame[i] - pedestal_mean[i]", + "rms = sqrt(pedestal_sum2[i]/n - pedestal_mean[i]^2)", + "if (v > «nSigma» * rms && v == max(3x3 window)) -> emit cluster", + "else -> update pedestal", +], title="THE KERNEL IN FIVE LINES") +callout(s, M, 5.55, COL, + "**Thesis of this talk:** the compute was fast almost immediately. " + "Five of six steps are about feeding it.") +rail(s, [ + ("label", "Baseline · same data, same threshold"), + ("gap", 0.15), + ("stat", "CPU, 1 thread", "1.75 ms", MUTED), + ("stat", "CPU MT, 48 threads", "210 µs", PALE), + ("gap", 0.1), + ("row", "That is the bar", "4 761 frames / s", TEXT2), + ("gap", 0.25), + ("note", "Every CUDA number in this deck is measured against the 48-thread " + "CPU on the same 100 000 frames."), +]) + +# =========================================================== 3 · THE LADDER +s = new_slide() +chrome(s, 3, "Roadmap", "Two acts: feed the GPU, then speed up the kernel") +rows = [ + ("opt1", "First CUDA port", "1 stream, one launch per frame", "×3.14", ACCENT), + ("opt2", "Streams + batching", "4 streams, 2 000-frame batches", "×4.86", ACCENT), + ("opt3", "Pipeline rework", "sync barriers removed", "×5.58", ACCENT), + ("opt4", "Pinned memory", "DMA-speed host transfers", "×7.73", ACCENT), + ("opt5", "CUDA Graphs", "one launch replaces six", "×8.29", ACCENT), + ("opt6", "FP32 pedestal + variance rewrite", "the first kernel change", "kernel −40%", AMBER), +] +y = 2.05 +for i, (tag, name, sub, gain, col) in enumerate(rows): + rect(s, M, y, 11.9, 0.72, PANEL if i % 2 == 0 else BG) + rect(s, M, y, 0.035, 0.72, col) + tf = tb(s, M + 0.28, y + 0.13, 1.0, 0.4) + run(para(tf, True), tag, 15, col, bold=True, font=MONO) + tf = tb(s, M + 1.45, y + 0.10, 5.0, 0.3) + run(para(tf, True), name, 13, PALE, bold=True) + tf = tb(s, M + 1.45, y + 0.38, 5.6, 0.3) + run(para(tf, True), sub, 10, MUTED) + tf = tb(s, 9.4, y + 0.18, 3.1, 0.4) + run(para(tf, True, align=PP_ALIGN.RIGHT), gain, 15, col, bold=True) + y += 0.78 +rect(s, M, 2.05, 0.012, 4.68, RULE) +caption(s, M, 6.72, 11.9, + "opt1–opt5 change only how work is scheduled and moved — the arithmetic is " + "byte-identical. opt6 is the first step that changes the kernel itself.") + +# =========================================================== 4 · METHODOLOGY +s = new_slide() +chrome(s, 4, "Before any number is believed", "Three ways a GPU benchmark lies") +items = [ + ("First-touch page faults", AMBER, + "Each run materialises ~10 GB of clusters. The first pass faults in ~2.6 M " + "pages at 0.7 µs each — up to 4 s of pure OS work inside the timer.", + "Fix: re-run until getrusage() minor faults plateau (< 200 k)."), + ("CUDA-event kernel timing", AMBER, + "avg_kernel_time_ms() measures elapsed time on a stream — including waiting " + "for other streams. Under 8-stream load it over-reads by up to 3.5×.", + "Fix: Nsight Systems per-instance times; 1 stream for exclusive numbers."), + ("The profiler itself", AMBER, + "Under nsys, wall time per frame inflates ~4× from API tracing.", + "Fix: GPU op times from nsys, wall times from unprofiled runs."), +] +x = M +for title, col, body, fix in items: + rect(s, x, 2.0, 3.83, 3.15, PANEL) + rect(s, x, 2.0, 3.83, 0.035, col) + tf = tb(s, x + 0.26, 2.28, 3.3, 0.6) + run(para(tf, True, line=1.15), title, 13, PALE, bold=True) + tf = tb(s, x + 0.26, 3.02, 3.3, 1.5) + run(para(tf, True, line=1.3), body, 10, TEXT2) + tf = tb(s, x + 0.26, 4.42, 3.3, 0.65) + run(para(tf, True, line=1.25), fix, 9.5, ACCENT) + x += 4.03 +code(s, M, 5.4, 11.9, [ + "# every timed cell in the benchmark notebook is bracketed with:", + "mf0 = resource.getrusage(resource.RUSAGE_SELF).ru_minflt", + "... t = time.perf_counter() - t0 ...", + "print(f'minor faults: {mf1-mf0:,}') # quote the run where this plateaus", +], title="THE FAULT PROTOCOL — python/tests/ClusterFinderCUDA_perf.ipynb") +callout(s, M, 6.52, 11.2, + "Validated: **wall = steady-state + faults × 0.68 µs** reproduced a 6.110 s " + "run to within **1 ms**. Kernel time stayed constant throughout — the GPU was never the variable.", + h=0.70, size=9.5) + +# =========================================================== 5 · OPT1 +s = new_slide() +chrome(s, 5, "opt1 · the first CUDA port", "One frame, one stream, fully synchronous") +bullets(s, M, 1.95, COL, [ + "Shared-memory tiling with **halo loading** for any cluster size; pedestal " + "subtraction fused into the tile load.", + "Cluster geometry is a **compile-time template parameter** → the 3×3 stencil " + "is fully unrolled.", + "One cudaMemcpy in, one kernel, one cudaMemcpy out — **the host blocks " + "on every frame**.", +]) +code(s, M, 3.62, COL, [ + "// one frame at a time — the host waits at every step", + "cudaMemcpy(d_frame, h_frame, bytes, cudaMemcpyHostToDevice);", + "find_clusters_in_single_frame", + " <<>>(d_frame, d_pd_mean, ...);", + "cudaMemcpy(h_out, d_out, out_bytes, cudaMemcpyDeviceToHost);", +], title="ClusterFinderCUDAOpt2.hpp · find_clusters()") +callout(s, M, 5.62, COL, + "The stencil was **already fast**: 23 µs of kernel inside a 67 µs frame. " + "The other 44 µs is the host standing still.") +rail(s, [ + ("label", "opt1 · 3×3 · 100 k frames"), + ("gap", 0.15), + ("stat", "End to end", "66.8 µs", PALE), + ("stat", "vs 48-thread CPU", "×3.14", ACCENT), + ("gap", 0.05), + ("row", "Kernel (GPU)", "23 µs", TEXT2), + ("row", "Host + PCIe", "44 µs", AMBER), + ("gap", 0.2), + ("note", "Two thirds of the frame is spent not computing."), +]) + +# =========================================================== 6 · OPT2 +s = new_slide() +chrome(s, 6, "opt2 · streams and batching", "A CUDA stream is a queue the GPU can overlap") +bullets(s, M, 1.95, COL, [ + "A **stream** is an ordered queue of GPU work. Work in **different** streams may " + "overlap — so a copy can run while another stream computes.", + "Each stream gets its own **StreamContext**: device frame buffer, output buffer " + "and pedestal. Frames are handed out **round-robin**.", + "The host now submits **2 000 frames per call** instead of one.", +]) +code(s, M, 3.62, COL, [ + "struct StreamContext {", + " cudaStream_t stream;", + " FRAME_TYPE *d_frame; ClusterType *d_clusters;", + " PEDESTAL_TYPE *d_pd_mean, *d_pd_sum, *d_pd_sum2;", + "};", + "auto &sc = v_sc[frame_idx % «n_streams»]; // round-robin", +], title="ClusterFinderCUDA.hpp · per-stream state") +callout(s, M, 5.72, COL, + "**Scaffolding, not yet the payoff.** The streams exist, but the host still " + "synchronises after every round — see opt3.") +rail(s, [ + ("label", "opt2 · 4 streams · batch 2 000"), + ("gap", 0.15), + ("stat", "End to end", "43.2 µs", PALE), + ("stat", "vs CPU", "×4.86", ACCENT), + ("gap", 0.05), + ("row", "Step gain over opt1", "×1.55", ACCENT), + ("row", "Host + PCIe", "21 µs (was 44)", AMBER), +]) + +# =========================================================== 7 · OPT3 +s = new_slide() +chrome(s, 7, "opt3 · remove the sync barriers", "Stop draining the GPU between rounds") +bullets(s, M, 1.95, 7.4, [ + "opt2 synchronised **all streams after every round** of n_streams frames. " + "The GPU drained to empty each time.", + "opt3 submits every frame's H2D → kernel → D2H **asynchronously**, then " + "synchronises **once at the end of the batch**.", +], size=10.5) +figure(s, "fig_streams", M, 3.05, 7.55) +code(s, 8.35, 1.95, 4.25, [ + "// opt2: barrier after every round", + "for (round) {", + " submit(n_streams frames);", + " «cudaDeviceSynchronize»();", + "}", + "", + "// opt3: submit everything, sync once", + "for (frame : batch) {", + " cudaMemcpyAsync(..., sc.stream);", + " kernel<<<..., sc.stream>>>(...);", + " cudaMemcpyAsync(..., sc.stream);", + "}", + "for (sc : streams)", + " «cudaStreamSynchronize»(sc.stream);", +], size=8, title="THE ONE-LINE IDEA") +callout(s, 8.35, 5.05, 4.25, + "**37.6 µs/frame · ×5.58**\nHost overhead 21 → **14 µs**", h=0.86, size=11) +caption(s, 8.35, 6.15, 4.25, + "Each lane is one stream. Removing the barrier lets a stream start its next " + "frame while its neighbours are still copying.") + +# =========================================================== 8 · OPT4 +s = new_slide() +chrome(s, 8, "opt4 · pinned (page-locked) memory", "What pinning is, and why the GPU cares") +bullets(s, M, 1.95, 12.0, [ + "Normal host memory is **pageable** — the OS may move or swap it. A DMA engine " + "cannot safely read that, so the driver first copies your data into a **hidden " + "pinned staging buffer**. Every transfer is copied twice.", + "**Pinning** locks the pages in physical RAM. The GPU's DMA engine then reads " + "host memory **directly** — no staging copy, and the transfer can be truly asynchronous.", +], size=10.5) +figure(s, "fig_pinning", M, 3.15, 7.6) +code(s, 8.5, 3.15, 4.1, [ + "// pin the whole dataset once", + "«cudaHostRegister»(ptr, bytes,", + " cudaHostRegisterDefault);", + "", + "// ... run the whole campaign ...", + "", + "«cudaHostUnregister»(ptr);", +], size=8, title="ClusterFinderCUDA.hpp") +callout(s, 8.5, 4.72, 4.1, + "**27.2 µs/frame · ×7.73**\nHost overhead 14 → **3 µs**", h=0.86, size=11) +caption(s, 8.5, 5.82, 4.1, + "Measured H2D: one 400×400 uint16 frame (312.5 KiB = 320 000 B) in 13.2 µs " + "= 24.2 GB/s — 77% of PCIe 4.0 ×16 theoretical, i.e. true DMA speed. " + "Pageable staging runs ~15 GB/s.") +caption(s, M, 6.62, 7.6, + "Caveat: pinned memory is a finite system resource — it cannot be swapped. " + "aare exposes a budget helper (print_pinning_budget) before you pin 31 GB.") + +# =========================================================== 9 · OPT5 +s = new_slide() +chrome(s, 9, "opt5 · CUDA Graphs", "Record the pipeline once, replay it with one call") +bullets(s, M, 1.95, 12.0, [ + "Every cudaMemcpyAsync / kernel launch costs the **CPU** a few microseconds of " + "driver work — per frame, per operation. At 39 k frames/s that is the budget.", + "A **CUDA Graph** captures the whole dependency DAG once. Replaying it is a " + "**single** cudaGraphLaunch — the driver already knows every node and edge.", +], size=10.5) +figure(s, "fig_graphs", M, 3.15, 7.6) +code(s, 8.5, 3.15, 4.1, [ + "// record once, at setup", + "cudaStreamBeginCapture(sc.stream, ...);", + " submit_h2d_kernel_d2h(sc);", + "cudaStreamEndCapture(sc.stream, &sc.graph);", + "«cudaGraphInstantiate»(&sc.graphExec, ...);", + "", + "// per batch — one call", + "«cudaGraphLaunch»(sc.graphExec, sc.stream);", +], size=8, title="ClusterFinderCUDA_graph.hpp") +callout(s, 8.5, 4.88, 4.1, + "**25.3 µs/frame · ×8.29**\nBest end-to-end result", h=0.86, size=11) +caption(s, M, 6.62, 12.0, + "Worth 7% here because opt4 already removed the transfer cost — what remains " + "is CPU launch overhead, which is exactly what graphs eliminate. " + "Trade-off: shapes are frozen at record time, so the batch geometry must be fixed.") + +# =========================================================== 10 · OPT6 why +s = new_slide() +chrome(s, 10, "opt6 · FP32 device pedestal", "The first change to the kernel itself") +bullets(s, M, 1.95, 7.5, [ + "~99.9% of pixels take the **pedestal-update** branch, which reads and writes " + "mean, sum and sum². In FP64 that is **32 bytes per pixel**; in FP32, 16.", + "The kernel is **bandwidth-bound**, so halving that traffic nearly halves the time.", + "Second effect: on a GeForce part, **FP64 arithmetic runs at 1/64 of FP32**. " + "The pedestal update was paying that tax on every pixel.", +], size=10.5) +figure(s, "fig_f32_kernel", M, 4.05, 7.5) +code(s, 8.5, 1.95, 4.1, [ + "// clusterfinder_kernel.cuh", + "using COMPUTE_TYPE = float;", + "using DEVICE_PED_TYPE = «float»;", + "// was: double", +], size=8.5, title="ONE TYPEDEF") +callout(s, 8.5, 3.05, 4.1, + "Kernel, 9×9, measured with Nsight Systems\n**43.0 µs → 25.6 µs (−40%)**", + h=0.92, size=11) +caption(s, 8.5, 4.20, 4.1, + "Exclusive per-instance kernel time, 2 000 instances, 1 stream. " + "σ = 0.4 µs. Transfers are unchanged, as they must be.") +callout(s, 8.5, 5.30, 4.1, + "But a faster kernel is **not automatically a faster frame** — and naive FP32 " + "is **wrong**. Both on the next slides.", h=1.05, size=10, color=AMBER) + +# =========================================================== 11 · OPT6 trap +s = new_slide() +chrome(s, 11, "opt6 · the correctness trap", "Why the obvious FP32 pedestal is broken") +bullets(s, M, 1.95, 7.9, [ + "The running variance was computed as **var = E[X²] − mean²**. With a pedestal " + "mean of ~4 655 ADU, both terms are ≈ 2.17 × 10⁷ while the answer is ≈ 2 000.", + "In FP32 the spacing between representable numbers at 2.17 × 10⁷ is **2 048** — " + "larger than the variance itself. This is **catastrophic cancellation**.", +], size=10.5) +figure(s, "fig_cancellation", M, 3.25, 7.5) +rail(s, [ + ("label", "What it looked like"), + ("gap", 0.15), + ("stat", "Extra clusters", "+28.06%", AMBER), + ("gap", 0.05), + ("note", "Quiet pixels got rms → 0, so their threshold became 0 and they fired " + "on every single frame — producing a large unphysical high-energy tail " + "in the spectrum."), + ("gap", 0.35), + ("row", "Affected pixels", "~1–2% of the sensor", TEXT2), + ("row", "Written up in", "docs/pedestal_precision_…", MUTED), +]) + +# =========================================================== 12 · OPT6 fix +s = new_slide() +chrome(s, 12, "opt6 · the variance rewrite", "Accumulate what is small, not what is large") +bullets(s, M, 1.95, COL, [ + "Freeze a per-pixel baseline **X₀ = round(mean)** once, at the end of pedestal " + "training, and never move it again.", + "Accumulate the **centred** value Y = X − X₀ instead of X. Now both sums are " + "O(rms)-sized — the huge common term is gone before the subtraction.", + "The reported pedestal mean is still the full value, **X₀ + sum/n**, so nothing " + "downstream changes.", +]) +code(s, M, 4.0, COL, [ + "// before — both terms ~2.17e7, answer ~2000", + "var = sum2/n - mean*mean;", + "", + "// after — centred on a frozen per-pixel offset X0", + "DEVICE_PED_TYPE resid = mean - «d_pd_off»[i]; // ~O(1)", + "DEVICE_PED_TYPE var_px = sum2[i]/n - resid*resid; // no cancellation", +], title="clusterfinder_kernel.cuh") +callout(s, M, 6.05, COL, + "Result: the 100% FP32 build now matches the FP64 build to " + "**3 × 10⁻⁷** — 70 clusters out of 233 million.") +rail(s, [ + ("label", "Why it works"), + ("gap", 0.15), + ("note", "Precision is relative. Floats resolve small numbers finely and large " + "numbers coarsely — so never let a small answer be the difference of " + "two large numbers."), + ("gap", 0.5), + ("row", "f32 vs f64 counts", "3 × 10⁻⁷", ACCENT), + ("row", "vs CPU", "0.0039%", ACCENT), + ("gap", 0.3), + ("note", "X₀ must never be updated — the accumulators are defined relative to it."), +]) + +# =========================================================== 13 · OPT6 when +s = new_slide() +chrome(s, 13, "opt6 · when does a faster kernel help?", "Only if the kernel was the tallest bar") +figure(s, "fig_bottleneck", M, 2.05, 11.9) +callout(s, M, 5.35, 11.9, + "**f32 buys exactly the distance between the kernel and the next-tallest bar.** " + "At 3×3 the kernel already hides inside the transfers, so end-to-end throughput does not move at all. " + "At 9×9 the kernel is on the critical path, and the same change is worth 8% of the frame.", + h=0.95, size=11) +caption(s, M, 6.55, 11.9, + "9×9, 20 000 frames, 8 streams, cap 1 500 (all CPU clusters recorded): " + "1.540 s → 1.423 s batched, 1.482 s → 1.372 s with graphs. " + "The FP64 build shows the classic kernel-bound signature — per-frame wall time shorter than per-frame kernel time, because kernels from different streams queue.") + +# =========================================================== 14 · RESULTS +s = new_slide() +chrome(s, 14, "Results", "The whole ladder, one dataset, one baseline") +figure(s, "fig_arc", M, 1.95, 11.9) +callout(s, M, 5.55, 5.85, + "**×8.29 over 48 CPU threads** — 21.0 s → 2.53 s for 100 000 frames.", h=0.8) +callout(s, 6.75, 5.55, 5.85, + "Every step is **monotonic**, and correctness is held constant at **0.004%** throughout.", + h=0.8, color=AMBER) +caption(s, M, 6.55, 11.9, + "3×3 clusters · nσ = 5 · 100 000 frames · batch 2 000 · 4 streams · warm run " + "(page faults plateaued) · CPU baseline = ClusterFinderMT with 48 threads.") + +# =========================================================== 15 · WHERE TIME GOES +s = new_slide() +chrome(s, 15, "Where the time actually went", "The kernel never changed — the overhead collapsed") +figure(s, "fig_overhead", M, 2.05, 5.9) +bullets(s, 7.0, 2.15, 5.6, [ + "The GPU kernel is a **flat ~23 µs** across opt1–opt5. Not one of those steps " + "made the arithmetic faster.", + "What changed is everything around it: **44 → 21 → 14 → 3 µs** of host and PCIe " + "time per frame.", + "By opt4 the pipeline is **fed almost perfectly** — which is precisely why opt5 " + "(launch overhead) is the only lever left, and why opt6 shows nothing at 3×3.", +], size=10.5) +callout(s, 7.0, 5.15, 5.6, + "The bottleneck moved from **the host**, to **PCIe**, and finally — only for " + "large cluster windows — to **the kernel**.", h=0.95, size=10.5) +caption(s, M, 6.4, 11.9, + "Per-frame GPU operation profile at 9×9 (Nsight Systems): kernel 43 µs · " + "D2H 19.4 µs · H2D 13.2 µs. At 3×3 the kernel is 13–24 µs against a ~25 µs " + "transfer-and-host floor — which is the entire story of this deck in three numbers.") + +# =========================================================== 16 · CORRECTNESS +s = new_slide() +chrome(s, 16, "Validation", "Same physics out of every variant") +figure(s, "fig_correctness", M, 2.0, 7.6) +bullets(s, 8.6, 2.05, 4.1, [ + "233 million clusters over 100 000 frames.", + "All CUDA variants agree with the CPU to **0.004%**.", + "The residual is **not** precision: it is the CUDA finder updating the pedestal " + "**once per frame** vs the CPU's per-pixel update.", +], size=10) +code(s, M, 4.85, 7.6, [ + "CPU (ClusterFinderMT) 233 085 343 2330.85 / frame reference", + "opt1 .. opt5 (f64 ped) 233 093 484 2330.93 / frame 0.0035 %", + "opt6 (f32 ped) 233 094 465 2330.94 / frame 0.0039 %", +], size=8.5, title="CLUSTER COUNTS · 3×3 · 100 000 FRAMES") +callout(s, 8.6, 4.85, 4.1, + "A local-maximum gate had to be **back-ported** into the opt1/opt2 snapshot so " + "the whole ladder is compared at identical correctness.", h=1.0, size=10, color=AMBER) +caption(s, M, 6.35, 11.9, + "Cross-checks: ClusterFinderFrozen (a CPU finder with the CUDA pedestal-update " + "timing) isolates that residual; energy spectra overlay within statistics.") + +# =========================================================== 17 · API 1 +s = new_slide() +chrome(s, 17, "For users · Python API", "The fast path in eight lines") +code(s, M, 1.95, 7.6, [ + "from aare import File, ClusterFinderCUDA", + "", + "cf = ClusterFinderCUDA(image_size=(400, 400), cluster_size=(3, 3),", + " n_sigma=5, «n_streams»=4,", + " «max_clusters_per_frame»=3000)", + "", + "for _ in range(1000): # 1. train the pedestal", + " cf.push_pedestal_frame(pd.read_frame())", + "", + "data = f.read_n(100_000) # 2. one contiguous array", + "cf.«register_input_buffer»(data) # 3. pin it once", + "", + "for s in range(0, N, 2000): # 4. batch through it", + " clusters = cf.«find_clusters_batched»(data[s:s+2000], first_frame=s)", + "", + "cf.unregister_input_buffer() # 5. release the pages", +], size=9, title="THE RECOMMENDED PATTERN") +bullets(s, 8.6, 2.0, 4.1, [ + "find_clusters_batched returns **one ClusterVector per frame**, in order.", + "register_input_buffer is what turns opt3 into opt4 — **one call**.", + "Swap in ClusterFinderCUDAGraph for opt5; the API is identical.", +], size=10) +callout(s, 8.6, 4.55, 4.1, + "GIL is released for both find_clusters and find_clusters_batched, so " + "reading the next file can overlap with the GPU.", h=1.05, size=10) +callout(s, 8.6, 5.80, 4.1, + "Pin **once**, outside the loop. Slices of a registered array inherit the pinning.", + h=0.85, size=10, color=AMBER) + +# =========================================================== 18 · API 2 +s = new_slide() +chrome(s, 18, "For users · choosing the knobs", "What to set, and what it costs you") +hdr = [("Parameter", 1.05), ("What it does", 3.6), ("Guidance", 5.2)] +y = 2.0 +rect(s, M, y, 11.9, 0.4, PANEL) +for lab, dx in hdr: + tf = tb(s, M + dx - 0.85 if dx > 1.05 else M + 0.28, y + 0.09, 5.0, 0.3) + run(para(tf, True), lab.upper(), 9, MUTED, bold=True, spc=1.3) +y += 0.44 +params = [ + ("n_streams", "How many frames are in flight at once.", + "4 is right for 3×3. Larger windows saturate the GPU — 8 helps at 9×9."), + ("max_clusters_per_frame", "Fixed size of the per-frame D2H transfer.", + "Must exceed the real maximum or clusters are silently dropped. Too high wastes PCIe."), + ("batch size", "Frames per find_clusters_batched call.", + "2 000 amortises launch overhead without a large pinned footprint."), + ("cluster_size", "Compile-time stencil geometry.", + "3×3 and 9×9 are registered; 9×9 shifts the bottleneck onto the kernel."), + ("register_input_buffer", "Page-locks the host array for DMA.", + "Always, if the data is already in RAM. Check the pinning budget first."), +] +for i, (p_, what, guide) in enumerate(params): + if i % 2 == 0: + rect(s, M, y, 11.9, 0.82, PANEL) + tf = tb(s, M + 0.28, y + 0.14, 2.6, 0.5) + run(para(tf, True, line=1.1), p_, 9.5, ACCENT, font=MONO, bold=True) + tf = tb(s, M + 3.0, y + 0.14, 2.9, 0.6) + run(para(tf, True, line=1.2), what, 9.5, PALE) + tf = tb(s, M + 6.15, y + 0.14, 5.4, 0.6) + run(para(tf, True, line=1.2), guide, 9.5, TEXT2) + y += 0.80 +callout(s, M, 6.55, 11.2, + "The single most common mistake: leaving **max_clusters_per_frame** too low. " + "It does not error — it truncates, and every frame quietly returns the same count.", + h=0.66, size=10, color=AMBER) + +# =========================================================== 19 · NEXT +s = new_slide() +chrome(s, 19, "Where this leaves us", "The bottleneck has moved — twice") +cards = [ + ("DONE", ACCENT, "×8.29 over 48 CPU threads", + "25.3 µs/frame end to end, at 0.004% cluster agreement. Five pipeline steps " + "and one kernel step."), + ("DONE", ACCENT, "FP32 pedestal, safely", + "−40% kernel, and correct — because the variance is now accumulated on a frozen " + "per-pixel offset instead of a raw second moment."), + ("NEXT", AMBER, "Attack the transfers, not the kernel", + "At 3×3 the kernel is already hidden. The remaining per-frame cost is PCIe: " + "492 kB of clusters out, 312 kB of frame in."), + ("NEXT", AMBER, "Keep results on the device", + "On-GPU reduction, eta/interpolation on device, or compressed cluster formats — " + "so the D2H bar stops setting the floor."), +] +x, y = M, 2.05 +for i, (tag, col, title, body) in enumerate(cards): + cx = M + (i % 2) * 6.05 + cy = 2.05 + (i // 2) * 2.35 + rect(s, cx, cy, 5.85, 2.05, PANEL) + rect(s, cx, cy, 5.85, 0.035, col) + tf = tb(s, cx + 0.3, cy + 0.26, 1.4, 0.26) + run(para(tf, True), tag, 8.5, col, bold=True, spc=1.5) + tf = tb(s, cx + 0.3, cy + 0.60, 5.2, 0.4) + run(para(tf, True, line=1.1), title, 14, PALE, bold=True) + tf = tb(s, cx + 0.3, cy + 1.12, 5.2, 0.85) + run(para(tf, True, line=1.3), body, 10, TEXT2) +callout(s, M, 6.58, 11.2, + "Full numbers, methodology and reproduction steps: **docs/benchmark_opt1_opt6_results.md** · " + "notebook **python/tests/ClusterFinderCUDA_perf.ipynb** · profiler probe **python/tests/nsys_kernel_probe.py**", + h=0.66, size=9.5) + +prs.save(OUT) +print(f"saved {OUT} ({len(prs.slides.__iter__.__self__._sldIdLst)} slides)") diff --git a/docs/deck/make_figs.py b/docs/deck/make_figs.py new file mode 100644 index 00000000..e147198e --- /dev/null +++ b/docs/deck/make_figs.py @@ -0,0 +1,363 @@ +"""Figures for ClusterFinderCUDA_optimizations.pptx — deck palette, dark, transparent.""" +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np +from matplotlib.patches import FancyArrowPatch, Rectangle +from pathlib import Path + +OUT = Path(__file__).parent / "figs" +OUT.mkdir(exist_ok=True) + +BG = "#0B1018" +PANEL = "#121A28" +RULE = "#1E2836" +ACCENT = "#1E90C2" # data 1 +AMBER = "#E8B25C" # data 2 +PALE = "#E7EDF4" # data 3 / primary text +TEXT2 = "#A5B2C4" +MUTED = "#6B7A90" # non-data only: grid, axes, annotation + +plt.rcParams.update({ + "font.family": "DejaVu Sans", "font.size": 9, + "text.color": PALE, "axes.labelcolor": TEXT2, + "xtick.color": TEXT2, "ytick.color": TEXT2, + "axes.edgecolor": RULE, "axes.facecolor": "none", + "figure.facecolor": BG, "savefig.facecolor": BG, + "axes.grid": False, "svg.fonttype": "none", +}) + + +def save(fig, name): + fig.savefig(OUT / f"{name}.png", dpi=220, transparent=False, + bbox_inches="tight", pad_inches=0.08) + plt.close(fig) + print("wrote", name) + + +def bare(ax, keep=("left", "bottom")): + for s in ("top", "right", "left", "bottom"): + ax.spines[s].set_visible(s in keep) + + +# ---------------------------------------------------------------- 1. the arc +def fig_arc(): + steps = ["CPU MT\n48 threads", "opt1\n1 stream", "opt2\nstreams+batch", + "opt3\npipeline", "opt4\npinned", "opt5\ngraphs"] + fps = [4761, 14968, 23134, 26588, 36810, 39472] + spd = [1.0, 3.14, 4.86, 5.58, 7.73, 8.29] + colors = [MUTED] + [ACCENT] * 4 + [AMBER] + + fig, ax = plt.subplots(figsize=(11.4, 3.5)) + x = np.arange(len(steps)) + bars = ax.bar(x, fps, width=0.62, color=colors, zorder=3) + for b in bars: + b.set_linewidth(0) + for xi, (f, s) in enumerate(zip(fps, spd)): + ax.text(xi, f + 900, f"{f:,}", ha="center", va="bottom", + color=PALE, fontsize=11, fontweight="bold") + ax.text(xi, f + 3100, ("baseline" if s == 1.0 else f"×{s:.2f}"), + ha="center", va="bottom", color=AMBER if s > 8 else TEXT2, fontsize=9) + ax.set_xticks(x) + ax.set_xticklabels(steps, fontsize=9, color=TEXT2) + ax.set_ylim(0, 47000) + ax.set_yticks([]) + bare(ax, keep=("bottom",)) + ax.spines["bottom"].set_color(RULE) + ax.set_ylabel("") + ax.text(0, 45500, "frames / second · 3×3 clusters, 100 000 frames, warm run", + color=MUTED, fontsize=9, ha="left") + save(fig, "fig_arc") + + +# ------------------------------------------------- 2. host overhead collapse +def fig_overhead(): + steps = ["opt1", "opt2", "opt3", "opt4"] + ovhd = [44, 21, 14, 3] + kern = [23, 22, 24, 24] + + fig, ax = plt.subplots(figsize=(5.6, 3.0)) + x = np.arange(len(steps)) + ax.bar(x, kern, width=0.55, color=ACCENT, zorder=3, label="kernel (GPU)") + ax.bar(x, ovhd, width=0.55, bottom=kern, color=AMBER, zorder=3, + label="host + PCIe overhead") + for xi, (k, o) in enumerate(zip(kern, ovhd)): + ax.text(xi, k + o + 1.6, f"{o} µs", ha="center", color=AMBER, + fontsize=10, fontweight="bold") + ax.set_xticks(x); ax.set_xticklabels(steps, color=TEXT2) + ax.set_ylabel("µs / frame", color=TEXT2) + ax.set_ylim(0, 78) + bare(ax) + ax.legend(frameon=False, fontsize=8.5, labelcolor=TEXT2, loc="upper right") + save(fig, "fig_overhead") + + +# ------------------------------------------------------ 3. streams timeline +def fig_streams(): + fig, axes = plt.subplots(3, 1, figsize=(7.7, 3.9)) + H, K, D = 12, 22, 12 + FR = H + K + D + LANE = 0.68 + + def frame(ax, lane_y, t0): + ax.broken_barh([(t0, H)], (lane_y, LANE), facecolors=AMBER, zorder=3) + ax.broken_barh([(t0 + H, K)], (lane_y, LANE), facecolors=ACCENT, zorder=3) + ax.broken_barh([(t0 + H + K, D)], (lane_y, LANE), facecolors=PALE, zorder=3) + + # --- opt1: one stream, strictly serial + ax = axes[0] + for i in range(3): + frame(ax, 1.0, i * FR) + ax.set_ylim(0.4, 2.3) + ax.text(3 * FR + 6, 1.34, "GPU idle between every stage", color=MUTED, fontsize=7.5, + va="center") + + # --- opt2: 4 streams, barrier after each round + ax = axes[1] + ROUND = FR + 3 * 8 + for r in range(2): + for st in range(4): + frame(ax, 3 - st * 1.0, r * (ROUND + 26) + st * 8) + ax.axvspan(ROUND, ROUND + 26, color=AMBER, alpha=0.13, zorder=1) + ax.text(ROUND + 13, 4.15, "barrier — GPU drains", color=AMBER, fontsize=7.5, + ha="center", va="bottom") + ax.set_ylim(-0.4, 4.9) + + # --- opt3: no barriers, continuous + ax = axes[2] + for i in range(11): + frame(ax, 3 - (i % 4) * 1.0, i * 11) + ax.set_ylim(-1.5, 4.5) + ax.text(0, -0.25, "streams never wait on each other — the GPU is continuously busy", + color=ACCENT, fontsize=7.5, va="top") + + titles = ["opt1 · 1 stream, synchronous", + "opt2 · 4 streams, sync barrier per round", + "opt3 · 4 streams, barriers removed"] + for ax, t in zip(axes, titles): + ax.set_xlim(-2, 190) + ax.set_yticks([]); ax.set_xticks([]) + bare(ax, keep=()) + ax.set_title(t, color=TEXT2, fontsize=9, loc="left", pad=4) + + handles = [Rectangle((0, 0), 1, 1, color=c) for c in (AMBER, ACCENT, PALE)] + axes[0].legend(handles, ["H2D copy", "kernel", "D2H copy"], frameon=False, + fontsize=8, labelcolor=TEXT2, ncol=3, loc="lower right", + bbox_to_anchor=(1.02, 0.98), handlelength=1.1) + axes[2].set_xlabel("time →", color=MUTED, fontsize=8.5, loc="left") + fig.subplots_adjust(hspace=0.75) + save(fig, "fig_streams") + + +# ------------------------------------------------------------- 4. pinning +def fig_pinning(): + fig = plt.figure(figsize=(7.7, 3.0)) + ax = fig.add_axes([0, 0.05, 0.63, 0.95]); ax.axis("off") + ax.set_xlim(0, 10.4); ax.set_ylim(0, 6.4) + + def box(x, y, w, h, label, sub=""): + ax.add_patch(Rectangle((x, y), w, h, facecolor=PANEL, edgecolor=RULE, lw=1)) + ax.text(x + w / 2, y + h / 2 + 0.26, label, ha="center", va="center", + color=PALE, fontsize=8.5, fontweight="bold") + ax.text(x + w / 2, y + h / 2 - 0.34, sub, ha="center", va="center", + color=MUTED, fontsize=7) + + def arrow(x0, x1, y, color, label): + ax.add_patch(FancyArrowPatch((x0, y), (x1, y), arrowstyle="-|>", + mutation_scale=10, color=color, lw=1.6)) + ax.text((x0 + x1) / 2, y + 0.22, label, ha="center", va="bottom", + color=color, fontsize=7) + + ax.text(0, 5.95, "PAGEABLE · before opt4", color=AMBER, fontsize=8.5, + fontweight="bold") + box(0, 4.05, 2.5, 1.1, "numpy array", "pageable") + box(4.0, 4.05, 2.4, 1.1, "driver staging", "hidden pinned buf") + box(7.9, 4.05, 2.5, 1.1, "GPU", "device memory") + arrow(2.5, 4.0, 4.60, AMBER, "memcpy") + arrow(6.4, 7.9, 4.60, AMBER, "DMA") + ax.text(0, 3.62, "every transfer is copied twice", color=MUTED, fontsize=7) + + ax.text(0, 2.75, "PINNED · opt4", color=ACCENT, fontsize=8.5, fontweight="bold") + box(0, 0.85, 2.5, 1.1, "numpy array", "page-locked") + box(7.9, 0.85, 2.5, 1.1, "GPU", "device memory") + arrow(2.5, 7.9, 1.40, ACCENT, "DMA — engine reads host RAM directly") + ax.text(0, 0.42, "no staging copy, no page faults, fully async", + color=MUTED, fontsize=7) + + ax2 = fig.add_axes([0.75, 0.16, 0.25, 0.66]) + v = [14, 3] + ax2.bar([0, 1], v, width=0.55, color=[AMBER, ACCENT], zorder=3) + for i, val in enumerate(v): + ax2.text(i, val + 0.5, f"{val} µs", ha="center", color=PALE, + fontsize=10, fontweight="bold") + ax2.set_xticks([0, 1]) + ax2.set_xticklabels(["opt3\npageable", "opt4\npinned"], color=TEXT2, fontsize=8) + ax2.set_ylim(0, 18); ax2.set_yticks([]); bare(ax2, keep=("bottom",)) + ax2.set_title("host overhead / frame", color=MUTED, fontsize=7.5, pad=6) + save(fig, "fig_pinning") + + +# -------------------------------------------------------------- 5. graphs +def fig_graphs(): + fig, ax = plt.subplots(figsize=(7.7, 2.6)) + ax.axis("off"); ax.set_xlim(0, 12.6); ax.set_ylim(0, 4.6) + + def node(x, y, w, h, t, fc): + ax.add_patch(Rectangle((x, y), w, h, facecolor=fc, edgecolor="none")) + ax.text(x + w / 2, y + h / 2, t, ha="center", va="center", + color=BG, fontsize=7.5, fontweight="bold") + + ops = [("H2D", AMBER), ("kernel", ACCENT), ("D2H", PALE)] * 2 + + ax.text(0, 4.15, "WITHOUT GRAPHS · one driver call per operation, every frame", + color=AMBER, fontsize=8.5, fontweight="bold") + for i, (t, c) in enumerate(ops): + x = 0.1 + i * 1.62 + node(x, 2.85, 1.4, 0.6, t, c) + ax.add_patch(FancyArrowPatch((x + 0.7, 3.72), (x + 0.7, 3.52), + arrowstyle="-|>", mutation_scale=7, + color=MUTED, lw=0.9)) + ax.text(12.5, 3.15, "CPU cost\n≈ 6 launches", ha="right", va="center", + color=MUTED, fontsize=7.5) + + ax.text(0, 2.18, "WITH GRAPHS · opt5 · record once, replay with one launch", + color=ACCENT, fontsize=8.5, fontweight="bold") + ax.add_patch(Rectangle((0.1, 0.72), 9.85, 1.15, facecolor=PANEL, + edgecolor=ACCENT, lw=1.2)) + for i, (t, c) in enumerate(ops): + node(0.38 + i * 1.58, 0.98, 1.34, 0.6, t, c) + ax.add_patch(FancyArrowPatch((0.8, 2.02), (0.8, 1.90), arrowstyle="-|>", + mutation_scale=8, color=ACCENT, lw=1.3)) + ax.text(12.5, 1.30, "CPU cost\n≈ 1 launch", ha="right", va="center", + color=ACCENT, fontsize=7.5, fontweight="bold") + ax.text(0.1, 0.32, "cudaGraphLaunch() — the whole DAG is submitted as one unit; " + "the driver already knows every dependency", + color=MUTED, fontsize=7) + save(fig, "fig_graphs") + + +# ------------------------------------------------ 6. f32 kernel (nsys truth) +def fig_f32_kernel(): + fig, (ax, ax2) = plt.subplots(1, 2, figsize=(7.7, 2.6), + gridspec_kw={"width_ratios": [1, 1.35]}) + v = [43.0, 25.6] + ax.bar([0, 1], v, width=0.5, color=[AMBER, ACCENT], zorder=3) + for i, val in enumerate(v): + ax.text(i, val + 1.2, f"{val} µs", ha="center", color=PALE, + fontsize=11, fontweight="bold") + ax.annotate("", xy=(1, 27.5), xytext=(0, 44.5), + arrowprops=dict(arrowstyle="-|>", color=MUTED, lw=1.2, + connectionstyle="arc3,rad=-0.25")) + ax.text(0.5, 37, "−40%", ha="center", color=PALE, fontsize=10, + fontweight="bold") + ax.set_xticks([0, 1]); ax.set_xticklabels(["f64 pedestal", "f32 pedestal"], + color=TEXT2, fontsize=8.5) + ax.set_ylim(0, 52); ax.set_yticks([]); bare(ax, keep=("bottom",)) + ax.set_title("kernel, exclusive (nsys, 9×9)", color=MUTED, fontsize=8, pad=8) + + labels = ["kernel", "D2H", "H2D"] + f64 = [43.0, 19.4, 13.2] + f32 = [25.6, 19.8, 13.5] + y = np.arange(3); h = 0.35 + ax2.barh(y + h / 2, f64, height=h, color=AMBER, zorder=3, label="f64 ped") + ax2.barh(y - h / 2, f32, height=h, color=ACCENT, zorder=3, label="f32 ped") + for yi, (a, b) in enumerate(zip(f64, f32)): + ax2.text(a + 1, yi + h / 2, f"{a:.1f}", va="center", color=TEXT2, fontsize=8) + ax2.text(b + 1, yi - h / 2, f"{b:.1f}", va="center", color=TEXT2, fontsize=8) + ax2.set_yticks(y); ax2.set_yticklabels(labels, color=TEXT2, fontsize=8.5) + ax2.invert_yaxis(); ax2.set_xlim(0, 56); ax2.set_xticks([]) + bare(ax2, keep=("left",)) + ax2.legend(frameon=False, fontsize=8, labelcolor=TEXT2, loc="lower right") + ax2.set_title("per-frame GPU operations (µs)", color=MUTED, fontsize=8, pad=8) + save(fig, "fig_f32_kernel") + + +# ------------------------------------------------------- 7. cancellation +def fig_cancellation(): + fig, (ax, ax2) = plt.subplots(1, 2, figsize=(7.7, 2.7), + gridspec_kw={"width_ratios": [1.25, 1]}) + names = ["E[X²]\n2.17e7", "mean²\n2.17e7", "variance\n2025"] + vals = [2.17e7, 2.17e7, 2025] + ax.bar([0, 1], vals[:2], width=0.5, color=[PALE, PALE], zorder=3) + ax.bar([2], [2025], width=0.5, color=AMBER, zorder=3) + ax.set_yscale("log"); ax.set_ylim(1e2, 2e8) + ax.set_xticks([0, 1, 2]); ax.set_xticklabels(names, color=TEXT2, fontsize=8) + ax.set_yticks([1e3, 1e5, 1e7]) + ax.axhline(2048, color=ACCENT, lw=1.3, ls="--", zorder=4) + ax.text(2.42, 3000, "f32 rounding step\nat 2.17e7 = 2048", color=ACCENT, + fontsize=7.5, ha="right", va="bottom") + bare(ax) + ax.set_title("var = E[X²] − mean² (f32, mean ≈ 4655 ADU)", + color=MUTED, fontsize=8, pad=8) + + rms = np.linspace(0, 12, 200) + ax2.fill_between(rms, 0, np.where(rms < 6.5, 1, 0), color=AMBER, alpha=0.16, + step="pre") + ax2.plot(rms, rms**2, color=PALE, lw=1.8, label="true variance") + ax2.axhline(42, color=ACCENT, lw=1.4, ls="--", label="f32 error floor") + ax2.set_xlabel("pixel rms (ADU)", color=TEXT2, fontsize=8.5) + ax2.set_ylabel("variance", color=TEXT2, fontsize=8.5) + ax2.set_ylim(0, 150); ax2.set_xlim(0, 12) + ax2.set_yticks([]); ax2.tick_params(labelsize=8) + bare(ax2) + ax2.text(1.0, 108, "quiet pixels:\nerror > variance\n→ rms clamped to 0\n→ fires every frame", + color=AMBER, fontsize=7.5, va="top") + ax2.legend(frameon=False, fontsize=7.5, labelcolor=TEXT2, loc="lower right") + save(fig, "fig_cancellation") + + +# ------------------------------------------------- 8. where f32 pays or not +def fig_bottleneck(): + fig, (a1, a2) = plt.subplots(1, 2, figsize=(11.4, 3.0)) + + for ax, title, kern, floor, gain in [ + (a1, "3×3 clusters — pipeline-bound", (24, 13), 25, + "kernel already hidden → 0% end-to-end"), + (a2, "9×9 clusters — kernel-bound (f64)", (43, 26), 32, + "kernel on the critical path → −8% wall"), + ]: + x = [0, 1] + ax.bar(x, kern, width=0.5, color=[AMBER, ACCENT], zorder=3) + ax.axhline(floor, color=PALE, lw=1.4, ls="--", zorder=4) + ax.text(1.62, floor + 1.2, "transfer + host floor", color=PALE, + fontsize=8, ha="right") + for i, v in enumerate(kern): + ax.text(i, v + 1.2, f"{v} µs", ha="center", color=PALE, + fontsize=10, fontweight="bold") + ax.set_xticks(x); ax.set_xticklabels(["f64 pedestal", "f32 pedestal"], + color=TEXT2, fontsize=9) + ax.set_xlim(-0.6, 1.7); ax.set_ylim(0, 55); ax.set_yticks([]) + bare(ax, keep=("bottom",)) + ax.set_title(title, color=PALE, fontsize=9.5, pad=10) + ax.text(-0.55, -9, gain, color=AMBER if "0%" in gain else ACCENT, + fontsize=8.5, fontweight="bold") + fig.subplots_adjust(bottom=0.22) + save(fig, "fig_bottleneck") + + +# ----------------------------------------------------------- 9. correctness +def fig_correctness(): + fig, ax = plt.subplots(figsize=(7.4, 2.4)) + names = ["CPU MT", "opt1", "opt2", "opt3", "opt4", "opt5", "opt6 (f32)"] + diff = [0.0, 0.0040, 0.0035, 0.0035, 0.0035, 0.0035, 0.0039] + colors = [MUTED] + [ACCENT] * 5 + [AMBER] + x = np.arange(len(names)) + ax.bar(x, diff, width=0.55, color=colors, zorder=3) + for xi, d in enumerate(diff): + ax.text(xi, d + 0.00022, ("reference" if d == 0 else f"{d:.4f}%"), + ha="center", color=PALE if d else MUTED, fontsize=8.5) + ax.axhline(0.01, color=PALE, lw=1.2, ls="--") + ax.text(6.4, 0.0104, "0.01% — well inside statistical noise", color=PALE, + fontsize=8, ha="right") + ax.set_xticks(x); ax.set_xticklabels(names, color=TEXT2, fontsize=8.5) + ax.set_ylim(0, 0.0125); ax.set_yticks([]) + bare(ax, keep=("bottom",)) + ax.set_title("cluster-count difference vs CPU · 233 million clusters, 3×3", + color=MUTED, fontsize=8.5, pad=8) + save(fig, "fig_correctness") + + +for f in (fig_arc, fig_overhead, fig_streams, fig_pinning, fig_graphs, + fig_f32_kernel, fig_cancellation, fig_bottleneck, fig_correctness): + f() +print("done ->", OUT) diff --git a/docs/figures/fig_arc.png b/docs/figures/fig_arc.png new file mode 100644 index 00000000..abb72fca Binary files /dev/null and b/docs/figures/fig_arc.png differ diff --git a/docs/figures/fig_bottleneck.png b/docs/figures/fig_bottleneck.png new file mode 100644 index 00000000..75019cb5 Binary files /dev/null and b/docs/figures/fig_bottleneck.png differ diff --git a/docs/figures/fig_cancellation.png b/docs/figures/fig_cancellation.png new file mode 100644 index 00000000..56aad03e Binary files /dev/null and b/docs/figures/fig_cancellation.png differ diff --git a/docs/figures/fig_correctness.png b/docs/figures/fig_correctness.png new file mode 100644 index 00000000..e32be63c Binary files /dev/null and b/docs/figures/fig_correctness.png differ diff --git a/docs/figures/fig_f32_kernel.png b/docs/figures/fig_f32_kernel.png new file mode 100644 index 00000000..bd17ff3f Binary files /dev/null and b/docs/figures/fig_f32_kernel.png differ diff --git a/docs/figures/fig_graphs.png b/docs/figures/fig_graphs.png new file mode 100644 index 00000000..fc89380f Binary files /dev/null and b/docs/figures/fig_graphs.png differ diff --git a/docs/figures/fig_overhead.png b/docs/figures/fig_overhead.png new file mode 100644 index 00000000..c9eabeb0 Binary files /dev/null and b/docs/figures/fig_overhead.png differ diff --git a/docs/figures/fig_pinning.png b/docs/figures/fig_pinning.png new file mode 100644 index 00000000..2b7242c6 Binary files /dev/null and b/docs/figures/fig_pinning.png differ diff --git a/docs/figures/fig_streams.png b/docs/figures/fig_streams.png new file mode 100644 index 00000000..c00b1b77 Binary files /dev/null and b/docs/figures/fig_streams.png differ diff --git a/include/aare/ClusterFinderCUDAOpt2.hpp b/include/aare/ClusterFinderCUDAOpt2.hpp new file mode 100644 index 00000000..ee77689e --- /dev/null +++ b/include/aare/ClusterFinderCUDAOpt2.hpp @@ -0,0 +1,435 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once +#include "aare/ClusterFinder.hpp" +#include "aare/clusterfinder_kernel_opt2.cuh" +#include "aare/utils/cuda_check.cuh" +#include +#include +#include +#include +#include +#include +#include + +namespace aare { + +// Per-stream device resources +template +struct StreamContextOpt2 { + cudaStream_t stream = nullptr; // handle to the stream + FRAME_TYPE *d_frame = nullptr; + PEDESTAL_TYPE *d_pd_mean = nullptr; + PEDESTAL_TYPE *d_pd_sum = nullptr; + PEDESTAL_TYPE *d_pd_sum2 = nullptr; + ClusterType *d_clusters = nullptr; + uint32_t *d_cluster_count = nullptr; + + // Pinned host staging buffers. These make cudaMemcpyAsync real async DMA + // transfers even when the caller's NDView points to pageable memory. + FRAME_TYPE *h_frame = nullptr; + uint32_t *h_cluster_count = nullptr; + ClusterType *h_clusters = nullptr; + + cudaEvent_t kernel_start = nullptr; + cudaEvent_t kernel_stop = nullptr; +}; + +template , + typename FRAME_TYPE = uint16_t, typename PEDESTAL_TYPE = double, + typename = std::enable_if_t::value>> +class ClusterFinderCUDAOpt2 { + using COMPUTE_TYPE = + device_opt2::COMPUTE_TYPE; // match the kernel's internal precision + + static constexpr int BLOCK_X = 16; + static constexpr int BLOCK_Y = 16; + static constexpr int col_radius = ClusterType::cluster_size_x / 2; + static constexpr int row_radius = ClusterType::cluster_size_y / 2; + + Shape<2> m_shape; + size_t nrows; + size_t ncols; + size_t m_image_size; // nrows * ncols + int n_streams; + size_t m_capacity; + + size_t m_image_bytes; + size_t m_cluster_bytes; + + COMPUTE_TYPE m_nSigma; + Pedestal m_pedestal; + ClusterVector m_clusters; + bool m_pedestal_dirty = true; + + using SC = StreamContextOpt2; + std::vector v_sc; + + float m_total_kernel_ms = 0.0f; + size_t m_frames_processed = 0; + + // Kernel parameters + dim3 grid; + dim3 block; + size_t shmem_bytes; + + public: + /** + * @brief Construct a ClusterFinderCUDAOpt2 + * + * @param m_image_size shape of the detector frame (rows, cols) + * @param nSigma threshold in units of per-pixel pedestal + * std + * @param capacity device-side cluster buffer size per stream + * @param n_streams number of CUDA streams for multi-frame + * overlap + */ + ClusterFinderCUDAOpt2(Shape<2> shape_, COMPUTE_TYPE nSigma = 5.0, + size_t capacity = 1000000, int n_streams_ = 1) + : m_shape(shape_), nrows(shape_[0]), ncols(shape_[1]), + m_image_size(nrows * ncols), n_streams(n_streams_), + m_capacity(capacity), m_nSigma(nSigma), + m_pedestal(shape_[0], shape_[1]), m_clusters(capacity) { + if (n_streams_ <= 0) { + throw std::invalid_argument( + "ClusterFinderCUDAOpt2: n_streams must be > 0"); + } + + if (capacity > + static_cast(std::numeric_limits::max())) { + throw std::invalid_argument( + "ClusterFinderCUDAOpt2: capacity must fit in uint32_t"); + } + + if (capacity == 0) { + throw std::invalid_argument( + "ClusterFinderCUDAOpt2: capacity must be > 0"); + } + + // Grid/Block dimensions + block = dim3(BLOCK_X, BLOCK_Y); + grid = dim3((static_cast(ncols) + BLOCK_X - 1) / BLOCK_X, + (static_cast(nrows) + BLOCK_Y - 1) / BLOCK_Y); + + // Shared memory: one tile of (BLOCK_X + 2*col_radius) x (BLOCK_Y + + // 2*row_radius) elements + // Mixed precision used -> shmem takes COMPUTE_TYPE = floats (not + // PEDESTAL_TYPE) + shmem_bytes = (BLOCK_X + 2 * col_radius) * (BLOCK_Y + 2 * row_radius) * + sizeof(COMPUTE_TYPE); + + m_image_bytes = m_image_size * sizeof(FRAME_TYPE); + m_cluster_bytes = m_capacity * sizeof(ClusterType); + + v_sc.resize(n_streams); + for (int k = 0; k < n_streams; ++k) { + auto &sc = v_sc[k]; + CUDA_CHECK( + cudaStreamCreateWithFlags(&sc.stream, cudaStreamNonBlocking)); + CUDA_CHECK(cudaEventCreate(&sc.kernel_start)); + CUDA_CHECK(cudaEventCreate(&sc.kernel_stop)); + CUDA_CHECK(cudaMalloc(&sc.d_frame, m_image_bytes)); + CUDA_CHECK(cudaMalloc(&sc.d_pd_mean, + m_image_size * sizeof(PEDESTAL_TYPE))); + CUDA_CHECK( + cudaMalloc(&sc.d_pd_sum, m_image_size * sizeof(PEDESTAL_TYPE))); + CUDA_CHECK(cudaMalloc(&sc.d_pd_sum2, + m_image_size * sizeof(PEDESTAL_TYPE))); + CUDA_CHECK(cudaMalloc(&sc.d_clusters, m_cluster_bytes)); + CUDA_CHECK(cudaMalloc(&sc.d_cluster_count, sizeof(uint32_t))); + + CUDA_CHECK(cudaMallocHost(reinterpret_cast(&sc.h_frame), + m_image_bytes)); + CUDA_CHECK( + cudaMallocHost(reinterpret_cast(&sc.h_cluster_count), + sizeof(uint32_t))); + if (m_cluster_bytes > 0) { + CUDA_CHECK( + cudaMallocHost(reinterpret_cast(&sc.h_clusters), + m_cluster_bytes)); + } + } + } + + ~ClusterFinderCUDAOpt2() { + for (auto &sc : v_sc) { + if (sc.stream) + cudaStreamSynchronize(sc.stream); + + if (sc.d_frame) + cudaFree(sc.d_frame); + if (sc.d_pd_mean) + cudaFree(sc.d_pd_mean); + if (sc.d_pd_sum) + cudaFree(sc.d_pd_sum); + if (sc.d_pd_sum2) + cudaFree(sc.d_pd_sum2); + if (sc.d_clusters) + cudaFree(sc.d_clusters); + if (sc.d_cluster_count) + cudaFree(sc.d_cluster_count); + + if (sc.h_frame) + cudaFreeHost(sc.h_frame); + if (sc.h_clusters) + cudaFreeHost(sc.h_clusters); + if (sc.h_cluster_count) + cudaFreeHost(sc.h_cluster_count); + + if (sc.kernel_start) + cudaEventDestroy(sc.kernel_start); + if (sc.kernel_stop) + cudaEventDestroy(sc.kernel_stop); + if (sc.stream) + cudaStreamDestroy(sc.stream); + } + } + + // Non-copyable, non-movable + ClusterFinderCUDAOpt2(const ClusterFinderCUDAOpt2 &) = delete; + ClusterFinderCUDAOpt2 &operator=(const ClusterFinderCUDAOpt2 &) = delete; + ClusterFinderCUDAOpt2(ClusterFinderCUDAOpt2 &&) = delete; + ClusterFinderCUDAOpt2 &operator=(ClusterFinderCUDAOpt2 &&) = delete; + + void set_nSigma(COMPUTE_TYPE nSigma) { m_nSigma = nSigma; } + COMPUTE_TYPE get_nSigma() const { return m_nSigma; } + + void push_pedestal_frame(NDView frame) { + m_pedestal.push(frame); + m_pedestal_dirty = true; + } + + void clear_pedestal() { + m_pedestal.clear(); + m_pedestal_dirty = true; + } + + NDArray pedestal() { return m_pedestal.mean(); } + NDArray noise() { return m_pedestal.std(); } + + /** + * @brief Move clusters out of the internal ClusterVector, optionally + * reallocating the internal one with the same capacity. + */ + ClusterVector + steal_clusters(bool realloc_same_capacity = false) { + ClusterVector tmp = std::move(m_clusters); + if (realloc_same_capacity) + m_clusters = ClusterVector(tmp.capacity()); + else + m_clusters = ClusterVector{}; + return tmp; + } + + /** + * @brief Find clusters in a single frame, appending them to the internal + * ClusterVector. + */ + void find_clusters(NDView frame, uint64_t frame_number = 0) { + if (m_pedestal_dirty) { // need to update the pedestal on the gpu + sync_pedestal_to_device(); + m_pedestal_dirty = false; + } + + auto &sc = v_sc[0]; + const uint32_t n_pd_samples = + static_cast(m_pedestal.n_samples()); + + // First, CPU copies frame into a reusable pinned buffer + std::memcpy(sc.h_frame, frame.data(), m_image_bytes); + + // Reset cluster counter + CUDA_CHECK(cudaMemsetAsync(sc.d_cluster_count, 0, sizeof(uint32_t), + sc.stream)); + + // Upload frame + CUDA_CHECK(cudaMemcpyAsync(sc.d_frame, sc.h_frame, m_image_bytes, + cudaMemcpyHostToDevice, sc.stream)); + + // Timed Kernel launch + CUDA_CHECK(cudaEventRecord(sc.kernel_start, sc.stream)); + device_opt2::find_clusters_in_single_frame + <<>>( + sc.d_frame, sc.d_pd_mean, sc.d_pd_sum, sc.d_pd_sum2, + n_pd_samples, m_nSigma, nrows, ncols, sc.d_clusters, + sc.d_cluster_count, static_cast(m_capacity)); + CUDA_CHECK(cudaEventRecord(sc.kernel_stop, sc.stream)); + CUDA_CHECK(cudaGetLastError()); + + // Read back cluster count into pinned buffer + CUDA_CHECK(cudaMemcpyAsync(sc.h_cluster_count, sc.d_cluster_count, + sizeof(uint32_t), cudaMemcpyDeviceToHost, + sc.stream)); + + // Synchronize to ensure count is available before the CPU reads + // clusters + CUDA_CHECK(cudaStreamSynchronize(sc.stream)); + + record_kernel_time(sc); + + // Clamp to max in case of overflow + uint32_t n_found = *sc.h_cluster_count; + n_found = std::min(n_found, static_cast(m_capacity)); + + // Read back clusters + m_clusters.set_frame_number(frame_number); + if (n_found > 0) { + append_device_clusters_to(m_clusters, sc, n_found); + } + } + + /** + * @brief Batched cluster finding across multiple frames, using n_streams + * CUDA streams to overlap H2D transfer, kernel, and D2H transfer. + * + * Returns one ClusterVector per input frame (with frame_number set to + * first_frame + i). + */ + std::vector> + find_clusters_batched(NDView frames, + uint64_t first_frame = 0) { + if (m_pedestal_dirty) { + sync_pedestal_to_device(); + m_pedestal_dirty = false; + } + + const size_t n_frames = frames.shape(0); + const uint32_t n_pd_samples = + static_cast(m_pedestal.n_samples()); + + std::vector> results; + results.reserve(n_frames); + for (size_t i = 0; i < n_frames; ++i) { + results.emplace_back(); + results.back().set_frame_number(first_frame + i); + } + + const size_t n_rounds = (n_frames + n_streams - 1) / n_streams; + for (size_t round = 0; round < n_rounds; ++round) { + + // Launch phase: fan out kernels on all streams for this round + for (int k = 0; k < n_streams; ++k) { + // OOB guard + const size_t frame_idx = round * n_streams + k; + if (frame_idx >= n_frames) + continue; + + auto &sc_k = v_sc[k]; + const FRAME_TYPE *h_src = + frames.data() + frame_idx * m_image_size; + + std::memcpy(sc_k.h_frame, h_src, m_image_bytes); + + CUDA_CHECK(cudaMemsetAsync(sc_k.d_cluster_count, 0, + sizeof(uint32_t), sc_k.stream)); + CUDA_CHECK( + cudaMemcpyAsync(sc_k.d_frame, sc_k.h_frame, m_image_bytes, + cudaMemcpyHostToDevice, sc_k.stream)); + + CUDA_CHECK(cudaEventRecord(sc_k.kernel_start, sc_k.stream)); + device_opt2::find_clusters_in_single_frame< + ClusterType, FRAME_TYPE, PEDESTAL_TYPE> + <<>>( + sc_k.d_frame, sc_k.d_pd_mean, sc_k.d_pd_sum, + sc_k.d_pd_sum2, n_pd_samples, m_nSigma, nrows, ncols, + sc_k.d_clusters, sc_k.d_cluster_count, + static_cast(m_capacity)); + CUDA_CHECK(cudaEventRecord(sc_k.kernel_stop, sc_k.stream)); + CUDA_CHECK(cudaGetLastError()); + + // Queue count D2H immediately after the kernel + CUDA_CHECK(cudaMemcpyAsync( + sc_k.h_cluster_count, sc_k.d_cluster_count, + sizeof(uint32_t), cudaMemcpyDeviceToHost, sc_k.stream)); + } + + // Drain phase: fan in results from all streams + for (int k = 0; k < n_streams; ++k) { + const size_t frame_idx = round * n_streams + k; + if (frame_idx >= n_frames) + continue; + + auto &sc_k = v_sc[k]; + + // Wait for memset -> H2D -> kernel -> count D2H + CUDA_CHECK(cudaStreamSynchronize(sc_k.stream)); + + record_kernel_time(sc_k); + + uint32_t n_found = *sc_k.h_cluster_count; + n_found = std::min(n_found, + static_cast(m_capacity)); + + if (n_found > 0) { + append_device_clusters_to(results[frame_idx], sc_k, + n_found); + } + } + } + + return results; + } + + float avg_kernel_time_ms() const { + return m_frames_processed > 0 ? m_total_kernel_ms / m_frames_processed + : 0.0f; + } + + void reset_timers() { + m_total_kernel_ms = 0.0f; + m_frames_processed = 0; + } + + private: + /** + * Upload the current host pedestal (mean, sum, sum2) to every stream's + * device buffers. Called lazily before a find_clusters call when the + * host pedestal has been updated. + */ + void sync_pedestal_to_device() { + // These return-by-value NDArrays must stay alive until the async + // copies complete, so we synchronise at the end before they go out + // of scope. + NDArray h_mean = m_pedestal.mean(); + NDArray h_sum = m_pedestal.get_sum(); + NDArray h_sum2 = m_pedestal.get_sum2(); + + const size_t bytes = m_image_size * sizeof(PEDESTAL_TYPE); + for (auto &sc : v_sc) { + CUDA_CHECK(cudaMemcpyAsync(sc.d_pd_mean, h_mean.data(), bytes, + cudaMemcpyHostToDevice, sc.stream)); + CUDA_CHECK(cudaMemcpyAsync(sc.d_pd_sum, h_sum.data(), bytes, + cudaMemcpyHostToDevice, sc.stream)); + CUDA_CHECK(cudaMemcpyAsync(sc.d_pd_sum2, h_sum2.data(), bytes, + cudaMemcpyHostToDevice, sc.stream)); + } + for (auto &sc : v_sc) + CUDA_CHECK(cudaStreamSynchronize(sc.stream)); + } + + /** + * Copy n_found clusters from sc.d_clusters into the given ClusterVector + * and block on the transfer. + */ + void append_device_clusters_to(ClusterVector &cv, SC &sc, + uint32_t n_found) { + + CUDA_CHECK(cudaMemcpyAsync(sc.h_clusters, sc.d_clusters, + n_found * sizeof(ClusterType), + cudaMemcpyDeviceToHost, sc.stream)); + + CUDA_CHECK(cudaStreamSynchronize(sc.stream)); + + for (uint32_t i = 0; i < n_found; ++i) + cv.push_back(sc.h_clusters[i]); + } + + void record_kernel_time(SC &sc) { + float ms = 0.0f; + CUDA_CHECK(cudaEventElapsedTime(&ms, sc.kernel_start, sc.kernel_stop)); + m_total_kernel_ms += ms; + m_frames_processed++; + } +}; + +} // namespace aare diff --git a/include/aare/clusterfinder_kernel_opt2.cuh b/include/aare/clusterfinder_kernel_opt2.cuh new file mode 100644 index 00000000..4eb4eb12 --- /dev/null +++ b/include/aare/clusterfinder_kernel_opt2.cuh @@ -0,0 +1,346 @@ +#pragma once +#include "aare/Cluster.hpp" +#include "aare/ClusterFinder.hpp" +#include +#include + +// OPT2 SNAPSHOT (commit 88e0e8d) — the pre-refactor kernel, kept for +// benchmarking the opt2 pipeline. Lives in namespace aare::device_opt2 so it +// coexists with the modern kernel. opt2 pipeline behaviour: f32 stencil / +// f64 pedestal, raw E[X^2]-E[X]^2 variance. The Test3 local-max gate has been +// backported from the current kernel so opt1/opt2 cluster counts match the CPU +// and current finders (holding correctness constant across the opt arc). +namespace aare::device_opt2 { + +// Implementing mixed precision for shared memory and stencil arithmetic +using COMPUTE_TYPE = float; + +template , + typename FRAME_TYPE = uint16_t, typename PEDESTAL_TYPE = double, + typename = std::enable_if_t::value>> +__global__ void find_clusters_in_single_frame( + const FRAME_TYPE *__restrict__ d_frame, + PEDESTAL_TYPE *__restrict__ d_pd_mean, PEDESTAL_TYPE *__restrict__ d_pd_sum, + PEDESTAL_TYPE *__restrict__ d_pd_sum2, const uint32_t n_pd_samples, + const COMPUTE_TYPE m_nSigma, const size_t nrows, const size_t ncols, + // const uint64_t frame_number, + ClusterType *d_clusters, uint32_t *d_cluster_count, + const uint32_t max_clusters) { + using CT = typename ClusterType::value_type; + + // Compile-time cluster geometry useful for unrolling loops + constexpr uint8_t CSX = ClusterType::cluster_size_x; + constexpr uint8_t CSY = ClusterType::cluster_size_y; + constexpr int col_radius = CSX / 2; + constexpr int row_radius = CSY / 2; + + // Squared threshold constants; avoids sqrt at runtime + // c2^2 is for the 2x2 quadrant test + // c3^2 is for the full-cluster total test + constexpr int pow2_c2 = ((CSY + 1) / 2) * ((CSX + 1) / 2); + constexpr int pow2_c3 = CSX * CSY; + + // Thread/pixel mapping + auto col_global = + static_cast(threadIdx.x + blockDim.x * blockIdx.x); + auto row_global = + static_cast(threadIdx.y + blockDim.y * blockIdx.y); + auto global_tid = static_cast(col_global + ncols * row_global); + auto local_tid = threadIdx.x + blockDim.x * threadIdx.y; + + // ==================== + // Shared memory layout + // ==================== + // The tile is laid out contiguously in a 1D configuration: + // [0 ... tile_size-1] = pedestal-subtracted frame values + // + // Each tile has (blockDim.x + 2*col_radius) x (blockDim.y + 2*row_radius) + // elements, with a halo of col_radius/row_radius pixels on each side. + + // CUDA prefers raw bytes + aligned cast + // Compile error happens when using: `extern __shared__ T sh[];` + extern __shared__ __align__(sizeof(COMPUTE_TYPE)) unsigned char smem[]; + COMPUTE_TYPE *shmem = reinterpret_cast(smem); + + // Stride includes halo on both sides + auto shmem_stride = static_cast(blockDim.x) + 2 * col_radius; + auto tile_size = + shmem_stride * (static_cast(blockDim.y) + 2 * row_radius); + + // Offset so that thread (0,0) maps to shared-memory position + // (row_radius, col_radius) i.e. past the top-left halo. + auto shmem_tid = + (static_cast(threadIdx.y) + row_radius) * shmem_stride + + (static_cast(threadIdx.x) + col_radius); + + // Cooperative zero-fill + for (int idx = static_cast(local_tid); idx < tile_size; + idx += static_cast(blockDim.x * blockDim.y)) { + shmem[idx] = COMPUTE_TYPE{0}; + } + __syncthreads(); + + // OOB flag + bool valid_pixel = col_global < static_cast(ncols) && + row_global < static_cast(nrows); + + // ====================================================== + // Load pedestal-subtracted frame data into shared memory (MIXED PRECISION) + // ====================================================== + + // Helper: read (frame - pedestal_mean) from global memory, or 0 if OOB. + // gr, gc are the global row/col of the pixel to load. + // Returns the pedestal-subtracted value. + auto load_pixel = [&] __device__(ssize_t gr, ssize_t gc) -> COMPUTE_TYPE { + auto gid = gc + ncols * gr; + return static_cast(d_frame[gid]) - + static_cast(d_pd_mean[gid]); + }; + + // A. Interior: every valid thread loads its own pixel + if (valid_pixel) { + shmem[shmem_tid] = load_pixel(row_global, col_global); + } + + // B. Halo regions (Boundaries) + // B.1 Top rows of the halo + if (threadIdx.y == 0 && valid_pixel) { + for (int i = 1; i <= row_radius; ++i) { + if (row_global - i >= 0) + shmem[shmem_tid - i * shmem_stride] = + load_pixel(row_global - i, col_global); + } + // Top-left corner rectangle + if (threadIdx.x == 0) { + for (int i = 1; i <= row_radius; ++i) + for (int j = 1; j <= col_radius; ++j) + if (row_global - i >= 0 && col_global - j >= 0) + shmem[shmem_tid - i * shmem_stride - j] = + load_pixel(row_global - i, col_global - j); + } + // Top-right corner rectangle + if (threadIdx.x == blockDim.x - 1) { + for (int i = 1; i <= row_radius; ++i) + for (int j = 1; j <= col_radius; ++j) + if (row_global - i >= 0 && + col_global + j < static_cast(ncols)) + shmem[shmem_tid - i * shmem_stride + j] = + load_pixel(row_global - i, col_global + j); + } + } + + // B.2 Left column of the halo + if (threadIdx.x == 0 && valid_pixel) { + for (int j = 1; j <= col_radius; ++j) + if (col_global - j >= 0) + shmem[shmem_tid - j] = load_pixel(row_global, col_global - j); + } + + // B.3 Right column of the halo + if (threadIdx.x == blockDim.x - 1 && valid_pixel) { + for (int j = 1; j <= col_radius; ++j) + if (col_global + j < static_cast(ncols)) + shmem[shmem_tid + j] = load_pixel(row_global, col_global + j); + } + + // B.4 Bottom rows of the halo + if (threadIdx.y == blockDim.y - 1 && valid_pixel) { + for (int i = 1; i <= row_radius; ++i) { + if (row_global + i < static_cast(nrows)) + shmem[shmem_tid + i * shmem_stride] = + load_pixel(row_global + i, col_global); + } + // Bottom-left corner rectangle + if (threadIdx.x == 0) { + for (int i = 1; i <= row_radius; ++i) + for (int j = 1; j <= col_radius; ++j) + if (row_global + i < static_cast(nrows) && + col_global - j >= 0) + shmem[shmem_tid + i * shmem_stride - j] = + load_pixel(row_global + i, col_global - j); + } + // Bottom-right corner rectangle + if (threadIdx.x == blockDim.x - 1) { + for (int i = 1; i <= row_radius; ++i) + for (int j = 1; j <= col_radius; ++j) + if (row_global + i < static_cast(nrows) && + col_global + j < static_cast(ncols)) + shmem[shmem_tid + i * shmem_stride + j] = + load_pixel(row_global + i, col_global + j); + } + } + + __syncthreads(); + + // ===================== + // Cluster-finding logic + // ===================== + if (!valid_pixel) + return; + + // Per-pixel variance from global pedestal arrays + // Variance = rms^2 = E[X^2] - E[X]^2 + // NOTE: Keep thresholds squared to avoid one sqrtf() per pixel. + PEDESTAL_TYPE mean_px = d_pd_mean[global_tid]; + PEDESTAL_TYPE var_px = + d_pd_sum2[global_tid] / n_pd_samples - mean_px * mean_px; + PEDESTAL_TYPE rms_sq = max(var_px, PEDESTAL_TYPE{0}); // variance = rms^2 + PEDESTAL_TYPE nSig_sq_rms_sq = static_cast(m_nSigma) * + static_cast(m_nSigma) * + rms_sq; + + // Pedestal-subtracted value of the center pixel (already in shmem) + COMPUTE_TYPE val_pixel = shmem[shmem_tid]; + + // Negative pedestal early exit: + // val_pixel < -nSigma * rms + // is equivalent to: + // val_pixel < 0 && val_pixel^2 > nSigma^2 * rms^2 + if (val_pixel < COMPUTE_TYPE{0} && + static_cast(val_pixel) * + static_cast(val_pixel) > + nSig_sq_rms_sq) + return; // NOTE: pedestal update for this pixel is skipped (same as + // sequential) + + // Stencil reduction: total, max, quadrant sums + COMPUTE_TYPE total = 0.0f; + COMPUTE_TYPE max_val = -HUGE_VALF; + + // // Quandrants + // PEDESTAL_TYPE tl = PEDESTAL_TYPE{0}; // top-left quadrant (ir<=0, + // ic<=0) PEDESTAL_TYPE tr = PEDESTAL_TYPE{0}; // top-right quadrant + // (ir<=0, ic>=0) PEDESTAL_TYPE bl = PEDESTAL_TYPE{0}; // bottom-left + // (ir>=0, ic<=0) PEDESTAL_TYPE br = PEDESTAL_TYPE{0}; // bottom-right + // (ir>=0, ic>=0) + +#pragma unroll + for (int ir = -row_radius; ir <= row_radius; ++ir) { +#pragma unroll + for (int ic = -col_radius; ic <= col_radius; ++ic) { + COMPUTE_TYPE val = shmem[shmem_tid + ir * shmem_stride + ic]; + + total += val; + max_val = fmaxf(max_val, val); + + // // Quadrant accumulation (pixels on the axes contribute to two + // quadrants) if (ir <= 0 && ic <= 0) tl += val; if (ir <= 0 && ic + // >= 0) tr += val; if (ir >= 0 && ic <= 0) bl += val; if (ir >= 0 + // && ic >= 0) br += val; + } + } + + // Three-way classification (mirrors ClusterFinder's logic) + // + // 1. Single-pixel significance: max_val > nSigma * rms + // -> only the pixel that IS the max gets recorded (local-max + // suppression) + // + // 2. Quadrant significance: max(tl,tr,bl,br) > c2 * nSigma * rms + // -> charge-sharing events where a 2x2 sub-region is significant + // NOTE: This test is absent in the serial ClusterFinder! + // + // 3. Total significance: total > c3 * nSigma * rms + // -> distributed events where the full cluster sum is significant + + bool is_photon = false; + + // Test 1: single-pixel significance + // max_val > nSigma * rms + // is equivalent to: + // max_val > 0 && max_val^2 > nSigma^2 * rms^2 + if (max_val > COMPUTE_TYPE{0} && + static_cast(max_val) * + static_cast(max_val) > + nSig_sq_rms_sq) { + // Local-max suppression: only the center-pixel thread records the + // cluster + if (val_pixel < max_val) + return; // some other pixel in the neighborhood is brighter + is_photon = true; + } + + /* // Test 2: quadrant significance (only if test 1 didn't fire) + if (!is_photon) { + PEDESTAL_TYPE max_quad = max(max(tl, tr), max(bl, br)); + if (max_quad > 0 && max_quad * max_quad > pow2_c2 * nSig_sq_rms_sq) { + is_photon = true; + } + } */ + + // Test 3: total significance (only if tests 1 & 2 didn't fire) + if (!is_photon) { + if (total > 0 && static_cast(total) * + static_cast(total) > + pow2_c3 * nSig_sq_rms_sq) { + // Local-max suppression: only the center-pixel thread records the + // cluster, so an extended charge-shared event yields one cluster, + // not one per pixel. (Backported from the current kernel to make + // the opt2 counts match; the historical snapshot lacked this gate.) + if (val_pixel < max_val) + return; // some other pixel in the neighborhood is brighter + is_photon = true; + } + } + + // Pedestal update (if not a photon) + // In the sequential code, non-photon pixels feed back into the running + // pedestal via push_fast(). In this kernel, the GPU updates all pixels in a + // frame simultaneously. So the updated pedestal will only be used starting + // from the next frame. -> This avoids a/serialization and b/global mem I/O. + if (!is_photon && valid_pixel) { + PEDESTAL_TYPE raw_val = static_cast(d_frame[global_tid]); + PEDESTAL_TYPE sum = d_pd_sum[global_tid]; + PEDESTAL_TYPE sum2 = d_pd_sum2[global_tid]; + + sum += raw_val - sum / n_pd_samples; + sum2 += raw_val * raw_val - sum2 / n_pd_samples; + + d_pd_sum[global_tid] = sum; + d_pd_sum2[global_tid] = sum2; + d_pd_mean[global_tid] = sum / n_pd_samples; + return; + } + + /* + if (!is_photon) return; // Debugging + */ + + // Delay building clusterData until we know this thread will write a photon. + // This avoids CSX*CSY conversions/rounds for the overwhelmingly common + // background pixels. + CT clusterData[CSX * CSY]; + int idx = 0; + +#pragma unroll + for (int ir = -row_radius; ir <= row_radius; ++ir) { +#pragma unroll + for (int ic = -col_radius; ic <= col_radius; ++ic) { + COMPUTE_TYPE val = shmem[shmem_tid + ir * shmem_stride + ic]; + if constexpr (std::is_integral_v) + clusterData[idx] = static_cast(lroundf(val)); + else + clusterData[idx] = static_cast(val); + idx++; + } + } + + // Write cluster to global output buffer using atomic index + // for coordination across all blocks + uint32_t write_idx = atomicAdd(d_cluster_count, 1u); + + // Guard against overflowing the pre-allocated cluster buffer + if (write_idx >= max_clusters) + return; + + ClusterType cluster{}; + cluster.x = static_cast(col_global); + cluster.y = static_cast(row_global); + + memcpy(reinterpret_cast(&cluster.data), clusterData, + sizeof(CT) * CSX * CSY); + + d_clusters[write_idx] = cluster; +} + +} // namespace aare::device_opt2 \ No newline at end of file diff --git a/python/aare/ClusterFinder.py b/python/aare/ClusterFinder.py index 6f53f131..53eae39a 100644 --- a/python/aare/ClusterFinder.py +++ b/python/aare/ClusterFinder.py @@ -166,7 +166,34 @@ def ClusterFinderCUDAGraph(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.i n_streams=n_streams) -def ClusterCollector(clusterfindermt, dtype=np.int32): +def ClusterFinderCUDAOpt2(image_size, cluster_size=(3, 3), n_sigma=5, dtype=np.int32, + max_clusters_per_frame=3000, n_streams=4): + """ + Factory for the OPT2 snapshot finder — the pre-refactor pipeline (per-frame + pinned staging, round-robin streams with sync barriers, variable-length D2H), + kept only for benchmarking the optimization arc against the current finder. + + It uses its own kernel snapshot (clusterfinder_kernel_opt2.cuh): f32 stencil + / f64 pedestal. The Test3 local-max gate has been backported so its cluster + counts match the CPU and current finders (correctness held constant across + the opt arc; only the pipeline differs). + + Only the 3x3 cluster size is registered. + """ + if not _cuda_available(): + raise RuntimeError( + "ClusterFinderCUDAOpt2 is not available in this build of aare. " + "Rebuild with -DAARE_CUDA=ON (and -DAARE_PYTHON_BINDINGS=ON)." + ) + + cls = _get_class("ClusterFinderCUDAOpt2", cluster_size, dtype) + return cls(image_size, + n_sigma=n_sigma, + max_clusters_per_frame=max_clusters_per_frame, + n_streams=n_streams) + + +def ClusterCollector(clusterfindermt, dtype=np.int32): """ Factory function to create a ClusterCollector object. Provides a cleaner syntax for the templated ClusterCollector in C++. diff --git a/python/aare/__init__.py b/python/aare/__init__.py index 7c6b3594..9059912a 100644 --- a/python/aare/__init__.py +++ b/python/aare/__init__.py @@ -32,7 +32,7 @@ from ._aare import corner from ._version import __version__ from .ClusterFinder import ClusterFinder, ClusterFinderFrozen, ClusterCollector, ClusterFinderMT, ClusterFileSink, ClusterFile -from .ClusterFinder import ClusterFinderCUDA, ClusterFinderCUDAGraph, _cuda_available +from .ClusterFinder import ClusterFinderCUDA, ClusterFinderCUDAGraph, ClusterFinderCUDAOpt2, _cuda_available from .ClusterVector import ClusterVector from .Cluster import Cluster diff --git a/python/src/bind_ClusterFinderCUDAOpt2.hpp b/python/src/bind_ClusterFinderCUDAOpt2.hpp new file mode 100644 index 00000000..4d8e6c54 --- /dev/null +++ b/python/src/bind_ClusterFinderCUDAOpt2.hpp @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once +#include "aare/ClusterFinderCUDAOpt2.hpp" +#include "aare/ClusterVector.hpp" +#include "aare/NDView.hpp" +#include "aare/Pedestal.hpp" +#include "np_helper.hpp" + +#include +#include +#include + +namespace py = pybind11; +using pd_type = double; + +using namespace aare; + +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wunused-parameter" + +namespace aare { + +// Binding for the OPT2 snapshot finder (pre-refactor pipeline: per-frame pinned +// staging, round-robin streams with sync barriers, variable-length D2H). Kept +// only for benchmarking the optimization arc; not part of the shipped API. +template +void define_ClusterFinderCUDAOpt2(py::module &m, const std::string &typestr) { + auto class_name = fmt::format("ClusterFinderCUDAOpt2_{}", typestr); + + using ClusterType = Cluster; + using CF = ClusterFinderCUDAOpt2; + using ContigArr = + py::array_t; + + py::class_(m, class_name.c_str()) + // ctor: (image_size, n_sigma, capacity, n_streams) — capacity is the + // per-stream device cluster buffer (upper bound on clusters/frame). + .def(py::init, float, size_t, int>(), py::arg("image_size"), + py::arg("n_sigma") = 5.0f, + py::arg("max_clusters_per_frame") = 3000, py::arg("n_streams") = 4) + + .def_property( + "nSigma", &CF::get_nSigma, &CF::set_nSigma, + R"(Number of sigma above the pedestal to consider a photon.)") + + .def("push_pedestal_frame", + [](CF &self, ContigArr frame) { + auto view = make_view_2d(frame); + self.push_pedestal_frame(view); + }) + + .def("clear_pedestal", &CF::clear_pedestal) + + .def_property_readonly("pedestal", + [](CF &self) { + auto pd = new NDArray{}; + *pd = self.pedestal(); + return return_image_data(pd); + }) + + .def_property_readonly("noise", + [](CF &self) { + auto arr = new NDArray{}; + *arr = self.noise(); + return return_image_data(arr); + }) + + .def( + "steal_clusters", + [](CF &self, bool realloc_same_capacity) { + return self.steal_clusters(realloc_same_capacity); + }, + py::arg("realloc_same_capacity") = true) + + .def( + "find_clusters", + [](CF &self, ContigArr frame, uint64_t frame_number) { + auto view = make_view_2d(frame); + self.find_clusters(view, frame_number); + }, + py::arg("frame"), py::arg("frame_number") = 0, + py::call_guard()) + + .def( + "find_clusters_batched", + [](CF &self, ContigArr frames, uint64_t first_frame) { + auto view = make_view_3d(frames); + return self.find_clusters_batched(view, first_frame); + }, + py::arg("frames"), py::arg("first_frame") = 0, + py::call_guard(), + R"(Process a 3D array (n_frames, nrows, ncols) round-robin across +n_streams. Returns a list of ClusterVector, one per input frame.)") + + .def("avg_kernel_time_ms", &CF::avg_kernel_time_ms) + .def("reset_timers", &CF::reset_timers); +} + +} // namespace aare + +#pragma GCC diagnostic pop diff --git a/python/src/cuda_bindings.cu b/python/src/cuda_bindings.cu index 0ca99e96..b15fee71 100644 --- a/python/src/cuda_bindings.cu +++ b/python/src/cuda_bindings.cu @@ -8,6 +8,7 @@ #include "bind_Cluster.hpp" #include "bind_ClusterFinderCUDA.hpp" #include "bind_ClusterFinderCUDAGraph.hpp" +#include "bind_ClusterFinderCUDAOpt2.hpp" #include "bind_ClusterVector.hpp" #include @@ -30,6 +31,10 @@ namespace py = pybind11; aare::define_ClusterFinderCUDAGraph(m, "Cluster" #N \ "x" #M #TYPE_CODE); +#define DEFINE_BINDINGS_CLUSTERFINDER_CUDA_OPT2(T, N, M, U, TYPE_CODE) \ + aare::define_ClusterFinderCUDAOpt2(m, "Cluster" #N \ + "x" #M #TYPE_CODE); + PYBIND11_MODULE(_aare_cuda, m) { // Types first — finders reference them in their signatures. @@ -83,8 +88,14 @@ PYBIND11_MODULE(_aare_cuda, m) { DEFINE_BINDINGS_CLUSTERFINDER_CUDA_GRAPH(int, 9, 9, uint16_t, i); DEFINE_BINDINGS_CLUSTERFINDER_CUDA_GRAPH(double, 9, 9, uint16_t, d); DEFINE_BINDINGS_CLUSTERFINDER_CUDA_GRAPH(float, 9, 9, uint16_t, f); + + // OPT2 snapshot finder (benchmark only) — 3x3 is what the deck uses. + DEFINE_BINDINGS_CLUSTERFINDER_CUDA_OPT2(int, 3, 3, uint16_t, i); + DEFINE_BINDINGS_CLUSTERFINDER_CUDA_OPT2(double, 3, 3, uint16_t, d); + DEFINE_BINDINGS_CLUSTERFINDER_CUDA_OPT2(float, 3, 3, uint16_t, f); } #undef DEFINE_CUDA_CLUSTER_TYPES #undef DEFINE_BINDINGS_CLUSTERFINDER_CUDA -#undef DEFINE_BINDINGS_CLUSTERFINDER_CUDA_GRAPH \ No newline at end of file +#undef DEFINE_BINDINGS_CLUSTERFINDER_CUDA_GRAPH +#undef DEFINE_BINDINGS_CLUSTERFINDER_CUDA_OPT2 \ No newline at end of file diff --git a/python/tests/ClusterFinderCUDA_perf.ipynb b/python/tests/ClusterFinderCUDA_perf.ipynb index 84fe1160..7902491a 100644 --- a/python/tests/ClusterFinderCUDA_perf.ipynb +++ b/python/tests/ClusterFinderCUDA_perf.ipynb @@ -1,617 +1,817 @@ - { - "cells": - [ - { - "cell_type": "markdown", - "id": "md-title", - "metadata": {}, - "source": [ - "# ClusterFinder — CPU vs CUDA performance\n", - "\n", - "Throughput comparison of the serial CPU finder against the CUDA variants:\n", - "per-frame, batched+pinned, CUDA-Graph, and the async double-buffered pipeline.\n", - "\n", - "CPU↔CUDA **correctness** (why the cluster counts differ) is analysed\n", - "separately in `ClusterFinderFrozen_vs_CUDA.ipynb`; here we only sanity-check that\n", - "the counts and spectra agree, and focus on timing." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "imports", - "metadata": {}, - "outputs": [], - "source": [ - "import sys; sys.path.append('/home/ferjao_k/aare/build')\n", - "\n", - "from pathlib import Path\n", - "import matplotlib.pyplot as plt\n", - "import numpy as np\n", - "import boost_histogram as bh\n", - "import time\n", - "from tqdm import tqdm\n", - "\n", - "from aare import (File, ClusterFinder, ClusterFinderFrozen, ClusterFinderMT, ClusterCollector,\n", - " ClusterFinderCUDA, ClusterFinderCUDAGraph)\n", - "from helper import print_pinning_budget" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "hist-helpers", - "metadata": {}, - "outputs": [], - "source": [ - "N_BINS = 200\n", - "\n", - "def make_hist(clusters):\n", - " h = bh.Histogram(bh.axis.Regular(N_BINS, -2, 4000))\n", - " h.fill(clusters.sum())\n", - " return h\n", - "\n", - "def make_hist_from_batch(result_list):\n", - " h = bh.Histogram(bh.axis.Regular(N_BINS, -2, 4000))\n", - " energies = [np.asarray(cv.sum()).ravel() for cv in result_list if cv.size > 0]\n", - " if energies:\n", - " h.fill(np.concatenate(energies))\n", - " return h" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "config", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Image size: (400, 400)\n", - "Pedestal frames: 1000\n", - "Data frames: 20000\n", - "Total in file: 100000\n" - ] - } - ], - "source": [ - "base = Path('/mnt/sls_det_storage/moench_data/2603_MaxIVBeamtime/2026032408/process/xrf/')\n", - "f = File(base / 'Cu_factor_10_data_master_0.json')\n", - "pd = File(base / 'Cu_factor_10_pedestal_master_0.json')\n", - "\n", - "n_frames_pd = 1000\n", - "N = 20000\n", - "cluster_size = (9, 9)\n", - "rows, cols = f.rows, f.cols\n", - "image_size = (rows, cols)\n", - "capacity = 1000\n", - "BATCH_SIZE = 2000\n", - "\n", - "N_STREAMS = 4\n", - "N_SIGMA = 5\n", - "\n", - "print(f'Image size: {image_size}')\n", - "print(f'Pedestal frames: {n_frames_pd}')\n", - "print(f'Data frames: {N}')\n", - "print(f'Total in file: {f.total_frames}')" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "pinning", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "── System RAM ──────────────────────────────────────────\n", - " Total RAM : 125.1 GiB\n", - " Currently available : 112.7 GiB (free + reclaimable cache)\n", - " Reserved headroom : 4.0 GiB (OS + CUDA context)\n", - " Safe pinning budget : 108.7 GiB\n", - "\n", - "── Frame layout ────────────────────────────────────────\n", - " Frame size : 400 × 400 × 2 B = 312.5 kB\n", - "\n", - "── Pinning estimate ────────────────────────────────────\n", - " Max frames pinnable : 364,859 (108.7 GiB)\n", - "\n", - " Note: no swap on this machine — exceeding available RAM\n", - " will trigger the OOM killer. Stay within the budget.\n" - ] - } - ], - "source": [ - "print_pinning_budget(rows, cols)" - ] - }, - { - "cell_type": "markdown", - "id": "md-build", - "metadata": {}, - "source": [ - "## Build finders\n", - "\n", - "`SERIAL` picks the CPU baseline: the sequential `ClusterFinder` or the\n", - "multi-threaded `ClusterFinderMT`. All finders are trained on the same pedestal\n", - "frames below." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "build", - "metadata": {}, - "outputs": [], - "source": [ - "SERIAL = False\n", - "\n", - "if SERIAL:\n", - " # cf_cpu = ClusterFinder(image_size, cluster_size, n_sigma=N_SIGMA, capacity=capacity)\n", - " cf_cpu = ClusterFinderFrozen(image_size, cluster_size, n_sigma=N_SIGMA, capacity=capacity)\n", - "else:\n", - " cf_cpu = ClusterFinderMT(image_size, cluster_size, n_sigma=N_SIGMA,\n", - " capacity=capacity, n_threads=48)\n", - " sink = ClusterCollector(cf_cpu)\n", - "\n", - "cf_cuda_v1 = ClusterFinderCUDA(image_size, cluster_size, n_sigma=N_SIGMA,\n", - " max_clusters_per_frame=1500, n_streams=N_STREAMS)\n", - "cf_cuda = ClusterFinderCUDA(image_size, cluster_size, n_sigma=N_SIGMA,\n", - " max_clusters_per_frame=1500, n_streams=N_STREAMS)\n", - "# cf_async = ClusterFinderCUDA(image_size, cluster_size, n_sigma=N_SIGMA,\n", - "# max_clusters_per_frame=2000, n_streams=N_STREAMS)\n", - "cf_graph = ClusterFinderCUDAGraph(image_size, cluster_size, n_sigma=N_SIGMA,\n", - " max_clusters_per_frame=1500, n_streams=N_STREAMS)" - ] - }, - { - "cell_type": "markdown", - "id": "md-ped", - "metadata": {}, - "source": [ - "## Pedestal (all finders trained on identical frames)" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "train", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Pedestal (1000 frames): 10.831s\n" - ] - } - ], - "source": [ - "t0 = time.perf_counter()\n", - "for _ in range(n_frames_pd):\n", - " img = pd.read_frame()\n", - " cf_cpu.push_pedestal_frame(img.copy())\n", - " cf_cuda_v1.push_pedestal_frame(img.copy())\n", - " cf_cuda.push_pedestal_frame(img.copy())\n", - " # cf_async.push_pedestal_frame(img.copy())\n", - " cf_graph.push_pedestal_frame(img.copy())\n", - "print(f'Pedestal ({n_frames_pd} frames): {time.perf_counter() - t0:.3f}s')" - ] - }, - { - "cell_type": "markdown", - "id": "md-io", - "metadata": {}, - "source": [ - "## Read all data frames into memory (I/O out of the timing loop)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "io", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Reading 20000 frames: 1.074s (18614 FPS, 5680.686 GB/s)\n" - ] - } - ], - "source": [ - "f.seek(0)\n", - "t0 = time.perf_counter()\n", - "data = f.read_n(N)\n", - "t_io = time.perf_counter() - t0\n", - "print(f'Reading {N} frames: {t_io:.3f}s ({N/t_io:.0f} FPS, '\n", - " f'{f.bytes_per_frame * N / 1024**2 / t_io:.3f} GB/s)')" - ] - }, - { - "cell_type": "markdown", - "id": "md-cpu", - "metadata": {}, - "source": [ - "## CPU clustering (baseline)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "cpu", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [00:09<00:00, 2184.48it/s]\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU clustering: 9.158s (2184 FPS, 28438072 clusters, 1421.90/frame)\n" - ] - } - ], - "source": [ - "t0 = time.perf_counter()\n", - "for frame in tqdm(data):\n", - " cf_cpu.find_clusters(frame)\n", - "t_cpu = time.perf_counter() - t0\n", - "\n", - "if SERIAL:\n", - " clusters_cpu = cf_cpu.steal_clusters(realloc_same_capacity=False)\n", - " n_clusters_cpu = clusters_cpu.size\n", - " hist_cpu = make_hist(clusters_cpu)\n", - "else:\n", - " cf_cpu.stop(); sink.stop()\n", - " clusters_cpu = sink.steal_clusters()\n", - " hist_cpu = bh.Histogram(bh.axis.Regular(N_BINS, -2, 4000))\n", - " n_clusters_cpu = 0\n", - " for cv in clusters_cpu:\n", - " hist_cpu.fill(cv.sum())\n", - " n_clusters_cpu += cv.size\n", - "\n", - "print(f'CPU clustering: {t_cpu:.3f}s ({N/t_cpu:.0f} FPS, '\n", - " f'{n_clusters_cpu} clusters, {n_clusters_cpu/N:.2f}/frame)')" - ] - }, - { - "cell_type": "markdown", - "id": "md-v1", - "metadata": {}, - "source": [ - "## CUDA — per-frame (1 launch/frame, pageable memory)\n", - "Simplest path: no batching, no pinning. Isolates per-frame launch + PCIe overhead." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "v1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CUDA per-frame: 3.659s (5467 FPS, 28445699 clusters, 1422.28/frame)\n", - " Kernel only: 0.034 ms/frame\n", - " PCIe + overhead: 0.149 ms/frame\n", - "Speedup (CPU/CUDA): 2.50x\n" - ] - } - ], - "source": [ - "cf_cuda_v1.reset_timers()\n", - "t0 = time.perf_counter()\n", - "\n", - "n_clusters_cuda_v1 = 0\n", - "hist_cuda_v1 = None\n", - "for frame in data:\n", - " cf_cuda_v1.find_clusters(frame)\n", - " clusters_frame = cf_cuda_v1.steal_clusters(realloc_same_capacity=True)\n", - " n_clusters_cuda_v1 += clusters_frame.size\n", - " h = make_hist(clusters_frame)\n", - " hist_cuda_v1 = h if hist_cuda_v1 is None else hist_cuda_v1 + h\n", - "\n", - "t_cuda_v1 = time.perf_counter() - t0\n", - "kernel_ms = cf_cuda_v1.avg_kernel_time_ms()\n", - "print(f'CUDA per-frame: {t_cuda_v1:.3f}s ({N/t_cuda_v1:.0f} FPS, '\n", - " f'{n_clusters_cuda_v1} clusters, {n_clusters_cuda_v1/N:.2f}/frame)')\n", - "print(f' Kernel only: {kernel_ms:.3f} ms/frame')\n", - "print(f' PCIe + overhead: {t_cuda_v1*1000/N - kernel_ms:.3f} ms/frame')\n", - "print(f'Speedup (CPU/CUDA): {t_cpu / t_cuda_v1:.2f}x')" - ] - }, - { - "cell_type": "markdown", - "id": "md-batched", - "metadata": {}, - "source": [ - "## CUDA — batched + multi-streamed + pinned dataset\n", - "Pins the whole input once, then submits `BATCH_SIZE`-frame chunks across `N_STREAMS` streams so H2D / kernel / D2H overlap." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "batched", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CUDA batched: 3.088s (6477 FPS, 28439289 clusters, 1421.96/frame)\n", - " Kernel only: 0.031 ms/frame\n", - " PCIe + overhead: 0.123 ms/frame\n", - "Speedup (CPU/CUDA): 2.97x\n" - ] - } - ], - "source": [ - "cf_cuda.register_input_buffer(data) # pin the whole dataset once\n", - "clusters_cuda_per_frame = []\n", - "\n", - "cf_cuda.reset_timers()\n", - "t0 = time.perf_counter()\n", - "for start in range(0, N, BATCH_SIZE):\n", - " stop = min(start + BATCH_SIZE, N)\n", - " clusters_cuda_per_frame.extend(\n", - " cf_cuda.find_clusters_batched(data[start:stop], first_frame=start))\n", - "t_cuda = time.perf_counter() - t0\n", - "\n", - "cf_cuda.unregister_input_buffer()\n", - "kernel_ms = cf_cuda.avg_kernel_time_ms()\n", - "n_clusters_cuda = sum(cv.size for cv in clusters_cuda_per_frame)\n", - "hist_cuda = make_hist_from_batch(clusters_cuda_per_frame)\n", - "\n", - "print(f'CUDA batched: {t_cuda:.3f}s ({N/t_cuda:.0f} FPS, '\n", - " f'{n_clusters_cuda} clusters, {n_clusters_cuda/N:.2f}/frame)')\n", - "print(f' Kernel only: {kernel_ms:.3f} ms/frame')\n", - "print(f' PCIe + overhead: {t_cuda*1000/N - kernel_ms:.3f} ms/frame')\n", - "print(f'Speedup (CPU/CUDA): {t_cpu / t_cuda:.2f}x')" - ] - }, - { - "cell_type": "markdown", - "id": "md-graph", - "metadata": {}, - "source": [ - "## CUDA — Graph (pinned dataset)\n", - "Pre-records the H2D->kernel->D2H pipeline as a CUDA Graph per stream, cutting per-frame CPU API overhead (~21 us vs ~60 us for the streamed version)." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "graph", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CUDA Graph: 3.132s (6386 FPS, 28439289 clusters, 1421.96/frame)\n", - " Kernel+PCIe+ovhd: 0.157 ms/frame (kernel not individually timed)\n", - "Speedup (vs batched): 0.99x\n", - "Speedup (CPU/Graph): 2.92x\n" - ] - } - ], - "source": [ - "cf_graph.register_input_buffer(data)\n", - "clusters_graph_per_frame = []\n", - "\n", - "t0 = time.perf_counter()\n", - "for start in range(0, N, BATCH_SIZE):\n", - " stop = min(start + BATCH_SIZE, N)\n", - " clusters_graph_per_frame.extend(\n", - " cf_graph.find_clusters_batched(data[start:stop], first_frame=start))\n", - "t_graph = time.perf_counter() - t0\n", - "\n", - "cf_graph.unregister_input_buffer()\n", - "n_clusters_graph = sum(cv.size for cv in clusters_graph_per_frame)\n", - "hist_graph = make_hist_from_batch(clusters_graph_per_frame)\n", - "\n", - "print(f'CUDA Graph: {t_graph:.3f}s ({N/t_graph:.0f} FPS, '\n", - " f'{n_clusters_graph} clusters, {n_clusters_graph/N:.2f}/frame)')\n", - "print(f' Kernel+PCIe+ovhd: {t_graph*1000/N:.3f} ms/frame (kernel not individually timed)')\n", - "print(f'Speedup (vs batched): {t_cuda / t_graph:.2f}x')\n", - "print(f'Speedup (CPU/Graph): {t_cpu / t_graph:.2f}x')" - ] - }, - { - "cell_type": "markdown", - "id": "md-async", - "metadata": {}, - "source": [ - "## CUDA — async double-buffered pipeline\n", - "Two pinned batch buffers: while the GPU processes `buf[cur]`, the CPU fills `buf[nxt]`, so H2D copies and kernels stay back-to-back." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "async", - "metadata": {}, - "outputs": [], - "source": [ - "# buf = [np.empty((BATCH_SIZE, rows, cols), dtype=np.uint16) for _ in range(2)]\n", - "# cf_async.pin_buffer(buf[0])\n", - "# cf_async.pin_buffer(buf[1])\n", - "\n", - "# clusters_async = []\n", - "# cf_async.reset_timers()\n", - "# t0 = time.perf_counter()\n", - "\n", - "# cur = 0\n", - "# n0 = min(BATCH_SIZE, N)\n", - "# buf[cur][:n0] = data[:n0]\n", - "# tok = cf_async.submit_batch(buf[cur][:n0], first_frame=0)\n", - "\n", - "# for start in range(BATCH_SIZE, N, BATCH_SIZE):\n", - "# nxt = 1 - cur\n", - "# stop = min(start + BATCH_SIZE, N)\n", - "# n = stop - start\n", - "# buf[nxt][:n] = data[start:stop] # fill next while GPU runs current\n", - "# next_tok = cf_async.submit_batch(buf[nxt][:n], first_frame=start)\n", - "# clusters_async.extend(cf_async.collect(tok)) # drain previous\n", - "# tok = next_tok\n", - "# cur = nxt\n", - "# clusters_async.extend(cf_async.collect(tok)) # drain final\n", - "\n", - "# t_async = time.perf_counter() - t0\n", - "# cf_async.unpin_buffer(buf[0]); cf_async.unpin_buffer(buf[1])\n", - "\n", - "# kernel_ms_async = cf_async.avg_kernel_time_ms()\n", - "# n_clusters_async = sum(cv.size for cv in clusters_async)\n", - "# hist_async = make_hist_from_batch(clusters_async)\n", - "\n", - "# print(f'CUDA async pipeline: {t_async:.3f}s ({N/t_async:.0f} FPS, '\n", - "# f'{n_clusters_async} clusters, {n_clusters_async/N:.2f}/frame)')\n", - "# print(f' Kernel only: {kernel_ms_async:.3f} ms/frame')\n", - "# print(f' PCIe + overhead: {t_async*1000/N - kernel_ms_async:.3f} ms/frame')\n", - "# print(f'Speedup (vs Graph): {t_graph / t_async:.2f}x')" - ] - }, - { - "cell_type": "markdown", - "id": "md-agree", - "metadata": {}, - "source": [ - "## Agreement sanity check\n", - "Counts should match to a hair. The residual comes from the CUDA finder updating the pedestal once per frame vs the CPU's per-pixel update (analysed in `ClusterFinderFrozen_vs_CUDA.ipynb`)." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "agree", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " CPU 28,438,072 diff vs CPU 0 (0.0000%)\n", - " CUDA per-frame 28,445,699 diff vs CPU 7,627 (0.0268%)\n", - " CUDA batched 28,439,289 diff vs CPU 1,217 (0.0043%)\n", - " CUDA graph 28,439,289 diff vs CPU 1,217 (0.0043%)\n" - ] - } - ], - "source": [ - "for name, n in [('CPU', n_clusters_cpu), ('CUDA per-frame', n_clusters_cuda_v1),\n", - " ('CUDA batched', n_clusters_cuda), ('CUDA graph', n_clusters_graph)]:\n", - " # ('CUDA async', n_clusters_async)]:\n", - " d = abs(n - n_clusters_cpu)\n", - " print(f' {name:<16} {n:>12,} diff vs CPU {d:>6,} ({d/max(n_clusters_cpu,1):.4%})')" - ] - }, - { - "cell_type": "markdown", - "id": "md-plots", - "metadata": {}, - "source": [ - "## Spectrum: CPU vs CUDA variants" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "plots", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": - "iVBORw0KGgoAAAANSUhEUgAAAxYAAAJOCAYAAAAqFJGJAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQABAABJREFUeJzsnQd4FEUbx//pvYeQSoCEFnpVEKQIIl1FERUEVFBBRECxK/pJUaoioiAgqEgRUEQsgIoivfdeE0ooKZBe9nv+c+xxOS4hQMoleX/Pc9nc3uzu7Mzs3fvOW8ZG0zQNgiAIgiAIgiAId4DtnRwsCIIgCIIgCIIgioUgCIIgCIIgCAWCWCwEQRAEQRAEQbhjRLEQBEEQBEEQBOGOEcVCEARBEARBEIQ7RhQLQRAEQRAEQRDuGFEsBEEQBEEQBEG4Y0SxEARBEARBEAThjhHFQhAEQRAEQRCEO0YUC0EogezatQv9+vVDpUqV4OzsDHd3dzRo0AAff/wxLl++bCzXqlUr9SosPv/8c3z99deFdn6h5FAax8K///6LHj16ICQkBI6OjvDy8kKzZs0wbdo0JCUlGctVrFgRNjY2xhefx7vuugtz587NcT6W69y5s8VrbdmyRR1rLW2Y3++YW72nkSNH5mgrV1dXhIaGon379pgyZQquXLmSa50yMjIQGBiojvvhhx9gjfz999+qftwWJqXxeRNKB6JYCEIJY8aMGWjYsCE2b96MV199Fb/99huWLl2KRx99FF988QWeeeaZIquL/LgJpXUsvPfee7j33nsRExOD//3vf1i5ciXmz5+P++67TwnHb7/9do7y99xzD9avX69ebAcKl3369FFKSEmjKL5jeE62Fbfjx49HhQoVMGLECNSsWRM7d+60eMzy5ctx/vx59f/MmTNhjVD54n1xW5iUtudNKD3YF3cFBEHIP/zBeuGFF9CuXTv8+OOPcHJyMn7GfcOHD1c/1CUZTdOQmpoKFxcXlARSUlJKTF2tBc48U/C2t7fOn6BFixbhgw8+UAI0hWzWVadDhw5KAOazaIq3tzfuvvtu4/u2bdsiPDwcEydOVM9sSaGovmOouPj7+xvf9+zZEy+++CJatmyJrl274tChQzmurSsTtByxzB9//IHo6Ghl7bCmMe3p6ZljHAhCWUMsFoJQghg9erT68Zo+ffoNP7qEP7r8Ub5VM/2JEyducFk4duyY+rEPDg5W1ypfvryard2xY4fRBWLv3r1Ys2aN0a2B+3QSExPxyiuvKFcK1ovuJC+//HIOFxLC4yhQcCa0Ro0a6lpz5szJsx0WLFiApk2bws3NTblo0I1i+/btOcr07dtXfXbkyBF07NhR/R8WFqYEo7S0tBxl09PT8eGHH6J69erq+uXKlVNuIBcuXMhRTnf7WLJkCerXr69cRN5//331Gdvi/vvvV64dPH7QoEH45ZdfcrQ3Z74pTJ8+ffqGe3r66afh5+enlKrcuFmfmNaRM8x16tRRdaxcuTI+/fTTG86X3z7Kzs5Wbir16tVTSpQuRC9btuymY0Efc998841qe16DdWe/6G4x5ugz/hyX5vfFWWu2PevB8cL3+jF8zzHRpEkT5YZzu1Cp8PHxUW1mqX4eHh6qr/OCbVStWjWcPHkSBQVn8lkfS7P1v/76q/pM7xOO3QEDBqgxr49pWlVWrVpVqN8xd0LdunXx1ltv4dSpU+oZN+XMmTNKoenSpYuyonBM5mfG/lbajGOSz32VKlXUc8yxyuvt3r07x3F5jWlL37Eci3xuOYY5brl9/PHHbxgb+rj/66+/lHJHxYvfCQ8//LC6f528nje2C7/LOPb0Z5XfA5988km++0EQ7gTrnC4SBOEGsrKy8Oeff6qZPgoLhQ2FcV6TPtV0U7h48SLWrVuH+Ph49TkF10ceeUT5ndMsT3RBJDk5Wc0qckbxzTffVD9s/CF899131Y80hRtTgY0zo/Rn5+f0oQ4ICMhT8KEbCgUAbqkUjBs3Di1atMCmTZsQFRWVYxaRQhBnnikA/PPPP0q4Z515Lf2HuFu3bur6nImmDz1/8OkKw/gUCgWmFolt27Zh//796toUyCnInj17Vt0v/6frC+v//fffK4XJlOeeew6jRo3Cl19+qX78deizTjcblqcicLt9okNFgwoCBXe253fffYchQ4aotqIicat9RCXt22+/Ve1IoZvCJdtBF/zzGgs6b7zxhlIGqUDa2trm2cd5CYk8D4VPXotKHYUu7lu9erVRKH7ttdeUEnL8+HFj37Gu7C+6J+UlkLIv9+zZg8cee0wJl7cLxx7HEQX6ghS8qVTNnj37Bnck3hPblGOE9O7dW/URx1vVqlXVGOH7S5cuWc13jCX4vPI55LP61FNP5bg/1o8KuG4NmjVrlhoLlpS/22kzCu8U5MeOHav6jc8lJzkYL8OJCwrrNxvT586du6EOHHs8lsqFr6+vGmP8nmjcuDH27duXw3JDnn32WXTq1Anz5s1TkxBUpHr16qX65mbPG78bdFc9uvJxHB44cOCG7whBKDQ0QRBKBOfOndP4yPbs2TPfx7Rs2VK9dP766y91Dm5NOX78uNo/e/Zs9f7ixYvq/eTJk/M8f82aNXOcX2fMmDGara2ttnnz5hz7f/jhB3XeFStWGPfxvZeXl3b58uWb3s+pU6c0e3t7bfDgwTn2X7lyRQsMDNR69Ohh3NenTx917oULF+Yo27FjR61atWrG999//70qt3jx4hzlWHfu//zzz437wsPDNTs7O+3gwYM5yr766quajY2Ntnfv3hz727dvf0N7s14BAQFaWlqacd9HH32k2ov9kBv57RPWkXXZsWNHjv3t2rXTPD09taSkpFvqo3/++Ue9f+utt25rLOhj7t57773hs/fee099Zg7HIfebtgfvy8XFRYuOjjbu4z2yXFBQkPG+yI8//qj2L1u2zLjvxIkTqu+efvrpPO9jw4YN6tjXX39dyy+sG8dVRkaGerHe+vjj2DAt16lTJ4vn0Meb/gzmxqeffqrKmY5BPjtOTk7a8OHDjfvc3d21l19+WSvs75hbvSe9zy9cuGDxmJSUFPV5hw4djPuys7O1yMhILSQkRMvMzMxxntWrV9+0jvltM3N4rfT0dK1KlSra0KFD8zWmc/uONT/v1atXNTc3N+2TTz65YdwPHDgwR/mPP/5Y7T979uxNn7fOnTtr9erVy/XaglDYiCuUIAg3wFm1iIgIZQmgjzhn6zizn1/onlKrVi3lOpOZmWl80WXJkitWmzZtlOvJzfj999/VeTiTaXpezvJz9t38vLwWXRlM4cy8qQsC60p3AZYzPSfrztl+83PyeM4Am0KXBN6vqbWE0N3BHFoOYmNjlR8/Ybty9pIzlKauZHfSJwyA5UytKU888YRyfeKs9a30Ed1FCF277oTu3bvjTmFd6XaiQ9cnQsuSqXVB32/az5zh5v0VVtDvihUr4ODgoF60jCxcuBCDBw/OYZkqCJ588kk1O21qdaF1jO59tOLp0B2MZXj9DRs2qJnrkoBhrgE3PF90M6K1yc7OTu3jvXKc0mpRUG3G8UGrF59jWuXotsjt4cOHlZXydsf01atXlRUtMjJSnZMvumbS5dDSec1dzfidQ/LjVsd+p2Vv4MCB6vuSz7wgFCWiWAhCCYHmcgpPdO8obPiDTdcSCpk0rTPDCV0DXnrppTzTQeowcwvTVeqClv6ibzoFB7rwmBIUFJSveukZYehCYH5u+mSbn5ftZe5aRAHDNI6B56SbAAUI83PSrSE/daV7CeMdzLG0j24ZdNuaOnWqUcCnq4S529Sd9AkVInP0fborTH77iL76FOYsnfNWyG8f5wWVK1PYZ3ntzyteJTfoYkZu9Tlr3ry5yqJE1zm6t3BMMUZDrwuhQEl3HktQqCXsg7zgvVLwZCpb/VwUmClQUqHU4fNAQfyrr75S7jo8jgq5JVedO/mOKYh7MkUXnhlHpKMrgw899JBqV77oBsQ2X7x48U3dfPLbZsOGDcM777yDBx98ED///DM2btyo+pRKOpM03O6YplL/2WefKRcnCvt02eR5+fxaOi/dsUzR3ZwslTWH7lnMskVlkokGeC7GYd1JzJEg3AoSYyEIJQQKd/yB4Azy7WZD0YVs8+Blc+FZn+HVf9CZoYUzsPTdpZ8+fYrzggIKfdtzm0009ynOy0fa0nHMYc/6FQR6gGRumW4oaN+srjxeV3pMyU2IozLA1J20HlDgoAWEGXduRn77xNJ19X260JLfPqLwQ2GMx9+JcmCp3UzHo2lMhqXxWFTwHmvXrq2yDjEOJb9xFhR0GzVqlGcZKppMX2sJfb8lZdQczrLT4sUUuFSEKKSap7Vl/02ePFm9GAzNAOXXX39dWctyG+u38x1TUPekowdS6+vvJCQkKOVBn1CwBGMROEN/p23GOCIqX7RamMLxSKumOfn53mL9OXnAmC22vw7HvOl6IAUFFT0qSHxR4WKsFGOoOCHBeI07iRsShPwgFgtBKEFwNoqzyf3791fCpDl0d+BMW27orjacqbb0Y54bFHwZDEiBS3elIRQGLc2iMXD26NGjSoilsGX+ysvlJy/448gfTp7b0nlvJthZgnXlLD6FZ0vnMw/YtATdsBjwy5lqUxiQbQnOvFK4YUA5f/gpFOVXubpZnxAGYZuvBUDhi0qSnl8/v33EWU9ys/UYchsLeZHbeMxrDBcFnLWOi4tTCqAl1xy6tlDxuFUYdGxpnBAqifrCejeDGanoEsaAZL6ooFlyu9PhWKNFjMqr+Vi50++YgronwjFLoZ7jggsT6uOW44pJF5gtyfxFBSo/7lD5aTM+g+ZJB5jZLTfFKT/wnGxP8/PSkpSbpSc/5Od5ozLEIG+6MVKJMc2yJgiFhVgsBKEEQZcGCngURJm5hSkJacrnjz197pkikn7z5nEFOnRnoSAwZswYFdPAGXC61zB9qikU9CiIcFadqRfpzsGMJNxvOutGoZbCM90umNKUP9bcx4xEnGVkVpKhQ4cqH2HGA3DmlAIZBer8ChumUOBgViJmgmHq1QceeEDdB60FdC9gViY9/Wt+YaYWZk1iZhjGP9A9gq4bnLGl4MKMUVQE8oL3S+GGQjjrxxlaCkTMxkKYMcZ8Zpg/9vS7Zp2Zdelm5LdPdDcSun7QmsEZeM7Ecqb2o48+Ms5Y5reP6LbFDEP01Wc7UyGhUMPxxnMxjiCvsZAXbHO6qejZpqg00kXFUjreO4UuNoxRoXvQzeIs2MZULijMsg9ZPx5LCwbdY5jVi1mjbpZy1hyOL7rjcDaes8hsHyowbDNa4Rg7Y24hswTHD2fWWZ7rJjAzFi0mprPkrVu3Vi44TKHMc3KGnpYKls2LW/2Oud172rp1q6ozz8tsTPweYvpWZlai4qK7kLGv+Iwzm5mljGl6O1ApMY8rupU2IxzbHH9sMz4PrCNjmu5krQxei88Yz0MliN9hjBnhfVmyguSX3J439gv7h5MDtDZy3NNqxe96fm8IQqFT6OHhgiAUOMyGw6wzFSpU0BwdHVV2kfr162vvvvuuFhsbm2tWKMLMIo888ojm6+ursjH16tVL27JlS47sLefPn9f69u2rVa9eXZ2bGWbq1KmjTZo0yZiVRc+0c//992seHh7qeGaI0WHWk7fffltlYGIdea3atWur7CrMPqPD4wYNGnRL98+sP61bt1ZZjpjZhdflPa1atcpYhu3DuucnExEz+YwfP16rW7eu5uzsrO6X9/7cc89phw8fzlcGnD179mht27ZVx7Ntn3nmGW3OnDnqWjt37ryhPNuOnz3//PP5uuf89oleR2Z3YuYYtn3FihW1iRMn3nDO/PZRVlaWuk6tWrWM5Zo2bar9/PPPNx0LepacRYsWWbyvTZs2ac2aNVP3xKw/7J+vvvrKYlYoS21vafzoWc7GjRt3wz6Oi/yyZs0aNa6YdcrBwUGNN943z5uYmHjTulmC7frCCy+oZ5cZzthezZs3z7V9cuPQoUPqfvhauXJljs9SU1PVuOL4YJ2ZTYt9zLY1zZ5VEN8xt3pP+vOnv/j8sn05dpghybRd+dywTF7ZrQ4cOKDKmGeKu9U2I3Fxceq5ZdY2V1dXdQ///vtvrtn1LN2fpaxQzGTWvXt3zcfHR7XNAw88oL4vOG5Mx6OeFco8U5ulc+b2vE2YMEE9T/7+/qrf2Ce8J5YXhKLAhn8KX30RBEEoe3CBMmafoauVaRAv4YJzdLWhG4lpAOmdwhlRzljqC8cJgiAIQlEhrlCCIAgFAF156IJEtwT64FOwpx814yBMlQq6kzDrDsvTzaoglQpBEARBKE5EsRAEQSgAGJdBP2rGZjDNJv2Z6c9NH3RTGK/BDEuMXbhZdi1BEARBKEmIK5QgCIIgCIIgCHeMpJsVBEEQBEEQBOGOEcVCEARBEARBEIQ7RhQLQRAEQRAEQRDuGAneLmK4ABUXA+KCQbe60q4gCIIgCIIgFCVcmeLKlSsq86H5gq/miGJRxFCpCAsLK+rLCoIgCIIgCMJtc/r06ZuuRC+KRRFDS4XeOZ6enkWucSYkJMDLy0usJdJuMt6sFHlOpd1kzJUM5FmVdisr4y0xMVFNiusybF6IYlHE6IOBSkVxKBZ88brihiXtJuPNOpHnVNpNxlzJQJ5VabeyNt5s8nFdCd4WBEEQBEEQBOGOEcVCEARBEARBEIQ7RhQLQRAEQRAEQRDuGImxEARBEEo1WVlZyMjIKO5qWC30205PT0dqaqrE30m7yXgrg8+pg4MD7OzsCuRcolgIgiAIpfaH+Ny5c4iPjy/uqpSINZYuXbpU3NUocUi7SbuVlvHm7e2NwMDAO1ZaRLEQBEEQSiW6UhEQEABXV1eZjc9DAaNVhzOWkjEw/0i73R7SbtbVbjxvcnIyYmNj1fugoKA7Op8oFoIgCEKpgz/AulLh5+dX3NWxakTQk3aT8Va2n1MXFxe1pXLB78w7cYuS4G1BEASh1KHHVNBSIQiCIOSN/l15p/FoolgIgiAIpRZx7REEQSi670pRLARBEARBEARBuGNEsRAEQRAEwWo5ePCgylZz5cqV4q5KqePrr79W2YBKIo888ggmTpxY3NUQzBDFQhAEQRCsLJvV4MGDUblyZTg5OSEsLAxdunTB6tWrjWUqVqyoXBf4om90rVq18OWXXxo/HzlyJOrVq3fDuRnQzmP+/vtvlBTeeustDBo0CB4eHuo9696tWzeVvcbNzU3d53fffXfDcdxXt25d1T4s269fv1xTdc6fP1+1y4MPPphj/7Rp01CnTh14enqqV9OmTfHrr7/mKHP16lW8+OKLCA0NVUGwNWrUUMeZkpaWpvrU399f1blr166Ijo42fs570vvT/LV582ZVZufOnXj88cfVeNCv88knn8Ca6Nu37w1tWFi8++67GDVqFBITE4vkekL+EMVCEARBEKyEEydOoGHDhvjzzz/x8ccfY/fu3fjtt9/QunVrJVyb8sEHH+Ds2bPYtWuXEuaef/55LFiwAKUJCt/Lli1TSoHOunXrlLC/ePFide9PP/00nnrqKfz888/GMmvXrlX7nnnmGezduxeLFi1SAvqzzz57wzVOnjyJV155BS1atLjhMyoLY8eOxZYtW9SrTZs2SqnhOXWGDh2q+ujbb7/F/v371XsqET/99JOxzMsvv4ylS5cqBYZ1ozLSuXNnleWHNGvWTPWl6Yt1pQLZqFEjVWbr1q0oV66cug6vT4XrjTfewGeffYbSBheCuxkcA2wfS0qlUIxoQpGSkJCgsdm5LWqys7O1uLg4tRWk3crMeEs8q2lxp7SSgtW0WwnDvN1SUlK0ffv2qW1JokOHDlpISIh29erVGz7j/emEh4drkyZNyvF5lSpVtJ49e6r/33vvPa1u3boWz8HfoL/++su4j22WkZFhlWNuwoQJWqNGjW5armPHjlq/fv2M78eNG6dVrlw5R5lPP/1UCw0NzbEvMzNTu+eee7SvvvpK69Onj9atW7ebXsvHx0eV19utZs2a2gcffJCjTIMGDbS3335b/R8fH685ODho8+fPN34eExOj2draar/99pvFa6Snp2sBAQE3nNecgQMHaq1bt86zDPu8f//+6nxOTk6qvj///LP6bPbs2ZqXl5exrKU2GDJkiNayZUvj+0WLFmm1atXSnJ2dNV9fX+2+++5T45VjjmPL9KWPs+joaK1Hjx6at7e3OqZLly7asWPHbrju6NGjtaCgIDW+ydSpU7XIyEhVb9a/e/fuOeo2cuRIrUWLFlpZILuQn9O8vjNvRXYVi4UgCKWP1ETgyCrDdstsYGoTw/szO4C4E4YyGamG9/Gni7u2gqC4fPmymvmmZYLuMubczBfe2dn5jlNFWhv//POPccY+LxISEuDr62t8TwsArR0rVqxQ+f/Pnz+PH374AZ06dbrB6kMrAC0bN4PWBVockpKSlEuUzj333KOsKjExMepaf/31Fw4dOoT27dsbLQ3sl/vvv994THBwsHJfo/XFEjzfxYsXlWvRrdy3pZWaO3TooK5DS8e+ffuUBeZ21ymgJYXuWLQS0TpDF66HH35Y3TetPj169MADDzxgtLqwH7j4Gi1u7u7uqj///fdfNb5ZL1PLBF39eM6VK1di+fLlykL00ksvqT5inA2fjXvvvTdHfZo0aYJNmzYpVzPBOpAF8gRBKDXEHNmBhEwGeF4CFj4LNHwfoVWbwxOfAt92NxSq3QPoPgNIjAGmtwQcXIFBmwDvsOKuvlDIpKRn4eiFq0XezhHl3OHieHNB7siRI0pAq169+i2dPzMzUwmNdJt64YUXUBpdw/KCCgPdnExjTCjQ0kXmscceQ2pqqmojxjVMmTLFWOa///7DzJkzsWPHjjzPz3alIsHzUDimS1NUVJTqK/Lpp59iwIABym3K3t4etra2+Oqrr9C8eXNjzIyjoyN8fHxynLd8+fLqM0uwXlRMGE+RG+vXr8fChQvxyy+/5Fpm1apVSvCmwF61alW1j7E7twuVBbYllYnw8HC1r3bt2sbPGftBIZ/B9jocm3qbMGaE7cb7Y7wJFRNd4aKywTJsK7JkyRK1jy5jjK/h9erXr5+jPiEhIep6bEe9PkIZViwY3MQXvzhIzZo1VTAOtVjCwff+++9j+vTpiIuLw1133YWpU6eqcjocUNSSv//+e6SkpOC+++7D559/rh5wHR5LrZczAET/cjGd/Tl16pSaJaJfKx+MJ554AuPHjzcOcP3LhQFafEg5Q/Dcc8/hnXfekTzpgmAFxEYfwrGnemNrxSzMbG+HCi6BGD/8M+yb+hbupuKQfC1o0+Xac+8ZAjw8A1jSHzi1XhSLMgCVis5T1hb5dZcPbo5aIV43LacLqvnNJ//aa6/h7bffVr+D/K169dVX1e9SaYK/67TE5AYFU87qz5gxI4dswJl5/u5TpqCAToGY7cM4FAq1zDDVq1cvdRwF3LyoVq2aUj4Y+M64jj59+mDNmjUqeFpXLDZs2KBkDAq3nJUfOHCgChhv27Ztnv1tqa9pafn999+V0pAbjLFgrAfvr127drmWY70pD+lKxZ3CYHjKWVQm2K5UCpidyVxpMoUWGyrNevC9DhW1o0ePGt/znKYyF++L7UlFiFYQvh566KEci17qK0bTKiJYB8WqWOhBUZGRker9nDlz1IOyfft29QXBwDWmEmM6ND4UH374oRpoNInpA5QBUQzYonnSz88Pw4cPV9otB7Ju6qOSwAeVZjTCmYXevXsbA71o3qR5lOZQBlUxawS/OPjQ67MbzDrAa9Ocx5kRmjn5ZUZtmtcUBKF4OHv1LOJOH4FtUipq/PAj/OY+hPtjztAxBNqUUShXvhL+eH8EvJ/pC/egMODoQbh7XUaFKg2Q4l8blzM9EELlok4P6cJSDi0HFPKL47r5oUqVKkrQ5OxyfjLrUFDm75Ce9chUSGUGI7rJmEPhmHh53VzRsQYo9HNy0BIU7pkti3ICA7VNGTNmjHJRYhvpgb78vWaANmUJukZxUpPHm7oNEVodKGdERESo9xR2dTmFblmUAZiN6YsvvlCKD4OoacXQ3ax4LQr0nJykYsHZe7r88D5MBfDY2FhlWTFn9uzZSp7hJKglqDQxiLx///5KscwLXfDOL7Qs6Aqujql7HeUquirRteqPP/5QMhLvf+PGjahUqZLFc7JdaXXSg6x5fspdPFdAQICxnLn7H+W8bdu2KeWR16ISxWxnbH99Ypjug4Tym2AlaFaGaVBUYGCgNnbsWONnqampKsjoiy++yHdAFANReJsbNmwwllm/fr3ad+DAAfV+xYoV6hgeq/P999+rYCE9UOXzzz9X12YddMaMGaMFBwffUiCNBG+XPCSY1vraLTouWdu8/6i2eOlX2t0zG2hjn6qh/dk0Stt3+Kh2aNdG7fCOf7WzJw8Zyh7erq1tVENr+nlNrdbXtbRfm1fXFr/QQX22Y/UCbV+16lr0ugWaduGQps24T9P2/axpMduLLeBbxlvZDt5+4IEHbjt425RffvlFs7e3186ePZtj/8KFC9XvXWJiYokI3h40aJDFgGoGBbu5uWmfffaZxeMefvhhFSxsyrp169RvP3/rOS52796d48XrtGnTRv2flpaWa51YhsHGbK9Lly6pc1KOMGXAgAFau3btcsgqCxYsMH5+5swZi8HbPGelSpW04cOHW7z2nj17VBDzq6++quWHv//+W13n4MGDFj83D94eMWKE1rhx4xxlmjVrliN42zz4neOVQfaEQeKdO3fOUWb69OlKttPlKUvjLT+B83wmOKYXL15s3Ed50Twgv7SSXUKCt60mxoLaK9PB6UFRx48fVz5zpsFOzOfdsmVLpSnT3HuzgCia6eiDyJkZulHp3H333Wofy9DEyTI8hsfq8Fial3kNWilYhtdmHUzLMNUbZz1y09QFQShYYuJT0HbCGlSK/w8TVv6A8k/bokX4ZfgGaCj3zb3okDYWsfCBi8NVfNHbG35uleA66xt8FXxt5q7mabh7GWa3tNBATHzQFkNDIhBi7wyc3wssePL6xV7eIy5SQpFCV17OYjMolUGrnP2mTztniek6TGtGfuDvIl11evbsqXL98/eNqVnpOkx3IHO3FGuFv7NMu6rPcBPOYNM6MGTIEHTv3t0Yp0DLgh7ITEsEZ/TZZrorFD0c2K76bz1/903RZ8FN97/55pvKPZuxDnSfoncEr697QNAyRNmAlhFaB+i6Q0vK3LlzjYu3Ud5gcDi9G2iJYB3ZD3T9MXeVojs25R9LweR0f6I8wr4dNmyY8b7ZLrnN2LNuDHhmO7E+tLwcOHBAWbfoWmQOLSHjxo1T9acsxviIPXv2GGMbaJlgkDXrQGsD31+4cMHoFsb0r3TjosWH98p7f/LJJ9U56ZHCMc24CMpNP/74I0aMGJHDdd0UBnAfO3ZM1Z+WHgbi0/pBuU2HgeCmMqBQ/BS7YpFbUJSeKYHBTabwPXNO5zcgiltTU5sO95mWMb8Oz8lzm5bhA2N+Hf2z3BQLKiem2Qr0hVxoCjQ3NxY2+jWL+rolHWk362i37ft2INomHacOH8Ezh2YirPtTONPsWUw9MBpZbSYgzTsCF5x9MdM9BJeS0jHw263oO2ujOtbFwQ6f92oIPzdHIDAYtm6Oql4Ovn44UMEWpw9uRY22/YGBGw2xGClxQOx+ICMFiNkOuPoVmYIh461g2s10W5K+8/g7wwktKgMURCkQU2ikKwmVDtN7yeveKGxSwKNgTMGObjcUeimwUpjL7ThraysK9Q4ODkqx0rMs0T2aPvV0d+LLVIhmRiZCd2b+3nKNB7YjlQYKzXS/vtk9mn7O33e6TrMfKCRT0eMCeaYKAWM89Xamaw7bme5WnADVz0Whnn3CrEl6PChdnsxdjxj/QcWSAfzm9WTMBYV4uhSZrt3A61EZySu4nYoMszlx8pbKBdvN0vNCIZ3uVRwjlMu4fgjvn8oFy1AhZQzJ5MmTVfvy2nT5opLCz6kEUvGiyxjX6qCi1KpVK6Vsvf766yromwoalQv2B89nPqZ12N4M4Kb7E+tCV8F58+YZA+e5jzIjlTxrG7eFTWHcb17fmbdyPRutMGp3C9DvkIHTelAUMwJwAPI9/SPPnDmjfEd1OANx+vRpNZA4wDjozdOMMRaCvpH0fxw9erSK3aD2bAoHKL9gOdAZc0FlhV/CplCxoNbOGR8+bFQeTLNOMLUcNW1aM2gFsQQfCAagm8PrcaajKGFX80GnApff4EBB2s0axtvBg3twYfxIzGxvyJA99MdsePeohwoPDAPijgM+lQDH6wF9JPZKGq6kZCA+JQNjVuxHaqbBf5o429sqRcPW7gp++HQwWuzMQM3vf7jxwnuXAmsnA7Rm9JgLeOScgCgM5DktmHbjbwuFMAo+eQX/CgY4E0wh1xqh1YExkZyxtjasud2smYJoNyraHBfmK6GXZrILcbxRUaNsyokM0yB6oiuRjNu6mexa7BaL3IKimO1Cny0wVSw466JbCvITEMUyDNIyhz84puehOc8UnpNuVqZlzNPC8TrE3NphCl2laLI07RyaVKmJF4diQXhtUSyk3UrCeKPbU1xSOo5eTEZceibu8+uHyPIOqHxsCEKavQmUCzK8LGAam1qtQnl1HnI09iqGLtyBfRcyEBHgj/uf/BBO3c7h4sldOPbacFT+aAIi6lxbgbduZ8DdDVg6ALBNznnSQkKe04JpN/5IMhEHZ4lvN2d/WcNa24muWxRoaKWwRhcua203a+dO242u6QweL2vtb1dI98vzUmnhM2Y+GXMrv+HFrlhY+nGgBYLWAQrzNH/qvn1UImjN+Oijj9R7moZ1EynNi4TmSprsmFGK0M2KX0hMEUvfSkIlgvt05YNlaHbmsboSwwwEHLR6/myWoamTddA1OZahr6a5i5QpPIdpXIZpJxWHcK9fVxQLaTdrH29UKtpN/AcpGVnqvUud17GqdUuEJB8EHNMBV19eIF/nCvVxVS/i6+4EZwd7vLxwp+G8DnZYNbwNEmI24pBXErKvHEP6+kQEVa0PH/8QoBzTNGqGaxXRMyvP6Z23mz7m5Pvu5pg6LljjbwN/52+W/ag4sPZ2s1YKqt1KW2rl4h5veX1nlhjFIq+gKN4EA63oykS3Jb74P9PqMX1sfgOiGFBE3z+6UOluTHR9YkpaPQCIbk702aMfIQOM6CPJ8/AY3arAa9Klian9WO/Dhw+r+jD9mXyhCELBQwtDqnYJI7qEIMg2GW4x++GDend83hBvF6wa3lKd/0jsVby8YIf63y+8GuZ2dYfTzvGYPiUL297rh/seHwE4eQARbQxbQRAEQRCsU7Ggi5J5UBSVCn2xFwYPMciJC83oC+TRSmBqCp00aZLKOW0aEMXALlNTEYOcuFCOnjmAuaEZ0KXDsly5ktdhXIfpAnk6rB8tI1xEjy5bdL2ii5Opm5MgCHeO7v60Y+92PHZpLObt1FDrhIYhy7IR5+8O18YPAY2fNQRU34FywZcpQe5B+KnbT4hLugCt/ik0qHpthVe/CKD30ju9LUEQBEEo9RR78HZZgzEWVFLyEwBT0LCreV2JsZB2s9bxpqeSpftTZPIOfPT3tzg56iVUD/GB18IXEfLCz0B4U0O2JodbW/jJEntiEtRKzJMfq4fIAHf4uDkqhePimaPYPn8q6vccBP/AikD8SUOmKBu7Qs8QJc9pwbQbYyyYKYdutRK8ffO209O5igX+1sactNvtPavSbtbXbnl9Z96K7Gp1MRaCIJRdaKmgUmEQ9JvD54NhaEjLwpkdgHPGdWWiAJQKQkWCMRZ0h7oeb9ESCSf3o/zMXxHbtA38XR2BT69ZL9S1XYFBm2R9C0EQBEEwQxQLQRCsDloPaoUUfgam3OItbKtWxpMj7DE+8wJczp5BJS6Ux/UtLh4ClvQ3/F9E61oIgiAIQklBFAtBEKyKgPSTOH7ZBxePH0fqB+MROW4yIgN8AEcPwKbg83dbirfwcfKBi70L4keNw05ff1Sa949BkShf0xDITbcoWlGKcOE8QRAEQbB2RLEQBMFquHxqN+asmIKhIXZIdAX6etugdvJ5wK488GZ0kdXDGMhdcy+c7F2QePkcXD18Ye/gCLj5Az+9CBz6VdyiBEEQBMEEWS5SEIRihQHbDKLm61y6C6Y0a4xn7h6NrzpMQefalxHye39gUZ8irxeVi6h6bZGWGIeYZq1xaPMf1z98YDTw8AwgI9ngFiUIwg0wfTyDTOPj4wu0dVq1aqXS0d8JXPCWGSjd3Nzg7e2N0sLMmTONGTCFgoXLDTz44IMlrlnT0tJQoUIFbN26tUiuJ4qFIAjFhp4FipmZ+Brx6zn8FfIEmtVshyhHHwRnXTUI8I/OKbI6MdaCSg7rpggKwMQHbdXWiG9lwJ8L5wlCwUOhd/DgwahcubJaYJVrPXXp0gWrV682lqHA/uOPP95wLAVuCt6mwpC+4BUXmitfvrwSqGfNmoXs7GyL16dgyswzGzZsuKP74CK0ejp5a4Op6lm3HTt24NChQygNUIDk2lrvvPOOcd+MGTPQokULlSKfL67xxQWDTcnMzFQLEDIbENPtc9x98MEHuY4PLkzH8TR58uQb9kdERKhzlCtXDt26dcOBAwdylNm2bZsaX/7+/urFdcWuXr1q/Hznzp14/PHH1ZjnebgW2SeffHJDHX7//XfcfffdavkBXqt79+4qo5EpU6dOVcfzPFy3bO7cubAmKlaseEMbFgb8DuHabK+99hqKAlEsBEGwiixQywc3x3ePBeN9t2VwSDJxe6IAH1C90OtimiGKSg4VHqVceLpjT7gNog9vQ3pW+vUDPIOB+0cZtoJQQJw4cQINGzbEn3/+iY8//hi7d+9W6zu1bt1araN0O3CRWArRPPevv/6qzjVkyBC1UCyFSlNOnTqF9evX48UXX1Sz33eCo6MjAgMDrTKF7dGjR1U7c/HdgACTSQMTMjIyUJJYvHgx3N3dlSJhajWioP7XX3+pfuXMNQX7mJgYY5mPPvoIX3zxhVrfa//+/WrccbHgKVOm3HANKrMbN25EcPCN33tsz9mzZ6tzUPBnelReiylSyZkzZ5RiQ+Xjv//+U2Nx7969SvnV4aw6FYVvv/1WffbWW2/hjTfeyLH22LFjx5TS0qZNG6UY8loXL17Eww8/bCwzbdo0ddzIkSPVebjAMZ+fn3/+GaWN9HST36VcePLJJ/Hvv/+qvil0uI6FUHQkJCRw3RC1LWqys7O1uLg4tRWk3axhvO2OjtfCX1uu/fXPam3v9pXa779P03ZXr67t/e9nTUuO07T9vxi2RUR0XLKq09Jt0ape/P/MlTPa0FfrafuqVdcuXjmf84Dky5oWs93wijtVIHWQ57Rg2i0lJUXbt2+f2pYkOnTooIWEhGhXr1694TPenw5/R5YuXXpDmSFDhmgtW7Y0vu/Tp4/WrVu3G8qtXr1anWPGjBmqzTIyMtR25MiRWs+ePbX9+/drHh4eFuthyokTJ7TOnTtr3t7emqurqxYVFaX98ssv6rO//vpLXUOv9+zZszUvLy/tt99+06pXr665ublp7du3186cOWM8H+sxePBgVc7X11cbMWKE9tRTT+W4B94f71MnLS1Ne/XVV7Xg4GBVhyZNmqhr50Z4eLiql/5iG+ltOm3aNK1r167qPO+++66WmZmpPf3001rFihU1Z2dnrWrVqtrkyZNVeb3d9DYeNWqUFhAQoOrOduRnr7zyiubj46P6dObMmTnqER0drfXo0UO1He+V1z1+/Ljxc95D48aNVV14zmbNmqn2zo0uXbqo6+UF74f9OmfOHOO+Tp06qXs05eGHH9Z69ep1Q315H3v27FFtOGnSpDyvtXPnTtWmR44cUe+//PJL1T6sgz7etm/frsocPnw41/MMHDhQa926tfH9okWLNHt7ey0rK8u4b9myZZqNjY2Wnp6u3jdt2vSGtuCYueeee/KsM++tY8eOqo3c3d215s2bG+tv/ixZaoO6detq7733nvE9/w8LC9McHR21oKAgNbb1MWw6Bk3F8f/++09r0aKFGm+hoaHqGD6H+njjdf/3v/+p+nh6eqrng8/AoEGDtMDAQM3JyUmVGT16dI66tWrVSnvnnXdyvfe8vjNvRXYVi4UgCMUWV0G3IxL/2fvY+cZLGH52Kp5+yw0+5Z2BuBNASEPApej8n5kdimlume7WNNZi+EvfQ5s1DvEnDmPFoy1wct9Gw4fH/gamtzS8pjYB4k8XWV2F0sfly5eVdYIzq/T9N6cgYwE421u3bl0sWbLEuI+yNWece/XqherVq6Nq1apYuHBhnudhXemC888//yjrCme/OWueG8nJyRg/fjy++eYbdQwtJHTT0OHx3333naoHZ7W5MJclly9T+vXrp8rOnz8fu3btwqOPPqqsNIcPH7ZYfvPmzerzHj16KEuOqavNe++9p2bDeS9PP/20cgcKDQ1V7bBv3z7lavTmm2/e0C60MHFGnvc0ceJENVNOixDdjzjD//zzz6vX6dOnje1AyxHbisesXbtW/c96cQaaliT687ds2VLdE60NdBvKy/rDGelGjRrl2Va8Li0xvr6+xn3NmzdXbna6SxjdkVifjh07GsuwHXr37o1XX30VNWvWxM1ISkpSfUj3Kro1EY4TWrFsba+LnnRTIrxebnBRNtP68h7pqsfz0xrCzzmeaB2hu59+LfNF3ngtuoHlZomiFefee+9Vx7E/aT3hGDC36uWXH374Qbncffnll2oschzXrl1bfcbnjuOKLmccg3wRjrv27dsr6wv7fcGCBaptaEE0hRalWrVqqTrS9e3TTz/FsmXL1Lg8ePCgsvjQ1cqUJk2aqDFS2EhWKEEQihTT1bVttXR4O1xB8NCRcHfOwgJvO/jM74Wgb7obCt/3LtBieLH1kK74+LiHI6pZdezb+ScupVxCUkaSoUDlVsCANbK+RUniyjnDyxQqrz4VgYxU4EJOn3BFcD3D9uJhIP1a3+t4VwBcfYGki0CCWeYyJw/ALyLfVTty5IgS7inUFwW8DoUXnVWrVinBk4INoYJBdygK7rlBxYD+7brARP/8vKBQR7cbusMQCkwUrnTofkMXloceeki9pwvMihUr8nRp+v777xEdHW10z6GiQgWNgufo0aNvOIauNvQ7p6BJVy1TnnjiCSVMmkI3Gh0KyuvWrVMCHBUYHQq+FO4oNNOfn+5EbEsqIYT3NHbsWKUA9ezZUylBLPvVV18ZlQXWl8oj3ZcoPFNgpnKitxXjBXKDAfJ8WXJRMuX1119HSEiIcknSoe89r8XxQIGdwvqoUaOUC5Wpwmdvb4+XXnopz/N//vnnGDFihFIseL6VK1cqZUJXZocNG6aEYvY7V3rW20cXrM2hQsW2/uWXX4z7KDD/8ccfqv0Z18H6Nm3aNMc44Rhm21I5a9CggRLAGVfE8Ue3qaCgoBuuxZgMxgOxb3QFhcr17XLq1Ck1vtjWPB/d0Cjc6+OFbc0YEdMxyLbhGNSTE9BVj+OKCibbVq8X29JUIee1WJZKIsdTeHj4DfVhv9MdsrARxUIQhGKLq3CN+Q9hr4yB1nEcoup1NqwNkXotYJuxFR43fvkXBbmtyI3QQHz8qB0WhF77IXDxMby4EnhgnQJbEVwoRLbMBtaMzbmvdg+g+wwgMcZgfTJnZIJh++MLQPTmnJ89NB2o+xiwdymw4voPvYJrnvRemu+qGbxxDIHZRQGvZ3otCl6PPfaYEiAJBUvOUHMGlMKyJShovvDCC0rQowBFJaNOnTq5XtPV1dUoKBMKeLGxsep/Crfnz583Cl+Ewhd993MLJGYwMO/DXADkjLWfnx9uFUsz/lSEKKSePHkSKSkpyqJQr941ZfManMU3nYlnkDxnlE3vg/XR75WCLhVJCpamUNimssTZd8YeUEBmsD3blhYWSwIxYb2I+Sy9KVR2qIRRcTEtx1lxznDPmzdP3QfjFijYUknp06ePqiutOmzrm41N+vKzvlQUaJlinalM8Xo895w5c5RyQYWCbcLxw7bi/+YwNoLWI1qJeE7T5AbPPvusqhvH6JUrV1SZRx55RCkyrCNn8VmOAd4cH7wG25NtYOlahPfN+BRdeL9THn30URWcTWWblihagJiEQX++LKGPC1rtdFh/jn8Gp+vj3Hyc8t7YRnxOeS0qpObZwahIU9ktbESxEAShyLHRMpVSke2djE+62uKloHKGD5y9gKhuQGhjwLdSsfVMrityO2mwzdb4TZ/zgHLVgOcL38QsFACN+gHVOuTcp7vbeYYYLFC58eA0yxYLUvMhw7g1t1jcApxxpFDEAMubpbWkQEpB3BzOWuc3CxOvwxl43Q2Lrhqc0WXgqw5ng6lwcMbaEhTwKPxyRpnKxZgxYzBhwgSV1coS5kIb71dXqEz3mWL+uSkUuCgoUiAzFxjzcsnKDXMXNM6WDx06VN0TZ8XZ7pxVpnvTze7L0j5dQeKWCpOpAGlqUdEtGBS8aX2h8M/MTRScKSybQ6WF54+Li7N4XxTyab2hVcpc8aPySEsGLSmE1icqUexLCu90n6FCxBl303ExfPhwJTibzoJz7PHFscx60hVs6dKlRusHZ+P5P93GPD09lTJG1zF9HOrQ7Yyz8v3791f3bW5Z4LFUEnSoGNHliv3C61KI5rilGxKVVSpk06dPV/3HbFSW0N2y8gvrbj42M0zcrFgfKuXsM7b7wIED1dhZs2ZNrsoLxwWtMJYsQ7pLmaVxSqsMFQ8GxPNaVOiojNIdS4fPuD62ChNRLARBKHLstUyEvfIepnSxxbZ6bvApd+0Hi8pED+tICWhpRW4cOo75H2VBq3Yc8L+5n7FghXgEGl6WcHC+7vZkCf8quX/GhRP5ugPoHkEhnYITBQtz4YFKgx5nQTcTxgpQ8NOhkEMBu0MHM8XJAvQhpz83hWbC2Wr6fJvHM9D3ngImXWNym2mlwKPHENDlhylOc1Ms8oICKWeW6QevZzaiALt9+/YbLAQ69evXV2Uo+JpmQyooKFQzbS6FQh1aFO4UCoJUFpiRikJybvD++GK7UrFhP1lSLOhuFBUVpQRy85lqCrMffvihyp5kySLDWWxTawuhkqYrQYytMHWdIhyn3J+Xm5w+Jmk9Mke3UlB5ojXD1CJBSwWVCo5tjjtL9TVXIvX35pYtCvAc14QuTpzJN79XHSpctKhQOciP1YJCuqkLF+OBjpulvKWy0rVrV/ViPBKfWz537H/2mZ4xS4f7ef+RkZEW29K8vCkcR7Q48kXrDS0XVCb0+JQ9e/aosVTYSPC2IAhFTqaNPU6Pfx8vDpylVrhmgLThg3QgIcawtUL8K9XAsUEd4eTpg33rluOM7lN/difwv3KGrSDcAfSjpvBAdyCmD2XQJy0L9LOmYKlD/2rGPzAGgUG3DLil3zqFXvO0tBTs6BbC4FS6s3Dmmi4mFLKeeuopVYYCHt2Y6L5j+mK8ARUaUx93U+gyQ4GVAhXPTYUlr1iAm0GFhIrMTz/9pGZ7mRaXs/C5ueDQNYTuN7wPBsSyHlS4aGHJKzYjv1DA27Jli7pHtjNdbHj+O4V15sw5+4HKC+vNmWzeL+NF+J7KBGMMaD2gNYjXz6ttKeybB0FzVp8z/py9Z2wCxwFfpmtH0D2HAjz7mNYHWhhoRdDjXGgNMR8XFLwZG6C7yDEFLPuNiq2espiz5hSsTYPAOV45TngvVKA5ZnmcrjBTqGZQOxUNukzp9b1w4YLxHJ06dVJ9wNgcPh88HxUcxhXogjPPTysGP6eiSmsMBWtLMTc6rAuVA5Zln/NYBoVzHFqCyg8/Z//x3FSE7EwUnq+//lo9o/yM7cOybA89/oH9wcB9PpeM+9DjXdh2fIbpmsU6MCj7Zoo6g8SpOHHdEN77okWLVP+YJnxgPYtk8cSb5o0SChRJN1vykPSfBdNueirXuT8t02Z1bKitXrPyxoOYtvU9T8PWStBT4jIFLf/nfayZM1aln20xs5FKR2us984Fhv/Tk2/7ejLeyna6WcL0q0wdyZSRTFPJFJ9MRWqeQnX+/Plao0aNVMpJpvFk6tYtW7bkKMOUlHo6S6boLFeunNa2bVtt1qxZxnSdmzdvVp9v3Lgx1zSmfFnixRdf1CIiIlSKS567d+/e2sWLF/NMN2sKU+aaiiJMp8lz8p6YpvW1117THn30UZUCN7d0s0wxytSwTAnr4OCgUm4+9NBD2q5du3JtY6YN1dPM5pXCNzU1Vevbt6+qN9PCvvDCC9rrr7+u0oqap5s1xbyOltKTnj17VqUK9ff3V+1XuXJlrX///kpOOHfunPbggw+qFKUcAzyW92iaYtUcpgh2cXHR4uPjc1zTPK0pX6YpURMTE1VdK1SooFKcsh5vvfWWSmGaG+b3EhMTo1IlcxyyD5gm9YknntAOHDiQ4ziOD6bW5T3VqVNHmzt3bo7PWS9L9eX1TPn++++1+vXrq5TFHHd8Pnj/Onz269Wrp9qDY4n9Y16X3FLk3n///SrFL1POMu3r0aNH1Wfm/cx+Yrpgnp8pZb/++usc6WY5lu666y71Oet59913a6tWrTIev379etUG7HvTZ2DTpk1au3btVLpbHscyTGVsmm7WPM3t9OnT1f2yPK933333adu2bTN+vm7dOjV+k5OTCz3drA3/FL76IuhQG6a5l76xeZk/CwN2Na/L61vjgkXWirTbnbfbmYRUYyaokIy9GHjga1R4awJaNLvmskHz9bld17Mr0c89L5eUYspipQdyL+kbidNHV+GVmM8wv+tCRNl5GNLNZlwLjBu4AQi4vVlbGW8okHZjECxnfem7nVdAq3DdxYKzrdb220DXFs7Sc/b7f//7H6wJa2w3tpPuOmWtWGO7lQS0O2g3BpJzXOhZuCyR13fmrciuEmMhCEKRZoKycy6HNzd9gwXVTNLhZWdcz8bj4Aq43no2l6IM5M5yLofQoGp4aVo20Og8UDsKGLQJSL5kOMjWHvjpRaDliOvBvYIg3BTd7YfpNenCRdcZCjsM+hVuDuMp6DojCDp8jrhmjR5PVdiIYiEIQpHBhedsnU0ytVw8Avw8BOgy+Xo2HioV3tezX1hrILejZo/ADFe13b/+F3iGVkZI2DUrC9Pmbv8GaPysKBaCcAswsJa+6Ywh4Qwt/fmZ5eZO4jbKEvTfv53AeaH04uTkdENmrcJEFAtBEIqWg8ewcEwmtCrHgIqhwMm1hhSeVuL6lF8i67RA5PIt6v9dNWvgs3ZOeOnDX68HoguCcMswwxTXPRAEoWQiWaEEQShSyleqiTMvPqi2JRW6RO2JSVDxFyR28lCsqZaJuDTLOeQFQRAEoSwgFgtBEAqN2CtpOH01AUcvJMHGPh7H4/bBNvES6vV4Hn4B4QaXoRJEbityB1etj8h/NOBqEqCHh9hcm7dJuwKkxFude5cgCIIgFDSiWAiCUChwNn/gt1txNCEbsE+AW8QEjFqbjtmTs7D19cdxf993S1zL57oid8x5vLEoG1qH80D4tTiRe4YAXqHAkVXAor7Ay3tEuRAEQRBKNaJYCIJQKFDgTs3MxqQe9eCUuBOnp6SiwtAR0Gb5o2GAv8FawQxQXT4FvErObL6lQO5K9Vsi+uc5CA29FmBK60SrNwB7J8CtnGEfM0aJ1UIQBEEoxYhiIQhCoRIR4A43By/YprqjilckKoeFX1/zgUpFwz4lvgdcXDxQpUqTnDupVBBHkyxYgiAIglCKEcVCEIRCp3Lt5qj8yybDG1oqqFQ8PAOIuK9UtH704e3YPep1eL80CF7hkcDhE/AKCkdIeMkNUBcEQRCEW0WyQgmCUOhka9lIz0pXWyP+VQE361kI707gKqiZp6MxfOObeGz5Y0h6fjg2fz1O3TPsHACPYMNWEEoZXHPC29u7WK7dqlUrvPzyywV+3pEjR6JevZunv37nnXfw/PPPF/j1hcLr28ImNjYW5cqVQ0xMDMoqolgIglDoHNjwK/bWq6u2yjUovHmpchEKiayHBnMW46seC7Gg8wK4fTEB9zz7NhztHIHyNYHh+w1bQcgH586dU4ucVa5cWS1uxbUdunTpgtWrV+dQZn/88ccbjqUwRqFMp2/fvqosXw4ODihfvjzatWuHWbNmITvbRNE34f7774ednR02bNhQZP1VsWJFTJ48GSWF8+fP49NPP8Xrr79u3DdmzBg0btwYHh4eCAgIwIMPPoiDBw/mOO7q1at48cUXERoaChcXF7Xw37Rp0yxegwsEdujQwWJfd+3aFRUqVICzszOCgoLQu3dvnDlzJkcZjpdmzZqp+rDMa6+9hszMTOPnf//9N7p166Y+c3NzU8rUd999d0M9uI8rN7u6uqqy/fr1w6VLl4yfZ2Rk4IMPPkBERISqD8v+9ttvsCZye14KmoCAANUX7733HsoqolgIglDg2aC4xsPR2Kuwz07Dme1/4FTWBXzf0hYo5wv4RwL9fjFsSwH6mhZwD0eUX5R6hVVvhNO71uFKfGxxV08oYZw4cQINGzbEn3/+iY8//hi7d+9WQlrr1q0xaNCg2zrnAw88gLNnz6pz//rrr+pcQ4YMQefOnXMImuTUqVNYv369En5nzpxZQHdV+mDbNG3aVClEOmvWrFF9RIVs5cqVqm2ppCUlJRnLDB06VPXnt99+i/3796v3VCJ/+umnG65BRYsCsSXYhwsXLlSKy+LFi3H06FE88sgjxs937dqFjh07qr7fvn075s+fj2XLluVQhNatW4c6deqo41n+6aefxlNPPYWff/7ZWGbt2rVq3zPPPIO9e/di0aJF2Lx5M5599lljGa7q/OWXX2LKlCnYt2+fsuI89NBD6rqlDSpRN6Nfv35KGYuLK6PrGmlCkZKQkKCx2bktarKzs7W4uDi1FaTdCoPouGSt+tu/auGvLdcqvvaz9vDImdre6jW0J9+N0hp/21g7c+WMpmVlaVpGqmFbSu6VL/7PfWTvfz9r+6pVV1vt3B5NG19d0/Ys0bSY7ZoWdyrP88pzenuYt1tKSoq2b98+tS1JdOjQQQsJCdGuXr16w2e8Px3+jixduvSGMkOGDNFatmxpfN+nTx+tW7duN5RbvXq1OseMGTNUm2VkZKjtyJEjtZ49e2r79+/XPDw8LNbDlNmzZ2teXl6qLlWqVNGcnJy0tm3baqdOXR/nR44c0bp27aoFBARobm5uWqNGjbSVK1caP2d9WRfTl87atWu1e++9V3NxcdG8vb21+++/X7t8+bLxuMGDB2uvvvqq5uPjo5UvX1577733ctQvPj5e69+/v1auXDl1P61bt9Z27NiRo8yYMWNU3dzd3bWnn35ae+2117S6devmed+1a9fWpkyZYmw3S8TGxqp7WbNmjXFfzZo1tQ8++CBHuQYNGmhvv/12jn2sY2hoqHb27Nlc+9qUn376SbOxsdHS09PV+zfeeEO1syk8h7Ozs5aYmJjreTp27Kj169fP+H7cuHFa5cqVc5T59NNPVd10goKCtM8++yxHGY65J5980uI19PH277//5tm3HMs6ltqA447jj6SlpWmDBg3SAgMD1RgMDw/XRo8erT7j/6Zji+91li1bptqfx1SqVEmNf9bN9LrTpk1T49fV1VV79913VR2feOIJzd/fX7VnZGSkNmvWrBx1q1ixojZz5kytIDF9TguDvL4zb0V2FYuFIAgFmmI2JSMLkx+rh2UvNscHve6D9tVHePOZOfip208Icg8Czu0CPgwwbEsw+poWywc3V/fL++b9KyqGYvBzdmoLJ08gNd6wlsX0lsDkWkD86eKuvmCFXL58Wc1mc9abrinmFGQsQ5s2bZTLypIlS4z7KEfNnj0bvXr1QvXq1VG1alU1K34zkpOTMWrUKMyZMwf//fcfEhMT0bNnzxzuP5w9X7VqlZrFbt++vXLtonWEsA50DaI7DS0rfJEdO3bgvvvuQ82aNZUVhbPnPC4rK8t4bl6TbbVx40Zl4eE5aC3Q76dTp07KtWzFihXYunUrGjRooM7Jtia8P7qtsP5btmxRrj6ff/55nvfLmeg9e/agUaNGeZZLSEhQW19fX+O+5s2bK8sBffBZv7/++guHDh1SbWLano8//jg+++wzBAYG3rT9eS+cIafbE93dSFpamnJLMoWuV6mpqaod8qqzaX15zujoaNV+rC9dwH744QfVrjq5XYv9lRvs27Zt2+bZt7cC3dLYrroVhxYh3ZpECwvh2ObY0t///vvvaqy/9NJLytJCqwtjhjgWTOH4oMsYrYe06jC2huVp/aPVia5s/v7+OY5p0qQJ/v33X5RJCkXtEXJFLBYlD5lBzj+7o+PV7D23xw9u1hYPf0Jtc8BZ+/c8DdtSeN9k78W9WtPPa2qr/v7aUIBWCt7v8X8NlovU3GcMZbwVrsUiNilW9Y/p63TiafVZambqDZ/xpXMs/tgNn8WnGvr8UsqlGz47kXDilu5h48aNalZwyZIlNy17pxYL8thjj2k1atQwzoT+/vvvamZfn7GdNGmSds899+RZD84Ysy4bNmww7qO1g/t4P7kRFRWlZvx1OIvM65ny+OOP53l93mfz5s1z7GvcuLGyOOhWGU9PTy01NTVHmYiICO3LL79U/zdt2lR7/vnnc3x+11135Wmx2L59u7q/kydP5jqDzH1dunS5oX6cWX/qqafU8fb29pqjo6M2d+7cHGUGDBigPfPMMzft6xEjRqhZdH5+9913axcvXjR+xr60tbXV5s2bp2VmZmrR0dGqLizLfZZYtGiRqs+ePXtu2E9rDuvL4zl7r1tG9H5ifx46dEjLysrS/vjjD2WF4LkswbahVexmfXsrFgtartq0aZPrbL6l41u0aGG0auh88803ygJjetzLL7+cowz71dSqY4mhQ4dqrVq10sqixULSzQqCUCgkXbkE+1Pn1dZIZjpw8VCpb3FvzRWPrbNF9vzxwJo+hoXxZHE8q2DRoUWYtjNnsGynyp0wtsVYnE86r7J6mbO7z261ffu/t7HrQk5L2+jmo9Elogt+P/E7Rm8cneOzZsHN8GW7L/NdN4McYwg0LQp4PdNrMaD7scceg729QTTgrPmrr76qZoCrVauW63lY3nT2ntYOWlc4m8uZW8YYvP/++1i+fLkKMGbsQUpKitFikdes9qOPPppnGcYImEKLAzPzEM7M01ri55cz+xyvzZgEwjqaZ3Zi7AQtCbnB44n5LL0pjFFh3IL5rD1n1hmDwdn18PBw/PPPPxg4cKCqN2fwuZ/xNfmJT2DfMPbh5MmTqn0ZC8E2Zp8ytmPcuHHq3hhMzCQAnGlnfRiYbw4DuRnoP2PGDGVF0OHMPGf03333XWVV4Yw/r8vz6jE4n3zyCfr376/6nddmEDfjDGghyI2dO3fmiAm5U1h3JiXgOGVcCeOH2AZ5wfFB64WphYIWE1p1aDVisDoxt0y98MIL6N69O7Zt26auwSB9WnbMLTbJyckoi4hiIQhC4RAeiqld7TAxPPT6vqQLwJL+hhW3XUtHqllLBPtXxCPvzcXVxIs4uPkPXBo0FH5TJ6FatVrApulAkwGAZ3BxV7NM8mjVR9Eq7HrWJOLp6Km25d3Kq6xeufHhPR8iJdMgVOqEuIeobfuK7VG3XN0cn7k53OjOlBdVqlRRghmFXQorecFMP7qrjSnx8fHw8vLK1/V4nUqVKhndaZg1h8GpplmKKGhR4fjoo4/yPJclZUjfR0GUbifjx49HZGSkErooVKanX3MdzAWWuxm664/pNfVsV9xSYKfQXJBuZbrbC12iTN2GdBiMTQWBSgNdvEwVkjfffBNLly41uhJRMaICxbahYkGlgkqPef0oyLZo0SLHvbAefNFljdmlmD2MSgsVIzJs2DAVHE5lwMfHRwXvv/HGG8Y+Nw06pxvSxIkTlXJiCjNd3XPPPaoP9frS9Yx1+fDDD1X7Mr0qxw4FcmaLCg4OVkHi5tcxJS+lzBLsV13xthRITRe348ePK/ckutz16NFDtSfdtnKD44MK2cMPP5xn/czdEpmpi8rcL7/8oq5F1zq6L7IPdfg8sV3KIqJYCIJQ+Fw+Dqx6D2jzDjBgjUGpKIUz+MwQpePjXw3VIuth38H/sKqOhkc83YCrscDaSUDUg6JYFBPlXMuplyWc7JxUVq/cqOSVu6Dk6+yrXncChVTOCk+dOlXNEpsLNFQadIGTs8Ocbe3T5/rK9RS8OAtLwedmUIClzzgFTzJv3jwlBJun5GTKUgqXnNXVLRnm0ALB+ARaJwgtHKwr60joa84ZZWYKIrQiUMg1xdHR8Qb/egqxvD6Fv9uBwibjK1hv0+xNplAgpzBuKlDfLM0uZ+Q9PT3VbD7/N21/KhVUHKgAmAvWFIT5srXNGd5KC4KuDFEgN824RGrXro1JkyYp4T83dKGb8Q7mAjkFffL9998r5YPtosN6cnafiuOAAQNuOC9n3c37Xbd4mAv6FMZDQkLUPTLTFIX73OA9cQwyJiY/UEjXY2/I4cOHb7AIsE9oceOLiistFxTw+VxRATUfX2wHjlUqu7cK68MxzReVLCpeporFnj17cqR9LkuIYiEIQuFw5ATemZcFRJ4AKoQA+34Cmg8Dgm++8FRJw8fNES4Odnh5wQ7jPr5ncDf8fbCohR0e8fcB0vKeoRUEBg7TrYJCOoUuCtcU3BmQTEsCrQzklVdeUUoFhXe6Y3A2fPr06Wq22zwtLYVNCtgUrBh8ywBxKgsUKHWBmm4rnBWvVatWjmPprsP1Dzg7ywBWS1Boo0BNNx/+Tzegu+++26hoUHBjgDYFYwq6dMkxX0ODgj9n+Bn0TbcdzsRzdp0CKF2F6HpD5YMuSnSPMg+WtQRnrDl7T+sPBWe6ydAVi4HI3EcXF6bdZTvyfwZWMwiaaVW5hkhuUDHgufWAYx22OxU0po6lRYltTmhBovWFgm/Lli2VEMr3bFtaC+bOnausBYTB2pYCtrlmha6obNq0Sb1YX1oijh07plyVqOTo1gpCVygK16wv23/s2LEquFlXDKhU0HLCNmDf6/VlO+uWGN4f3Zw49nRXKK6Vwr7VFRYGzjMYnetgcMsFBtm/I0aMyLUNOabq16+f775lsgEGs3Nc8dw83tRaRcWL1hPWgffLtLhsR10R5/iikkrrC8cX241txmeAyhavy+PovkaFm9aY3OBxTAlNlzE+W3Q/o4Kqk5ycrBT80aNzukaWGQolAkTIFQneLnlIMO3NYZpVBi4v3RZtDGI+d3K/tvzj4WpbGgO2c2sD83bYe3qr1m10TbU1tsPh66k2zZHxVrbTzZIzZ86o1JkMaGYALNPPMmD2r7/+ylFu/vz5KqUoA5SZLrV9+/bali1bcpRh8LaeZpPBtwzOZjpYpsdkoC3ZvHlznsHWDFblK690s4sXL1ZpSVlfBtGeOHE9cP348eMqzSsDesPCwlRqUvPg3PXr12t16tRRaT9NRZO///5ba9asmdrPlKS8Rz3trvk5CAPVec86TK3KwN7g4GDNwcFBXZ9pUE3T4Y4aNUqlDmWAMo9lUPTN0s3+9ttvql8YjK2POfOUufpLDzAmTB/bt29fVR+mKq1WrZo2YcKEPANyzQOPd+3apdrT19dXtQtTmzIAnQHaprAM+4bXYUD6ihUrch0bpi/T4H89vSyDs9l/DGxm+5lei33EJACsi5+fn9a7d28tJibmpkHIHM/57Vuej+loma6YaY15L6bB29OnT9fq1aunPufzcN9992nbtm3LkVaWaWH5DJimm2U/sg68Nx7XpEkTda7c2p7873//U/fLY9gHHHPHjh0zfj5v3jzVrwVNSQnetuGf4lZuyhJMw8fZC/rGcvaiKGFX87q8flEFB5YGpN1uviBe2wlrVLpV4uKSiGmPV0a5C7HwrBCF0NDKsDm705BqlW5QpdBiYQ4XzOs8Za1KRWt78l/YPP0qtFnjEBVVF5jaBKjVHeg4Dog7AfhUBByu+5LLeLs9zNuN/t70ueYs7636c5fFtqM1gzPZ8tuQ/zbj7DktNU8++aS0m4w3I02aNFFWnSeeeKJEPad5fWfeiuwq61gIglBga1d83T8SbhETMXr5M7B59jUc3fVPmW/dsNpNkTzzQ7VVcSWDNgEtRwAXDgKf323YCoJQoqBgx3UPzFcuF8o2sbGxKr6DGdXKKhJjIQhCgRAZ4A7bxGN47Pck1H5uBLS7/BBR4VogrEcQcN+7hm0Zw8PDDw3v6X59hx60nmxYoEsQhJIJFxg0j0kRyjYBAQF5xpaUBcRiIQhCgeGYmonGJx0Q6RaOqGad4YFM4MwO4MpZoNEzgEf5MtfaZ4/vwe8vdsfGXb9i36V9iDm5t7irJAiCIAiFglgsBEEoMCLrt0bkPzvV/1rcKWDhU8AVuvpoQM/vgeody1RrM/3s5dR4OG3dj6H/GfLAf/hNNjB3HkIMay8JgiAIQqlBFAtBEAqH5EtAZirw0HSgXFVDkHIZwTz9bGDLoRjboREc46NxaecX0Bh4x9g7O0c6axviLLhoYClc20MQBEEoO4hiIQhCgbF/8++4+OJQ+H82CdUrVDAoE4G1gfLXc3yXBUK8XdQaFgxsp9Xi5QWAv0Nl1KpXH/jCZJGrdy4Ytl+0AC4dMQR2e5msVC4IgiAIJQiJsRAEocDQPNywpqZhi4DqQI85hm0ZhMpFrRAvFdSuk3w1Hoe3rlbbHDBLVEaywcojCIIgCCUUUSwEQSg4yvliQUs7tRVu5MSutch88kW1zYGXuEAJgiAIJZ9iVSzGjBmDxo0bw8PDQ6XoevDBB3HwYM6c7n379lX5ok1fXJTGFC6pzkVquAy8m5sbunbtiujo6Bxl4uLi0Lt3b7XAB1/8Pz4+56zhqVOn1PL1PAfP9dJLLyE9PT1HGS713rJlS7i4uCAkJAQffPCBWrREEMriwnhcCI6uPkZS01DpnKa2OLsLmNXBsBUMhAXhrd52aisIgiAIpY1iVSzWrFmDQYMGYcOGDVi5cqVaaOb+++9HUlJSjnIPPPAAzp49a3ytWLEix+dc4XDp0qWYP38+1q5di6tXr6Jz585qhUIdroC4Y8cO/Pbbb+rF/6lc6LBsp06d1LV5Dp5r8eLFGD58eI6VB9u1a4fg4GBs3rwZU6ZMwfjx4zFx4sRCbSdBsNbVtrm6NAOUw7JPI+PSYeBkDD6anaW20LIN7j3cCgZcXXDBG4g5tD1ni/iEA49+bdgKgpBv/v77bzXhaD5ReDtcunRJTXKeOHFCesCK+6moeeWVV9REs5BPNCsiNjaWU//amjVrjPv69OmjdevWLddj4uPjNQcHB23+/PnGfTExMZqtra3222+/qff79u1T592wYYOxzPr169W+AwcOqPcrVqxQx/BYne+//15zcnLSEhIS1PvPP/9c8/Ly0lJTU41lxowZowUHB2vZ2dn5ukeei9fVz1mUsI5xcXH5rqsg7ZYbu6PjtfDXlmtLt0Vrazat19Y0qaEtG9RNu5p4Sdv738/a1YsxWvaOBVrc2DpadvS2Mj2UTNvqryMHtA8G1NH+a1A91/LynN4e5u2WkpKivvu5LWmcPXtWe/HFF7VKlSppjo6OWmhoqNa5c2dt1apVxjL8HVm6dOkNxw4ZMkRr2bJljt9QQ75naPb29lpAQIDWtm1bbebMmVpWVpYqwzbLyMgwtl27du3U7yF/J62dv/76S90b+/5OGT58uPb0008b3+/YsUPr2bOnan9nZ2etevXq2uTJk42f6+3266+/anfddZfm7u6u+fv7aw8//LB27Ngxi9dYu3atZmdnp9WtWzfH/sWLF2sNGzZUMoarq6v6fO7cuTnKJCYmqv6tUKGCqk/Tpk21TZs2GT9PT0/XRowYodWqVUudIygoSOvdu3cOuUYfX7169dLKly+vytWvX19btGhRjjJbt25V44T18fX11fr3769duXKlQPrJfLzdDu+9994NbVhYnD9/XvVtbn1aVGQXQLvlRV7fmbciu1pVjEVCQoLa+vr63qDpchahatWq6N+/v1oyXWfr1q3IyMhQlg4dWhS4Gua6devU+/Xr1yv3p7vuustYhu5U3GdahsfwWJ327dsrNyteQy9DNygnJ6ccZc6cOZPrDAePp6XD9EX4uyAvaYOSPAZsoCGinBv8Knngjb528Hv2Wbi6+6BG005wTT0Pbelz0OxdoLn4Fntdi/Pl7eoAVwdbDF2wHf1mHMZvPoOQ9fFYHN72F1a1baC22pXz0P77zLC1gjqXlldJ/K49fvw4GjZsiD///BMfffQRdu3ahV9//RWtWrVSFv783JvpZ7rVn79TPDct/jzXkCFDlGWfv5+mnDx5Uv3W8VpfffVVodwjXYytrZ+Tk5Mxc+ZMPPPMM8Z9W7ZsUW7R33zzDfbs2YM333wTb7zxhvJW0K977Ngx5cbdunVrbN++XXlEXLx4EQ8//PAN1+Bs/VNPPYX77rvvhjr7+Pio81Mm2blzp3ID79evnzqfXubZZ59V3h1z585V44IeFG3btlWu3/ycHhfbtm3D22+/reQWel0cOnRIuYebXoveGnQ7/+mnn9R5HnroITz22GPqWH4eExOjzhsREaE8Sjj+9u7dq+pUEP1kSnH3e37GYrly5ZSMOW3atEJ5HrRbuN+CaLfbbdMSl26WlR42bBiaN2+uBHydDh064NFHH0V4eLj6UnznnXfQpk0b9dBQwD937hwcHR3VQ2lK+fLl1WeEWyom5nCfaRkeYwrPyXOblqlYMWcufv0YflapUiWLcSTvv/++RSXqVjqqIOD16CZGaJIUpN1ul/TkqwhxM2xtstLg7hEMZ59gJEQfAnbOA2p1h/bEb7iqOQM2nrC5NmlQFnG3AX4c0ABXUjJwOi4Z4/8AXMPq4cqVgzhUzRe+WjYSzp0A1s4AAhpD83eU57QAvt8oMGRnZys3V1O3WGtn4MCBqv4UMBnvp1O9enX06dMnx73o92eKLgTo+1mGv2MUjkhgYCDq1q2LJk2aKGFp1qxZSphmOcL3dAseMGAAmjVrhgkTJuSohyWogIwaNUq5EvGc99xzj3pPAZswFpFC7Isvvqh+EzkRx0m3P/74A6NHj1ZCq52dnZrwo2sxBVrCclWqVMG3336Lzz77TAnu/OzTTz9Vk3z6/RG6J1Po379/v7o/1qlatWr5bvdffvkF9vb2ql30tmN786VDOYT9smTJErzwwgtqH4VxlufvvK2tYa526NChSrFITU2Fg4OD8fjnnnsOPXv2VPfK9jDtuxYtWuSoD9tqzpw5+Pfff5WQn5KSohQFXpvtSygP/fjjj/j8889VG7u7uyslwJRJkyapfqT8VIEpwK9NkrI9qcASttvkyZOVIlWnTh0sW7ZM1ZvtrN/TJ598omJiqZBERkZabEP26XvvvYcFCxaoCWBe79VXX8XTTz9t7Cf9eWR78Tr6xK1+DSptR44cMbrLv/7669i3b5+qT1RUlFLyONnM+yV6/djf7CvKVq+99po6N9uf90iXdY6JvMYi2/V///sfjh49CldXV9SrV0/t08c+n4mRI0eq8VqcZF9rx8KA/cLzX7lyRbWJKfqkeIlSLNjJ1JwZ32AKtWgdKhyNGjVSDze/BPjg5ga/WE2FZ0uCdEGU0ZWD3AR1PrBUmEw7JywsTFlLPD09UZTodeW1RbGQdrsTHK8CMUmAo6s7bM/uwbAvT8OpWiy8gssDWz4B6j8IrUJjatAy3tQzd63dYhIQk3TI0G6+gfiiSSxahgfCKy0dSDkJJJ+EluUHuPtJu93h9xuFCgq6FOL4KglcvnwZv//+Oz788EOLvw9+fn453lOoMr83PcmJvp9lTN/rUFilsEUhi0qEXpbCLIXOmjVrKi8BCrOcOc+N//77T1k3xo4dq2bGV61ahXfffVd9pl+T16fAxnP98MMPxj6hsMzfx9q1a6vZdgqlnEikAmF6bxQuKSBTsKTiwRl2WgrYHrpgyWtSCaICRaGf92QuT+QFy1K+uNlYodDF6+rl9GNoReCMPpXbefPmKQXL2dnZeNzs2bNVnakksX8t9YnpWKbFitYGWq1YTlcWKfSaHsf3VHZyOxfrw2uZ1pkTuOwHJqvx9vbGwoULlSDJSVuWoRWLyqipUkSlRVdKclPYqEDwcyoIHFtUZqhc8px6P+l9bz5OiWkZxtx2795dWWm+//57NVGwadMmpfw9/vjjStngs0ILjv7c8/hu3boprxfKiNz35ZdfKs8SKkTcb2ksXrhwAb169VJtzbHFPqZCZzoGqfSePn1aWYcogxYndoX0fab3ExMqmY5dcisyo1UoFszoRO3yn3/+QWho3otDBQUFqU49fPiwcfaFA45Zn0ytFtSWqaXrZc6fP3/DuTiYdIsDy2zcuDHH5zwnHzDTMrr1wvQ6xNzaoUOriqnrlI7+UBU1ptm1BGm3OxlHdIbi1scvBMcbRqmtjQ1n4DTDatJm2dyEnO1mk5EJ//hstbVx8wccXICl/YGqHWHTcZq02x1+v+ljznz8ZcTGIisuHs7Vqqr3aUeOwNbNDQ5BQchOS1PvHcMrws7dDZkXL6qXc3XDWixpx47D1skRDiEh0DIykHroEBwrVICdhwcyL19G5rlzcI6KUmXT6R5rbw/Hm/ymmUKBhwJkjRo18vXMWHq2TO/b0n5TaAXhhJ4OlQK6BNF1iuUpbNGCQYExN6iE0LOAM9OEQieFy+XLl+eoC3+nOdusW07II488kuNcdEWiJwGtDpxI1I/nxKNe9osvvlACJes1YsQIYxlaSOjipSsinGGmsGwuIOUGXcDoCp1Xu/O+KIRTaFXPsqYpLwbWp0ePHnj++eeV8N+0aVPlcqafi/IKJxkprFJYz62PONvObJOsN4U8WiJ0N28qmjwvlRIqWJQ5KHBTbqFVx1K9qVzzukxeQyFbhxYFTtrSzYuCOpUTJsDRLRF01WLiGs7002WOSt9bb72lPqMMZOlaVILYNhT0qbQS3fKU33FpWobCPduDyo9eL963DoVf1p0yoQ6VMWbupFymy11UNqk8U5GgsmlpLNLqpCsyutJAy40pumzKcWLuuVJUaCZeLoXxm5rbd+atXq9YYyzYSPzCoLmJA8KSK5E5nIGi1qgPJpq5+KDqWith5ij6Q+qKBR9GDlBquzp8GLnPtAyP4bE6NNNycOrmQpah8mOagpZl+GVUXANNEIqb4Mq10XnyD2or3ALHT2PqtCy1hXeYYdXtAWuAdu8DKfFAVk7fd6FgiF+wEKevzdCTmGHDcWnmLPU/FYMT3R9B6t696n3CTz/hZJ++xrJn33gDF6ZNM5SNi1Nlk6+5ciT++itOPNbzetn338eFiZNuqW43s4AXNOYWeQrrFDgpsBHODPO30jwNvCn8jO5Dppi/JxTYTJUKXZGi0Fu5cmUlOOsyAFO/m8LfXh3WjVYCKh+mmAqCunxgGo95M2g9yUsJobsWZ8NpGWFsgw4FbcZ+0g2H7lh03+FsPxUh3crAe6TrDy1AeUFhmRkreR4qSrTm0O1Hh8Iwz0nlg7IJXZV4bksz2JwUpdsVXVuooJjCGAxOnFKRpPsTr0NLEYVyQmsVLVcUyql0cFKVfURlJrfZctabn+kuancKrQu0ANHaQOWCVhBT+cwSdKuihYbWGVpY9BctJxxruY1FWleoTNFyxnaYMWOGah9TuMQAoeItWLHFguZTmgypTfKB0q0B1KzZiRwg9GmjFskvCvrCMbiJWjbNVXpZ+odSu+Zg4mBkajAOEF1r5uwPZ2D48NMsRqi5MnBNN+lxVoDaMIOaxo0bp0zSPA+P0U3S+pcDBzvrwVkI+tvxi0ZmZIWyho19PI4nHkTqlSSkHjuC+vc+grw9sQVTgqo1wM7RLyDM0wf71v8Cn9oNEBRcD4jZDnz7KNDrayCkvjRaAeP9WA94mCT7CJk4QVksiH1gICou/kFZLIhXt25wu+bPrvpszBhlsVBlfXwMZa/5rXt26ADX+tf7K+i995TF4lbQZ54pNDMgOC/4m6knPDGFAcKms9N5wevowjx/8+ivT4GUQao6FIypcNBNJD/Kib7PHEtxGhQY6RpMQY4TdBSCaakwXz/KEubXNHXb0T+7FX90yhXmwqQO3W7oJkR5gEK5KWwryggff/yxcR/dnXhfVMpoFaLwTvcuTqTq9WIbUUni5CTPTeiGos/O08ef/cM4AN0SQwsAFRdaEOhWTbmIiqD5pCz7kBYUCtSctDV1q6OATSsTJ1KpQOiCNa0pU6dOVRYhXd7hi94e7Du2Kd3QcpsA1gXv/MJ7NR8n5okE6D7GNK8MYKeVhW3PSWTztcx02K5sE1NlTIcuX7mNRSpEPC9dytgfjPOghYb9Z/p8EHPlWLDQt8XZKHwg+cXIh4aDQX9xAOmdTQ2aswTU9DkjwC3NkfxS1aHvJb+E+SAxqIka9s8//5xDs/7uu++UskEFgi/OblD712FZmjc5Y8Fz8Fw8J02BOvyy5uCjjx1nTBhkR03fNIZCEMrCong792zDk7Fj8dGfT+PzuS/Cb9hExJ87CbiVA+4eZNgKeeLjF4xWD7+EIz/PQ8pzr6DbT91w9mreM3LCneMQEGB0gyJOkZHKDYrYOjnBpWZN5QZF7P39jW5QqmzlSsoNitg4OBjKXvstsvf1NbpBEceKFW/JDYpwYowztBTwzNdzIqZrAFBg5cy2KRTUOGubn6Bl3W2EE3eEk3x092BGIs4+6y8G9XL2mq4ilmA9TL0BCAXp/HgfUHCmsMjZYk4A5ibYMzORDuvBe+R1C5L69esrBcKSpYIZnyh/0IpgDmewzWfx9fcUdCnUs51N25QuU+wj/m+ardIc9qd5EK0uGFNWYnvRDYsykrlSwYlPWiTM43L0GXc9nsG0zpYUMVopOOtPuYzykam1xhTKVzyeik9+FTlOJpsqF2wPS/1Cdy4K/VQ6OU4JrULmiQsaNGigzkmFjQqa6YvXywsqTpT9OHlMJZDnp3uYDhUxKq+6MibkQaEkwxVyRdaxKHnIugIGouOStepv/6rWZGj90mhtQ70a2uLlX2h7T23Rog9vl3a7hTUtuNU5f/qg9vsfX2i1vq6l7b24V637Iet/lN11LJgrPzAwUIuKitJ++OEH7dChQ+o+PvnkE7WOgs6CBQvUWgZTpkzRDh48qNZcGDhwoObi4qKdOHEixzoWDzzwgFq7IDo6Wq1PMGrUKJWXn2tjZGZmqjarU6eOWgPBHK6dwPWcfvzxx1zXZeCaFxMmTFB1/eKLLzQ/Pz/N29s7zzUHuIYGy3E9hcOHD2urV6/WGjdunGN9juPHj6v3XLdhyZIl2v79+7UBAwaoul+4cCHXdSy2b9+u9vH4/LJr1y61zsfly5eN+/bs2aOVK1dOe/LJJ1X76S+uuUXYbn/88YdmY2Ojvf/+++r+2b7t27fXwsPDteTkZIvXstQeo0ePVuc6evSouk+2J+szY8YMYxmuzcU1MzhGWJbnaNKkiVq/gnCNg65du6p1NzgeTOuclpamyrBsZGSk1qJFC23jxo3akSNHtPHjx6t7+OWXX4zX4rjivXBsffbZZ2pccQzmRd++fbWwsDDVf6wj+4bj1Lyf2G5sb15z7Nixqg68ho+Pj2o3wuNff/11bd26dWo8//7772o9Da4nRr777jvNzc1N9TXHAtcX43mbN2+u2oVtxf7/77//tLfeekvbvHlzrm3PNc74TLDMyZMntYULF6r1Y7i+mWmftWnTRitOskvIOhaiWBQxoliUPESxuHGhN/5PReMGUq9o2qmNaivtlv823Lv1D23JfdXVVhSLsq1YkDNnzmiDBg1SQhYFnJCQECUwUjgzhQvDNmrUSPP09FQL31Gg3bJlS44y5gvkUVDmwmezZs0yLpBHgYqfU9C0RJcuXdQrN6ZPn67qSOHzwQcf1D788EOlHN1sMbOVK1dqNWrUUIoLFZu///7bomIxb948tQAd24LlqYToFJRiQe6++26lGJnWW28705cu/OqCHuvHReYo6LJ92VdUDnLDUntQ+KXAT2WRAjYXvzNd+JdQSK9cubJqB7YvxwgXCdbR28vSy3TsUAHiIn4cM1wgj21vvhgfF9ajIM9rWfrcEnzWhg4dqhbm43G8H46z3BbIo5JARYTt9tRTTynhXm/bc+fOqbGkn4v73333XeOYpSLRvXt3pcDyvLNnzzYqwoMHD1YLF3PxZJ6fiuGpU6dybXt+T/DZYd9xLFatWlUpVqZwHxdNLk6yS4hiYcM/eVk0hIKFfpF0qaILWHGkm+V1Jd2stNvtQBeozlPWYvng5qgVkosP95kdwPSWKghZC6or482CK1nbCWuQkmEw4bs42GHV8JbIPLcH28a8hgZvfIQKHvZI+LYvvHp9DRtnT8DO0RDcLdzS9xsz4tDHnD7S+c0MVFbRg4z1NKB3CmMRDhw4oPz27wTGVbL/6JrCmIPChpmcGFtJtxdzV6GiaLeyQklrN7rJM+sZM6jpiQ1KY7ul5vGdeSuyq1WkmxUEoWRxePtfOPbKy6g8fjKq1G9t2HluD3DxUHFXzaoJ8XZRikRcUjqOxF7Fywt2qP9rVW+M8Dl/qjJaVibQdwXgXx6Y9yhwaoMhY5QoF4KVwlhE+t7T958LtDEmwzwTUUmgY8eOKjaBK08z+FoQCOOdGEhenEpFSUJaSRCEWybD0RYHy2UgzNFkVm9uVyD5EuDgCrjmDBgUcioXfJmSnpaMxEtn4ekXBAdHF8DJDbC1A5oMAI7+aWhXUSwEK4XB28yKxLUHmJaUaVC5sFlJhOs2CIIpDIYX8o8oFoIg3DqB5TCjgx3aBppkf3pqGZCdaVAqKASLl2W+ObL1T9g8/SpiZ41Djaadrn/gcX3xJ0GwVrgwWmHA9aHEW1sQShbFmm5WEIQSSkYGfBM1tUXsfuCTeoYZdq7DIDPrt05weXz0iC1OZV/EvnXLkZWdM42iIAiCIJQERLEQBOHWOXYaX0zNUltkpgFxxw1b4ZZhrEVslif2VHfF7z+MQ3b/15CWda0tvUKBjuMNW+G2kBlvQRCEovuuFFcoQRBumdDqjXBg7MuoXr0RkHpOWvA28HFzVFmhGMBNXFyG4cG+wbB9NBFOdk6GQm7+QJP+0r63gb4SMxcEu9VVgQVBEMoaydcWTzRdxf52EMVCEIQ806Mya5E+s25jH4/jiQdhe+oCKjdpC0/fQOCMKBYFlSHKr1w9uNjtwcZXn0X1l19B5YgawOGVQJV2gKuvjNRbgCkZvb29ERsbq967urqWiNSWxUFJS/9pLUi7SbuVhvGmaZpSKvhdye9M85XkbxVRLARByNeaC1Qq3CIm4M1NGZg7PhP/PXEvOr72Ge5sbqNsYylDVEp2Ki7gCsKzU4H4U8DSAcDDMwD/qtcD44V8ERgYqLa6ciHkTnZ2dr7WbhCk3QoCGW/W125UKvTvzDtBFAtBECzCmXQqFZMfq4fIAHdE71+Dqx+nwOPtEXCaHoBGIZXgYOsA+FYGei02bIU7JyQQ81rboVFIIGDvaUjfu+SaO1RoY+CJhWK9yCec1QsKCkJAQAAymGhAyHXGkqliPTw8xGJxC0i73R7SbtbXbnR/ulNLhY4oFoIg5AmVCq607XklCLuCQlEjoBYqVG8MxJ82rLRNguoBXCVauHOysuCQoakt/MMMi+NxHQvCIG5xibpl+INZUD+apVVgSUtLU6vtiiuUtJuMN+tEKyHPqSgWgiDkCyoTFeasNLyhUjG1CZCRDNjYAfe+Ctz7CmAnjlF3zJGT+N+32UDVk4iz8UZy4mWERNaTUSoIgiBYPeJQKQhCvkhNS8LJ03vUVs2gU6mg73//P4H6vUSpKCACImrhzJOt1XbTrI+R2PlxxBzZYbAOjfS6biUSBEEQBCtDFAtBEPLFsa1/Ibndo2oLF2+gdg8grIksileAMDvUuUwfVGn3LPzLV4RXt654fpAdErzEuCwIgiBYP/JrJQhC/ggKwPiHbTE8KADwqQh0nyEtVwhrWthAQ4SXLb5+vhU8A0Jx2dMGkGw9giAIQglAFAtBEPKHpzs2VbNVW2SkAokxgGcI4OAsLViQa1qcv4Lxv+xQ/9tdPofhi7OA+ucAf1nHQhAEQbBuxBVKEIT8EZ+INjuy1RYXDgBTGhi2QoEpF8y+FRHgbtznZOsEXzsPtc1B3AlDrAWD6AVBEATBShCLhSAIFlfbpr9/Ds5fxPO/ZkN79CJQMVRarQiIqHUPIpZuNLyhlWjwNoOVaNlgYPdCwxoXTEcri+YJgiAIVoAoFoIg5LratotLIi5mHMO+XXFw8vRBxIH9hoKSmajoocuZX4Th/zZvAVXaGRbOY4YuUSwEQRAEK0AUC0EQLK627e2eiNf/642X/k7FsCXZyiXHOHsuFAn71i2HzdOvQps1DlHNOl//gMHzKfHSC4IgCIJVIYqFIAgWV9u2Pbkds8cm4fSnQxD6YdUb/fyFwqe8P77oYIsO2ReBdctRuWFrODu5GT7zCARavm7YCoIgCIIVIIqFIAgWCYqsgx3De6JxvY7wCahg2MlgYbreOLoBIxOk5QqRo7FX4e3lg/8auuLqinF4ZUk2zq9chPCwWoYCVChavyF9IAiCIFgNolgIgmARKhOt+793fQeViqlNDCtuhzYGnl0lLVdIa1o429ti6MId0GADF5dheOrxUGidE1A+oNL1gqmJQPQmILQJ4OwpfSEIgiAUO5JuVhAEi1w8fwK/Tn9LbRW0VFCpeHgG8OA0abVCTDv7ea+GWPZicxXrkpLiCT/fOnD3LY9VAx7EqQObDQUvHwO+7W7YCoIgCIIVIIqFIAgWiT26BxUnLlHbHPhXBfyrSKsVIgEeTmpNC8a66FxNSUDq2Wi1zcHFQ0DstWxdgiAIglCMiGIhCIJlIsPRe7id2iq8KwAPTTdshaKnQjDef9JebRWufoZ1LJhy9vvHpUcEQRCEYkdiLARBsIydHdIcbdRW4eoL1H1MWsta4NoVXByPLmr2krFLEARBKH7EYiEIgmVizmHo0iy1VSRdBDbNMGyFoufQccwdn6m2iZfPqTUuzmgaEFwPyM4CPq4MnDNzWxMEQRCEIkQUC0EQLOJi64xgeKutIiEaWPGKYSsUKUdir+KCoz8O97gHAeHVceCfn9TCef3mP4yzV88C2ZkGywW3giAIglBMiCuUIAiIiU9Rq25TgLWxj8fJ05tRKTsV7Zb8Czvba65QQrGknnVxsMPLC3ao9y4OD+Ee12BUv7cbNnyq4Xz0Z4hLi0OQ9I0gCIJgBYhiIQhlHCoVbSesQUpGllIq3CImYNX8NDzzezZSdraCu+P1zERC0aeeXTW8pVHpo4LB/0NCAhEaUgNPz8oG7roA+HhJ1wiCIAjFjigWglDGoaBKpYJrJjhkHcKiH9PQ/pFXYdvDHy72LsVdvTIPlQu+zHFIz0a1Cw5qm4OEGEDLNgR3C4IgCEIRIoqFIAgKrplgezIJr/2QDa2jP6KadTastk3ffeLoBkS0AZw8pMWsgCr1W6PK6p2GN2lXgWdWAn6RwF+jga2zDRmjRLkQBEEQihBRLARBMBLZsA0S/1wOT78gg1IxtYlhtW3y+mmg91JpLWvEyR0Ia2L4v0o7YMNUg0IoioUgCIJQhEhWKEEQjDg6ucI/OEJtlWBKpeLhGcCANQaLhWA1HNj0OzY1iFLbHLj4FFeVBEEQhDKOKBaCIBg5eWAzlvZpo7ZG/Ksa1kqQ7FBWRba3O35ubIMT6WfVmhZJV+OKu0qCIAhCGUcUC0EQjCQlxwMxZw3b8jWBV48atoLVwOxQe2ISEGvnj19buOKrv8erNS2OHFhvKODqBzR+1rAVBEEQhCJEYiwEQbhOeAje7W2PBeEhgJ0D4OYvrWOla1oQF5dhGNyjIrQ2lxBZvalhJ+MqOk0ovooKgiAIZRaxWAiCYJnLx4B5PQ1bwWrWtFg+uLl6MT1wSoon/Dyj4B0Yjr/e7o8zx3YD6cnAmR2GrSAIgiAUIaJYCIJwnUPH8fWETLVFaiJw6FfDVrAa5aJWiJd6MT2wTvzlM3DcsldtcfEQML0lcGC5QcFgdi9BEARBKALEFUoQyvCK2/qKzjr+YVWw/fE2qB9WBYDMeJcYKoVheH97LKgUBth5AA6uwJL+hs8cPYBBGwGvkOKupSAIglDKEcVCEMqoUtF2whq14jZxcUnExaSDsL0YiyYD3oKPX7BhtlsoMXgka0iIPg7U7WRYHE9f2NArVGJlBEEQhCJBXKEEoQxCSwWVCvrpf90/Em4RE/HBr0+r7EI71/xQ3NUTbgFanC4eS8TYr7NxccYMJF25jH37diLJo4IhTbAE4AuCIAhFhFgsBKEMQz992/gj6LUiCdUHjoA2yx91qzUwfOgZDNw/yrAVrD5LVFjLl/Bpr0Y4uXudUhBPzhqHqMhIYFEf4NE5QED14q6yIAiCUMopVovFmDFj0LhxY3h4eCAgIAAPPvggDh48mKOMpmkYOXIkgoOD4eLiglatWmHv3r05yqSlpWHw4MHw9/eHm5sbunbtiujo6Bxl4uLi0Lt3b3h5eakX/4+Pj89R5tSpU+jSpYs6B8/10ksvIT09PUeZ3bt3o2XLlqouISEh+OCDD1QdBaGk4piuofZ5J0S6VURUs87wscsyuEElngHqPwm4BxR3FYWbZImi5em0bRgc/KqolMGv9bNTW2SmAhcOGLaCIAiCUJoVizVr1mDQoEHYsGEDVq5ciczMTNx///1ISkoylvn4448xceJEfPbZZ9i8eTMCAwPRrl07XLlyxVjm5ZdfxtKlSzF//nysXbsWV69eRefOnZGVZfAfJ0888QR27NiB3377Tb34P5ULHZbt1KmTujbPwXMtXrwYw4cPN5ZJTExU16aSw7pMmTIF48ePV/UThJJKZP1WuG/1drVVGYSmNjFkFeLr2N/FXT0hH1miTDNEwdkJxwNt1FYQBEEQyowrFAV8U2bPnq0sF1u3bsW9996rLAGTJ0/GW2+9hYcffliVmTNnDsqXL4958+bhueeeQ0JCAmbOnIlvvvkGbdu2VWW+/fZbhIWFYdWqVWjfvj3279+vrkUF5q677lJlZsyYgaZNmyoLSbVq1fDHH39g3759OH36tFIcyIQJE9C3b1+MGjUKnp6e+O6775Camoqvv/4aTk5OqFWrFg4dOqQUi2HDhsHGxqbI21AQChQG/GYkAw/PAPyrAj7h0sAljQuX8diaLOCuy4DnNYUjNaG4ayUIgiCUAawqxoJKAvH19VXb48eP49y5c8qKoUOBnq5I69atU4oFlZCMjIwcZagYUOhnGSoW69evV+5PulJB7r77brWPZahYsAyP0ZUKwmPpZsVrtG7dWpXhtVkH0zJvvPEGTpw4gUqVKt1wTzyeL1OrB6HSVNQuVPo1xXVL2o1jwAaGsbBvwwrEvfgKfD4bjxqVKwOBdYHAOkC5avrAkfFm5c+paX8i8Sru3Wejtlr5CgBsgLiTQOI5YMd3QL0nAY/yKG3I95u0nYy5koE8qyWv3W7lmlajWLDSnPVv3ry5EvAJlQpCC4UpfH/y5EljGUdHR/j4+NxQRj+eW1pCzOE+0zLm1+E5eW7TMhUrVrzhOvpnlhQLxpG8//77FpWo4lAs6CZGxLpSttstPfkqQtwMWxsXN/zbKgT3u7ghwTEQ6LnMUOiaon+7lMZ2Kwpup91M+zOi6t0I/u0/nD64BdtPHIXnk7/C3zMUOH8SWD8XCGoOZDujtCHjTdpOxlzJQJ7Vktdu+qR4iVIsXnzxRezatUvFN5hj3oBqdu4mjWpexlL5giijKwe51YfWDCpMpp1DNy1aS+heVZTodeW1RdAr2+3meBWISQIcXd1h6+uPebXPo0uIv7rHgqI0tltRcDvtpvfn6avsU0PGqIsjXsGWChn44QF3LO26FEGZjkDKScDVkSdHaUPGm7SdjLmSgTyrJa/dbuV6VqFYMKPTsmXL8M8//yA0NNS4n4HaujUgKCjIuD82NtZoKWAZZm5i1idTqwXLNGvWzFjm/PnzN1z3woULOc6zcePGHJ/znHSzMi2jWy9Mr0PMrR06dJsydZ0y7aTiELb064qgV7bbjfdB5xl1TympiIzOVlubc7uAr9oCz64CguoWyHVKU7sVFbfabr7uTnB2sMfLC3eq90xDO/vDsahtcw4rN41H/JVYBKtzaTy54VUKkfEmbSdjrmQgz2rJardbuZ5tcWtftFQsWbIEf/755w2uRHxPYZ4Zo3SoRDCblK40NGzYEA4ODjnKnD17Fnv27DGWYZA2XY82bdpkLEMlgvtMy/AYHqvDgG4qBbyGXobKj2kKWpZhXIa5i5QglBhOn8Wob7LUVsVTZKXfdlyFYB2pZ7n4oXvlpqhg649PpmcBJ3Km3xYEQRCEwqBYFQummmUGJ2Z44loWtAbwlZKSYtSQmEp29OjRKp0sBX9maXJ1dVXpY3WT0DPPPKPSwq5evRrbt29Hr169ULt2bWOWqBo1auCBBx5A//79VWYovvg/U9IycJsw+DsqKkqloOU5eK5XXnlFldNdlnhNKhqsA+vCOrFukhFKKCnExKdgT0yCWq3Zxj4eMXtWw8nTB/bffYaKdZoXd/WEAk49GxzVCJcmDlNbOHsBUd0MW0EQBEEoBIrVFWratGlqy0XvzNPOUngnI0aMUIrGwIEDlWsSMzvRSkBFRGfSpEmwt7dHjx49VNn77rtPpYS1s7MzlmGqWC54p2eP4iJ6XBtDh2V/+eUXdZ177rlHLYBHRYLrVOhQiaFlhApRo0aNlOsVlQrTGApBsGalou2ENWo2m0qFW8QE2L+fgp1NquHhyT8aCuU/PksoAXh7B6J5x/7Xdzz8FRC7z7AAoqsf4B1WnNUTBEEQShnFqljkJysSrRZceZuv3HB2dlaL1fGVG0xhS+tIXlSoUAHLly/PswwtIXSHEoSSRlxSulIq6Cpj5xyDNzdlIHPUcDSp0Li4qyYUErGnD2Lr7PFo2O8VBIRVA5IuGBY+JA6uwKBNolwIgiAIpcMVShCEooeuMhFXL+PTaZkIcw9FaIRJkDbXrhi44foaFkKJhe5u2/fshu/StdhxaA32bfsDMefOAAPWGBZA5EKIXBBREARBEAoIq8gKJQhC0eLjH4rjLeqprSL+9HUh0ysUcHCRLimhMNUss0K9vGAHbOzT4PaSC2yip2D0/zKRUSUMbcfPQ9EmuhYEQRDKCqJYCEIZJKhSLXQa9/11pWJqE8MMNqnfG+jwEeDoVqx1FO4sQxRd32i1GLoY+PTJKoioHY+AwAh4evkDHr7A27GArYM0syAIglBgiGIhCGWQK1cu4dCuv1G1Tit40FJBpYLuMf5VDUG9olSUeOWCL6JleqOSZzVEVnTAxbNHkZ6SDEcXV8D2xvV1BEEQBOFOkBgLQSiDnN69Hq7PvK22cA8Amg8FKjYHgutJMG8p5cj2P5HQsYfa4uIRYHYnw1YQBEEQCghRLAShLFIhGG/0sVNbeAYDbUcatkLpJTQQox6zVVukXwVOrgXObDOknk2SIG5BEAThzhHFQhDKIi7OOBpso7ZIuwIc/9ewFUoljLU4mpSN4+VtEH1wK7JdfAzpZpf0N6SfPZB3mm1BEARByA8SYyEIZWBhPD2Q18jFODz6bxZwdxyQlg7M6WxIQ0pXKKGUZoiKxyNxtig3bTIyd/SDI9ewMGYCk4XyBEEQhDtHFAtBKCOrbZOw7NO4cmwdfJ3t0HaXDewSkwAPyf5UVjJEjfwuCW0/8oHnvo048vpwVBk7EZUr1zBYLKp3Btz8irvKgiAIQglGFAtBKCOrbXs7XcLOMW/g6tsamq7ZDWzaayhEH3uhTGSIircPQUjd5ki7vA0nna8iDBlAwmng55eAoLqiWAiCIAh3hMRYCEIZWW07oLw9FjTLhu0HrxR3dYTiJrg8pnS1U1tBEARBKChEsRCEskJaOlxTgaAqdXPut3MAPIINW6FskJkFtxRNbY0kninOGgmCIAilAFEsBKGscCIaU77MUlvjitu6G9Tw/UD5msVaPaEIOXoSsydnqa1aEJEZotZPBbKzgcw0w1YQBEEQbhGJsRCEMkJYVBMcnvQGqkQ1MSgVU5sYVtymtYKKhVAmYBB3efeKODvsUdT19Ma+fTvh0285ghy9gHO7DOlnJUOY9cBnVc/eRagIeksWL0EQrBNRLAShjODhHYAGHZ4yvKGlgkrFwzPEUlEGU88SF4dm+PiPnxA5fTm6veOBn7r9hCAunCcUGTGn9yPh9FFUu7sD7BLP4PjOv5GiZQCBvoCLP3DoKAL+Hgp/BxOXtdDGwBMLgfhTomQIgmB1iGIhCGWEs6f2Y+Psj3BXv9cQpD/5/lVFsSijqWepYPi36Y3TtSKQcnYq4tLiEFTclSylbN++EQlJKfCvVB+Z6amI3bsGl/0csXXhSPT5LQ27fghE3aWdcfAfL5xxtcOkh+zglK7hmwlZ+O/xe3FvxydwNvYCUCkI8AwEjuyA03c9EeFpBzz2DeDqD5SrDjg4A3EnABs7sWoIglByFItt27bBwcEBtWvXVu9/+uknzJ49G1FRURg5ciQcHR0Lup6CINwhcWeOI2TpRsS1P46g+i2APssBvwhp1zKaepZ4lAuHl80FvP5JFlDvLFBO1rEo6IUpj57ch+x3nsUlFw/0rfUOqmQewKfLv8K8R2xxONIBsQ/ci1PfbsX79pFYXqs2Ltq6IvC4Oy5le+Kl+5MRnxKOkZv2oPKURXjiVTtk2ttg+OIs+NqEIaLJGeDb7oYLDt5meJ7/HGVYl4QLIIrLlCAIJUGxeO655/D6668rxeLYsWPo2bMnHnroISxatAjJycmYPHlywddUEIQ7o0pFPD3UHguqVAScPIBKLaRFBTg7uMLdzVttc3D5OJCaIO42t6FMXEpKx/PfbIVzcjT8bfYh6REbPF+uB36vXQmVvnsCRzq5YIR9FnwuZyHI6zj29XkH2Y498bzZ+XTL0tWoGjg9vhY+qFQBHm5OCKgfBydbJ1x0AGITTgCebkDcWeDgflSu0RXOuxcCp9Zfy/gWKKNcEATrViwOHTqEevXqqf+pTNx7772YN28e/vvvP6VkiGIhCNYh4FAwsQhTi26aDjQZAHgGF3X1BCuB4yMyIAr1p/2OqwfXYJ+jD3ye+xtB5aKAJc8C+34yZIyS2e/8rXL/yY/IyoxDeMIluLhHYLjLRlRfthbPD7VDu7S1CIocDLy4EVGMZbG9/vMbFRgE2NrlGhfz2u+xANyAtZfU+y96N4TNpYNI79cf37e0xY/NbNHkYDZeWZKN80s+Qzj7bEl/oOXrQOs3CnrYCIIgFKxioWkasq+lI1y1ahU6d+6s/g8LC8PFixdv55SCIBSkgDNhjVpxm1AQoYCScCwGH87JBGrHAIHpwNpJQNSDoliUQcwDuT1sM7BwyRuY0sUWW+q5GQK5274PVO9sEFCZlUjcanIo7STuzH74BNdQ/6//70f4+E+Gc2YGPl2UhSPv9sddjZ7BxdqBWHZoKoIem2CwIARUv624GKJbQvrM2qTeh3Udgg+frIHHQ8sB916F1jkWmdlOWH6gOuo8PwAVqjUCdi4wPOO0UkpGKUEQrFGxaNSoET788EO0bdsWa9aswbRp09T+48ePo3x5WclVEIoTCiFUKiY/Vk+tuJ2RdBoJ+1bDydMHdhGV4OnhLx1UxjEP5B46fwtOj38fXb2Tkb1sHOKanUJQpbsMrlCCRaU9IP0kJq+Zgidbv4oE+/KYum4CeoRko9qAF6BNAJrXawOfk6sRsP9Tg9UnIKpA4mJyBuADcdmR8Et1h4+PI0IquWDf9lVIPX8O+7NScXXnenj9ORQhDpmGg8X6JAiCNSoWkyZNQq9evfDjjz/irbfeQmRkpNr/ww8/oFmzZgVdR0EQbgMqFX5eyXh3xlMYNj8F2T/PQecZKwwf6gvjCWUWU4FVs7FHcP1WsD35L4Ysy4b2IDMQ0ZQRBNz3LuDoZhgzZXTG29S1UFfaHZMdsPaiDcY8FIoKFe/CpRavoPu6NxC25k3DQRW/Bur0ACo2L9B20/vtxvTBdkrp8KlSEx/38UDK8YmYOz4TW3p0QkC/l+CQEA2sGgnQDevcHsDZq0z2pSAIVqhY1K1bF7t3775h/7hx42BvLxlsBcFaYArRnUHpOP3pELQINbhsCIIlKDRXDG4MrFgKj+RL2LdjFRAWDJ/6TyAoejsw//EyOeNtybWwdnk7HN1zDAtb2KJz9VBE+XkBNnWAzVcNa8MwjbNPOODiA7gHFFn64M3HL6sJhS9bL4CTUwpQ9TjuCa8Oh+AIILQRUPNBw8EfVwYyUspcXwqCUPjclhZQuXJlbN68GX5+OVMTpqamokGDBipTlCAIVkBcAu7dqyH0vrpwcfG4vt/VF6jf27AVyjSWZr4/OjwWSQmxGN3TDi72Lvip/TcIosBchuItTK0UqdoljOgSgoirlxFasTqyz+5F2EufoEpfZ/jEnwHSMwAHFyCwDhBUFyhXrUjqmJf1ggHeLl4OOL50DgLuawW3coHw1lwQ7F8J6DgO+OHpMtOXgiBYuWJx4sQJZGWZrAR6jbS0NERHRxdEvQRBKABsLlzGU6uzYfP4ZYAei/GnDcIEoXBBYUgo01ia+fYc9C4iPbIxIfYIgod8grhKhxFUsSrKopXCxj4ebhETMO1IBiZOz8TBxtXQ6s0pODfldUzY8CaCvukO2DkC71wAnv+32PvQNMCbcSBzVizCiKtLcMkT6PWvLbp9MBchvtfWr0lPMmwzUoul3oIglHHFYtmyZcb/f//9d3h5eRnfU9FYvXo1KlWiY64gCNZAjaadgL2dDG+oVExtAmQkG94P3AAEiHuUcGOAsH/lhogK8YJ/TDg29t+Lu8KrAfZZhtWd7Z3LhJXCI+kY3n/kXnhdSYP9+ynIHDUcIZND4eXuBd+AMPhWqwb8d831qYgsFPntw+uKYj30AfBOh3pwSziKjB/GI/lKHA6cPguPdAeEMDg/6SLwz0SgVh/A5DddEASh0BWLBx80+Gfa2NigTx9+XV2HK3FXrFgREyZMuK2KCIJQyNBSQaXC6ANeUZpcsMj19U8C0ODpjxGgKx2DNgJnd5XKQG7zjE9zVkzBkcapaNiyGzbeUw/1whohxN8PWPMRUKEKkJVhOLBCU6trB1MlI9YxHJX8aqJWnWbA6t5q36YGUVh4VzD6B1RFEIO6N04D/OsDtsmAm7/V3Y8gCKVUsdDXrqBVgjEW/v6StlIQrJkju/7FgTeHovroSYgs52lw2eDsKv3ABcEMc199t8zLaJi0GW++OgTVK19zn5ndEUi/YpWB3LFX0nD6aoKa/OK9mFph8pum+X8NriDL1RZj3GwxuH49+AdHoO2wkXDSUgyrWW//Bmj8rCEY+uU9VnX/N1MU9Ta5Mv4V/LF/Ih7R0hFEBZEukX9+CKScBBo/A3SSCUJBEIowxoLrVQiCYP2k22TinEMyKttkGpQJ+oELQj4XZNu25ic0/mAltrSJQHZyXXg5eyOk3wrgwgGrC+SmxWHgt1txNCEbGmyM6VdvplyYr1IfMe5/WFVXw4HW7ghlJrXMNDjt/R747xNAyzYoVLowbiX3nhu5paQNqlIXgetoxUwBwqIMbpEXzwKujoCTu8EiRaumo2tx34IgCCWM284Ny3gKvmJjY42WDJ1Zs2YVRN0EQbgFzAUkRXB5fPKgHZoFy8KVwq270Vxs2BDPDHZG8pUZGPxONoKTneH49Q8oR6HTyuDYT83MxqQe9eivq4Rp7rOkWOjPih7obJpK1mb8x3gk0Bv9PZwRlHge4KthX8Mq9aQEuYDllpLWNeYo/vdtFrQ2Z3EmYzeyszLhTksmYyzO7gSmtwQGrAGC6xX3LQiCUBYUi/fffx8ffPCBWoE7KChImZ0FQbCuXPucrUxIz4JLmsbsCsCFg8DiZ4HuX1lFsKlg/fi7h+Ls2VfwaocQBEckoUJgJZQLqgykJxsETytTMGxsE+ESsw7uYdVu6VmZ83QTpJ9cD5sPX4fvQ5NQrXyFnIkO+iwHKrVAScRSSlqHbA0Vuj6FL8IbYNf/XoDd3qMInzEXXl71rx948RAbFAiqU5zVFwShLCgWX3zxBb7++mv07m0IBBMEoXjR/cO5IjAXyMqIPQCHlDPA8ZOYMzELWq2TQMVQ4Nwuw8JYgpAPKIw62/jh458NaUnLa5sxrmcs/CsEwMfBAUEUPq1kFv9i6jm4hXyLsBEnsKNfWwDt8/WsOGYmIDNmI065XMa+0CwEOzvemOjA71p8SQnmRuuFO9LgBu8XBmD4utfxzuldcHW0R6gv3bxcDa5ujh7Am5JCXhCEQlYs0tPT0axZs9s5VBCEQoSCkk/KcSQ+3hfrHr8LLQf+D7vf7IvaVeoCGdfWrxCE2xBGd+3agmqvjcWvF2yw7G5bRJ2xwejsMwi2zbSKQO4r6QnQ7DKx7YPnEFwuCi3n/YCrlysBIV43uApybQqXs+vh51cTZ/dthsvQMRj/nB0S29ni8XJ+gGt5wCPYKjM+FWRaYRIe0QCZe1zh8Nm32FT1F/i8+yVONhmL8IqV4ObuXWx1FQShZGJ7Owc9++yzmDdvXsHXRhCEOybB2wEDBtvB/5Ee8CkXhnufek1tBeF2oCBaK8QLdeo0wgttB6P8I2Pxkk9/vDY/E/F3fWCY1efsvr7wYjFhf/a8WgzS390fmVfj8Po//yDx5J4c7k+dp6zF0MV/qwXvfN98Bxtnf4QqDZpDm/AiPm3YDz+dPoOgjAxDytUBf5UqpcISVLIuJbhiWqvv4TNiGJq88A5O7l4Hm8FjcfJCoiGL3NS7gH3LDAHdXAtHEAShoC0WqampmD59OlatWoU6deqoNSxMmThx4u2cVhCEgsDGBvHuNvANlHUqhIJ1i7riVhmj/smCQ3Ywgh58FrPqP8KctMDdgwBHt2Jd38IjIwVaVjbKn/4LMU0eQ59hdnjWzRl7YhKUAK27P2Vc/heTDqUjbvxQ3FW1BTySYxC1+U3DSfSMT+qEgSitmGeLcnWwxY8DGiIkNACJTll4rZ8dxoaHGPo0/hSw0MTtuYSk2BUEoQQpFrt27UK9eoZsEXv2GGaEdCSQWxCKmehzGLEoC6h3DvCLur7fJxx49GvDVhDu0Ef/ve+u4Mie9UhvWBU+zQcjKO40MLNdsblFBQYGo3atS4ir/RKSsuOQ4mSDRWu2YvTvdrDTUhFsdxa1g5vg0Cez8cZpDTXn3Iug0OpA2hVDILqVxIoUR18OXbAdV1KuLfjn7ITjgTZqq9qCfUlrFFfpjj8JOHkUd/UFQShtisVff/1V8DURBKFAcHNwg5+Ln9oq6L6gu6mE3wO4+EhLC3fko5+enIinjn2LrFFn8djT9nCxd8FPHeYhiG5RxbS+RUZ2JhJggxQnL/heOY4hP2ahYr8ABDZsjjPbf0fYK1OR1rkiar81FjZXYxFimwoc+9vwajIA8AxGWcI83uJ0XDIcYxJw+uRp9PsjC7jrMuAHXEhKQ7kQSTsrCEIhr2MhCIJ1Eh51F8IX/XtdqTBNm3n/KKDZi8VaP6Hk4+jqiQUVH8HAx30wwu4iIkZ9jrjIQwgKK/r0s3pQ9oFtGxDxhz8uV9iG8CA/BCZlo6KnGzJOrUW2dzLGPGqL18v7INTPH1jY8fozQQtLo2dQVtHdosb/cQgxSYdQLv0Uxh0GNBtfXDxzFHsf744qs+YgpHyQYZHAyq0A94AyY90RBKEIFIvWrVvn6fL0559/3s5pBUEoANKz0nElJR4eLt5wNE+bWcZmZYXCjbcYuTULvhmJ6FyzEjprLti3cz28nf0RbGsPxO4HHN0LVfg0XZOiqk0qGja3QccKYYisVBWRzc4CVethV7vHseo+WxyoD/ikXwQcqgG1ugM1HwRc/cu8gEyrxR/DWuL8hctwdHXH0QtJGGprj7nuIYhN2IXdwRlIPbcDCcd3Ivi/afDeMNXQ+FaQCUwQhFKiWOjxFToZGRnYsWOHirfo06dPQdVNEITb4MjGP2Dz9Ks4P2scoqpHAVU7AKGNAN/K0p5CIa2JEIqIg4fRZMyneGGAF4YmOaDpT4/CITkWti9uLnDh0zR1rB6Ubevkj6nbNqBXtZqAZg8E1gEcXOAwfjCetkvE8I0fI8itvCHjU4ePDIHJgrE/3W3c4eXlpSYNL9uHqP0+YZH4pos7vDeNxyfTs7Bn1Fto3rQtcDUW2PEtkJ0pLSgIwp0rFpMmTbK4f+TIkbh69ertnFIQhNsUrggFLCOB5TC1ky0GBpYzKBNPzJe2FQp9RecxezwQ3qkD0hz+wAdfzEQ1hwcxzfETXD6wBr71ugLOngUy3i8lpeP571chTbuC4CtxCLdzReNKbXD53BHcvT8buDseiLwHeN7gDlhjS3MgK90ww+5pEJhFqbg5hu8Ud3zZegGckADt7mjUimoEeF/LllX1/jvqT0EQSicFGmPRq1cvNGnSBOPHjy/I0wqCkIcLiA6FOwp5Cc4eWFPHFgO9PICsDEM2F2cvwC5nWmhBKGjrxcXEWtjy0SrUP7ICmR/Nxunl03Fl+TBsS70Cv6oPoH5wpdtXJr7ZqsY7F7fjOhRuthkY/H0W7H38EOI9GBm7ovHgBg1enWKASJOTPLsK0LQy7/J0u6lo+T/71yHMBes/fQcN+72CgKBKwIX9QGa64XtF4i0EQSgMxWL9+vVwdnYuyFMKgmABClu6CwhX2yZ2GZeVkBd3JB73b80G7r0KpO0Fprc0pNMMlswuQiFnGAqpjSpvz8WV+AvIdHXGkb/8sLOyhrlHpqLams/Qu+5zqFqpAdw8/BBUuYZF65uOqTJBwrJP481HmsH2QjKCRqTg3OSXETo2Ai6Ormr9jBB3G6S3ikNIeJWclQuqK91+R25uO9T/tmeOwnfpWuy4pzFCT2yD1y+DEeJ8LUWtxFsIgnAnisXDDz+c472maTh79iy2bNmCd95553ZOKQjCbUClgqsixxzZgc3vvoRy42fDI90Oz/6RDc+XHKVNhSInJPK6Art92ixUTYrDyxeOoPH4qThzaRw8PJKw+YQvKg38FDWbtEHs8Z3oO2c3cOUsznj4INnGB1qmt1GZGHxfMOCQjNoDP8GxsCNo3P0FbH+mAxrUbAv/4AggNREYGwZ72CDAo9r1Be6EAktFSwXD2z8Crwx3R1r0FIyenYmMiDpo+9q78EyNLbYUw4IgWB+2t3MQA7xMX76+vmjVqhVWrFiB9957L9/n+eeff9ClSxcEBwergLEff/wxx+d9+/ZV+01fd999d44yaWlpGDx4MPz9/eHm5oauXbsiOjo6R5m4uDj07t3bWF/+Hx8fn6PMqVOnVF14Dp7rpZdeQnp6zhm03bt3o2XLlnBxcUFISAg++OADpVQJQnGTcPUiMi5eRGzscYRUqQ/P5d/nEPAEoTioX/8udGz+AJ566EVkzPwY5XpOw/qKg+BxxAF/Ht6K1b/MQersTpiS8R4+/WMeajlNQ6WQiZjYLBaLO9vgne3TEf/l6xh94kO88owdKt/XHv64gnY9n1JbtdJ3ZqrBItf/L6DHXBFuC8ktqu+MI0g6OgxTWs5FxMRPcO//ZsCzaiug1iPA27HXV15nimtBEMost2WxmD17doFcPCkpCXXr1kW/fv3QvXt3i2UeeOCBHNdzdMw5C/vyyy/j559/xvz58+Hn54fhw4ejc+fO2Lp1K+zs7FSZJ554Qikbv/32m3o/YMAApVzwOJKVlYVOnTqhXLlyWLt2LS5duqSyW1FpmDJliiqTmJiIdu3aqVS7mzdvxqFDh5TiQ0WE1xSEYiUsGB88aYcFYcGwd3AUpUKwOurc00VtsyrWxFP2vwA2CzFuQiZ+DozE49WP4VynOLxkq+GPPUEI+/JjNHwxBBebnUdo81fRPKwifPyrIejYWmDhIzlPHNEGeGQW4OwNJCQUz82VUiy5Rfk7VEZkjWq4ePYo0lOS4ejiCtg6AT+/DJxcK25RglDGuaMYCwrv+/fvV5aEqKgo1K9f/5aO79Chg3rlhZOTEwIDr2WhMCMhIQEzZ87EN998g7Zt26p93377LcLCwrBq1Sq0b99e1Y8KxYYNG3DXXXepMjNmzEDTpk1x8OBBVKtWDX/88Qf27duH06dPK+sJmTBhglIcRo0aBU9PT3z33XdITU3F119/repUq1YtpVxMnDgRw4YNy3NdD0EQBMEAA7jntl+M6IQLQFAManr4Iz0kEFH2SerzihkOSDh7Egj2hf83D8H/v2tW8IemAzUfAkIb52xKuj5xNXmxHheJWxQ5sv1PldI6nimtm3U27OzyCXBmm7hFCUIZ57YUi9jYWPTs2RN///03vL291cw+hXzO5tNywJn/goLXCAgIUNehGxIFfb7XFRuuoXH//dfT3lExoNC/bt06pVgwoJzuT7pSQehOxX0sQ8WCZXiMrlQQHks3K16D98UyvD6VCtMyb7zxBk6cOIFKlW4t24kgFCiHjuG7jzOBqseAplEGdwT6PNM94fXTkl5TsDrlQmWIuh6/bYQJYUPCaxrecAE2jmPiXQFw9TWsQyEUL6GBGPWYLXplXwTW/4Lqd3eArX8kkC7p5gWhrHNbigVjGugatHfvXtSoYfhl4Iw/3YcYm/D9998XSOVozXj00UcRHh6O48ePq8DwNm3aKGGfAv65c+eUa5SPj0+O48qXL68+I9zqiogp3GdahseYwnPy3KZlKlaseMN19M9yUyyonPClw3YjVMaKOj5Dv6bEhZT8dmNdbHC9XuUqVFcBrfUrVIcWdwr4/G7Ditt0E+m1WD8IZb3dSgLSbiZ4hRpe1xtH2q2Yxpzpd463TzAOV3XHn/PHo+c/2cjY1haOdo6AZyjQ+VPDtgw89/KsSruVlfGm3cI1b0uxoGsRXY10pYLQFWrq1Kk5rAd3ymOPPWb8nxaFRo0aKSXjl19+uSEz1Q1fgCauSZbclAqijN7QeblBjRkzBu+///4N+2nhKQ7FQl/AUFy3Sna7pSdfRYibYUu3cgc3fzR5+l31WcKFg4B9OeD+t4GAGsXmd26N7VYSkHaTdrPGMad/5xyNiUV6sis+bTILDnUuAT0SkXQlCSk2KQaRIqQVcPYogKOGuBePnJN2pQl5VqXdysp4S7w2KV5oikV2djYcHG5cbIv7+FlhERQUpBSLw4cPq/eMvWDmJmZ9MrVa0FWrWbNmxjLnz5+/4VwXLlwwWhxYZuPGjTk+5znpZmVaRrdemF6HmFs7TKGrFGMwTDuHMSB0xWLsRlGiKzK8tgh6JbvdHK8CMUmAo6u7qlfcxRhsWzUPDdo+AR9XRyCFPuqRQPA1l5JiwBrbrSQg7SbtZo1jrrzmiMvpdhi69JB6z2xRfwxriTRtKzb17oAqo8ejcu3mwJGfgOVDrq9vMXBDqc3UJc+qtFtZGW82t3C921Is6I40ZMgQ5fKkxyXExMRg6NChuO+++1BYMFsTA6ypYJCGDRsqZWblypXo0aOH2sf1NPbs2YOPP/5YvWeQNq0DmzZtUquCEyoR3KcrHyzD2A0eq5+bAd10t+I19DJvvvmmUmT0zFQsw/s3d5EyhecwjcvQ0dPnFjWmqXuFktdu+kJiRy8kgY4Jep3OHd6BkPdn41x4FHzr3g10HGf4MS/m+lpLu5U0pN2k3axtzIX6uGLl8FY5MkTFJ2fA1iYTp5yuAIlHkbY+AeWD68FvwN/AxUOGQO6Uy4BPBZRW5FmVdisL483mFq53W+tYfPbZZ7hy5YoSqCMiIhAZGaliDLhPT8+aH2jS2bFjh3oRxlHwf64pwc9eeeUVFTTN4GgGcXOdCa4x8dBDDxm1tmeeeUale129ejW2b9+OXr16oXbt2sYsUXTXYsra/v37q8xQfPF/pqRl4Dah+xZduZiClufguXhtltOtCkxZSwWBmaKouCxduhSjR4+WjFBCkSoVbSesQecpa9WPOmcMmWdeEVEBAwbbqa0Kbm3SX4JcBUEoUJgdigtycmFOQgUj1jUY0x9yw8g941WmqK1//wQE1wOCGwDhzQFHQ1lBEMoGt2WxoCvPtm3blKXgwIEDyjxDwVwX5vMLV+pmxiUd3WWIQeDTpk1TC9LNnTtXLWZHSwLLLliwAB4eHsZjJk2aBHt7e2WxSElJURYTpoTV17AgTBXLoHI9/oOL6FE50mFZxm0MHDgQ99xzj1oAj4rE+PHjjWWoxPB+Bw0apGI96HrF+pq6OQlCYcKZwpSMLEx+rJ76Yc+4eMiYBjL2yB5kcZrA3h5IvgwcXglUaWfIoiMIglBIC+cRF5dhmNarCrQ659AwsrahELNE9ftF2l0Qyhg22i1EEP/555948cUX1ay/eXyA7lr0xRdfoEWLFoVR11IBYyyopLC9iiPGgtcVn/eS2W57YhKUtWL54ObwSTmOM488jkorlsM/OAIr72+Io74Z6DrlRwRnJALTWxpWI+bMYRlvt5KGtJu0W0kYc7pbpu4Wxe+lINvL2LF4Oup1HwC/8uHAlbPA1fOAja1hvZFSFmshz6q0W1kZb4m3ILveksVi8uTJOdyDTOEFn3vuObVgnCgWglC4JGjJWFnfBg8mnYc/IlB5/GREuXoguFxl4IxhFlEQBKEoF847f3wvgj/7Eecb3AM/F3tgcq3rHzKQm+uSlDLlQhCEO4ix2Llzp4pXyA26GnGNCUEQChlfb3zfyk5tSUSdFgiJLD7rhCAIAqpVRo837HEq+yKOnYkGXt5jsJw+PMOwro6+2KEgCKWWW7JYMG2rpTSzxpPZ26s0roIgFDJJKYg6ma228JPWFgSheKFLlLenHZzsnHFpzMfY7eaNygvXGSwUAVFA+D2AWznpJkEo5dySYhESEqICqpkFyhK7du0ypmsVBKEQiT6LkfOyobU9C+iZHONPG2YEHd2A0MaGrSAIQhEHcld4yxvBnp5IS7kKe0dn2Nk7AvZOQOy+6weWwpgLQRBuUbHo2LEj3n33XXTo0AHOzs45PmNGpvfee0+lcRUEoXCpVO9enF88C+Ur1byuVExtYnA3GJkAPLtKukAQhEKHcRarhrfMEcjtE94MaSf/xbH6jaHNGoeoBvcCvwwD9v2U82C6SolyIQhlV7F4++23sWTJElStWlVlh+I6EIxM379/P6ZOnYqsrCy89dZbhVdbQRAULq6eqFiz6fXWoKWCSgV9mQVBEIo5kBtB5TCliy26Zl8Etv2DyNbvwLH5tfTsV2OBY38DdtfW4REEoWwqFuXLl8e6devwwgsv4I033jAuL07lon379vj8889VGUEQCpdTh7dhy6S30Wjoh6hQpcH1D/yrStMLgmAF8Rbe2FTXFdnLxmHIsmxcWrsCQabpr6sa1pUSBKGML5AXHh6OFStWIC4uDkeOHFHKRZUqVdSCcYIgFDym+eJ1riZcgOvu42orCIJgrfEWj/cOhfZgAvw8TeIvU+KB/csA73DA2UviLQShrK+8TahING7cuGBrIwjCDUpF2wlr1IrbhD/a/PFOqBSGV5+1x4JK14Ify1UHBm8DPEOkBQVBsJp4Cz/fOnCzP4iVfTug1jsfIbxGEyDtCrBs8PUDZY0LQSg13LZiIQhC4cMfaCoVkx+rh8gAd6VU8Mc7Zs9peCQbXBEVDs6AX4R0iSAIVhdvkZSWiMS48zhwcS+S1sXCP7waAhi4zdiw+FPAX6OA9KRiq7MgCMW0QJ4gCMUDlYoKbuk4vXUxEuMvIH7m1xg3KxteV7INBeJOAIv7G7aCIAhWAi0XsR5hmPCkG145NhHZz7yKjQs+NWSDYsxFVFdg0EYgoHpxV1UQhAJALBaCUEKI3rcRXoNGIXqWN+q+/C4yB6UhpGKt6z7LuxcCTQcBEu4kCIIVxltM61MdtlViEOXpg3ULPkHUfY/A21/cNwWhNCGKhSCUFCqG4JVn7DCuYgjKB9co7toIgiDcUryFv0NlRDWrj33rlsPnvS9wJiwC3hmXgNkdgS6TAb9ICeQWhBKOKBaCUFJwcsKpABu1NS6Kx4wqzp5AakJx104QBOHm61uQiAp4ZogdvoqoADj5AFoWsPiZ65/LwnmCUGIRxUIQSgoXLuHxv7OAuy4BdqeBybWAh6YDdR8DTq4zZFZx9SvuWgqCIFhET5l9PDFVbWMP70ZU8yeBQZuuLfKZAmQkyfeYIJRgRLEQhBKCQ1I6Why0U1v1I0w8gw3bRv2A+r0MAZGCIAjWHG+BWLx0BEhZOh7440nD95Z8dwlCqUAUC0EoIVRpeB+qrNtteHPG8AMNJw/D1iOw+ComCIKQz3gL3XIx+psUjO5UEfs3/IqUAcPgMn0ialSvBfw3GajW0WC14EsUDkEoUYhiIQglZLVtQRCE0hJvEesYjsDq90BL3ouFLWzxlJ+X4YMd84DNXxn+l4XzBKHEIYqFIJSQ1bYPbFmJs0OGIuiTSaheIRTwqQTYXwvkFgRBKGFw4sTO2R5/1rVBm+O7UDGkJlz1eIuUOODwSsBGltsShJKEKBaCUEJW297n5oxNkdno4uYMBNQAhlxzhxIEQSihMRc29vG4284OYRM/wYkfqiOqVqvr7k8RrYu7qoIg3CKiWAiClUKlolaIwT3gasJFRJ/eiwUtbNElQDI/CYJQeta4eHN+Gk6Pd0cEHLB8YFfUH/YBQsIigdh9QEAU4ORe3FUWBCGfiI1REEoAR7f+ibCXPkH4FSf4MO/7uT3Ax5UNW0EQhBKoXHDihBMoyTblEVy/PdLSkmCz9zASEmKBS0eAme2AgysMySq4bo8gCFaPKBaCYMWcObEXS996Au4BIdBmjcPU3osR5B4EZGca/JC5FQRBKA1UDMXrT9sjOv4Ezly+bAjeXtIfmN7S8Eo8W9w1FAThJogrlCBYMfHnT6L8b9uR1jUBUc06F3d1BEEQCgW6RHl72sHJzhnuIyZh4wP18dCr1wK5iVco4OYvrS8IVo4oFoJgzURWxHMv2WNBZEXD+4xUwMEZSE8q7poJgiAU/OJ5LsOQPc4VdQL8sXPbekQ0bAN3L1EoBKGkIK5QglBSoI/x6g+ApItAaoJhHxeQEgRBKOGB3MsHN1eZ8FJSPOFdrRXSEuPgOPA9nNq7AYjdD3xSD9j7o8RbCIKVIxYLQbBmTkRj7KxMoFY0EBQAbPgcqNMDqHQv8PIeWZVWEIRSt3ieomIIhj1rhwkVQwBHd+DqeWBRn+ufy/efIFglolgIghWvtu3lFQCtZhW1NaAZNky/KCkYBUEotQvnZSDBDTh7aAeiWvUD9IXz0q4AiWcA52srdQuCYFWIYiEIVrzadkhIPYR8vsxQiCkXBUEQykC8haPNRTwTYwOb7yYAa/sZrLP6wnmCIFgtolgIghWutp1hcxkJWcdx/uAFpEefRp2mXeBa3JUUBEEowoXzRn+ThPq9QnB462qcG/wyAqdMRpWqNYEts4HwZgarBePMROEQBKtBFAtBsCKoVPhkRuOL8U/irxqZqHVCw5Bl2YhbXguu/n5A42clYFsQhDIRbxHrGI7gms2RcWUX/q2WhQYpp5GxIRqh66bAUxtrOIBrXdBNSpQLQbAKRLEQBCsj8ewpPLQmDa26DUdQ53rwHGCPkMh6hg87TSju6gmCIBTx+hYBWHKfG/5bPx7jZ2Zh34Q3cHf9FkDSBWDXQkAzuJAKglD8iGIhCFZGjbs7ALs63PhBejJw8RDgXxVwFMcoQRDK1voWr/asDO2eC4iKugvwLmcoWKVd8VZUEIQcyDoWglBSoFIxvaVhKwiCUMbWt/B3rw6/kAisHfcKzp/cb1gw9NJRw1YQBKtAFAtBsDKO7FyDX9s3UFsjZ3eJQiEIQplTLmqFeKnYM51L50/A849NaosLB4ApDYD9y2ThPEGwEsQVShCsjHQ7DSc8UxFud23NCjK7I5B+xRCoKKttC4JQVte38PDDmy/aY3zyGbift0EFficu6X+9kCycJwjFiigWgmBli+Jxhe3PO9uhJVfa1um3AtCyJbWiIAhlOt7Cxj4ebhEO0N4Zjy21K6HC6GsL5xGvUMDNv7irKwhlGlEsBMHKFsVLSMuER7IGZGYCsQeARX2AR+cAAdWlrwRBQFlf32LoYgDvuKJucDmciIlG+Uo14eLqWdzVFARBFAtBsJ5F8dRK294uSFh3CjM/yYJW9xRQMdTgR5wpwYmCIJRdTNe30DK9EVqzOdJO/gubp1/F8VnjEFW1GvDjC8A9LxssF8S7AuDqW7wVF4QyhlgsBKGYoVLBAMWzV89i37rVcPL0wfn3n0dU1fpA+oXirp4gCIJ1EhqEkU/Yol/2RWBXHKLO7wUWPHn984emA3UfK84aCkKZQxQLQbACqFR0+6kbZn14Bfv6t0WXYVMMH5wRxUIQBMHywnnuOFrZFf98Nw4Prc8G/jaJt9AtFoIgFCmiWAiCFRCXFoeUzBSc+WQI7qp5X3FXRxAEocQsnPfAgCC49UkGvMMML5J0EdizGKj5kAR0C0IRIutYCII1cPoM3pyfhdCASASEVLm+36ci0PN7w1YQBKGMY2nhPN+A+nBy9cDK9o1wdNe/hoIJ0cCKV4Cjf8oaF4JQhIjFQhCsADcnT3j6lFdbRfzp6yb9kIaAi3ex1k8QBMEaA7l1tyinxCs46p2MrMSjSFuXgKDA8vAxXeMiog3wyCzA5f/t3Qd4U1UbB/B/dvfeu+y99xAQBVRcqCgoggNFEUVAFNRPcOFEXMhwg1tEUVBBBVzsPWSPlg7a0r3bJN/znpDSQoGy2qT9/54nJLm5ubk9OTfc955z3uNbcztOVAcwsCByANFNOyH68+Ungop3OwEl+bbnff8H9Bxfo/tHROTw81sMcIHvltcw520zNj5zF/qOPmmOCwYVRLW7K9Sff/6Ja6+9FmFhYdBoNPj+++8rvG61WjFlyhT1uqurK3r37o0dO3ZUWKeoqAhjxoxBQEAA3N3dcd111+HIkSMV1snIyMCwYcPg7e2tbvI4MzOzwjpxcXFqX2Qbsq2HH34YxcXFFdbZtm0bevXqpfYlPDwczz77rNpHonOdv2J7QlaFSfGKiwuRlHZQ3av/CCWoGDQXuG8l0OYOFjAR0Rm6Rb1xU2/k7R+PiZfNgfXDVxHTsjuWffgG0uAJhLQEirKBhA3sFkVUmwOLvLw8tG7dGu+8806lr7/yyiuYPn26en3dunUICQnBlVdeiZycnLJ1xo4di4ULF+LLL7/E33//jdzcXAwcOBBms23SMTF06FBs3rwZv/zyi7rJYwku7GTda665Ru2PbEO2tWDBAowff+IqcXZ2tvpsCXJkX95++2289tprav+IznVSvIFv/62ustknxdu3/jdk9rha3cPFG2h2PRDREQhrA3gGs4CJiE4TXEi6bknbLfNbxPo1R7NuA1GUk4HgD35GyuH/gOxE4K22wNzLgTm9bC3C0jJMRBef1UHIrixcuLDsucVisYaEhFhfeumlsmWFhYVWb29v66xZs9TzzMxMq8FgsH755Zdl6yQkJFi1Wq31l19+Uc937typtr169eqydVatWqWW7dq1Sz1fsmSJeo+81+6LL76wmkwma1ZWlno+c+ZM9dmyD3bTpk2zhoWFqX2tKtmefLZ9u9VJ9jMjI+Oc9pcubrltO5JpjX78J+vCjUfU4yMZ+Wr5jgOrrfc/1kzd1xasbyw31jfnUBuO1ZN/WxftXGNt8XEL6y9/fmg9sHO11ZoRZ7UmbLJat3xltT7jZXt8gWpDudUElpvzldu5nLs6bFaogwcPIjk5Gf369StbZjKZVFekf//9Vz3fsGEDSkpKKqwjLQotWrQoW2fVqlWq+1Pnzp3L1unSpYtaVn4deY+8165///6qm5V8hn0d+WzZh/LrJCYm4tChQ5e0LKj2ToonV9vM5lIc2bsJm+tpAC9PoLQYyEqw3RMR0TmNt5AW4Yc/2wurxYDMF17Fluces6WhlRZgST/72H4guDlLlaguDd6WoEIEB1fsBiLPDx8+XLaO0WiEr6/vKevY3y/3QUFBp2xflpVf5+TPkW3KtsuvExNTMeWn/T3yWmxsbKV/hwQncivfpUpII011j8+wfybHhdRcuck2NKi4vZKiAkQ88ha6XOcCH6MPrDJ77Nw+wMjltv8InRTrG8uN9c051IZjNczbBcvGXYaMPNsFmf0puRj3HeA1wQMNg/2RdSwJbp5+0BuM8gcDydttb3TzPzH3RR0st5rAcnO+cjuXz3TYwMJOBnWfcmJ20rKTnbxOZetfjHXsBX2m/Zk2bRqmTp16yvKsrKwaCSxkDIo4WxnSpSm34vxchLvb7ndu2op9c95A/ZFjUTzrZTwYGgU3sxuy8osB12hA7rOynParYH1jubG+OYfacqx6aAAPD9vj4nwgzOSNIs9G2HdoC3LGTsHRx25DUHA0PNZ/hqD0TbYV9S7A4E/PayxbbSm36sZyc75ys18Ud+rAQgZq21sDQkNDy5anpKSUtRTIOpK5SbI+lW+1kHW6detWts7Ro0dP2X5qamqF7axZs6bC67JN6WZVfh1760X5zxEnt3aUN2nSJIwbN67ClxMZGam6Ynl5HZ+zoJrYAxn5bP4I1ky5GXOBhDzA6OahAofcI3ug0ZWg7WUDT6yUZwQKDgNuRvlQOCvWN5Yb65tzqI3HarDViPRiHR5duAeelix06pCGHYfeRJ8frLhjhQW5H0xRF3mw8D5Am39ev7W1sdyqA8vN+crtXD7PYQML6VokJ/PLli1D27Zt1TIJIlauXImXX35ZPW/fvj0MBoNaZ/DgwWpZUlIStm/frjJKia5du6rWgbVr16JTp05qmQQRsswefMg6L7zwgnqvPYhZunSpGk8hn2FfZ/LkyWofpIuUfR0Zl3FyF6nyZBvlx2WU/5Jq4ofI/rn8EayZcpP3S2cota3IMLx0ixZfRdrSLSM/HXDzk58P202WOfl/VqxvLDfWN+dQ247VCF83LBvfW3WNktTejy5wxVu9GiK6bzGM400Iim0NTUEm0O95wOQBJG05r25Rta3cqgvLzbnK7Vw+r0YHb0uTjqR+lZt9wLY8ljkl5I+QVLIvvviiSicrwcKIESPg5uam0sfao7Z77rlHpYX9/fffsWnTJtxxxx1o2bIlrrjiCrVO06ZNMWDAAIwcORKrV69WN3ksKWkbN26s1pHB382aNVMpaGUbsq0JEyao9eytCvKZEiDIPsi+yD7JvklrBH9Q6IJJ6sOvhgF5aYDH8RYw+U+OiIguSipac2E4tG4tUWR2x77tf2NnURqSWt0EHN3ONLREF0mNtlisX78effr0KXtu7zI0fPhwfPzxx5g4cSIKCgrw4IMPqq5JktlJWgk8PT3L3vPGG29Ar9erFgtZt2/fvuq9Op2ubJ3PPvtMTXhnzx4lk+iVnztD1l28eLH6nO7du6sJ8CSQkHkq7CSIkZaR0aNHo0OHDqrrlexv+W5ORGeav8J+5azM7gP4eloprA0PADERwOG/gawjQEgrYOz28x5QSERElWeMcjNn4ontb0KvycW0wVq461zx3YBPESYTkn430jZBKX97ic6bRnLOnv/b6VzJGAsJUqQrVk2MsZDPZX/Q6i03+6R4BSW2SRvlPziZLdalIBmbF8xBm5vug781y3bFTGbaduJMUOWxvrHcWN+cQ104Vstf3Hlx3g8Y1TMQxtJEtH96lpqpO9TLiPz5QxH+8G+232C5yOMdgbpebpcCy835yu1czl0ddowFUW0h/5lJUDHj1jaqSb5Ek46son3IOngYbQaPgn9QNJBo6w5IRESXpluU3KT1Ise9Hp5dY4ZXqSd69mqLB/zrYf33byPi+0DgjkyEe6fZZuoes5GtF0TniIEFUTWRoMLfOx/X/3ArtDn5+GiGGRueGIJ+I/7H74CIqBpIcCEtxvbWi7FfBeMBj3B4XT8IY0x/4a0gf4RLa4W5mN2iiM4DAwuiapRxcBfu/S4XUY9OhPXDALRv0NL2gnckcO1btnsiIrrkrRd2EmDoXLxx1Fcjgy6BUlu3VaTtAQIbA4YT6xLRmTGwIKpGLjAgutADDb0boF7LHidecPcH2g/nd0FEVAODukNLd2L8HjPSGscDTZoh0eyJMBnM/eBqIKgpvxOiKmJgQVSNJJiot3htxTSzkoVEbkmbgXYjbEEGERFVW7eorVuM0G0xwmAKRlp+MdK+84T1/ekI1+qBHx4Cek0EfKL4jRCdBQMLompksVpQaimFXquHNisBeLcTUJJ/YoWWMtEjAwsiourtFtUJQ9qMxXMIRHrcf1jXTYsu5nxkrVmK2I3z4Rp72fFJTM99Ej2iuoSBBdElTG0oys9dsWv1zyi6bwJMc15Ds4aNgbB2QNfRgFcY/8MiIqrBblGlLiGqW5RGnwn3biYc+PVVTPrGgiO3eqKhdIvSaIHujwC9JwF6E78rokowsCC6yE6et0JIP175jysr0A9f9NJiRKAfENAAuGsxy5+IyMGyRT26AOjeyw/xnXPQqHEjwP34iq4+QMp/tscyDZjFTWbQrcldJ3IoDCyILvG8FUKCCvmPK8vsjZ87ajHC1xuwWABLCaA1AFotvwciIgeZ68JF449pv5sRXJSKe2a/jfDHH0Zs0+bw3vEnwn8bc/wdGsCzMXD3N4Avx18QCQYWRJeIBBUtwm1XsrKz0uRSF0rSj6HndgvQJx8o2lrrZtsmIqptg7rdtxzDC9umAtuAN2ebgffmIjy6EZB+EFjxLlCUDRzdAZi8OP6C6jwGFkSXWMK+zVjz8kRcNvU9GNOyMeZHC7zu1zOsJyJy+NaLHrihy3gcO2qGb9ExLG36J67UBCLr0BFYo2NgGjQL3sH1genNgMJMYPRaBhdUpzGwILrEsrJTYNgbj5Tk/WjQpg9Sf/oC4Q3aAImbWfZERA5MgovvJw4tN1N3K7Q5tAeeU2bgkfv1cA2JxayBsxA24EXgmxGcrZvqPAYWRJdadAQm36XHV9ERMLq6ITyinm25uYRlT0TkZDN154d3QfxrU3G7NR0li79FRqu9CPONxdFiPYKLj2cBLC1i5iiqkxhYEFUnmRBv6ZPANdMBvdG2TPKiExGRU8zU/djiI5A0UcHFaXgmVQurLgBpOZnYvzQIB7ouhXdJCbzXL0D4VY+zWxTVOQwsiC7y3BXl561Q9h3C+zNKgSaHgMgwYOcPQI9xQFAzYOx2/sdDRORkg7rFvqOt8dYiN4zK90a6ORtrm2rwQ9x3iF27QM1/Ad9YhLfqyzmKqE5hYEF0CeausM9bIQLC6uPwjT0QG1YfQEG5o8/IoIKIyEm7Rfm4GVBqCsKjX2+GFRq4NngaT3aNgmfmPhzNewYR/zwHrH4WaDsMuPJZIDOOQQbVegwsiC7B3BUlJcnIMh9EVmIxEH8EXR9+Dj4+IRywTURUS0iAMfOO9ijSmJCeX4JR8zbgqa8z5XISWhjG4smOeng3igA8goB/F8Lr98cR4WEAbp0HuAcCoa1r+k8guugYWBBdRBJU+BYcxNfP3YlvuljgUQC8OceM7dPHocfVI1nWRES1SJCnCd7e3tBoNGXdpHbHH8XON95B/l+HcJ+kFgcwY3YptrZohcHuO6GdfxOgMwJPp9b07hNddAwsiC6ygpwMtDmkQdt7HkdQg5awdjmCFs062F70DAX6/s92T0REtXLm7hnR1+P3sHTkHQxQrz3XOQ3FcMXSlBa46oqG8AoMgPu/38M/OwlNW3U9sRHfaMDVt+b+CKILxMCC6CJr0LYPGqzccmJB+Vm1PYOBnuNZ5kREtZQEF19OHl42yNvu4O6NaPDgHLzs9Ts2NNRi0D8W9N9gAa59AmnFOqRAj+TLH4d/bC+0di13eiaZA30iq/8PIToPDCyIqivNbP4xoDALyDwMNL0OcPVh2RMR1YG5L0SL8D7Y9MXHuNlahCugw5e7f8ehnul4Pc8P96z5BoeDzZhT/yNEr/sILxUcRRNtMVKK9EgtdUXckFkodfFDYPJhBPjbggwP32CERDVUjzdtWgNdYENoNTroClPRtH6DGvm7iRhYEF1k/637FWkPPYqAd95A0479bUHFu52AkvwTK9Xrw8CCiKiOadu2M9oef9ynXruyVo30uMsRU5KDMbnp6Prq85jbqyd2+V2OHgmL0G9zPCa2mATPfCse/9aMyI6pCDSWYF+eK/4e8DqyS4BO45/H09feDK3VgKGbFmLT/bfBu34bGBKz4B0cAX+jBZkFxUhPS0NRbBSg1cJ1zzq4aPQIiW6u9iFu5xoURjeD2dcfxpQDMCQmoGGrDoj09Ubcwf+QofWBLqobtMU58NMVwM3DE0d2rYdP01YI841CyZblgMUMQ3BA5S0t9gtsrn6Axqvay56qBwMLoos8d4XV0x0rmwODPN1tC4rzAJ8ooM+Ttns2axMR1XkVWjXCe5SVx6bQBrjF6A0Pn1BkJXfDsYTteLFhExQk7IXF7x0k938Wh7KOwe+JN/GT64vYE67BoZ5avGmcCV2pBstdAjEzbR6KM+dh3utmfN0rGs+ErsJvSX6IXOmCoY/pUKrXYPwCM/xKLOjROUl9bsmXYZh1lRZ/tNGi024LJnxngeWWaYCuBDvWBSO/WI9nOw/DVbk7McFtCVYE3o6Gs5dh+BhXjOj5Eto9+RBc0s2I7JWOfSVGeBrNCJ+4CdBogdT/kPDxCGQVl8Da/TqYOj6uBr1T7aOxWq3Wmt6JuiQ7O1sdTFlZWfDyqt6IXb5q+Vx7Bgu6sHKrbO4KyQoiaWZv/elWfDXwKzTzb1Zni5n1jeXG+uYceKw6T7llZR6Ft08wiovysX7pZ8gIqQ+LhweiUYBWx0/Ut8UfQKKlEKUm/7IWi2BzGjx/HoWjuRYUNmiAhN7Tz9piUXo4Dh/uMaLQrIdH6UHEx/yBHI9SfPpaKeI6FaFnWCb2FLrgkca+yDdoEZTmjrCjt2DsoHYwPzgaP3fSofGd/0Psmu/Q8Niv+HdvAAyJOjx2rwHNSiLwdMMb0bxhR9sfJsFHaCvb45RdQGkhL8I50HF6LueubLEgukhzV7jqitTVp6O7U9H6gAUoLGLZEhHRRSNBhTCa3NDt2spTmLcMa4OW9iftyr3QpC38pSuS0R3tAhoCTTtVeF+D1idaTeyvtTneKi9KNA/BZCoAGh1ErE6HdJMHpNPT6xoLMgwuyCsCpi5Ix03f56N+r5tRVO9HbPtjKp6bb8ZD1wxDamQUisOKEbI7E/dt34D9Lq8ipnEKdmR5QB+kg8sDP8NH64Gwb4YDqbsAg5ttzo/wDuw67EQYWBBdpLkr1sx+Ht6PvozS/3bhya8s8Bp2/IpC0lbgo6uBu5acuCJDRERUnWS8wzlml6o4CP1416WuFVviyw8T7xljD0R6IK3wduRmHUV8m2SMa9wZLm62hCVpuUWY9+XnyMk5hs/yLZiy9Du8cKsWB74ejJv/tuDWsS8i3NcD+z64G8UfDQUGvQDfiM4INVtsH8LuxA6NgQXRRZCVkQzfv3YgZfBeNO97CzJ/6ozwBsfTzFotQHGO7Z6IiKiWOjUQaVyx1eR4l56wu4eUzVh+ny4SaWYzvA+no+3+L7ClJAIZrg2R+rcP1seU4ovNb6Lez8DzRUfRRFdsa8kY/hMQ0b4m/kQ6CwYWRBdDvSg8/IAeX9WLgrunH9xdjv+wFucDaXtYxkRERJXMWG6f8+NYXjGesIYiadFRAEcR3e4O5Jt0MO7Nx3Pff47dD9wKvw4dsWHO82j/yzQEjfhcDQpHaTGgM9g2zNaMGsfAgugiZIKqQFLq/TMD6D0JyDoCfDfSdoVFfvCIiIio0laObyfdXm5iQduYD/m/dnixP8a374ZN2fvgt1WPzTd3RMSqBXBfMgbRphIUm4F9+a7wcS9B2LjNQGkRoDdxYsEawMCC6BxUlgnK192IrH3xeH1uKdA8HggLBta9D7QdBgQ0Au5byasoRERE5zGxoPwfm+NeD4//lACNvgjuD7tCkz4fL75eisKYloh+8Q2kHdkPzZ1P4akhJryg0yJ04SNA4kbb4G+3AMA7EnDnxb3qwMCC6AIyQckPnvwIavzCcKBDc/j4hQGwBR2K0Q0IOz7WgoiIiM6J/B8rqdzt3aVGfaFFkTUHr7ZLh1XvhaDccHgHBGP7c6MQd2Qutq3+CcGXPwXt/EHA/JtsG7n2LaDJQCArnhf6LjEGFkTnQYKKFuHeSMpNws74nUByEvo8PxfuHr5A4maWKRER0SVoyfjtkRtOBBnzNmD4h2vVco0+ADdnaBH44QyUbt6C7Ju/RUrybiDYD77+jRC66yfgx4dtXZNHr2U3qUuEgQXReZKg4vofrkfUwXyVp3vf/NfRusPVLE8iIqLqCDKOt2TYx2JM+SwPLSe7omjHHmQ88SiSDbmYcYMWfmZXfHbNhwgbNNc27jF5GwOLS4SBBdF5ykg8gP5/5qH7sMdgvTwADZp0tb3gHgh0GW27JyIiokseZEjX5CLXKPxvixnYchiRLe7GmMtD8FjOYbR/ehYymx2GyT8ChaWeCF/1LtBoAJCTBOQetc38zYxSFwUDC6LzpM3MxbXrrPC6JxRNOvW3ZYNKPGx7sc9kwOTBsiUiIqqJsRjzdJi4wgxPsw869O2OwboI7PjkTbT4zhNxc26A9+pv4L1oNMJdSmwbYBepi4KBBVEVpOQUIT43C/tT88qWqWBi407bEwkq3u0ElOTbnt+zDIjsxLIlIiKqwW5StiAjGMsXJsBH0wZuj6xDzsG38OQXZrh6NIDrlKlI3v4XvLe/h/D8Y4ClFNDq2VXqPDGwIKpCitkH52/A/iwLrNCUpZitQH6MJKiQ/puSYta/AcuViIjIgYKMfSlt8OiCADx2VTh8JuYgMrw+3KOioLl5LF69PhKPu/si9Nengb3LTqSq9QwFPIP5PVYRAwuis5Afo8JSC94Y3AYNgj1RkrILWYV7sPuvrSiZ8hrqvTYDDUMDbf0zg5oBIS1YpkRERA4WZMhFQReNP175UXofaOFqiMPMob5IeG4Utkuq2jU/w7/jAzBKYGFPVdv3f0DP8TX9JzgNBhZEVVRf5q0oOIjsISMw8S4djnkBdwRr0cLT1xZMTDzAsiQiInKacRgbcNcnG1Wq2n5FWkTOfhPH/l6CUElHKz0RhNG9Yhp5DvI+IwYWROfA0y8Ee8bfhv/17An3wBD43OKKsIBYliEREZHTdpHKxdNfFOCm17xgTE7EL89OhseEh+FXvym8Ny9D+F+TTmyAg7zPiIEF0Tnw8gtBn5HPnFigMkFtBjIOAb9NAYZ8AQQ1ZZkSERE5UZCRow1FfngbJOQcRFbuMTy74RlgAzDnLTPw4Vy4e/kicee/8En+EWHFeUDSVsDVl4O8T8LAgugMg7blSsb+lNyyZUeP7MHqea+jy7DxCPZwBWa0qHgVw8gUs0RERM5Exl5IYpaxX9m6PAW0HYexnRtDl5eE7dcvwOVRbbHz7yXwfeZLDBvnjvlu3gh9qyNgNZ8Y5C2JW4xuqOsYWBCdJqi44vWVKCgxQwMr6ntr1Q/PsV17EfX5nzjW+zoEt+0J3DgH8AoDTJ7sd0lERFRLxl489XUmAFe4Goajp3sEmvW9GWt9TSje+w52b/4DoRJQfDn0xCDv+1bazgNkbEYdHofBwIKoEvLjIkHFjFvboH6gO0zWIpiTtyO+JAUTJujxVaNYwMULaH0ry4+IiKiWjr2QVox1B9PRIMgDprAuGPb+uyj84TVg6Sak3fwtUjIOAD6e8HX1Qug/M4B179fpcRgMLIjOQH5Imod5ISsrC3899RSKkuLhOtwTviZfICcZWP8R0OEuwDOE5UhERFTL0tOW7yIlImNGY1zvMPy1ZhXcR96LJR00WNRFi2ZHjXjxtjcRFtkZ+G6kreWiDgYWWjiwKVOmQKPRVLiFhJw4gbNarWqdsLAwuLq6onfv3tixY0eFbRQVFWHMmDEICAiAu7s7rrvuOhw5cqTCOhkZGRg2bBi8vb3VTR5nZkoT2AlxcXG49tpr1TZkWw8//DCKi4svcQmQIygsykPS0f1oOuFpNHv5bfxw/Q8I9Qi1BRYrX7LdExERUa3sIvXTmB7q9sndnZBmisGjf+owbGE6HhnwCBoMew0P+47E458VIDM1yzbWAhrbBopybUle6hCHDixE8+bNkZSUVHbbtm1b2WuvvPIKpk+fjnfeeQfr1q1TQceVV16JnJycsnXGjh2LhQsX4ssvv8Tff/+N3NxcDBw4EGazuWydoUOHYvPmzfjll1/UTR5LcGEn615zzTXIy8tT25BtLViwAOPHc8KUuuDAhuUofPBxFGVnolnbK2xBhWSBSttT07tGRERElzi4aBHurW69GgWWBRr2IOOJX62Yui4M999wL3K17ti5byeSOo4AvCOAg3/akrzUoeDC4btC6fX6Cq0U5VsrZsyYgSeffBKDBg1Syz755BMEBwfj888/x/3336+6r3zwwQeYN28errjiCrXO/PnzERkZid9++w39+/fHf//9p4KJ1atXo3PnzmqduXPnomvXrti9ezcaN26MpUuXYufOnYiPj1etI+L111/HiBEj8MILL8DLy6tay4QufSYo6VdZJjQI8/pocE9o0Illf7wAbPva1o9SBmkRERFRHR6L4YZ9c16F9/6DuHOkB360liDUxdv2prhVQGGWbTLdWs7hWyz27t2rTuZjY2Nx22234cAB2+zGBw8eRHJyMvr161e2rslkQq9evfDvv/+q5xs2bEBJSUmFdWRbLVq0KFtn1apVqvuTPagQXbp0UcvKryPvsQcVQoIS6WYln0G1KxPUwLf/Vv0ppV+l9K+Elwd2xGjVPUoKgWP7gcses2WAqKODs4iIiOo6e2tGx1g/dc4ww2cgXu08EMacAmzavg7wibJdgJQxF59eZ3tTyn+2+a9qaSuGQ7dYyMn+p59+ikaNGuHo0aN4/vnn0a1bNzWOQoIKIS0U5cnzw4cPq8eyjtFohK+v7ynr2N8v90FB5a5EHyfLyq9z8ufINmXb9nVOR4IPudllZ2eXtbjIrTrZP7O6P9dZpOcWobCkFDMGt0H9IA/oClMQ5u2CjH1Z6L3FCmvnLFiL/gPm9gFGLgfC2tjeyPKsFOvb+WG5sdyqG+scy4317cLIucKycZchI68Ltm7fgJinFuNg+w+xbuzrSA+6D37No+HmHwCftEMI+2IokHHQFnA8uLrKFydr8jg9l8906MDiqquuKnvcsmVL1T2pfv36qsuTtCoIGdB98h9/8rKTnbxOZeufzzqVmTZtGqZOnXrKcummVROBhYwxEWfb77qoOD8X4e5ApAdgytiN/958EW6PvwB3qycuj/eHoUCHLBQDrtFAfrF8iTW9yw6N9Y3lxvrmHHisstxY3y6chwbwkPOH5i0w+dpxKLBooPviD0z95x/MHvAvjvpq0He7Fv1ueQZBxhLgj+eBxP2Axsvhj1P7RXGnDyxOJhmZJMCQ7lE33HCDWiYtBqGhoWXrpKSklLUuyNgMydwkWZ/Kt1rIOtLyYV9HWkNOlpqaWmE7a9asqfC6bFO6WZ3cknGySZMmYdy4cRW+HBnjIV2tqntshj2Qkc9mYHEqYy6QkCcTZ3qgpCAXOcn7kJd/FE1b9kHx5EfQsEVHaLZ9CxQcBtyMUpDV+v05G9Y3lhvrm3PgscpyY327eLy9gRkPD1VjL/an5GJUbgnual8fnjkHELJhJsy3uyElPwNuGYkIL0kBtAXA+o+BDiPOmLq+Jo/Tc/k8pwospEuRDLbu2bOnGnMhJ/zLli1D27Zt1esSRKxcuRIvv/yyet6+fXsYDAa1zuDBg9UyySy1fft2lVFKSCuItB6sXbsWnTp1UsskiJBl9uBD1pFB2vJeexAjA7plTId8xpnIOnI7mT19bnUrn7qXKg7Y3p+aByuOl09EKF6+RYuvIkLV8+CoJrblC0fami/dA6QwWYSsbzxOHQh/31h2rHPOobYfqxG+burm52FCtns9vLS0FEAUXHu/it8a9sS+gZ2xtlEouutc4PPXd4hY/Qq8AurbUtW6+gC+MQ5VbrUmsJgwYYKaOyIqKkq1MsgYC7niP3z4cPVHSirZF198EQ0bNlQ3eezm5qbSx9qjunvuuUelhfX394efn5/aprR62LNENW3aFAMGDMDIkSMxe/Zstey+++5TKWklI5SQwd/NmjVTKWhfffVVpKenq+3Ie5gRqnYM2JZZtoV9wHZWiQX6UitgsQBpe4HFzwDXTLUN2JYsUBywTURERFWYByPjpFm8NZMfx5L4N7Dilyfx2gdm7LzeG11kgLdoORi4aa7TlqtDBxYykd2QIUOQlpaGwMBANa5C0sJGR0er1ydOnIiCggI8+OCDqmuSDPaWlgRPT8+ybbzxxhsqZa20WMi6ffv2xccffwydTle2zmeffaYmvLNnj5JJ9GRuDDtZd/Hixepzunfvribjk+Dltddeq9byoItPDnYJKmbc2kbNsl2SsgumoqPA/kP4/FUzrE0PAdHhwNGdQHEeEG5rHSMiIiI6v1m8feDqOh5P9Q9BfNNU+HnpsDM/FYgJAfQu8F7/DcKDogC9yekuZmqsTBFUraTFRVpSpKtVTYyxkM/lGIsTtidkqfSyMtmNb8FBZA8cgj3DL8Nl9zyJzT98hDbX3wWfkmPImj8C3nd8DA0DC9Y3HqcOib9vLDvWOedQl4/VhONdr4/lFWPUvA3qwqbOYsadCXPRe98+PPSgHp75Vkz53Izm3VMR7lJie+PY7bB6RyDraBy8g6NqZPB2Vc9dHbrFgqg6ZXnp8NADOjx/5Y3wDYpCn5HPqOXWffv4RRAREdElmVzvxXmJiLnOEy+2aoj8xP3Ii52LwqsmYWd6Anw2vYYwgyuQlwb88gRw/cuAb5TDfhMMLKjOsV8xEBVm2NbpkOqjgU9wlO0A3rEQaH6jbTIbwRm2iYiI6CJ3kcpxj8Ezm8zAplQAXnCNnoh5Lq5wnzQNT9wRhVEH1sI7/jDCpVt2/jEGFkSOOlhbRFri4eWmQ258MiZ+YwbaJgMBxcCSCUBER6D1bUDkFU7Vx5GIiIica4C3sA/yPmQJgfalCThw7D38NPMx3LDaCtxmgKNjiwXV6cHa6XHb4Hf3BGB0dwQFRCNQ7wNvN7+Kb5LnJScG+xMRERFdii5SvscHeU/4Qbphh6hB3q0GeyOn9y7g0CcOX+gMLKhOkqCiRbg3dhbo8GVPLQYXZyMquiU6z/0KvoGRQKJkbSAiIiKqmRaMY8cHeT/xqxk9ipJRmhiAgJYb0N6BE8kwsKC6zdcbC7trcZufbWZ2X2lltJzoJkVERERU04O8N6xJx6GlG2E1eDv0l6Gt6R0gqq6xFZJaVvouBhUfRtrBTWp5aXo6+myxADl5QGY8MKMFkHkY8I4A6l/OAdtERERUY8J9XFUPC98mjfBdDx2Kwx03I5RgiwXVqQHbXuZETN3yLjKTAoAeK+GWW4IHlljg9aDBlmlBFGQAfvWAmz8EXH0l6XZN/wlERERUh2nz0tUcF3LvyBhYUJ0asK1zCcS0ehq80vER9VpUs85I/ekLhDdoc2Jcheb4QG0JKoiIiIhqmOu+nXjyKwvSPXcCHa+Co2JgQbV+vgr7XBUyYFvr4oEcNw38YpuoZUaTG8IjYm1vsJTW5O4SERERVco3vD5SO+cjILw+HBkDC6oT81VI6jZJ4Zb130F8/Hop0Ogg0LWZbVzFkseA698BjB62N3MiPCIiInIgnl7+qOedh1QvfzgyBhZUJ+arKCk5iqydv8Pk5Yv4IZejbWRD24oyrmLPz0DWESC4OTB2OyfCIyIiIoeSlZ6MA8me8E9PhiNjViiq1SSo8PfOx5OLh0Jz92OI37MB/Se9i6CIRqeurDMwqCAiIiKHk3ksFX7bXdS9I2NgQbU2raxdRlEGklwKEf/WI2jd62Zb9yfJ/CTyHPsAJSIiIspv3h5PDdepe0fGrlBUa8dVhCHRNq4iIRX3/WJBxJSm8NWZbXNVXP0a0GkkcPgf25s5roKIiIgcldVS8d5BscWCauW4ik+v98Zry6bDlJ+IIN8INEo3wdvkfWKuioDjYyy6PMhxFUREROTQ3Pasxz2/mNW9I2NgQbVyXEWAnwZ/N9cgJScZAWH10Xret7a5KrLibSu5+NjuPYI4roKIiIgcmsngCqPBou4dGQMLqpXjKhDgi3l9depePQ07nvd55SuAwY1dn4iIiMhphEQ0Ru8mx9S9I+MYC6pV4yq8dIVqXEVKSira77UA+QVA8Rbg/SuAe38DbnrfFlj4RNb0rhMRERFVSam5FHlmjbp3ZAwsqNaMq/ArPIikz2bDlN8G1v2H8fi3Fnhdsx9wB2AuBqxWINCxI30iIiKikyXu2YjC3wOQHrsRTdv3gaNiYEFO21IhQYW9+5OMq9Am5qJo016kJOxFk57XIWOeF8IXD7e9gd2fiIiIyEkV1GuJz3r/gt71WsKRMbAgp+/+5GrQwXxsHxJyj2DCfXp8VS8Kbh4+cGvTHwhfeSKdLLs/ERERkROyeHpjW6wWl3l6w5ExsCCnbKWwd3+SlgoZU7H5hbvh9tdmuD7kDt/MRCAjEdjxPdBrIuATVdO7TkRERHTeTMn70G6vBaawfUCzznBUDCzIOVspXLPh45uCgpzdSFu7A02GP4Siaw7hh9UTETrvphPdn3o9XrM7TkRERHSB9EeTMfhvK9KbJMORMbAg5xukXRqHv+ZMwtOLLYhNtmLSNxa4/PQF6rfrA5SOBOr1ts1Twe5PREREVAuENm4H937zYGrcDo6MgQU53SBtQ2IO+mwxo+vQCQi5vjW8BqUi3A1AbgrQczxg8qzp3SYiIiK6aAxaPbxhRbHWsU/dHXvvqE6rrPtTeuI6BOu06LZ2h22lzHjgk2uAknzb8+E/AbE9a3CviYiIiC6uo3G7sWe7P7yb7kbDNo57nsPAghyyhUKUH6Tt45WLCavuwPoZeWh1RIfGPy0DPEOA/GO2oGLQXCCgEeB/fIZtIiIioloi1+CKTJ0WOoMrHBkDC3LYFgrharCiY6wfEpN3I+pwAZo9MBGRVhdgTh9g1F+AdwTgGQZEdWU6WSIiIqqViqMb4NMrdBgd3QCOjIEFOWwa2fSkXUj64V0Y85qgYO06vDDPDK/L3RAuM2nnJAJZR4CwNsB9y22tF0RERES1kKY4GxqLVd07MgYW5HCT3UVb46E9moPCzIOo9+M6pF6zB22vuwvZMSEI/3XkqTNpM6ggIiKiWsztv62Y9okF6fduBdpcCUfFwIIcbrK7TY/dBPP+g3hquB6uE9zxQ6ARnvmJ8GzZC2jAmbSJiIiobvEKjkJCu0IEBzv2pL8MLMhhJrsryduH5J170PShx1BkLsVXbrnw/f4h24R3Gi3Q/RGg9yRAb+K3RkRERHWGj18IGgbkINXPsbt+M7CgGmml0BcmY8bQK+FrTcDy9yZi8s8atNtnxejFFrjMeQL1LxsO/PAQYLYAdywA3AJsXZ8YVBAREVEdk5uVhvh0d/hmpcGRMbCgagkmjuUVY9S8DaqVIqj4MJ7aNhcN7+2EYE9/+P5rQf8ho+HdOB9ehlcQHtPY9uZeE4FejzPbExEREdVp6Ufj4bfeDent4+HIGFhQtXV5CtBk4vmOGuSYtCjYVwRN8moE+HVH8aKv0OrzvoC5GPB0A7zCbRvwcex+hERERETVIa9pR7x5+0+4t2lHhy5wBhZ0Sbs8FVrS8GwrC+o3a42sdZsR++R8DB+nA27Uov+fE4D/miNM5qO49zfAarV1d/KJ5LdCREREZKfTosioUfeOjIEFXZIuTz65u6E1ZMKz8ffo+GwhDoy+Gt1uG4uk5q3xsdsx+P74KEKvnwWEtrZtwH5PRERERBW47t2AISvMcPXcADTrDEfFwIIuTjDxxW8wFqYgKC8b8GqLxw99jZySDLzVwhVZL92FjiFh8N3yAXy7PAjs/AEwmzlbNhEREVEVGKGFf4FV3TsyBhZ0zoGEsLdMFBUXoUHxWng1WoKO+4oxeqkF+KEf3EpeQV52AhatGIPQ4hdObKTDPUDLm4FGA9jliYiIiKgKwqKbo3Wr95Aa3RyOjIEFVblV4sn3FyDRzRYph+VkIEzvhYm3XIaoO57A265GDB00BtYeGWjweT8YR68BYhsDh/oAne49kS7WPn7C1ZclT0RERFSLMLCgU1oj7CSYeOr9BThmdYeHOQfv//o2Rj2kQ46bBo98ZobeXYeOgd2QPOtZPPnbKIQve9z2Ro3ULBPgHgAMmg24eLOUiYiIiM5T/K51yPwtEDnR69CwTU84KgYWdUxKThHic7Og0cjZ/4kAYvIH3yFBE6GeRxTuQonBCo2uAB8t+hwbbuuIRlfdh0PF4XgtZTt8DKVAOx28tRr4efvB77JBQKOGJz6kfMsEgwoiIiKiC1IQ0QR/dvoZHSOawJExsKhjrRIPf/IXdue6QG8pQEzhLiR6+sLTnIG3f/0cedOeRkCra5Bw1/+w17cA7/fXYsI9OryW8zOaulwDvPwlkHWk8gAirE2N/V1EREREtZnZzx+rmmrRzs8fjoyBRR2y/78NuDb+Czww8A54efki5P7PMWWoFgkBGixrq8Hwvx5G+FVD4fraDEQum4TLk3fBV2tC6J1fAGHtADc/W/cmIiIiIqo2hrQ4NI63wBAVB8Bx0806ds4qBzVz5kzExsbCxcUF7du3x19//QVnUJy6BzHHshC692N06dgJ1rcm4BlrCuZkJuOuxlkIv/tLtV79Vj3RbPhHaDbiN4SOWg00uMIWVBARERFRtTMmxOGu36zq3pGxxeIcffXVVxg7dqwKLrp3747Zs2fjqquuws6dOxEVFQVHZg50xVs36DAhvD9c3bzQrM/tQIv2thdPnvE6oNyYCSIiIiKqMcH1W8Jy+Rfq3pGxxeIcTZ8+Hffccw/uvfdeNG3aFDNmzEBkZCTee+89OAuzycf2wOBiGxsht/JBBRERERE5DJPRFVH6EnXvyNhicQ6Ki4uxYcMGPPHEExWW9+vXD//++2+l7ykqKlI3u6ysrLJ7q9WK6lS05zDGfFWMolsPI6u5bT/o7OR7sn9v5bNpEcvtUmB9Y7lVN9Y5lhvrm+M7tHc7Nuzwgn+97QiKbVWtn52dna3uq3LeysDiHKSlpcFsNiM4OLjCcnmenJxc6XumTZuGqVOnnrK8RrtN/bICuGtyzX0+EREREZ27eX8DeBA1IScnB97eZ56bjIHFeTj5qrVEcKe7kj1p0iSMGzeu7LnFYkF6ejr8/f2r/eq3RJzSbSs+Ph5eXl7V+tnOjOXGcmN9c3w8Tll2rHPOgceq85WbnOdKUBEWFnbWdRlYnIOAgADodLpTWidSUlJOacWwM5lM6laej8/xMQ41RCokAwuWG+ubY+NxynJjnXMOPFZZbnWhvnmfpaXCjoO3z4HRaFTpZZctW1ZhuTzv1q3buX1DRERERES1CFsszpF0axo2bBg6dOiArl27Ys6cOYiLi8OoUaMuzTdEREREROQEGFico1tvvRXHjh3Ds88+i6SkJLRo0QJLlixBdHQ0HJ10yXrmmWdO6ZpFLDfWN8fB45TlxjrnHHisstxY306lsVZ3zlMiIiIiIqp1OMaCiIiIiIguGAMLIiIiIiK6YAwsiIiIiIjogjGwICIiIiKiC8bAgoiIiIiILhgDCyIiIiIiumAMLIiIiIiI6IIxsCAiIiIiogvGwIKIiIiIiC4YAwsiIiIiIrpgDCyIiIiIiOiCMbAgIiIiIqILxsCCiIiIiIguGAMLIiIiIiK6YAwsiIiIiIjogjGwICIiIiKiC8bAgoiIiIiILhgDCyIiIiIiumB1NrCYNm0aOnbsCE9PTwQFBeGGG27A7t27z/q+lStXon379nBxcUG9evUwa9asatlfIiIiIiJHVmcDCwkQRo8ejdWrV2PZsmUoLS1Fv379kJeXd9r3HDx4EFdffTV69uyJTZs2YfLkyXj44YexYMGCat13IiIiIiJHo7Fardaa3glHkJqaqlouJOC47LLLKl3n8ccfx6JFi/Dff/+VLRs1ahS2bNmCVatWVePeEhERERE5Fn1N74CjyMrKUvd+fn6nXUeCB2nVKK9///744IMPUFJSAoPBcMp7ioqK1M3OYrEgPT0d/v7+0Gg0F/VvICIiIiK6mKQNIicnB2FhYdBqz9zZiYHF8QIbN24cevTogRYtWpy2sJKTkxEcHFxhmTyXblRpaWkIDQ2tdCzH1KlTz//bJCIiIiKqYfHx8YiIiDjjOgwsADz00EPYunUr/v7777MW6smtDPaeZKdrfZg0aZIKWsq3jERFReHw4cPw8vJCdZLWEgmAAgICzhpxEsuN9a1m8DhlubHOOQceqyy3ulLfsrOzER0drRIenU2dDyzGjBmjxk38+eefZ43CQkJCVKtFeSkpKdDr9aprU2VMJpO6nczHx6dGAovi4mL12QwsWG6sb46JxynLjXXOOfBYZbnVlfqmPf55VenCX2cvW0tLg7RUfPfdd/jjjz8QGxt71vd07dpVZZAqb+nSpejQoUOl4yuIiIiIiOqKOhtYSKrZ+fPn4/PPP1dNO9ISIbeCgoIK3ZjuvPPOChmgpAuTdG2SzFAffvihGrg9YcKEGvoriIiIiIgcQ53tCvXee++p+969e1dY/tFHH2HEiBHqcVJSEuLi4spek1aNJUuW4NFHH8W7776rRse/9dZbuOmmm6p574mIiKguMpvNKhOlM3fpkf0vLCxkt2wHKzfpfaPT6S5oG3U2sKjK9B0ff/zxKct69eqFjRs3XqK9IiIiIqpcbm4ujhw5UqVzGEcl+y4nyZK+lGn3HavcZLsy3tjDw+O8t1FnAwsiIiIiZ2qpkKDCzc0NgYGBTntSLifIkqZfEt84699QEy51ucn2ZbJoqWMNGzY875YLBhZEREREDk66wcjJnwQVrq6ucFYMLBy33KRuHTp0SNW18w0s6uzgbSIiIiJnw6v85Mh1i4EFEREREZ0zuYL+7LPPokmTJmjevLm6v++++5CZmYkVK1aodPzlydVwuSpe/kS2VatWaN26NRo1aoQhQ4Zg586dp1ypj4mJwRVXXHHGfZFkPD/99NM5/w1TpkxR80NUZb2LkQV0ylm288033+CBBx6o9LWvv/4aHTt2ROPGjdGsWTNce+212LZtm3pNykjKv02bNuo1STIkTvc9yER7lwK7QhERERHRObvnnnuQnp6OVatWwdfXVw0uXrBggVpWVf/++68aLCzvlRT+PXr0wIYNG8rmF/vtt9/g7e2NzZs34+DBg1Wad+xcTJ06VZ3oG41GOILvv/++wlQH5bOWTps2Tb0uQZyQckpMTETLli3V82+//RYtWrRAfHy8WtazZ89q33+2WBARERHROdm3b5+6ui4nvBJUqJNKrRa33HIL6tWrd+4npFotRo4ciQEDBmDmzJllyyXYkFaQoUOHqs86EwlCpOVCBh8/9thjZdmzpk+frq70t23bFp06dcKaNWvK5icT3bp1U1f6U1JSkJWVhXvvvVedmEtLyt133122fTmJl1YCaRG4/PLLKwRQr732mtp2u3btcPXVV6uTeyHbu/nmm9V7+vfvr8rtdGRswz///IM+ffpU2tLx+uuvq+3YtW/fXm3zZJGRkaoFaM+ePahubLEgIiIickY5ybZbea4+gG8MUFIIpO469T1hbWz3aXuB4ryKr/lEAW5+VfpoSb0vJ/AXu0uNBADLli1Tj+XE/ZdfflFzj8m8YgMHDlQn2Kebx0G6Ucl75QT9sssuU4HP4MGDMWzYMDW5sVi9erVqadm+fTtmzZqF2bNnl7WaiLvuuks93rJli/ocyZRkJwHJunXr4Ofnh9tuu029VyZTlsmW5SR+1apVatDzvHnz8NBDD+GHH35QXcW8vLzUvqWlpanAQ/apMsuXL1dBzsmtJxLwSKDSpUuXKpWhdI/atWuXCowSEhJQnRhYEBERETmj9R8BK1+quKzlYOCmuUB2AjCn16nvmZJlu//+AeDIuoqv3TgHaH3rJR0IfLYBwuXn6Jg/fz6uuuoq1SIit+DgYCxdulS1alRm+PDhapI3ud1xxx2qBUNO4jdt2oQXXngBx44dU1mV5CRfxlVU1v1JxmlIFyN78FJ+TIjsiwQVomvXrmXjG6R70vr161ULgj01sD2rkgQLb7/9tnosQdigQYNO+7fLdm688UacL2kZcXFxUSmJP/zwQxX4SStLdSYBYGBBRERE5Iw63AU0vurUFgvhFQ7ct/L0773hvcpbLKpIrrzv3btXnaz7+/uf8rqckMsV+vLkeVBQ0Bm3Ky0CMk5AyMlxcnKyGpgsZHI46Rp1usCispNnCSBuuukmNYhZTvyzs7PVmI3TBRZnIiftdhI4yOB1ezD01FNPVeg2ZVfVyQxlvV9//RWvvPLKKa9JmcnEddLaIl2xTsc+xuJifA+1foyFNIFVdpO+a0RERER1jmeIrWtT+Zt0gxIGl1Nfs3eDEgENT32tit2gRIMGDdQJu3QrkixQ9pPjTz/9FPv371dXy+Wq/5IlS9RrMjhbug5deeWVlW5PXn///fdV1yfJiiQtANINSa64SxYjuUkgI6+X755UnnRBkpP9goIC1T1JMkkVFhaqrlEy7kDYWw/KitDTs8K55HXXXYdXX31V7Y843WeVJ++RcSHpx8dcyOdJK4no27dv2dgQeX3hwoWVbmPt2rVo2rTpaWe9fuaZZ9S4EeniZCddr37++ecz7tvpvod+/fqhTgcWEq1KJoCTb9IkFRISogbmEBEREVH1kBYF6cffuXNnlalIbjJeQVowpDuSnETLgGMZGC3ryQmujDkoT8YUSMpZSZUqXZdk8LKc30nLhIxjKD+eQs75JFiQAOJ0rSjyumyvV69eqmuQjG+Qz5SB1TLuwmQyVXjP+PHj1UBs++DtN954A/n5+erKvyybPHnyWctBxnBI16vevXurv1PeJ12gxNNPP42MjAw16Pr2228/bWAlZXXDDTec9jMkgHvyySfV50i6WSlryRIVFXXmVqaqfg8Xi8Za1TaaGiaDaCojUbJEeS+99JLqP2cf4e+o7E1wEh1LZa9OEqXKQSPNX6cb+EQsN9a3msXjlOXGOuccqvtYlSvv9nSr5bvkOBvOvF05CRQkGDldF6XqKLfT1bFzOXd1mjEWEmGdjkSloaGhKtWXowcWRERERETl7dixA7VBrblsLU1pBw4cqOndICIiIiKqk2pNYCH913x8jmdCICIiIiKialUrAgtJGSbpuao6cQgREREREV1cTjPG4nQTishAEpk9UQaz/PXXX9W+X0RERERE5ESBhYxGr4zkJZZ0YpLCq7qzLBERERERkZMFFvbJRYiIiIiIyPE41RiLNWvWqMlBJk6ciKVLl9b07hARERHVWTKvgky0JpPbyTwMcn/fffepOcZWrFiBDh06VFhfZs8ODAwsey7zMchkdjKlQKNGjTBkyBDs3LnzlPkbZJJkmfiupn388ceql0xVpaSkqIn2KrNhwwYMGDAA9erVU+t07doV33//vXptxIgRiIiIUBPayWzc999/v5rNW8icKbm5uRW2JeUjwwIcgdMEFjJrYPfu3fHmm29izpw5uOqqqzBjxoya3i0iIiKiOklmg163bh1WrVql5mGQoEBmlk5PT6/yNmSmbpkEedeuXWoG7B49eqhJ2uxkNm7pDr958+YKyy8kGKouP/zwA6699tpTlktZ9e/fH6NHj1ZTJUhQ8O2336pxw3ZPPPGE+ps3bdqErVu3YtasWXAGThNYvPjiiyqCkyhYblOnTsXzzz9f07tFREREVOfs27cP33zzjeqq7uvrW3Y1/ZZbblFX4c+VvHfkyJHqKv7MmTPLln/wwQeqFWTo0KFn7BYvQU3nzp3V1X9ZVzKF/vTTT+q13r17qx4vffv2VSf0ElzIvbSoSEuLjNPNz88va5WQ4Oimm25SLQYyCXNcXFzZ5+Tk5KiWlZYtW6r3n2kOte+//x433njjKctfeukl3H333RWCjvDwcAwfPvyUdWUG7J49e2L37t1wBk4TWEiBShcoyf4kHnvsMRVgpKWl1fSuEREREVW7kpQUFO7eU/a8aN8+lCQlqceWoiIU7NgBc26eel6alobCXbtOrHvgIEoSEtRja0mJbd2cnCp/9saNG9GwYUMEBARcxL8I6NixY9ks1NLy8csvv6hAQVpHJLCwWCyVvm/YsGEYM2aMuvo/fvx41ZJSnlz9l239/vvv0Ol0+Pzzz7F+/Xq1viT/KR/M/P333+qCtrznmmuuwahRoyp0y5fAYNu2bap71ssvv1zp/uTk5KhWGPl7KusGJV2fqjpP26+//or27dvDGThNYCH9ycpPgGcymeDq6ors7Owa3S8iIiKimpD51deIv+++sucJ48bj2Acfqselyck4dNPNKDx+kp71ww84PHxE2bpJkyYh9b33bOtmZKh18zdsuGj7JuMnzmV5+TEVdvPnz1dd36VFRMZhBAcHVzrGVs4FJUCQAETISbiM3Tg58DAYDGWf8cYbb6Bt27ZqvcWLF6sgwk66YzVu3Fg9ltaS5cuXl+2XvBYdHa0eS3Cwf//+Sv+On3/+WbW+nO3vPR0JXmT/pJVFWk+k186ZnO/n1NmsUEIitvJpZyVqlciz/ICV6667rob2joiIiKj6+Nw6GJ79+pU9D5/+OrTu7uqxPiQEMQu+hTE6Rj33vv56uHfvXrZu6LRp0JqMtnV9fW3rRkVV+bPbtWuHvXv34tixY/D39z/ldRmkfXKvEnkeFBR0xu1KS4N9wPOHH36I5ORkNTjZ3gogXaPkhL08OemXE+sznVx7eHiUPZbWipUrV+LPP/+Ep6cn3nrrLfW4KqRrkp20fJxuzMbChQtVd6fKSOAj41Iq6yZVfozFQw89VPbcHtjYy7X831OVcq0uThVYVNb3TEbK20mFMpvN1bxXRERERNXPEBSkbnamBg3KHmulZ0fz5mXP9QEB6la2br3Ysscag6HCulXRoEEDdSVduijJuATpVSInv/PmzVPJdqSblIybWLJkCa6++mp1MXj27Nlq/EJl5HUJJKS7knSzkm5KqampSExMVNuxd42S1gJZXj67lFx0btasGb744gvVaiEDnqWr0pm6F0kwJEGFBCuy/+XHhfzzzz/Ys2ePylT1/vvvq0Hl59IiUFxcrAKHTz/9tNLXpWt/nz591PgN6Woljhw5ospKWkjORMaGvPfee2VdsOQzZD/Ll0dNcpquUFLhznZjUEFERERUPSQQkC5KMmhaBkHLTbI8yUm7dDuSq/avv/66GgQt60mAIOlpy+vWrZvqjiSpaiUDlJzUx8bGqpaJ2267rSyoEH5+fmpcgwQvJ5MTbOneJK0B7777rvq8002ufOedd6ou9hKMDBo0SA2OLk9O+KdMmaL2+8cff6ww/qIq/vjjD9Vlyt716mQy8Fu6SkmmUwlo5Pmtt96KkJCQs25b/kYJtqTMZP8kmPrqq6/gKDTW8p3Z6JKTfoBS0SWlWHXPFC7Bl+RUluay8gcqsdxY3xwHj1OWG+ucc6juY7WwsFClW5WT7vLdcZyNnHZK9yFJxnMxxwXk5eXBzc1NbVMyREkmKEn8Y89YVVXSeiHZpCT96/kaNWqUCoDOZc6Lmiq3qtSxczl3dZqzSxlBL81GlQ3Wlj9UXpM8yERERERUt0hLh1zBlyv50tIxd+7ccw4qLpZZs2Zd1KDCmThNYCFNadLHrbJISaIo6bP36quvVnl7MkhH8geHhYWpyM8+2+HpyAyS9oFB5W+SSoyIiIiIak6/fv3UBWaZTE5u119//XltR7IvXUhrRV3nNIGF5A0+UyWRIEH69Z1Lk5n0v3vnnXfOaT+kWS0pKansJoOTiIiIiIjqOqfJCpWQkKBG75+OpN2SE/2qkrzIcjtX0pey/HwaRERERNWFQ2PJkeuW0wQWkkZLWgtkQEllpEvSxZ79sTIyWYkMbpFMAk899ZQa23EmRUVF6mZnHyNiz2RVneTzpNJU9+c6O5Yby431zfHxOGXZ1fY6J3MmSBdsSbUq5zuOMiHa+SgpKTltxiSqmXKTuizzYUi9krpWvl6fSx13msBCRte/8MILp0yKYi8MmXpd1rlUQkNDMWfOHJXGTAIFSXUmsyHK2IvLLrvstO+bNm0apk6despy+WGQAKU6ScWQge5SXswKxXJjfXNMPE5ZbqxzzqEmjlXJeiQXKCtLZONsZcfzEMcsNxnLLJMelidzfdS6dLMyZbqc1MsU6+PHj1f3ElX9999/amC3TGQik6nIhC3nSrYjuZZvuOGGc3qfjOuQ9y5atOicWiwiIyPV5Cw1kW7WPqkMD2iWG+ubY+JxynJjnXMONXWsypxdcuXamctNJrqTOSl4LuJY5SatIdJacTI5d5UMW1VJN+s0LRb169dXE6fIaH1JI2ZvApS4SLolLVu27LyCigvRpUsXzJ8//4zrmEwmdTuZVIqaOKCk3Grqs50Zy43lxvrm+HicsuzqQp2Tz3LmbkRygiyT00nrC89FnKPczuXznCawEB06dMD27duxefNm7N27VwUVMo255C2uCTJlvHSRIiIiIiKq65wqsLCTQOJCgwmJ+vbt21f2XGYalIBFmpiioqIwadIklYlKpogXM2bMQExMjJquvri4WLVULFiwQN2IiIiIiOo6pwwsLgYZj1E+o9O4cePU/fDhw9V07pK6Ni4urux1CSYmTJiggg1XV1cVYCxevBhXX311jew/EREREZEjqbOBRe/evc+Yr1eCi/ImTpyobkREREREdCqO4CUiIiIioroTWMgcEsnJyTW9G0RERERE5MyBxRdffKEGT3fu3FlNhrdjx46a3iUiIiIiInK2wGL58uVqQPWYMWNU9qZu3bqpuS1k0LXMfn0u040TEREREVEdDSyEzPp3xx134Ouvv1azXb777rsoLCzEsGHD1MyXd955J7799lvk5eXV9K4SEREREdUpThVYlGc0GjFgwADMnDkT8fHx+PXXX1VXqeeeew7Tp0+v6d0jIiIiIqpTak26WZmVW27PPvssSkpKanp3iIiIiIjqFKdtsTgTg8FQ07tARERERFSn1MrAgoiIiIiIqhcDCyIiIiIiqnuBRVxcHKxW6ynLZZm8RkRERERE1c/pAovY2FiVavZk6enp6jUiIiIiIqp+ThdYSMuERqM5ZXlubi5cXFxqZJ+IiIiIiOo6p0k3KzNsCwkqnn76abi5uZW9ZjabsWbNGrRp06YG95CIiIiIqO5ymsBi06ZNZS0W27ZtUxPk2cnj1q1bY8KECTW4h0REREREdZfTBBbLly9X93fddRfefPNNeHl51fQuERERERGRswUWdh999FFN7wIRERERETl7YJGXl4eXXnoJv//+O1JSUmCxWCq8fuDAgRrbNyIiIiKiusrpAot7770XK1euxLBhwxAaGlpphigiIiIiIqpeThdY/Pzzz1i8eDG6d+9e07tCRERERETOOo+Fr68v/Pz8ano3iIiIiIjImQOL5557Dv/73/+Qn59f07tCRERERETO2hXq9ddfx/79+xEcHIyYmBgYDIYKr2/cuLHG9o2IiIiIqK5yusDihhtuqOldICIiIiIiZw8snnnmmZreBSIiIiIicvYxFkRERERE5HicrsXCbDbjjTfewNdff424uDgUFxdXeD09Pb3G9o2IiIiIqK5yuhaLqVOnYvr06Rg8eDCysrIwbtw4DBo0CFqtFlOmTKnp3SMiIiIiqpOcLrD47LPPMHfuXEyYMAF6vR5DhgzB+++/r1LQrl69usrb+fPPP3HttdciLCxMzd79/fffn/U9MuN3+/bt4eLignr16mHWrFkX+NcQEREREdUOTtcVKjk5GS1btlSPPTw8VKuFGDhwIJ5++ukqbycvLw+tW7fGXXfdhZtuuums6x88eBBXX301Ro4cifnz5+Off/7Bgw8+iMDAwCq9n4hqocx4mFPiUZx4FC6d+kLj5lX2UsmuDdCYTNDHtoA5PRnFW1fBFBMOrdGIkpRjsBSVwNR5gFq38N8l0Hm6wuDvC0t+AYoOJsPSoTu0vkEo2bcFlqNxMMVGqnWLDsZDGxgGQ6P2FXalZP8WGOq3Vo9Lj+yFPqLhKbtbsmcjDB6260mlJQboo5tf0uIh52DJTEXRtlWw1A+D1sUEi84T2uD6p6wndVoXHAGtbzBKDu6AOWE/XOpFqdeKDh2B1i8YhiYdYcnNQtH6P2CMCIHOzRWl6ZkozciCS8/rbetuXA4tSmEICYS1tBSFB+JhbNoeuuAoVXdL92+HS8NYwM0f8Iks+2ytpgA6Tw+UZuWg5GgaXHpcA43eiOKt/wBFuTCGB8NqtaJw7yEYGrSo9Bg4l2OY6hZVx7z9oAuNrbBc6ocuKNJWV//+EcaQQOi8PGDOzkVxcipcug6AxuSK4m3/AgVZMEaEqvcV7DkIQ71m0Ec1ttWxnevhUi8SGr1evc9q0cLUoa9at/CvRdD7ekHv5wNzXj6KE47C1KEPtB4+ar8sGckwRIUBBXogKAiOzOkCi4iICCQlJSEqKgoNGjTA0qVL0a5dO6xbtw4mk6nK27nqqqvUraqkdUI+c8aMGep506ZNsX79erz22msMLIicmPnILui0haeckJiPHkbxfxvhUj8aGp0WxUkpgMYAY7s+6n2FSz+B/q9JKDhaiiN/+qPBd+8DGiMsWanQIh8H7nsSvr2bI+it71D410+Ie/x11L/mKIyeZhzb4IX8Yx6o968tsIgfMx4+MTkIbJmDonQTUuJ7IPChQuj7DkHGm1ORvWobGgxMUeseWRwEj7b1EPj6lyj4YwFM9aJhzcnAwQefRr2fflLrxA25EZEv/w+6sFh1wmds3hGWY0dx4OahqG/fzt/+CH//K+hjmqJw9a8whgVB5+GO0sxsFfi4XnYdoNWpctAFR9fY90NVo04+cjPLTlTKL9dF1K9wgmKKDlevFe4/bKsje7cjZfa3CIr9CzCWInWbJwJeXwhDk/YoWvMrtC4GIC8N++5+BpFPDIXHiKeR+c6zyPxjExpef1RtK+HnQLg1i0LIR3+gdP9WHBo1GVF90uAeXIys/9yR9p8nGm+1BRZJ4x+B0TUTYZ2yUFqgxaEfQhDx+FB43vU0sj+dgZT5v6PJ4CSUmt1huXYuNF4h2HfDHQhunwm/hvnIPeCKpLW+aLKmK+AdgqNPjQNyUxB5WTpgAQ59HYbQuy+H5/CJ530Mm+Qk7ngZ6cNioa/fylagmfEoObRHXRgwdOxnW2fVzzB4udkuDBQUoig+Caa2PaD1DmT1vciK1i6D1qSFIdAfluJiFB1KgKllZ2j9w1B6cCesRQWq3tpZC/JQuGapCjp17m4qwC09lgGXHvL7pkXRxpXQWIugKcpU9Tvkju7wHDEJme9Ogc/NN8NaVITkF15G6OwvofULUfU6rEsGvGMKkHvYFYmrfNF4RUtoQhogZcoEWI4lIaq3bazvoa9CEXJ7D/g+9T7yf/kCR178CA1vSIbexYK01T4oLvJEzPLtat3DD05EQPNs+DfJQ2GyEXErAlD/87dgbHcljr0yGfn/7UNM/3TAryMw7H3A1xbQOyKnCyxuvPFG/P777+jcuTMeeeQR1RXqgw8+UAO5H3300Uv2uatWrUK/frYfEbv+/furzy4pKTlloj4iqgGZ8UD+MRQfSQJcvWBs2R3WkmIU/rMYxtDAU652lu7bhqQHBiOsczp0RkvZCYnPxHeRt+gTJLz+BRrdlASdwYrUf31QavVG9LKt6qMOjZ2GoA4GeI+fjZihpdDHtETS/Tei+MAhxFyZhvrXuwNDbL9JLj0HImaWP/Qx4YDRCP+UY/AtKinb7ci3X1ctFvD3hSm/AEEHk2Fs11295vvIM/C+LQ443mIRcaWtxcKccEAFKxE9j8HFrwTe0R6wFuYBpWa4uOdAs+h+lJZqcOinYET9726Yul8L374tgEFDYS01Q79nOjQeXjCnJaj/LGU7nuFFyNnnhuQN3mi6qS9K044i4d7bEPa/x6GPbKBOsoyNWkMXVg/mpEMo3r0JLg2iodFq1RVf6F1gbNOLVf88VLgaGhJd4Uq9NTvljFdBjT46pE5+FMX5rohZsV1d9UdmgnqfnCzFTJ8A16vvKTtBqdc/TW0r7vtg+F3eAj5PvYcgnSsM9R+HNfkw8pc/reqGiB8zFl7hOQhonoOo3kaYul6plvs89D943rgfON5iEd7P1mIh5CQ8ZtaLqsUCbq7wTs+Ee4atd4EIff1N1WKBkEDoS0sRc5WtxUJ43TkWbt2vgNlbh4Sx46DbOgXh3/6D+l+/D62+BPDxgUdWDmKOpgHufuo9wc9PVy0WCA8GrFbEXG67QHChx7A4/F0IAq5uC5/xr6J4w28wrZ2MY+v06sJAzN871DpHHpkI3+gs24WBYwYcWhaI2Jn/gy6qma21sXmbsu/z5N+rkvh9MGfl2E50a1JmPIr37az6MXz8t7Zw3yHoI+pBX69c4HVwNyylVpg62upKhSA3LAbw8EVpwgEU7V8H1+O/a4UHba0ChoZtYMk6hqJNf6rgTlrQStLSofGLVBd8jowdC4/AbAS3zUZJtg6HlgQj+sXRcBv0EI69+gSy/tmF2DnTYLG6QBfRCNb8PPX7FtnrGDxCi5C9xx0pWzzRZMtVgNaE5MfHQq9NR1jnTNS7WgfdTXfCmpuJ1IXr4WP9CRo5NvN8Yc3NhiYk1lavQwIBLw94ZOciJjkVGl9boB405TXVYoHjx2pMb9uxKtwGDEFMVEPo6kUCej0CrrO1WNhFz3xFtVjAzwcuefmIue0o9I06qtf8J74I34xkFKbuR8L83+HVZhPcLnPcwEJjlct0TkzGVfz777+q9eK6687vwJQxFgsXLjzj5HuNGjXCiBEjMHny5LJl8rndu3dHYmIiQkNtFelkRUVF6maXnZ2NyMhIZGRkwMureptcLRYLUlNTVfctGexOLDdHr2+lB7ahNPEQXHpcq54XbVpxUheKIzA2baf+Qyr9709g3k3qalDcCn/o/f0R9s0qmI8lYW+vfgjrmg6vyEJkHXRF0jpfNP7nN5hzc5D0wDCETBoPfXQDFO49DEOD5tCHN4A5JQ7F/22CS/2oilc72/Y+0XQdWQ/6mBZl+1uya/2Jq53SjcM74pKWW8nujdAZSqB1danweZaj+6G15MJSXKKu6Bmbd4AuILzi52QkQ+sbAmthPgrXLIMxLLBci0U6XHsORMneLTg65jaEdMiAVm/Fnu9CEXr/AHiPeRVZM59C0swf0OimRGnYQMI/vrAYfBD503oUrPwBhmA/6L29YM7JRXFSmq2biasHineugTU3A6ZI229m4b7D0EfEQh9r6+LqVL9vWUfUyZXtKmgmXLoPtF0F3fSnugpqDA2C1WxG4f54GBu3hi4kRp1QlezbBtdGMWoTxQnJgNED+gatsKdTN4R2zoBHpBbFXZ+DsW0fJD90O8ypRxHV55haf9c3oQgZ0h0+k2cjZ/5rSHjpEzS4LhkwuMJ85ZvQBtfDwcG3w7dBLvya5CI3yQXuT/wAfYP2qr5UbLGIUyd6upjmFcqu9NBO6GNsJ0RFa5eqFgtDoN951+nzJftrLSmEsUW383r/BR/Dx8tIH90ERZv/QvykN1Dv2mxorp8Oi9YDhg5XqnLz3L/hpBaLZJjadMexZx5A9qrtqH9DLoo6vwRtZFMYGraDJScTRf/+ANOap5CxU4v0fR5ouH4Hitb/Dq0eMAQHnHLslsbtRunh3SqQF0VxidB4+sLYrNPn+w0AACOjSURBVMsZj2E5OIu3/AWUFsAYFgyr2aL+JmOjVqrLjzn5MIq3rIDLhqeQuNJUtWN4wy/QLroPel0edi8IRdANHeH10HMo3vg7XNY9haNrDSjM9kDMiq2qm6fe2x3W7FTsHzkVUc8/AJfrRmHv1EegX/Y76g+wtXjt/zEIXt0aIfDNH9RnHx79FKKvTIWrbwmObvRCbqIbor/6Cua05JNaLBJhatkJWr9QlG77A/jiVvV/wKFlAXCpF4bAVz9RdcAYHlSuxaLyY7V8/Za6Z/DS2eqR2QRdRBPUtKL9WxH39UeIGnwXTPYWtGoi566+vr5q+MHZzl2dPrC4GKoaWMh4jEmTJpUtk3EWPXr0UF2zQkJCKn2fZKqSTFYnmzlzJlxdXVGd5KsuLS1Vg97lbyaWW03VN/mP0FpSBK3JqJ5LtwKN3gCNqzus5lJY87Kh0WvVVfGSfC1cW9p+RIv+2waN1gyjeymsVg0KMwwwhvpDFxiOkrh9KM3Mh0u9CFglL4VWB62bJ6xWC6w5mdAYDOrkQv5jtZaUQOPpA41Gq1o0NAbbfjgCRztOrcUF0MAiF4JVtwCNizs0RhOsxUWqhUTjYlJX9SwlpfJrCq27Fwq2boXB3Qy9yQxzsRbFuXq4NK4PjcldXRW1FpfA5GVrsSnIMMLg66767ku3BRmXIn+2taQU8p+T1t1brScnY/L9afQ6WC1WWIuLoXH3gkanV/sh36NZr4devuMi23cq+yoBqDU/GxqTUZWntNZIHdB6+ti2m5sFjVZj267Vanuvm4fquy/dKqzFhSfqaXGJ+jw5uZLPNCfuUS1d5iItSvJ1cG3RAtBoUbRrOzQohdGjXD0N8bUFwElxKEnNhKufLVV6UY5ebdPQsLnq0qbRaWBOOYySHC1cmjaCFILaB6OtVdxSWKT6c8tNjqGyMtPpAb2tO7Asg9Ws/iZ1pnp8ubPUOUdkLS2BNT8HGlc3aAwuVSo39TtXnA9tbjwKM7TQuZlgiG1iG4Ny4DBM3mZo/KNVHdF6+qJo5zZo9WYY3EphsWhQlGmAMTwQOv9QlB45iJKMHLj6Hq832QZojAYYGzRT9aNw1x4YPUuhM1hQWqRDSZ7Ux+bqd7Bo93ZorMfrIzQoTDfAEOQDfUgUSo/Go+RoBlz8zbB6RQFaw9mP4b07VXkYo6Kkmqm6KH+TjAtw8TUDPtGwyu+vuzcKt22F3tUMnYsFlhINtKGNAJM7SvJzobeUlvs/4MzHLDQ69f/DWZUWAZbS48eq7f+U2sJag8dpQUGBGldclcDCKbpCLVq0SI2HkO5G8vhMzrfV4mwkcJCB4+WlpKSoL9jf3/+075NARFLintxiIV242GLhHNjSUzm5ogODAYZ6Fa802692Srkl/v0rAr1NMAX5q/84ig4nwtisIzJemYCsFZtQ/1pbf/+Dv0j/7EgEv78MxVv/xoGhDyCydxpMfgaU9nseLlfeefYWi4R9qrna0LgdnFltqG/n3GIR2wyFq37FkTc+QP2ByTC4WZC83rvsqqfY26EF/BrmwL9pLvLTjIj7IwCxc1+Eqeu1SHnweuRs3guPwSEIzN6Ggz8GnHT1842yq58yfiD7sDvqr7V1YznQs7XqWhHUJhtF2Toc/CVYXVV1u+FeHHtyBNJ/WVs2jsB+FTTkk+Xqaveht2ai/txnoPEOuuAWC2MrW9c3YY7bieKDe+HSpZ86abvUakOdqwlVLresIyjasdmWdMHeYrFxJUyNmkEbemKAeY21WOzeDJdWbaEp12//bMewnLSX/62VluHi7Wvh0rwVNAEnBj/bWyxkULK9RYD17fyUZqQgcfVKhHXpBb1v9Q7glnNXCSyqwilaLOSAlZP6oKCgMx68EsHJBHqXosXi8ccfx48//oidO3eWLXvggQewefNmNf7iXL4cb2/vKkV9F5sczBIMna0cieVmZ07YheK9/52SnUKrK8TB+yfDu0sjBL74AfJ+/BiurVuploa4R59G9IKfoA2rjy3DbkZYwXoEt8pCYaYeB38JUv29DS26ozRuz9kzygREVN43uRarq8epOS0RxdvXwCU2EhqDXo2DKd9PW8bJ2E9QygbItrsMWi9/lOzZjNLUeGT5+iPI2xXFh46csb+2OTcfLt0GVmkwqIxXkMG/at24BGi9AmBo1lm1CpQe3lVhoKizqqt17kKx3Fhu1Slv+ULsfGsemj08DO59bqzWzz6Xc1e9sxy8lT2+ELm5udi3b1+FdLISJPj5+ansT9LSkJCQgE8//VS9PmrUKLzzzjuq9UFSzkowIQO3v/jii4uyP0TVqUqp71xKUfTJI4j7zeOU7BQy+LPeQHdYb50A89F4le1CMsCYvEvhFuCpWg5EwH33wsd7NBDkD2NRMWJuSICpdTeVrlIfdaLPqimsTdljrYc3XHtX748m1TxdQFiF791g6+JexqX7NWWP5dTXtVw2VEOjNtA1aCXNyCoVo0t42xPrevufst3yqTZMna6suF1bQ4IiLSlyq6yeSheL2hBUEJFzMEWHIaR9prp3ZE4RWNhJ9iXJzDR79mw15uFCSKrYPn1sKeeEvbvS8OHD8fHHH6txE5Jpyi42NhZLlixRmafeffddNbHeW2+9xVSz5DgkH7s0a58mr7Z0wcj84jP4jH0RBSt+qFLqO5/GVsTMmHhKdgpEh0NfLsd8g+/nQ2cyq/6ywcdz4MtFAGkqN0jOba3WdtJWMT04ERERVYH8/2pws/0/68icKrCQMRbbt2+/KINWevfurQbCnI4EFyfr1asXNm7ceMGfTXTB6VTt/bJb97CNNfhtHoz/PonceMvp82oXapH6czC878utcuo7yX/vWq4r0umu0JZfzo4UREREF1dJ6jFkHnSDb+oxmCom+XMoThVYiDvvvFN1QXrppZdqeleILljpga0oPXLglMl6Tgz6jIOxcRs1yE6lU/3ketXCkLzSD1pPP0Qs2qCy9hwa+wrCuuvh8fDbiBluqTSvthzsDUaYy4IA16ATAYPx5G4nPWs4pzoRERGVseQXoyDdpO4dmdMFFsXFxXj//fexbNkydOjQAe7uFVOJTZ8+vcb2rTay5GSgaMMKlcFF6x9e5wbSnq+zDfpUM4H66ZH9ykNI2eRyymQ94V0zYSnS4NDCUISPvRFeo15E5qdzkfp9CBp8OBUh13ipFguhMbraJu2JbQRddHOUzyFjbFkx//vJ/daJiIjI8Zm6DEBovXYwSfdiB+Z0gYV0hWrXzpbibM+ePRVeY/7tcyMZflCSrSb1UWngKgkaSnauVbNWRl+RCrdQAzB6rVMHF2fKLHNGuamAOQklxzKqlFEmc840pC9Zg0Y32tJUJi6VNJXhCJ3/J0rj95bNBOoVo4fbnVNV7nAR8vIM1WKB0CBozWbE9Le1WAif0VPgPTzzlO5IMviag52JiIiopjldYLF8+fKa3gWnVnpkH4oObYI+OAqHbr8DXuG5CGyVg/wUI1wm/Ah9o86w5OeoicWEoVkndTVcV5yMtLnPwzt+HwzHAwtLyn4U7dwKY/OO0AVGqNSLcnNp1b4s+FD5tVf/CmNYULn82sfgepl0/dGhePOfakKc8ifLMila8e5NKl+3xiOw0kBG5Xk/sBsuna9UebXLB0sygZE+ummlf3/c6PHwa5SDgGa5KEw14vDvAaj34UvQBkTDknYEphjbrJuFB+PL0lWW7tuAgnmTYLFuRMZON5UDv8EGW2BxZOxYlQM/uG02SrJ1OLQkGNEvjobPfZPgcfVe4HiLRdgAW4uF0Ec2tLUwhAdDFxgJfbm/z9SuV9ljGUnkGnmiXPQR5dLgEBERUZ1RtHYpkt6eA+8x98G1ywA4KqcLLOj8WY7uR87HL6No0wHU/2cHYn9YBGvGEZhzUhF/9zOI6vMn4OaHtKnj4H/XMECrQdp7sxHw5Mswp+iQvtcDHlk5sKz7DdqSdFh/fBSHFnoh6n93w33oY8j68DWkLfgLjYdko6j7a9AE1oPWO0BdnY/oeQye4UXI2eeG5A3eaLqpL+DihaQnHoXOkonQV95AcaYZxkatkfvd+0h8ZyEa35IIM9xhufo9GLter/6GghULYfDUIH/Ow0hY6Y5GP8fCanCHOWEftNZc7L/nafgPaA2/SW8g98t34N6jB8xHjyDx+RkIm/4mot59XbVYwM8HLgWFiLklCYYWPZD62HDkbtqF+lenqs85sigIXt0aI+idn1C4aTXStnshbMIs+A4OgFduflmZRsyYoVosEOgPQ3ExYq47kQP/TGkq2cJAREREVaV1McDVL1/dOzKnDCzWrVuHb775RqWDlTEX5X333Xc1tl+OTmvJhVdkLvyuGaOe6yMaAnJT6ULrQxcaBXNSHPK27EHAovvV8vz/AgEL1ODiRpttA3oP9GwBN+9sBHWyqFSkxva2PPDed0+Ae/cewLpJSHrqaRjDQxD6yW+2q/NhQYCHOzwzs+GScgww2sbGhD7zDLRLRkHz7Z049G0owh64Gh6DxyCmRUdYfXRIfuxRWNY+g6hfbYGFBCkhHTPhFatRnw03Xxx9eBhKEo4g5opjaDDIHRj6MCxpSSo4id07G3pXC1x8fNVg55OzF9lz4fuNfx4+aUeA4y0WEVfaWiyEW78hCK3fFoZ23dTkUVXNgU9ERER0MRgC/eATW6DuHZnTBRZffvmlygwl81nIAG6537t3r5qZ+8YbOanWmcgYAEuppAY9MXmUnf2EW+sdiOgvvlEn4yJ6pBb6+q0qrBs+fQa0ekAb27hCKlJ9dBN1Q6e+CO27DRpPX2hc3CpcndeHAfoTF/Jh7Hod0LQtrDmpiLnisGqx0IXGwDXUdoYe/HY4rIW5ZetLkGII9ldZkuyfHTT1DViyUoDI0FPmVtC7WtUsvsFad2hDbEFUZWSCLcjtOJfyE7Z5+0Ef0eCMZUtERER0qViKilGcp1P3jpzW3ekCixdffBFvvPEGRo8eDU9PT7z55ptq8rr7778foaGhNb17Dq3oUCKS1/vCr1si9GeYqKx8IFFZBTF1vOLMH+QTCVOXcxjg7RMJjU8kXCNtg/LLO3mgcmVdiDi3AhEREdVmRYcTcXSjD/x7JEJfDw7LkYOeSu3fvx/XXHONemwymZCXl6eyQcmM2HPmzKnp3XNoxlZdEDjiBnVPRERERM7B1Lorgu65Ud07MqcLLPz8/JCTk6Meh4eHq/SzIjMzE/n5JwbV0ql0vsFw6dxP3RMRERGRc9D6BMHU4Up178icLrDo2bOnGlshBg8ejEceeQQjR47EkCFD0Ldv35rePYdWemgHsj95Rd0TERERkXMoObQTWR+8ou4dmdOMsdi8eTPatGmDd955B4WFhWrZpEmTYDAY8Pfff2PQoEF4+umna3o3HVrpkQPIXn8ApV0PwFivZU3vDhERERFVgTlhP/K27of5sv1AvRZwVE4TWMhs223btsW9996LoUOHqmWS+nPixInqRmfnUj8KEV0z1D0REREROQeXelEI65yp7h2Z03SF+ueff1Rw8cQTT6jsT3fccQdn4SYiIiIichBOE1h07doVc+fOVfNVvPfeezhy5AiuuOIK1K9fHy+88IJ6TmdWFJeIo5u91D0REREROYeiQ0eQvMFb3Tsypwks7FxdXTF8+HCsWLECe/bsUYO2Z8+ereayuPrqq2t69xyaTH5nCAlW90RERETkHLR+wTBFBap7R+Y0YywqI60V0jUqMjISkydPxq+//lrTu+TQDE06wG9cFAxBjp2qjIiIiIhOMDTuAN9HHP8czulaLOxWrlypWi5CQkLU4G3JCiXjMOj0rPnZKI3bqe6JiIiIyDlYcrNQcnCHundkThVYxMfH47nnnlMtFX369FGzcL/99ttITExU4y+6dOGM0mdSuG45kqa9o+6JiIiIyDkUbViB5FfeVfeOzGm6Ql155ZUqC1RgYCDuvPNO3H333WjcuHFN75ZTMYYFIaB5tronIiIiIudgjAhGYMtsde/I9M40aHvBggUYOHAgdDpdTe+OU9J5uMHVr0TdExEREZFz0Lm5wsWnRN07MqcJLBYtWlTTu+D0SjOzkZNogl9mNozhNb03RERERFQVpemZyD7iAr/0TIc+h3OqMRZ0YUrTMpB1wF3dExEREZFzKM3IRna8q7p3ZE7TYkEXzqXndYho3AUuDp6qjIiIiIhOcOlxLSIadXb4czi2WBARERER0QVjYFGHFG9agZSnx6h7IiIiInIORZtW4OjkMerekTGwqEusJdBp8tU9ERERETkHLUphMOWpe0fGwKIOMYYGwb9JrronIiIiIudgCAmEX8M8de/IGFjUIVazBRazRt0TERERkXOwlpaitFir7h1ZnQ8sZs6cidjYWLi4uKB9+/b466+/TltYK1asgEajOeW2a9cuOIPC/XFI+NdP3RMRERGRcyg8cARJa3zVvSOr0+lmv/rqK4wdO1YFF927d8fs2bNx1VVXYefOnYiKijrt+3bv3g0vL6+y54GBjt0sZWds2hb+NxxW90RERETkHIxN2yHgJjmHawdHVqdbLKZPn4577rkH9957L5o2bYoZM2YgMjIS77333hnfFxQUhJCQkLKbTqeDM9AFRcGt/23qnoiIiIicgy4oEq5XyDlcJBxZnQ0siouLsWHDBvTr16/Ccnn+77//nvG9bdu2RWhoKPr27Yvly5fDWZQm7EPuglnqnoiIiIicQ2nCPuR87fjncHW2K1RaWhrMZjOCg4MrLJfnycnJlb5Hgok5c+aosRhFRUWYN2+eCi5k7MVll11W6XtkPbnZZWVlqfvMzExYLNU7iDpv8zrE/bIRrlEN4e7JzFBVJd9TdnY2jEYjtNo6G4ufM5Yby431zTnwWGW5sb45vvwt63Dktw1wr9cQbtV8DifnQMJqtZ513TobWNjJ4OvypNBOXmbXuHFjdbPr2rUr4uPj8dprr502sJg2bRqmTp16yvLo6GjUmJ9+rLnPJiIiIqLz8+NPqCk5OTnw9vY+4zp1NrAICAhQYyNObp1ISUk5pRXjTLp06YL58+ef9vVJkyZh3LhxFa4Mpaenw9/f/7QBzKWMOGUMiQRD5QefE8uN9c1x8DhlubHOOQceqyy3ulLf5KK7BBVhYWFnXbfOBhbSrUW6NC1btgw33nhj2XJ5fv3111d5O5s2bVJdpE7HZDKpW3k+Pj6oSVIhGViw3FjfHBuPU5Yb65xz4LHKcqsL9c37LC0VqOuBhZCWhGHDhqFDhw6qW5OMn4iLi8OoUaPKWhsSEhLw6aefqueSNSomJgbNmzdXg7+lpWLBggXqRkRERERUl9XpwOLWW2/FsWPH8OyzzyIpKQktWrTAkiVLysY/yDIJNOwkmJgwYYIKNlxdXVWAsXjxYlx99dU1+FcQEREREdW8Oh1YiAcffFDdKvPxxx9XeD5x4kR1c1bSJeuZZ545pWsWsdxY3xwHj1OWG+ucc+CxynJjfTuVxlqV3FFERERERERnwKT8RERERER0wRhYEBERERHRBWNgQUREREREF4yBRR0yc+ZMxMbGwsXFRc3h8ddff6GumjJlipqgsPwtJCSk7HUZeiTryGQwkgGsd+/e2LFjR4VtFBUVYcyYMWqyRXd3d1x33XU4cuQIapM///wT1157rSoHKaPvv/++wusXq5wyMjJU6mfJky03eZyZmYnaWm4jRow4pf7JZJt1vdymTZuGjh07wtPTE0FBQbjhhhuwe/fuCuuwzp1fubHOneq9995Dq1atyuYFkLTzP//8M+vaBZYb61rVj1v57R87dmzt+n2TwdtU+3355ZdWg8FgnTt3rnXnzp3WRx55xOru7m49fPiwtS565plnrM2bN7cmJSWV3VJSUspef+mll6yenp7WBQsWWLdt22a99dZbraGhodbs7OyydUaNGmUNDw+3Llu2zLpx40Zrnz59rK1bt7aWlpZaa4slS5ZYn3zySVUO8nOxcOHCCq9frHIaMGCAtUWLFtZ///1X3eTxwIEDrbW13IYPH67+5vL179ixYxXWqYvl1r9/f+tHH31k3b59u3Xz5s3Wa665xhoVFWXNzc0tW4d17vzKjXXuVIsWLbIuXrzYunv3bnWbPHmy+n9SypF17fzLjXXt7NauXWuNiYmxtmrVSp2P1abfNwYWdUSnTp1UZSyvSZMm1ieeeMJaVwMLORArY7FYrCEhIeoAtyssLLR6e3tbZ82apZ5nZmaqH1IJ2OwSEhKsWq3W+ssvv1hro5NPkC9WOUmgK9tevXp12TqrVq1Sy3bt2mV1dqcLLK6//vrTvoflZiPBvpTfypUr1XPWufMrN9a5qvP19bW+//77rGvnWW6sa2eXk5NjbdiwoQoMevXqVRZY1JbfN3aFqgNkYr8NGzagX79+FZbL83///Rd11d69e1Vzo3QPu+2223DgwAG1/ODBg0hOTq5QXpKvvFevXmXlJeVZUlJSYR3ZlkyyWFfK9GKV06pVq1RTbefOncvWkW5Bsqw2l+WKFStUt5VGjRph5MiRSElJKXuN5WaTlZWl7v38/NQ969z5lRvr3NmZzWZ8+eWXyMvLU117WNfOr9xY185u9OjRuOaaa3DFFVdUWF5b6lydnyCvLkhLS1MHf3BwcIXl8lwqcV0kB9ynn36qTuqOHj2K559/Ht26dVN9Ge1lUll5HT58WD2WdYxGI3x9fetsmV6scpJ7OcE+mSyrrWV51VVX4ZZbbkF0dLT6z+Tpp5/G5Zdfrv7TkP9IWG62vsbjxo1Djx491H+agnXu/MqNde70tm3bpk6ICwsL4eHhgYULF6JZs2ZlJ2D8fTu3cmNdOzMJwjZu3Ih169ad8lpt+X1jYFGHyCChk/8DOnlZXSEndnYtW7ZUP5D169fHJ598UjaI9nzKqy6W6cUop8rWr81leeutt5Y9lpO/Dh06qCBj8eLFGDRo0GnfV5fK7aGHHsLWrVvx999/n/Ia69y5lxvrXOUaN26MzZs3q4GtCxYswPDhw7Fy5UrWtfMsNwkuWNcqFx8fj0ceeQRLly5VSXROx9l/39gVqg6QzAE6ne6USFW6XpwcGddVkllBAgzpHmXPDnWm8pJ1pIuZZF443Tq13cUqJ1lHWo1OlpqaWmfKMjQ0VAUWUv9EXS83yXiyaNEiLF++HBEREWXLWefOr9wqwzpnI1d/GzRooIJ7ydLTunVrvPnmm6xr51lurGunJy3S8hsuWTn1er26STD21ltvqcf2321n/z+VgUUdID8AUpGXLVtWYbk8l+4/ZEvf9t9//6n/bGXMhRyY5ctLDmT5AbCXl5SnwWCosE5SUhK2b99eZ8r0YpWTtBZJn/C1a9eWrbNmzRq1rK6U5bFjx9TVLKl/dbnc5IqaXHH/7rvv8Mcff6g6Vh7r3PmVW2VY505flvL/Aeva+ZUb69rp9e3bV3Uhk5Ye+00Cs9tvv109rlevXu34P/WSDw8nh0o3+8EHH6iMAWPHjlXpZg8dOmSti8aPH29dsWKF9cCBAypzgqRhkxRv9vKQrAySieG7775TKd+GDBlSacq3iIgI62+//aZSvl1++eW1Lt2sZK/YtGmTusnPxfTp09Vje5rii1VOkhpP0u5J5gq5tWzZ0qnTpp6p3OQ1qX+SAvDgwYPW5cuXW7t27arSB9b1cnvggQdUfZJjs3wq3vz8/LJ1WOfOvdxY5yo3adIk659//qmOw61bt6q0qZJdZ+nSpaxr51lurGvnpnxWqNry+8bAog559913rdHR0Vaj0Wht165dhVSEdY09N7QEW2FhYdZBgwZZd+zYUfa6pH2TlLSS+s1kMlkvu+wydZCXV1BQYH3ooYesfn5+VldXV3XQxsXFWWsTOemVE+OTb5Iu9WKWk8zhcPvtt6vgTm7yOCMjw1oby01O9vr162cNDAxU9U/mG5DlJ5dJXSy3yspMbjJHgx3r3LmXG+tc5e6+++6y/xPleOzbt29ZUMG6dn7lxrp2YYFFbfh908g/l75dhIiIiIiIajOOsSAiIiIiogvGwIKIiIiIiC4YAwsiIiIiIrpgDCyIiIiIiOiCMbAgIiIiIqILxsCCiIiIiIguGAMLIiIiIiK6YAwsiIiIiIjogjGwICKiWq93797QaDTqtnnz5preHYwYMaJsf77//vua3h0ioouCgQUREZ32pLf8bcCAAU5dUiNHjkRSUhJatGhxymv9+vWDTqfD6tWrz1geBoMBwcHBuPLKK/Hhhx/CYrFUWPd0gcLYsWNVcGP35ptvqn0hIqpNGFgQEdEpJIiQE9/yty+++OKSllRxcfEl3b6bmxtCQkKg1+srLI+Li8OqVavw0EMP4YMPPjhjeRw6dAg///wz+vTpg0ceeQQDBw5EaWnpOe+Lt7e32hciotqEgQUREZ3CZDKpE9/yN19f3wpX5t9//33ceOON6oS9YcOGWLRoUYVt7Ny5E1dffTU8PDzUVf5hw4YhLS2t7HW5gi8n8+PGjUNAQIBqBRCyHdmeq6urOoH/5JNP1OdlZmYiLy8PXl5e+Pbbbyt81o8//gh3d3fk5OSc87f50UcfqQDhgQcewFdffaU+43TlER4ejnbt2mHy5Mn44YcfVJDx8ccfswYRETGwICKi8zV16lQMHjwYW7duVQHE7bffjvT0dPWaXN3v1asX2rRpg/Xr1+OXX37B0aNH1frlSdAgLQj//PMPZs+erVoEbr75Ztxwww1qLMT999+PJ598smx9CR5uu+02FQyUJ8/lfZ6enuf0N1itVvXeO+64A02aNEGjRo3w9ddfV+m9l19+OVq3bo3vvvvunD6TiKi2YosFERGd4qefflItDeVvzz333CljD4YMGYIGDRrgxRdfVFf6165dq15777331JV9WS4n7G3btlVjEpYvX449e/aUbUPe+8orr6Bx48ZqvVmzZqnHr776qrqXIEI+p7x7770Xv/76KxITE9VzaQWR/b377rvP+Zv87bffkJ+fj/79+6vnEmCcrjtUZWSfJRgiIiIGFkREVAnpgiQtBuVvo0ePrrBOq1atKrQkSGtBSkqKer5hwwYVRJQPTOQkXOzfv7/sfR06dKiwzd27d6Njx44VlnXq1OmU582bN8enn36qns+bNw9RUVG47LLLzvm7lCDi1ltvLRt3IYHSmjVr1H5UtcVDumkRERFQcQQbERHR8UBBWhPORDIklScn2PYsSXJ/7bXX4uWXXz7lfaGhoRU+52wn6rLsZNJq8c477+CJJ55QXZnuuuuucz7Bl25bksGppKREtbDYmc1m1bpS2b6f7L///kNsbGzZcwmusrKyTllPxofIgG0iotqMXaGIiOiik25QO3bsQExMjApQyt9ODibKk1aNdevWVVgmYzROJl2WJJvTW2+9pT5n+PDh57yPn332GSIiIrBly5YKLTMzZsxQYz/Olu3pjz/+wLZt23DTTTedcf8lMJIWHOnaRURUmzGwICKiUxQVFSE5ObnCrXxGp7ORblPSIiBdi2TcxYEDB7B06VI1DkJaBE5HBmvv2rULjz/+uBqLIQOp7VmXyrdISIaqQYMG4bHHHlNzUEiAcD7doGTAt8xrUf4m+ygtDIsXLz6lPBISErBx40Y1duT6669X2aTuvPPOsvUmTJigtiutKbL/ErRI5ivp/nVyVzIiotqGgQUREZ1CsjhJl6Xytx49elS5pMLCwlSmJwkiZGC0nLDLvA/SHUirPf1/PdKtSFLJSqYlGcMhXZTsWaEk5Wt599xzj5r74nwGbUsLgpz0l29tKN+dSYKV8oO47eUhLTAyp4WMH5HWEkk5KxPr2UnWKwmEpMVDxorIdiSo+OuvvxAdHX3O+0lE5Ew01so6rxIRETmIF154QWWLio+PP6UrkwQrkh3KaDSecRsyZ4akvpVuTo5EWmEWLlyo0usSETk7tlgQEZFDmTlzphqnIN2nJOOTpJ4tP4ZC0sPKuIpp06aprlNnCyrKb1eyU8m4iJo2atQotS9ERLUJWyyIiMihPProo2oGbBmjIWlkZcbuSZMmlaWEnTJlimrFkPSy0hWpKifoMjaioKBAPZZtVjUYuVQkLW92drZ6LF2szjSgnYjIWTCwICIiIiKiC8auUEREREREdMEYWBARERER0QVjYEFERERERBeMgQUREREREV0wBhZERERERHTBGFgQEREREdEFY2BBREREREQXjIEFERERERFdMAYWRERERESEC/V/b8wOOcw+RZQAAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "fig, (ax_spec, ax_ratio) = plt.subplots(\n", - " 2, 1, figsize=(8, 6), sharex=True,\n", - " gridspec_kw={'height_ratios': [3, 1]})\n", - "\n", - "edges = hist_cpu.axes[0].edges\n", - "cpu_vals = hist_cpu.values()\n", - "cuda_vals_v1 = hist_cuda_v1.values()\n", - "cuda_vals = hist_cuda.values()\n", - "# async_vals = hist_async.values()\n", - "graph_vals = hist_graph.values()\n", - "\n", - "ax_spec.stairs(cpu_vals, edges, label=f'CPU ({n_clusters_cpu} clusters)')\n", - "ax_spec.stairs(cuda_vals_v1, edges, label=f'CUDA single frames ({n_clusters_cuda} clusters)', linestyle='--')\n", - "ax_spec.stairs(cuda_vals, edges, label=f'CUDA batched ({n_clusters_cuda} clusters)', linestyle='--')\n", - "# ax_spec.stairs(async_vals, edges, label=f'CUDA async ({n_clusters_async} clusters)', linestyle='-.')\n", - "ax_spec.stairs(graph_vals, edges, label=f'CUDA graph ({n_clusters_graph} clusters)', linestyle=':')\n", - "ax_spec.set_ylabel('Counts')\n", - "ax_spec.set_title('Cluster energy spectrum: CPU vs CUDA variants')\n", - "ax_spec.legend()\n", - "ax_spec.grid(alpha=0.2)\n", - "\n", - "with np.errstate(divide='ignore', invalid='ignore'):\n", - " ratio_cuda = np.where(cpu_vals > 0, cuda_vals / cpu_vals, np.nan)\n", - " ratio_graph = np.where(cpu_vals > 0, graph_vals / cpu_vals, np.nan)\n", - "\n", - "ax_ratio.stairs(ratio_cuda, edges, label='CUDA batched / CPU', color='C1', linestyle='--')\n", - "ax_ratio.stairs(ratio_graph, edges, label='CUDA graph / CPU', color='C3', linestyle=':')\n", - "ax_ratio.axhline(1.0, color='gray', linewidth=0.5)\n", - "ax_ratio.set_ylabel('Variant / CPU')\n", - "ax_ratio.set_xlabel('Energy [ADU]')\n", - "ax_ratio.set_ylim(0.5, 2.0)\n", - "ax_ratio.legend(fontsize=8)\n", - "ax_ratio.grid(alpha=0.3)\n", - "\n", - "plt.tight_layout()\n", - "plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "cbd03f58-0068-4ad6-aec6-51ed9c8f8f1e", - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" +{ + "cells": + [ + { + "cell_type": "markdown", + "id": "md-title", + "metadata": {}, + "source": [ + "# ClusterFinder — CPU vs CUDA performance\n", + "\n", + "Throughput comparison of the serial CPU finder against the CUDA variants:\n", + "per-frame, batched+pinned, CUDA-Graph, and the async double-buffered pipeline.\n", + "\n", + "CPU↔CUDA **correctness** (why the cluster counts differ) is analysed\n", + "separately in `ClusterFinderFrozen_vs_CUDA.ipynb`; here we only sanity-check that\n", + "the counts and spectra agree, and focus on timing." + ] }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.15" + { + "cell_type": "code", + "execution_count": 1, + "id": "imports", + "metadata": {}, + "outputs": [], + "source": [ + "import sys; sys.path.append('/home/ferjao_k/aare/build')\n", + "\n", + "from pathlib import Path\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import boost_histogram as bh\n", + "import time\n", + "from tqdm import tqdm\n", + "\n", + "from aare import (File, ClusterFinder, ClusterFinderFrozen, ClusterFinderMT, ClusterCollector,\n", + " ClusterFinderCUDA, ClusterFinderCUDAGraph, ClusterFinderCUDAOpt2)\n", + "from helper import print_pinning_budget\n", + "\n", + "import resource\n", + "\n", + "def _faults():\n", + " r = resource.getrusage(resource.RUSAGE_SELF)\n", + " return r.ru_minflt, r.ru_majflt" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "hist-helpers", + "metadata": {}, + "outputs": [], + "source": [ + "N_BINS = 200\n", + "\n", + "def make_hist(clusters):\n", + " h = bh.Histogram(bh.axis.Regular(N_BINS, -2, 4000))\n", + " h.fill(clusters.sum())\n", + " return h\n", + "\n", + "def make_hist_from_batch(result_list):\n", + " h = bh.Histogram(bh.axis.Regular(N_BINS, -2, 4000))\n", + " energies = [np.asarray(cv.sum()).ravel() for cv in result_list if cv.size > 0]\n", + " if energies:\n", + " h.fill(np.concatenate(energies))\n", + " return h" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "config", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Image size: (400, 400)\n", + "Pedestal frames: 1000\n", + "Data frames: 100000\n", + "Total in file: 100000\n" + ] + } + ], + "source": [ + "base = Path('/mnt/sls_det_storage/moench_data/2603_MaxIVBeamtime/2026032408/process/xrf/')\n", + "f = File(base / 'Cu_factor_10_data_master_0.json')\n", + "pd = File(base / 'Cu_factor_10_pedestal_master_0.json')\n", + "\n", + "n_frames_pd = 1000\n", + "N = 100000\n", + "cluster_size = (3, 3)\n", + "rows, cols = f.rows, f.cols\n", + "image_size = (rows, cols)\n", + "capacity = 1000\n", + "BATCH_SIZE = 3000\n", + "\n", + "N_STREAMS = 4\n", + "N_SIGMA = 5\n", + "\n", + "print(f'Image size: {image_size}')\n", + "print(f'Pedestal frames: {n_frames_pd}')\n", + "print(f'Data frames: {N}')\n", + "print(f'Total in file: {f.total_frames}')" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "pinning", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "── System RAM ──────────────────────────────────────────\n", + " Total RAM : 125.1 GiB\n", + " Currently available : 101.2 GiB (free + reclaimable cache)\n", + " Reserved headroom : 4.0 GiB (OS + CUDA context)\n", + " Safe pinning budget : 97.2 GiB\n", + "\n", + "── Frame layout ────────────────────────────────────────\n", + " Frame size : 400 × 400 × 2 B = 312.5 kB\n", + "\n", + "── Pinning estimate ────────────────────────────────────\n", + " Max frames pinnable : 326,220 (97.2 GiB)\n", + "\n", + " Note: no swap on this machine — exceeding available RAM\n", + " will trigger the OOM killer. Stay within the budget.\n" + ] + } + ], + "source": [ + "print_pinning_budget(rows, cols)" + ] + }, + { + "cell_type": "markdown", + "id": "md-build", + "metadata": {}, + "source": [ + "## Build finders\n", + "\n", + "`SERIAL` picks the CPU baseline: the sequential `ClusterFinder` or the\n", + "multi-threaded `ClusterFinderMT`. All finders are trained on the same pedestal\n", + "frames below." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "build", + "metadata": {}, + "outputs": [], + "source": [ + "SERIAL = False\n", + "\n", + "if SERIAL:\n", + " # cf_cpu = ClusterFinder(image_size, cluster_size, n_sigma=N_SIGMA, capacity=capacity)\n", + " cf_cpu = ClusterFinderFrozen(image_size, cluster_size, n_sigma=N_SIGMA, capacity=capacity)\n", + "else:\n", + " cf_cpu = ClusterFinderMT(image_size, cluster_size, n_sigma=N_SIGMA,\n", + " capacity=capacity, n_threads=48)\n", + " sink = ClusterCollector(cf_cpu)\n", + "\n", + "cf_cuda_v1 = ClusterFinderCUDA(image_size, cluster_size, n_sigma=N_SIGMA,\n", + " max_clusters_per_frame=3000, n_streams=N_STREAMS)\n", + "cf_cuda = ClusterFinderCUDA(image_size, cluster_size, n_sigma=N_SIGMA,\n", + " max_clusters_per_frame=3000, n_streams=N_STREAMS)\n", + "# cf_async = ClusterFinderCUDA(image_size, cluster_size, n_sigma=N_SIGMA,\n", + "# max_clusters_per_frame=3000, n_streams=N_STREAMS)\n", + "cf_graph = ClusterFinderCUDAGraph(image_size, cluster_size, n_sigma=N_SIGMA,\n", + " max_clusters_per_frame=3000, n_streams=N_STREAMS)" + ] + }, + { + "cell_type": "markdown", + "id": "md-ped", + "metadata": {}, + "source": [ + "## Pedestal (all finders trained on identical frames)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "train", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Pedestal (1000 frames): 10.754s\n" + ] + } + ], + "source": [ + "t0 = time.perf_counter()\n", + "for _ in range(n_frames_pd):\n", + " img = pd.read_frame()\n", + " cf_cpu.push_pedestal_frame(img.copy())\n", + " cf_cuda_v1.push_pedestal_frame(img.copy())\n", + " cf_cuda.push_pedestal_frame(img.copy())\n", + " # cf_async.push_pedestal_frame(img.copy())\n", + " cf_graph.push_pedestal_frame(img.copy())\n", + "print(f'Pedestal ({n_frames_pd} frames): {time.perf_counter() - t0:.3f}s')" + ] + }, + { + "cell_type": "markdown", + "id": "md-io", + "metadata": {}, + "source": [ + "## Read all data frames into memory (I/O out of the timing loop)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "io", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Reading 100000 frames: 39.114s (2557 FPS, 780.217 GB/s)\n" + ] + } + ], + "source": [ + "f.seek(0)\n", + "t0 = time.perf_counter()\n", + "data = f.read_n(N)\n", + "t_io = time.perf_counter() - t0\n", + "print(f'Reading {N} frames: {t_io:.3f}s ({N/t_io:.0f} FPS, '\n", + " f'{f.bytes_per_frame * N / 1024**2 / t_io:.3f} GB/s)')" + ] + }, + { + "cell_type": "markdown", + "id": "md-cpu", + "metadata": {}, + "source": [ + "## CPU clustering (baseline)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "cpu", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|███████████████████████████████████████████████████████████████████████████████████████| 100000/100000 [00:20<00:00, 4805.12it/s]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + " minor faults: 2,570,943 (~1.80s est.) major: 1\n", + "=========================================================================================\n", + "CPU clustering: 20.815s (4804 FPS, 233085343 clusters, 2330.85/frame)\n" + ] + } + ], + "source": [ + "mf0, Mf0 = _faults()\n", + "t0 = time.perf_counter()\n", + "for frame in tqdm(data):\n", + " cf_cpu.find_clusters(frame)\n", + "t_cpu = time.perf_counter() - t0\n", + "mf1, Mf1 = _faults() # bracket the timed loop only; MT drain + hist below allocate too\n", + "\n", + "if SERIAL:\n", + " clusters_cpu = cf_cpu.steal_clusters(realloc_same_capacity=False)\n", + " n_clusters_cpu = clusters_cpu.size\n", + " hist_cpu = make_hist(clusters_cpu)\n", + "else:\n", + " cf_cpu.stop(); sink.stop()\n", + " clusters_cpu = sink.steal_clusters()\n", + " hist_cpu = bh.Histogram(bh.axis.Regular(N_BINS, -2, 4000))\n", + " n_clusters_cpu = 0\n", + " for cv in clusters_cpu:\n", + " hist_cpu.fill(cv.sum())\n", + " n_clusters_cpu += cv.size\n", + "\n", + "print(f' minor faults: {mf1-mf0:,} (~{(mf1-mf0)*0.7e-6:.2f}s est.) major: {Mf1-Mf0:,}')\n", + "print(\"=========================================================================================\")\n", + "print(f'CPU clustering: {t_cpu:.3f}s ({N/t_cpu:.0f} FPS, '\n", + " f'{n_clusters_cpu} clusters, {n_clusters_cpu/N:.2f}/frame)')" + ] + }, + { + "cell_type": "markdown", + "id": "md-v1", + "metadata": {}, + "source": [ + "## CUDA — per-frame (1 launch/frame, pageable memory)\n", + "Simplest path: no batching, no pinning. Isolates per-frame launch + PCIe overhead." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "v1", + "metadata": {}, + "outputs": [ + { + "ename": "KeyboardInterrupt", + "evalue": "", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[9], line 8\u001b[0m\n\u001b[1;32m 6\u001b[0m hist_cuda_v1 \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 7\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m frame \u001b[38;5;129;01min\u001b[39;00m data:\n\u001b[0;32m----> 8\u001b[0m \u001b[43mcf_cuda_v1\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfind_clusters\u001b[49m\u001b[43m(\u001b[49m\u001b[43mframe\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 9\u001b[0m clusters_frame \u001b[38;5;241m=\u001b[39m cf_cuda_v1\u001b[38;5;241m.\u001b[39msteal_clusters(realloc_same_capacity\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[1;32m 10\u001b[0m n_clusters_cuda_v1 \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m clusters_frame\u001b[38;5;241m.\u001b[39msize\n", + "\u001b[0;31mKeyboardInterrupt\u001b[0m: " + ] + } + ], + "source": [ + "cf_cuda_v1.reset_timers()\n", + "mf0, Mf0 = _faults()\n", + "t0 = time.perf_counter()\n", + "\n", + "n_clusters_cuda_v1 = 0\n", + "hist_cuda_v1 = None\n", + "for frame in data:\n", + " cf_cuda_v1.find_clusters(frame)\n", + " clusters_frame = cf_cuda_v1.steal_clusters(realloc_same_capacity=True)\n", + " n_clusters_cuda_v1 += clusters_frame.size\n", + " h = make_hist(clusters_frame)\n", + " hist_cuda_v1 = h if hist_cuda_v1 is None else hist_cuda_v1 + h\n", + "\n", + "t_cuda_v1 = time.perf_counter() - t0\n", + "mf1, Mf1 = _faults()\n", + "kernel_ms = cf_cuda_v1.avg_kernel_time_ms()\n", + "print(f' minor faults: {mf1-mf0:,} (~{(mf1-mf0)*0.7e-6:.2f}s est.) major: {Mf1-Mf0:,}')\n", + "print(\"=========================================================================================\")\n", + "print(f'CUDA per-frame: {t_cuda_v1:.3f}s ({N/t_cuda_v1:.0f} FPS, '\n", + " f'{n_clusters_cuda_v1} clusters, {n_clusters_cuda_v1/N:.2f}/frame)')\n", + "print(f' Kernel only: {kernel_ms:.3f} ms/frame')\n", + "print(f' PCIe + overhead: {t_cuda_v1*1000/N - kernel_ms:.3f} ms/frame')\n", + "print(f'Speedup (CPU/CUDA): {t_cpu / t_cuda_v1:.2f}x')" + ] + }, + { + "cell_type": "markdown", + "id": "md-batched", + "metadata": {}, + "source": [ + "## CUDA — batched + multi-streamed + pinned dataset\n", + "Pins the whole input once, then submits `BATCH_SIZE`-frame chunks across `N_STREAMS` streams so H2D / kernel / D2H overlap." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "batched", + "metadata": {}, + "outputs": [], + "source": [ + "cf_cuda.register_input_buffer(data) # pin the whole dataset once\n", + "clusters_cuda_per_frame = []\n", + "\n", + "cf_cuda.reset_timers()\n", + "mf0, Mf0 = _faults()\n", + "t0 = time.perf_counter()\n", + "for start in range(0, N, BATCH_SIZE):\n", + " stop = min(start + BATCH_SIZE, N)\n", + " clusters_cuda_per_frame.extend(\n", + " cf_cuda.find_clusters_batched(data[start:stop], first_frame=start))\n", + "t_cuda = time.perf_counter() - t0\n", + "mf1, Mf1 = _faults() # before make_hist_from_batch, which allocates GBs itself\n", + "\n", + "cf_cuda.unregister_input_buffer()\n", + "kernel_ms = cf_cuda.avg_kernel_time_ms()\n", + "n_clusters_cuda = sum(cv.size for cv in clusters_cuda_per_frame)\n", + "hist_cuda = make_hist_from_batch(clusters_cuda_per_frame)\n", + "\n", + "print(f' minor faults: {mf1-mf0:,} (~{(mf1-mf0)*0.7e-6:.2f}s est.) major: {Mf1-Mf0:,}')\n", + "print(\"=========================================================================================\")\n", + "print(f'CUDA batched: {t_cuda:.3f}s ({N/t_cuda:.0f} FPS, '\n", + " f'{n_clusters_cuda} clusters, {n_clusters_cuda/N:.2f}/frame)')\n", + "print(f' Kernel only: {kernel_ms:.3f} ms/frame')\n", + "print(f' PCIe + overhead: {t_cuda*1000/N - kernel_ms:.3f} ms/frame')\n", + "print(f'Speedup (CPU/CUDA): {t_cpu / t_cuda:.2f}x')" + ] + }, + { + "cell_type": "markdown", + "id": "md-asyncpipe", + "metadata": {}, + "source": [ + "## Async pipeline — is the wall clock host-bound?\n", + "\n", + "Diagnostic for the \"next moves\" slide. Both loops submit **slices of the already-pinned\n", + "`data`**, so neither pays a staging memcpy (the two-buffer pattern in the cell below is\n", + "only needed when frames are streamed in, not when the whole dataset is pinned in RAM).\n", + "\n", + "- **serial** `submit(b); collect(b)` -> GPU + host, serialized\n", + "- **pipelined** `submit(b+1)` before `collect(b)` -> max(GPU, host)\n", + "\n", + "`serial - pipelined` = min(GPU, host), i.e. how much time is hideable. Compare\n", + "`pipelined` against the exclusive GPU floor from nsys (`nsys_kernel_probe.py`) to see\n", + "which of the two is the real limit. Re-run the cell until minor faults plateau." + ] + }, + { + "cell_type": "code", + "id": "asyncpipe", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "# Async pipeline vs serial — no staging memcpy, slices of the pinned dataset.\n", + "cf_async = ClusterFinderCUDA(image_size, cluster_size, n_sigma=N_SIGMA,\n", + " max_clusters_per_frame=3000, n_streams=N_STREAMS)\n", + "pd.seek(0)\n", + "for _ in range(n_frames_pd):\n", + " cf_async.push_pedestal_frame(pd.read_frame().copy())\n", + "\n", + "cf_async.register_input_buffer(data) # pin once; both loops reuse it\n", + "bounds = [(s, min(s + BATCH_SIZE, N)) for s in range(0, N, BATCH_SIZE)]\n", + "\n", + "# ---- 1. SERIAL: submit then immediately collect (no overlap) ----------------\n", + "cf_async.reset_timers()\n", + "mf0, Mf0 = _faults()\n", + "t0 = time.perf_counter()\n", + "res_serial = []\n", + "for a, b in bounds:\n", + " tok = cf_async.submit_batch(data[a:b], first_frame=a)\n", + " res_serial.extend(cf_async.collect(tok))\n", + "t_serial = time.perf_counter() - t0\n", + "mf1, Mf1 = _faults()\n", + "f_serial = mf1 - mf0\n", + "n_serial = sum(cv.size for cv in res_serial)\n", + "del res_serial # free before the next loop\n", + "\n", + "# ---- 2. PIPELINED: keep one batch in flight while collecting the previous ---\n", + "mf2, Mf2 = _faults()\n", + "t0 = time.perf_counter()\n", + "res_pipe = []\n", + "tok = cf_async.submit_batch(data[bounds[0][0]:bounds[0][1]], first_frame=bounds[0][0])\n", + "for a, b in bounds[1:]:\n", + " nxt = cf_async.submit_batch(data[a:b], first_frame=a) # GPU starts batch N+1\n", + " res_pipe.extend(cf_async.collect(tok)) # host marshals batch N\n", + " tok = nxt\n", + "res_pipe.extend(cf_async.collect(tok)) # drain the last one\n", + "t_pipe = time.perf_counter() - t0\n", + "mf3, Mf3 = _faults()\n", + "f_pipe = mf3 - mf2\n", + "n_pipe = sum(cv.size for cv in res_pipe)\n", + "\n", + "cf_async.unregister_input_buffer()\n", + "\n", + "us = lambda t: t * 1e6 / N\n", + "hidden = us(t_serial) - us(t_pipe)\n", + "print(f' minor faults: serial {f_serial:,} pipelined {f_pipe:,} '\n", + " f'(re-run until both plateau)')\n", + "print(\"=========================================================================================\")\n", + "print(f'serial (GPU + host): {t_serial:.3f}s ({N/t_serial:8,.0f} FPS) {us(t_serial):6.1f} us/frame '\n", + " f'{n_serial} clusters')\n", + "print(f'pipelined (max of both): {t_pipe:.3f}s ({N/t_pipe:8,.0f} FPS) {us(t_pipe):6.1f} us/frame '\n", + " f'{n_pipe} clusters')\n", + "print(f' hidden by overlap: {hidden:6.1f} us/frame = min(GPU, host)')\n", + "print(f' speedup pipelined/serial: {t_serial / t_pipe:.2f}x')\n", + "try:\n", + " print(f' vs find_clusters_batched: {t_cuda / t_pipe:.2f}x '\n", + " f'({us(t_cuda):.1f} -> {us(t_pipe):.1f} us/frame)')\n", + "except NameError:\n", + " pass\n", + "print(f' kernel (event timer): {cf_async.avg_kernel_time_ms()*1000:6.1f} us/frame '\n", + " f'(inflated under {N_STREAMS} streams — use nsys for the true value)')\n", + "print()\n", + "print('Interpretation: if pipelined ~= the nsys GPU floor, the remaining cost was host-side')\n", + "print('and is now hidden. If pipelined stays well above it, the host is the binding limit.')\n" + ] + }, + { + "cell_type": "markdown", + "id": "md-graph", + "metadata": {}, + "source": [ + "## CUDA — Graph (pinned dataset)\n", + "Pre-records the H2D->kernel->D2H pipeline as a CUDA Graph per stream, cutting per-frame CPU API overhead (~21 us vs ~60 us for the streamed version)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "graph", + "metadata": {}, + "outputs": [], + "source": [ + "cf_graph.register_input_buffer(data)\n", + "clusters_graph_per_frame = []\n", + "\n", + "mf0, Mf0 = _faults()\n", + "t0 = time.perf_counter()\n", + "for start in range(0, N, BATCH_SIZE):\n", + " stop = min(start + BATCH_SIZE, N)\n", + " clusters_graph_per_frame.extend(\n", + " cf_graph.find_clusters_batched(data[start:stop], first_frame=start))\n", + "t_graph = time.perf_counter() - t0\n", + "mf1, Mf1 = _faults() # before make_hist_from_batch, which allocates GBs itself\n", + "\n", + "cf_graph.unregister_input_buffer()\n", + "n_clusters_graph = sum(cv.size for cv in clusters_graph_per_frame)\n", + "hist_graph = make_hist_from_batch(clusters_graph_per_frame)\n", + "\n", + "print(f' minor faults: {mf1-mf0:,} (~{(mf1-mf0)*0.7e-6:.2f}s est.) major: {Mf1-Mf0:,}')\n", + "print(\"=========================================================================================\")\n", + "print(f'CUDA Graph: {t_graph:.3f}s ({N/t_graph:.0f} FPS, '\n", + " f'{n_clusters_graph} clusters, {n_clusters_graph/N:.2f}/frame)')\n", + "print(f' Kernel+PCIe+ovhd: {t_graph*1000/N:.3f} ms/frame (kernel not individually timed)')\n", + "print(f'Speedup (vs batched): {t_cuda / t_graph:.2f}x')\n", + "print(f'Speedup (CPU/Graph): {t_cpu / t_graph:.2f}x')" + ] + }, + { + "cell_type": "markdown", + "id": "md-async", + "metadata": {}, + "source": [ + "## CUDA — async double-buffered pipeline\n", + "Two pinned batch buffers: while the GPU processes `buf[cur]`, the CPU fills `buf[nxt]`, so H2D copies and kernels stay back-to-back." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "async", + "metadata": {}, + "outputs": [], + "source": [ + "# buf = [np.empty((BATCH_SIZE, rows, cols), dtype=np.uint16) for _ in range(2)]\n", + "# cf_async.pin_buffer(buf[0])\n", + "# cf_async.pin_buffer(buf[1])\n", + "\n", + "# clusters_async = []\n", + "# cf_async.reset_timers()\n", + "# t0 = time.perf_counter()\n", + "\n", + "# cur = 0\n", + "# n0 = min(BATCH_SIZE, N)\n", + "# buf[cur][:n0] = data[:n0]\n", + "# tok = cf_async.submit_batch(buf[cur][:n0], first_frame=0)\n", + "\n", + "# for start in range(BATCH_SIZE, N, BATCH_SIZE):\n", + "# nxt = 1 - cur\n", + "# stop = min(start + BATCH_SIZE, N)\n", + "# n = stop - start\n", + "# buf[nxt][:n] = data[start:stop] # fill next while GPU runs current\n", + "# next_tok = cf_async.submit_batch(buf[nxt][:n], first_frame=start)\n", + "# clusters_async.extend(cf_async.collect(tok)) # drain previous\n", + "# tok = next_tok\n", + "# cur = nxt\n", + "# clusters_async.extend(cf_async.collect(tok)) # drain final\n", + "\n", + "# t_async = time.perf_counter() - t0\n", + "# cf_async.unpin_buffer(buf[0]); cf_async.unpin_buffer(buf[1])\n", + "\n", + "# kernel_ms_async = cf_async.avg_kernel_time_ms()\n", + "# n_clusters_async = sum(cv.size for cv in clusters_async)\n", + "# hist_async = make_hist_from_batch(clusters_async)\n", + "\n", + "# print(f'CUDA async pipeline: {t_async:.3f}s ({N/t_async:.0f} FPS, '\n", + "# f'{n_clusters_async} clusters, {n_clusters_async/N:.2f}/frame)')\n", + "# print(f' Kernel only: {kernel_ms_async:.3f} ms/frame')\n", + "# print(f' PCIe + overhead: {t_async*1000/N - kernel_ms_async:.3f} ms/frame')\n", + "# print(f'Speedup (vs Graph): {t_graph / t_async:.2f}x')" + ] + }, + { + "cell_type": "markdown", + "id": "md-agree", + "metadata": {}, + "source": [ + "## Agreement sanity check\n", + "Counts should match to a hair. The residual comes from the CUDA finder updating the pedestal once per frame vs the CPU's per-pixel update (analysed in `ClusterFinderFrozen_vs_CUDA.ipynb`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "agree", + "metadata": {}, + "outputs": [], + "source": [ + "for name, n in [('CPU', n_clusters_cpu), ('CUDA per-frame', n_clusters_cuda_v1),\n", + " ('CUDA batched', n_clusters_cuda), ('CUDA graph', n_clusters_graph)]:\n", + " # ('CUDA async', n_clusters_async)]:\n", + " d = abs(n - n_clusters_cpu)\n", + " print(f' {name:<16} {n:>12,} diff vs CPU {d:>6,} ({d/max(n_clusters_cpu,1):.4%})')" + ] + }, + { + "cell_type": "markdown", + "id": "md-plots", + "metadata": {}, + "source": [ + "## Spectrum: CPU vs CUDA variants" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "plots", + "metadata": {}, + "outputs": [], + "source": [ + "fig, (ax_spec, ax_ratio) = plt.subplots(\n", + " 2, 1, figsize=(8, 6), sharex=True,\n", + " gridspec_kw={'height_ratios': [3, 1]})\n", + "\n", + "edges = hist_cpu.axes[0].edges\n", + "cpu_vals = hist_cpu.values()\n", + "cuda_vals_v1 = hist_cuda_v1.values()\n", + "cuda_vals = hist_cuda.values()\n", + "# async_vals = hist_async.values()\n", + "graph_vals = hist_graph.values()\n", + "\n", + "ax_spec.stairs(cpu_vals, edges, label=f'CPU ({n_clusters_cpu} clusters)')\n", + "ax_spec.stairs(cuda_vals_v1, edges, label=f'CUDA single frames ({n_clusters_cuda} clusters)', linestyle='--')\n", + "ax_spec.stairs(cuda_vals, edges, label=f'CUDA batched ({n_clusters_cuda} clusters)', linestyle='--')\n", + "# ax_spec.stairs(async_vals, edges, label=f'CUDA async ({n_clusters_async} clusters)', linestyle='-.')\n", + "ax_spec.stairs(graph_vals, edges, label=f'CUDA graph ({n_clusters_graph} clusters)', linestyle=':')\n", + "ax_spec.set_ylabel('Counts')\n", + "ax_spec.set_title('Cluster energy spectrum: CPU vs CUDA variants')\n", + "ax_spec.legend()\n", + "ax_spec.grid(alpha=0.2)\n", + "\n", + "with np.errstate(divide='ignore', invalid='ignore'):\n", + " ratio_cuda = np.where(cpu_vals > 0, cuda_vals / cpu_vals, np.nan)\n", + " ratio_graph = np.where(cpu_vals > 0, graph_vals / cpu_vals, np.nan)\n", + "\n", + "ax_ratio.stairs(ratio_cuda, edges, label='CUDA batched / CPU', color='C1', linestyle='--')\n", + "ax_ratio.stairs(ratio_graph, edges, label='CUDA graph / CPU', color='C3', linestyle=':')\n", + "ax_ratio.axhline(1.0, color='gray', linewidth=0.5)\n", + "ax_ratio.set_ylabel('Variant / CPU')\n", + "ax_ratio.set_xlabel('Energy [ADU]')\n", + "ax_ratio.set_ylim(0.5, 2.0)\n", + "ax_ratio.legend(fontsize=8)\n", + "ax_ratio.grid(alpha=0.3)\n", + "\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "5e4a3d4b", + "metadata": {}, + "source": [ + "## Optimization arc for the deck (opt1 → opt2 → opt3)\n", + "\n", + "Same pedestal and `data` as above, scored against the CPU baseline (`t_cpu`, `n_clusters_cpu`). Each cell is self-contained: it builds its finder, trains the pedestal, and runs.\n", + "\n", + "- **opt1** — first CUDA port: `ClusterFinderCUDAOpt2`, single stream, one launch per frame (pre-refactor pipeline, pageable H2D).\n", + "- **opt2** — multi-stream + batching: `ClusterFinderCUDAOpt2`, `n_streams=4`, `find_clusters_batched` (still pre-refactor, pageable H2D).\n", + "- **opt3** — current finder, batched but **without** input pinning: `ClusterFinderCUDA.find_clusters_batched` on pageable memory (no `register_input_buffer`). The pinned + Graph + async steps are the batched/graph cells above.\n", + "\n", + "Correctness is held constant across the arc: the Test3 local-max gate has been backported into the opt2 kernel, so opt1/opt2 counts match the CPU and current finders — only the pipeline changes from step to step." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9c456e2e", + "metadata": {}, + "outputs": [], + "source": [ + "# opt1 — first port: ClusterFinderCUDAOpt2, single stream, one launch per frame\n", + "cf_opt1 = ClusterFinderCUDAOpt2(image_size, cluster_size, n_sigma=N_SIGMA,\n", + " max_clusters_per_frame=3000, n_streams=1)\n", + "pd.seek(0)\n", + "for _ in range(n_frames_pd):\n", + " cf_opt1.push_pedestal_frame(pd.read_frame().copy())\n", + "\n", + "cf_opt1.reset_timers()\n", + "mf0, Mf0 = _faults()\n", + "t0 = time.perf_counter()\n", + "n_clusters_opt1 = 0\n", + "for i, frame in enumerate(data):\n", + " cf_opt1.find_clusters(frame, frame_number=i)\n", + " n_clusters_opt1 += cf_opt1.steal_clusters(realloc_same_capacity=True).size\n", + "t_opt1 = time.perf_counter() - t0\n", + "mf1, Mf1 = _faults()\n", + "kernel_ms = cf_opt1.avg_kernel_time_ms()\n", + "\n", + "print(f' minor faults: {mf1-mf0:,} (~{(mf1-mf0)*0.7e-6:.2f}s est.) major: {Mf1-Mf0:,}')\n", + "print(\"=========================================================================================\")\n", + "print(f'opt1 per-frame: {t_opt1:.3f}s ({N/t_opt1:.0f} FPS, '\n", + " f'{n_clusters_opt1} clusters, {n_clusters_opt1/N:.2f}/frame)')\n", + "print(f' Kernel only: {kernel_ms:.3f} ms/frame')\n", + "print(f' PCIe + overhead: {t_opt1*1000/N - kernel_ms:.3f} ms/frame')\n", + "print(f'Speedup (CPU/opt1): {t_cpu / t_opt1:.2f}x')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f32b821f", + "metadata": {}, + "outputs": [], + "source": [ + "# opt2 — multi-stream + batching: ClusterFinderCUDAOpt2, pageable H2D (no pinning)\n", + "cf_opt2 = ClusterFinderCUDAOpt2(image_size, cluster_size, n_sigma=N_SIGMA,\n", + " max_clusters_per_frame=3000, n_streams=N_STREAMS)\n", + "pd.seek(0)\n", + "for _ in range(n_frames_pd):\n", + " cf_opt2.push_pedestal_frame(pd.read_frame().copy())\n", + "\n", + "cf_opt2.reset_timers()\n", + "mf0, Mf0 = _faults()\n", + "t0 = time.perf_counter()\n", + "clusters_opt2 = []\n", + "for start in range(0, N, BATCH_SIZE):\n", + " stop = min(start + BATCH_SIZE, N)\n", + " clusters_opt2.extend(cf_opt2.find_clusters_batched(data[start:stop], first_frame=start))\n", + "t_opt2 = time.perf_counter() - t0\n", + "mf1, Mf1 = _faults()\n", + "kernel_ms = cf_opt2.avg_kernel_time_ms()\n", + "n_clusters_opt2 = sum(cv.size for cv in clusters_opt2)\n", + "\n", + "print(f' minor faults: {mf1-mf0:,} (~{(mf1-mf0)*0.7e-6:.2f}s est.) major: {Mf1-Mf0:,}')\n", + "print(\"=========================================================================================\")\n", + "print(f'opt2 batched (no pin): {t_opt2:.3f}s ({N/t_opt2:.0f} FPS, '\n", + " f'{n_clusters_opt2} clusters, {n_clusters_opt2/N:.2f}/frame)')\n", + "print(f' Kernel only: {kernel_ms:.3f} ms/frame')\n", + "print(f' PCIe + overhead: {t_opt2*1000/N - kernel_ms:.3f} ms/frame')\n", + "print(f'Speedup (CPU/opt2): {t_cpu / t_opt2:.2f}x')\n", + "print(f'Speedup (opt1/opt2): {t_opt1 / t_opt2:.2f}x')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cf8e1f9b", + "metadata": {}, + "outputs": [], + "source": [ + "# opt3 — current finder, batched but WITHOUT input pinning (pageable H2D)\n", + "cf_opt3 = ClusterFinderCUDA(image_size, cluster_size, n_sigma=N_SIGMA,\n", + " max_clusters_per_frame=3000, n_streams=N_STREAMS)\n", + "pd.seek(0)\n", + "for _ in range(n_frames_pd):\n", + " cf_opt3.push_pedestal_frame(pd.read_frame().copy())\n", + "\n", + "cf_opt3.reset_timers()\n", + "mf0, Mf0 = _faults()\n", + "t0 = time.perf_counter()\n", + "clusters_opt3 = []\n", + "for start in range(0, N, BATCH_SIZE):\n", + " stop = min(start + BATCH_SIZE, N)\n", + " clusters_opt3.extend(cf_opt3.find_clusters_batched(data[start:stop], first_frame=start))\n", + "t_opt3 = time.perf_counter() - t0\n", + "mf1, Mf1 = _faults()\n", + "kernel_ms = cf_opt3.avg_kernel_time_ms()\n", + "n_clusters_opt3 = sum(cv.size for cv in clusters_opt3)\n", + "\n", + "print(f' minor faults: {mf1-mf0:,} (~{(mf1-mf0)*0.7e-6:.2f}s est.) major: {Mf1-Mf0:,}')\n", + "print(\"=========================================================================================\")\n", + "print(f'opt3 batched (no pin): {t_opt3:.3f}s ({N/t_opt3:.0f} FPS, '\n", + " f'{n_clusters_opt3} clusters, {n_clusters_opt3/N:.2f}/frame)')\n", + "print(f' Kernel only: {kernel_ms:.3f} ms/frame')\n", + "print(f' PCIe + overhead: {t_opt3*1000/N - kernel_ms:.3f} ms/frame')\n", + "print(f'Speedup (CPU/opt3): {t_cpu / t_opt3:.2f}x')\n", + "print(f'Speedup (opt2/opt3): {t_opt2 / t_opt3:.2f}x')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cbd03f58-0068-4ad6-aec6-51ed9c8f8f1e", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8457b726-b4ce-444f-a8af-8cd911e77f06", + "metadata": {}, + "outputs": [], + "source": [] } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" }, - "nbformat": 4, - "nbformat_minor": 5 - } + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.15" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/python/tests/nsys_kernel_probe.py b/python/tests/nsys_kernel_probe.py new file mode 100644 index 00000000..9f96abfc --- /dev/null +++ b/python/tests/nsys_kernel_probe.py @@ -0,0 +1,32 @@ +# Minimal probe for nsys: train pedestal, run one batched pass, print summary. +# Usage: nsys_kernel_probe.py [n_streams] [n_frames] +import sys +sys.path.append('/home/ferjao_k/aare/build') + +from pathlib import Path +import time +from aare import File, ClusterFinderCUDA + +n_streams = int(sys.argv[1]) if len(sys.argv) > 1 else 8 +N = int(sys.argv[2]) if len(sys.argv) > 2 else 2000 + +base = Path('/mnt/sls_det_storage/moench_data/2603_MaxIVBeamtime/2026032408/process/xrf/') +f = File(base / 'Cu_factor_10_data_master_0.json') +pd = File(base / 'Cu_factor_10_pedestal_master_0.json') + +cf = ClusterFinderCUDA((f.rows, f.cols), (3, 3), n_sigma=5, + max_clusters_per_frame=3000, n_streams=n_streams) +for _ in range(1000): + cf.push_pedestal_frame(pd.read_frame().copy()) + +data = f.read_n(N) +cf.register_input_buffer(data) + +t0 = time.perf_counter() +res = cf.find_clusters_batched(data, first_frame=0) +t = time.perf_counter() - t0 + +cf.unregister_input_buffer() +n = sum(cv.size for cv in res) +print(f'n_streams={n_streams} N={N} wall={t:.3f}s ({N/t:.0f} FPS) ' + f'clusters/frame={n/N:.2f} event kernel_ms={cf.avg_kernel_time_ms():.3f}') \ No newline at end of file