Files

T

fische_r ef6c236b10 updated example notebook with fixes after dask update

2023-08-29 11:01:48 +02:00

1.2 MiB

Raw Blame History

Segment water in GDL to quantify influence of liquid water on electrochemistry

This is an example notebook how I segmented the water within the cathode GDL. Segmentation of the membrane cavities works analogous with different cropping and label sets. There was an update of dask that broke the code. The feature stack setup and training functionalities have been fixed, but the full volume segmentation has yet to be reviewed. However, there is no reason why this should not work after some adjustments, probably even much better than before.

In [1]:

# modules
import os
import xarray as xr
import matplotlib.pyplot as plt
import numpy as np
import dask
import dask.array
from scipy import ndimage
from skimage import filters, feature, io
from skimage.morphology import disk,ball
import sys
from itertools import combinations_with_replacement
import pickle
import imageio
import json
from dask.distributed import Client, LocalCluster
import socket
import subprocess
import gc
import h5py

# get the ML functions, TODO: make a library once it works/is in a stable state
pytrainpath = '/mpc/homes/fische_r/lib/pytrainseg' #path of git repo
cwd = os.getcwd()
os.chdir(pytrainpath)
from filter_functions import image_filter
import training_functions as tfs
from training_functions import train_segmentation
from segmentation import segmentation
pytrain_git_sha = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip()
os.chdir(cwd)

#paths
host = socket.gethostname()
if host == 'mpc2959.psi.ch':
    gitpath = '/mpc/homes/fische_r/lib/co2ely-tomcat'
    toppath = '/mpc/homes/fische_r/NAS/DASCOELY'
    toppathSSD = '/mnt/SSD/fische_r/COELY'
    temppath = '/mnt/SSD/fische_r/tmp'
    temppath_2 = '/mpc/homes/fische_r/NAS/tmp'
    training_path = '/mpc/homes/fische_r/NAS/DASCOELY/processing/05_water_GDL_ML/'
    memlim = '420GB' #1/2 of available RAM for 2 workers
else:
    print('host '+host+' currently not supported')

# fetch githash
cwd = os.getcwd()
os.chdir(gitpath)
git_sha = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip()
githash = subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip()
os.chdir(cwd)

functionalities for interactive training

In [2]:

from ipywidgets import Image
from ipywidgets import ColorPicker, IntSlider, link, AppLayout, HBox
from ipycanvas import  hold_canvas,  MultiCanvas #RoughCanvas,Canvas,

def on_mouse_down(x, y):
    global drawing
    global position
    global shape
    drawing = True
    position = (x, y)
    shape = [position]

def on_mouse_move(x, y):
    global drawing
    global position
    global shape
    if not drawing:
        return
    with hold_canvas():
        canvas.stroke_line(position[0], position[1], x, y)
        position = (x, y)
    shape.append(position)

def on_mouse_up(x, y):
    global drawing
    global positiondu
    global shape
    drawing = False
    with hold_canvas():
        canvas.stroke_line(position[0], position[1], x, y)
        canvas.fill_polygon(shape)
    shape = []
    
def display_feature(i, TS):
    print('selected '+TS.feature_names[i])
    im = TS.current_feat_stack[:,:,i]
    im8 = im-im.min()
    im8 = im8/im8.max()*255
    return im8

fire up dask

In [3]:

tempfolder = temppath  #a big SSD is a major adavantage to allow spill to disk and still be efficient. large dataset might crash with too small SSD or be slow with normal HDD
dask.config.config['temporary-directory'] = tempfolder

# here you have the option to use a virtual cluster or even slurm on ra (not attempted yet)
cluster = LocalCluster(dashboard_address=':35000', memory_limit = memlim, n_workers=2) #settings optimised for mpc2959, play around if needed, if you know nothing else is using RAM then you can almost go to the limit
# # maybe less workers with more threads makes better use of shared memory 

# # scheduler_port = 'tcp://129.129.188.222:8786' #<-- if scheduler on mpc2959; scheduler on mpc2053 -> 'tcp://129.129.188.248:8786'
# # cluster = scheduler_port

client = Client(cluster)
# # client.amm.start()
print('Dashboard at '+client.dashboard_link)

2023-05-16 09:28:37,202 - distributed.diskutils - INFO - Found stale lock file and directory '/mnt/SSD/fische_r/tmp/dask-worker-space/worker-aqx4i1f3', purging
/mpc/homes/fische_r/miniconda3/lib/python3.10/contextlib.py:142: UserWarning: Creating scratch directories is taking a surprisingly long time. (3.69s) This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)

Dashboard at http://127.0.0.1:35000/status

Test ROI selection

Data preparation

create dask array

In [12]:

sample = '4'
file = '02_'+sample+'_registered_3p1D.nc'
imagepath = os.path.join(path_02_4D, file)
data = xr.open_dataset(imagepath)
images = [im for im in data.keys() if im[3:7] == 'imag']
images.sort()

In [13]:

t_utc = data['t_utc'].data
time = data['time'].data

In [14]:

# define some cropping dimensions


# here with obtained cropping data from file
jsonpath = os.path.join(training_path, 'cathode_cropping_and_aligning.json')
crop_dict = json.load(open(jsonpath, 'r'))

(a,b,c,d) = crop_dict[sample]
(e,f) = (50,-50)

#corrections to crop coordinates
# f = e+1750
# e = e+100

define border to crop to GDL

In [16]:

GDL_crop = 0
plt.figure(figsize=(16,9))
plt.imshow(test_im1[600,GDL_crop:, :], vmin =10000, vmax=15000, cmap='gray')

Out [16]:

<matplotlib.image.AxesImage at 0x7f5fa9009b70>

In [17]:

shp = data[images[4]][a:b, c:d, e:f].shape
# shp = data[images[4]][a:b, c+GDL_crop:d, e:f].shape
shp = shp + (len(images),)
print(shp, test_im1[:,GDL_crop:, :].shape)

(750, 340, 1916, 71) (750, 340, 1916)

In [23]:

data.close()

let dask load the data

needs to be a 4D-hdf5, for example created with xarray as .nc

In [ ]:

filename = '02_'+sample+'_registered_4D.nc'
imagepath = os.path.join(path_02_4D, filename)

In [ ]:

file = h5py.File(imagepath)

In [ ]:

da = dask.array.from_array(file['image_data'][a:b, c+GDL_crop:d, e:f], chunks= chunks) #do the cropping within the dask array creation, otherwise dask only adds a graph layer and might crash ?!

get data into image filter class

In [24]:

# TODO: include this routine into pytrainseg

IF = image_filter(sigmas = [2,4,6])# sigmas define the kernels to be used for the Gaussian Blurs
IF.data = da
shp = da.shape
coords = {'x': np.arange(shp[0]), 'y': np.arange(shp[1]), 'z': np.arange(shp[2]), 'time': np.arange(shp[3])}
IF.original_dataset = xr.Dataset({'tomo': (['x','y','z','time'], da)},
                                 coords = coords
                                )
# IF.data = IF.data.rechunk('auto')

prepare features

In [25]:

IF.prepare()

In [26]:

IF.stack_features()

In [28]:

IF.feature_stack #shows the feature stack

Out [28]:

	Array	Chunk
Bytes	16.15 TiB	25.27 MiB
Shape	(750, 340, 1916, 71, 64)	(36, 36, 36, 71, 1)
Dask graph	903168 chunks in 465 graph layers
Data type	float64 numpy.ndarray

In [29]:

IF.make_xarray_nc()

Training

set up objects

In [31]:

training_path_sample = os.path.join(training_path, sample)
if not os.path.exists(training_path_sample):
    os.mkdir(training_path_sample)

In [32]:

TS = train_segmentation(training_path=training_path_sample)

In [33]:

#quick and dirty fix to pass the dask client to the filter and training objects
TS.client = client
IF.client = client

In [34]:

TS.import_lazy_feature_data(IF.result, IF.original_dataset)

In [35]:

IF.combined_feature_names = list(IF.feature_names) + list(IF.feature_names_time_independent)

In [36]:

TS.combined_feature_names = IF.combined_feature_names

interactive training

check for existing training sets

these training sets only work if the cropping has not changed

In [38]:

existing_sets = os.listdir(os.path.join(training_path_sample, 'label_images'))
existing_sets.sort()
existing_sets

Out [38]:

['label_image_x_157_time_5_.tif',
 'label_image_x_172_time_17_.tif',
 'label_image_x_172_time_2_.tif',
 'label_image_x_252_time_0_.tif',
 'label_image_x_264_time_29_.tif',
 'label_image_x_270_time_50_.tif',
 'label_image_x_307_time_25_.tif',
 'label_image_x_368_time_45_.tif',
 'label_image_x_456_time_0_.tif',
 'label_image_x_503_time_15_.tif',
 'label_image_x_531_time_50_.tif',
 'label_image_y_241_time_0_.tif',
 'label_image_y_245_time_8_.tif',
 'label_image_y_250_time_23_.tif',
 'label_image_y_250_time_33_.tif',
 'label_image_y_255_time_20_.tif',
 'label_image_y_255_time_40_.tif',
 'label_image_y_255_time_64_.tif',
 'label_image_y_270_time_47_.tif',
 'label_image_y_280_time_57_.tif',
 'label_image_y_71_time_27_.tif',
 'label_image_z_307_time_0_.tif',
 'label_image_z_307_time_10_.tif',
 'label_image_z_307_time_25_.tif',
 'label_image_z_320_time_10_.tif',
 'label_image_z_320_time_30_.tif',
 'label_image_z_320_time_45_.tif',
 'label_image_z_468_time_11_.tif',
 'label_image_z_546_time_0_.tif',
 'label_image_z_564_time_0_.tif']

In [39]:

training_path

Out [39]:

'/mpc/homes/fische_r/NAS/DASCOELY/processing/05_water_GDL_ML/'

In [40]:

# you can load a compatible pickled training dict, check feature names
TS.training_dict = pickle.load(open(os.path.join(TS.training_path, pytrain_git_sha+'_training_dict.p'),'rb'))

In [ ]:

TS.training_dict

re-train with existing label sets. clear the training dictionary if necessary (training_dict)

In [41]:

TS.train()

training with existing label images
label_image_z_546_time_0_.tif
label_image_y_255_time_40_.tif already done
label_image_z_307_time_25_.tif already done
label_image_x_157_time_5_.tif already done
label_image_y_71_time_27_.tif already done
label_image_x_270_time_50_.tif already done
label_image_x_456_time_0_.tif already done
label_image_z_320_time_10_.tif already done
label_image_x_307_time_25_.tif
label_image_z_564_time_0_.tif already done
label_image_z_320_time_30_.tif already done
label_image_x_264_time_29_.tif already done
label_image_y_280_time_57_.tif already done
label_image_y_250_time_23_.tif already done
label_image_x_172_time_17_.tif

2023-05-16 09:45:00,371 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 09:45:14,621 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-05-16 09:45:35,188 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:46:04,314 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-05-16 09:49:22,459 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:50:54,694 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:52:31,246 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:53:45,077 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:54:17,198 - distributed.worker - ERROR - Timed out during handshake while connecting to tcp://127.0.0.1:40027 after 30 s
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/core.py", line 328, in connect
    handshake = await asyncio.wait_for(comm.read(), time_left())
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
    return await func(*args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 1566, in close
    await r.close_gracefully(reason=reason)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1224, in send_recv_from_rpc
    comm = await self.pool.connect(self.addr)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1468, in connect
    return await connect_attempt
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1389, in _connect
    comm = await connect(
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/core.py", line 333, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://127.0.0.1:40027 after 30 s

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
    return await func(*args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 1518, in close
    await self.finished()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 494, in finished
    await self._event_finished.wait()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError
2023-05-16 09:54:47,223 - distributed.worker - CRITICAL - Error trying close worker in response to broken internal state. Forcibly exiting worker NOW
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
    return await func(*args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 1518, in close
    await self.finished()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 494, in finished
    await self._event_finished.wait()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 230, in _force_close
    await asyncio.wait_for(
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
2023-05-16 09:54:47,590 - distributed.nanny - WARNING - Restarting worker
2023-05-16 09:55:24,784 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-05-16 09:55:27,257 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:30,068 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:33,322 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:36,585 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:39,938 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:43,304 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:46,738 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:51,772 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:56,134 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:00,644 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:06,342 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:11,303 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:17,016 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:22,878 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:29,303 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:36,880 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:43,433 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:50,846 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:59,606 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:07,562 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:16,888 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:26,843 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:37,066 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:47,981 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:59,347 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:58:11,759 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:58:25,270 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:58:40,363 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-05-16 09:58:54,131 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-05-16 09:59:11,643 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-05-16 09:59:30,384 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-05-16 09:59:49,345 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-05-16 10:00:11,604 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-05-16 10:00:50,460 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-05-16 10:01:38,505 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 10:02:19,995 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 10:03:03,981 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-05-16 10:03:44,879 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-05-16 10:04:16,568 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-05-16 10:05:04,492 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-05-16 10:06:05,845 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:06:44,248 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:07:48,311 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:08:33,565 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:09:17,781 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:10:02,956 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:10:56,186 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:11:48,945 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:13:03,665 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:14:37,086 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:16:28,883 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 10:17:57,100 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 10:20:21,831 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 10:40:24,306 - distributed.worker.memory - WARNING - gc.collect() took 11.618s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
/mpc/homes/fische_r/lib/pytrainseg/training_functions.py:375: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if not X == 'no labels':

label_image_x_503_time_15_.tif

2023-05-16 11:00:29,991 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-05-16 11:02:55,018 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-05-16 11:04:08,520 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-05-16 11:21:07,596 - distributed.worker.memory - WARNING - gc.collect() took 11.310s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 11:34:20,746 - distributed.worker.memory - WARNING - gc.collect() took 12.740s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.

label_image_y_255_time_20_.tif

2023-05-16 11:58:42,997 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 12:00:56,045 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 12:03:19,904 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 12:05:08,338 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 12:40:23,700 - distributed.worker.memory - WARNING - gc.collect() took 15.942s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 12:54:44,026 - distributed.worker.memory - WARNING - gc.collect() took 16.627s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 13:06:40,104 - distributed.worker.memory - WARNING - gc.collect() took 18.170s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 13:23:04,136 - distributed.worker.memory - WARNING - gc.collect() took 19.639s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 13:39:07,655 - distributed.worker.memory - WARNING - gc.collect() took 25.502s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.

label_image_y_250_time_33_.tif

[0;31m---------------------------------------------------------------------------[0m
[0;31mKeyboardInterrupt[0m                         Traceback (most recent call last)
Cell [0;32mIn[41], line 1[0m
[0;32m----> 1[0m [43mTS[49m[38;5;241;43m.[39;49m[43mtrain[49m[43m([49m[43m)[49m

File [0;32m~/lib/pytrainseg/training_functions.py:374[0m, in [0;36mtrain_segmentation.train[0;34m(self, clear_dict, redo)[0m
[1;32m    372[0m     [38;5;28;01mcontinue[39;00m
[1;32m    373[0m [38;5;28mprint[39m(label_name)
[0;32m--> 374[0m X, y [38;5;241m=[39m [43mtraining_set_per_image[49m[43m([49m[43mlabel_name[49m[43m,[49m[43m [49m[43mpath[49m[43m,[49m[43m [49m[43mfeat_data[49m[43m,[49m[43m [49m[38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43mlazy[49m[43m)[49m
[1;32m    375[0m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m X [38;5;241m==[39m [38;5;124m'[39m[38;5;124mno labels[39m[38;5;124m'[39m:
[1;32m    376[0m     [38;5;28mself[39m[38;5;241m.[39mtraining_dict[label_name] [38;5;241m=[39m X,y

File [0;32m~/lib/pytrainseg/training_functions.py:138[0m, in [0;36mtraining_set_per_image[0;34m(label_name, trainingpath, feat_data, lazy)[0m
[1;32m    130[0m [38;5;66;03m# if lazy:[39;00m
[1;32m    131[0m [38;5;66;03m#     print('Need to actually calculate the features for each slice, seems inefficient')[39;00m
[1;32m    132[0m [38;5;66;03m# #   not sure how efficient this is[39;00m
[0;32m   (...)[0m
[1;32m    135[0m [38;5;66;03m#     feat_stack = feat_stack.compute()[39;00m
[1;32m    136[0m [38;5;66;03m# else:[39;00m
[1;32m    137[0m [38;5;28;01mif[39;00m [38;5;28mtype[39m(feat_stack) [38;5;129;01mis[39;00m [38;5;129;01mnot[39;00m np[38;5;241m.[39mndarray:
[0;32m--> 138[0m         feat_stack [38;5;241m=[39m [43mfeat_stack[49m[38;5;241;43m.[39;49m[43mcompute[49m[43m([49m[43m)[49m
[1;32m    139[0m [38;5;28;01mif[39;00m [38;5;28mtype[39m(feat_stack_t_idp) [38;5;129;01mis[39;00m [38;5;129;01mnot[39;00m np[38;5;241m.[39mndarray:
[1;32m    140[0m         feat_stack_t_idp [38;5;241m=[39m feat_stack_t_idp[38;5;241m.[39mcompute()

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/base.py:314[0m, in [0;36mDaskMethodsMixin.compute[0;34m(self, **kwargs)[0m
[1;32m    290[0m [38;5;28;01mdef[39;00m [38;5;21mcompute[39m([38;5;28mself[39m, [38;5;241m*[39m[38;5;241m*[39mkwargs):
[1;32m    291[0m [38;5;250m    [39m[38;5;124;03m"""Compute this dask collection[39;00m
[1;32m    292[0m 
[1;32m    293[0m [38;5;124;03m    This turns a lazy Dask collection into its in-memory equivalent.[39;00m
[0;32m   (...)[0m
[1;32m    312[0m [38;5;124;03m    dask.base.compute[39;00m
[1;32m    313[0m [38;5;124;03m    """[39;00m
[0;32m--> 314[0m     (result,) [38;5;241m=[39m [43mcompute[49m[43m([49m[38;5;28;43mself[39;49m[43m,[49m[43m [49m[43mtraverse[49m[38;5;241;43m=[39;49m[38;5;28;43;01mFalse[39;49;00m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m[43m)[49m
[1;32m    315[0m     [38;5;28;01mreturn[39;00m result

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/base.py:593[0m, in [0;36mcompute[0;34m(traverse, optimize_graph, scheduler, get, *args, **kwargs)[0m
[1;32m    585[0m     [38;5;28;01mreturn[39;00m args
[1;32m    587[0m schedule [38;5;241m=[39m get_scheduler(
[1;32m    588[0m     scheduler[38;5;241m=[39mscheduler,
[1;32m    589[0m     collections[38;5;241m=[39mcollections,
[1;32m    590[0m     get[38;5;241m=[39mget,
[1;32m    591[0m )
[0;32m--> 593[0m dsk [38;5;241m=[39m [43mcollections_to_dsk[49m[43m([49m[43mcollections[49m[43m,[49m[43m [49m[43moptimize_graph[49m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m[43m)[49m
[1;32m    594[0m keys, postcomputes [38;5;241m=[39m [], []
[1;32m    595[0m [38;5;28;01mfor[39;00m x [38;5;129;01min[39;00m collections:

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/base.py:366[0m, in [0;36mcollections_to_dsk[0;34m(collections, optimize_graph, optimizations, **kwargs)[0m
[1;32m    364[0m [38;5;28;01mfor[39;00m opt, val [38;5;129;01min[39;00m groups[38;5;241m.[39mitems():
[1;32m    365[0m     dsk, keys [38;5;241m=[39m _extract_graph_and_keys(val)
[0;32m--> 366[0m     dsk [38;5;241m=[39m [43mopt[49m[43m([49m[43mdsk[49m[43m,[49m[43m [49m[43mkeys[49m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m[43m)[49m
[1;32m    368[0m     [38;5;28;01mfor[39;00m opt_inner [38;5;129;01min[39;00m optimizations:
[1;32m    369[0m         dsk [38;5;241m=[39m opt_inner(dsk, keys, [38;5;241m*[39m[38;5;241m*[39mkwargs)

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/array/optimization.py:57[0m, in [0;36moptimize[0;34m(dsk, keys, fuse_keys, fast_functions, inline_functions_fast_functions, rename_fused_keys, **kwargs)[0m
[1;32m     54[0m [38;5;28;01mif[39;00m config[38;5;241m.[39mget([38;5;124m"[39m[38;5;124moptimization.fuse.active[39m[38;5;124m"[39m) [38;5;129;01mis[39;00m [38;5;28;01mFalse[39;00m:
[1;32m     55[0m     [38;5;28;01mreturn[39;00m dsk
[0;32m---> 57[0m dependencies [38;5;241m=[39m [43mdsk[49m[38;5;241;43m.[39;49m[43mget_all_dependencies[49m[43m([49m[43m)[49m
[1;32m     58[0m dsk [38;5;241m=[39m ensure_dict(dsk)
[1;32m     60[0m [38;5;66;03m# Low level task optimizations[39;00m

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/highlevelgraph.py:813[0m, in [0;36mHighLevelGraph.get_all_dependencies[0;34m(self)[0m
[1;32m    811[0m [38;5;28;01mif[39;00m missing_keys:
[1;32m    812[0m     [38;5;28;01mfor[39;00m layer [38;5;129;01min[39;00m [38;5;28mself[39m[38;5;241m.[39mlayers[38;5;241m.[39mvalues():
[0;32m--> 813[0m         [38;5;28;01mfor[39;00m k [38;5;129;01min[39;00m [43mmissing_keys[49m[43m [49m[38;5;241;43m&[39;49m[43m [49m[43mlayer[49m[38;5;241;43m.[39;49m[43mkeys[49m[43m([49m[43m)[49m:
[1;32m    814[0m             [38;5;28mself[39m[38;5;241m.[39mkey_dependencies[k] [38;5;241m=[39m layer[38;5;241m.[39mget_dependencies(k, all_keys)
[1;32m    815[0m [38;5;28;01mreturn[39;00m [38;5;28mself[39m[38;5;241m.[39mkey_dependencies

File [0;32m~/miniconda3/lib/python3.10/_collections_abc.py:638[0m, in [0;36mSet.__and__[0;34m(self, other)[0m
[1;32m    636[0m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m [38;5;28misinstance[39m(other, Iterable):
[1;32m    637[0m     [38;5;28;01mreturn[39;00m [38;5;28mNotImplemented[39m
[0;32m--> 638[0m [38;5;28;01mreturn[39;00m [38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43m_from_iterable[49m[43m([49m[43mvalue[49m[43m [49m[38;5;28;43;01mfor[39;49;00m[43m [49m[43mvalue[49m[43m [49m[38;5;129;43;01min[39;49;00m[43m [49m[43mother[49m[43m [49m[38;5;28;43;01mif[39;49;00m[43m [49m[43mvalue[49m[43m [49m[38;5;129;43;01min[39;49;00m[43m [49m[38;5;28;43mself[39;49m[43m)[49m

File [0;32m~/miniconda3/lib/python3.10/_collections_abc.py:880[0m, in [0;36mKeysView._from_iterable[0;34m(cls, it)[0m
[1;32m    878[0m [38;5;129m@classmethod[39m
[1;32m    879[0m [38;5;28;01mdef[39;00m [38;5;21m_from_iterable[39m([38;5;28mcls[39m, it):
[0;32m--> 880[0m     [38;5;28;01mreturn[39;00m [38;5;28;43mset[39;49m[43m([49m[43mit[49m[43m)[49m

File [0;32m~/miniconda3/lib/python3.10/_collections_abc.py:638[0m, in [0;36m<genexpr>[0;34m(.0)[0m
[1;32m    636[0m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m [38;5;28misinstance[39m(other, Iterable):
[1;32m    637[0m     [38;5;28;01mreturn[39;00m [38;5;28mNotImplemented[39m
[0;32m--> 638[0m [38;5;28;01mreturn[39;00m [38;5;28mself[39m[38;5;241m.[39m_from_iterable(value [38;5;28;01mfor[39;00m value [38;5;129;01min[39;00m other [38;5;28;01mif[39;00m [43mvalue[49m[43m [49m[38;5;129;43;01min[39;49;00m[43m [49m[38;5;28;43mself[39;49m)

File [0;32m~/miniconda3/lib/python3.10/_collections_abc.py:883[0m, in [0;36mKeysView.__contains__[0;34m(self, key)[0m
[1;32m    882[0m [38;5;28;01mdef[39;00m [38;5;21m__contains__[39m([38;5;28mself[39m, key):
[0;32m--> 883[0m     [38;5;28;01mreturn[39;00m [43mkey[49m[43m [49m[38;5;129;43;01min[39;49;00m[43m [49m[38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43m_mapping[49m

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/highlevelgraph.py:540[0m, in [0;36mMaterializedLayer.__contains__[0;34m(self, k)[0m
[1;32m    537[0m     [38;5;28msuper[39m()[38;5;241m.[39m[38;5;21m__init__[39m(annotations[38;5;241m=[39mannotations)
[1;32m    538[0m     [38;5;28mself[39m[38;5;241m.[39mmapping [38;5;241m=[39m mapping
[0;32m--> 540[0m [38;5;28;01mdef[39;00m [38;5;21m__contains__[39m([38;5;28mself[39m, k):
[1;32m    541[0m     [38;5;28;01mreturn[39;00m k [38;5;129;01min[39;00m [38;5;28mself[39m[38;5;241m.[39mmapping
[1;32m    543[0m [38;5;28;01mdef[39;00m [38;5;21m__getitem__[39m([38;5;28mself[39m, k):

[0;31mKeyboardInterrupt[0m:

import training dict of other samples

(replace sample name and repeat for multiple samples), if necessary check features for overlap

In [39]:

oldsample = '4'
oldgitsha = 'ec4415d'
if oldsample == '4':
    training_dict_old = pickle.load(open(os.path.join(toppathSSD, '05_water_GDL_ML', '4', 'ec4415d_training_dict_without_loc_feat.p'), 'rb'))
else:
    training_dict_old = pickle.load(open(os.path.join(training_path, oldsample,  oldgitsha+'_training_dict.p'),'rb'))
oldfeatures = pickle.load(open(os.path.join(training_path, oldsample,  oldgitsha+'_feature_names.p'),'rb'))
    
    # pickle.dump(TS.training_dict, open(os.path.join(TS.training_path, pytrain_git_sha+'_training_dict.p'),'wb'))
# pickle.dump(TS.feature_names, open(os.path.join(TS.training_path, pytrain_git_sha+'_feature_names.p'),'wb'))

for key in training_dict_old.keys():
    TS.training_dict[oldsample+key] = training_dict_old[key]

suggest a new training coordinate

currently retraining with new feature stack not properly implemented. Workaround: choose from the exiting training sets and train with them (additional labeling optional)

In [42]:

TS.suggest_training_set()

You could try  x = 55  and  z = 1058
However, please sort it like the original xyztimetime_0feature

In [70]:

c1 = 'y'
p1 = 240
c2 = 'time'
p2 = 14

In [ ]:

TS.load_training_set(c1, p1, c2, p2)

In [ ]:

im8 = TS.current_im8 #for canvas

In [73]:

feat_data = TS.feat_data
[c1,p1,c2,p2] = TS.current_coordinates
newslice = True

if c1 == 'x' and c2 == 'time':
    feat_stack = feat_data['feature_stack'].sel(x = p1, time = p2)
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(x = p1, time_0 = 0)
elif c1 == 'x' and c2 == 'y':
    feat_stack = feat_data['feature_stack'].sel(x = p1, y = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(x = p1, y = p2)
elif c1 == 'x' and c2 == 'z':
    feat_stack = feat_data['feature_stack'].sel(x = p1, z = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(x = p1, z = p2)
elif c1 == 'y' and c2 == 'z':
    feat_stack = feat_data['feature_stack'].sel(y = p1, z = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(y = p1, z = p2)
elif c1 == 'y' and c2 == 'time':
    feat_stack = feat_data['feature_stack'].sel(y = p1, time = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(y = p1, time_0 = 0)
elif c1 == 'z' and c2 == 'time':
    feat_stack = feat_data['feature_stack'].sel(z = p1, time = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(z = p1, time_0 = 0)

calculate the feature stack of the current slice for training

In [ ]:


if type(feat_stack) is not np.ndarray:
        fut = client.scatter(feat_stack)
        fut = fut.result()
        fut = fut.compute()
        feat_stack = fut
        client.restart()

if type(feat_stack_t_idp) is not np.ndarray:
        fut = client.scatter(feat_stack_t_idp)
        fut = fut.result()
        fut = fut.compute()
        feat_stack_t_idp = fut
        client.restart()
    

 
feat_stack = np.concatenate([feat_stack, feat_stack_t_idp], axis = 2)

In [ ]:

TS.current_feat_stack = feat_stack
if type(TS.current_feat_stack) is not np.ndarray:
    TS.current_computed = False
else:
    TS.current_computed = True

canvas for labeling

In [92]:

alpha = 0.15

# legacy function to zoom by selecting ROI
# better: use laptop trackpad (or sth similar) to zoom in webbrowser, not the CTRL+mousewheel zoom
zoom1 = (-500,-1)
zoom2 = (600,1400)

zoom1 = (0, -1)
zoom2 = (0, -1)

# im8 = TS.current_im8
#trick: use gaussian_time_4_0 to label static phases ()
# im8 = display_feature(-2, TS)
# im8 = display_feature(-20, TS)
# print(IF.combined_feature_names[-20]) #helps to find the correct image filter
print('original shape: ',im8.shape)
im8_display = im8.copy()[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]]
print('diyplay shape : ',im8_display.shape,' at: ', (zoom1[0], zoom2[0]))

resultim = TS.current_result.copy()

resultim_display = resultim[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]]


width = im8_display.shape[1]
height = im8_display.shape[0]
Mcanvas = MultiCanvas(4, width=width, height=height)
background = Mcanvas[0]
resultdisplay = Mcanvas[2]
truthdisplay = Mcanvas[1]
canvas = Mcanvas[3]
canvas.sync_image_data = True
drawing = False
position = None
shape = []
image_data = np.stack((im8_display, im8_display, im8_display), axis=2)
background.put_image_data(image_data, 0, 0)
slidealpha = IntSlider(description="Result overlay", value=0.15)
resultdisplay.global_alpha = alpha #slidealpha.value
if np.any(resultim>0):
    result_data = np.stack((255*(resultim_display==0), 255*(resultim_display==1), 255*(resultim_display==2)), axis=2)
else:
    result_data = np.stack((0*resultim, 0*resultim, 0*resultim), axis=2)
resultdisplay.put_image_data(result_data, 0, 0)
canvas.on_mouse_down(on_mouse_down)
canvas.on_mouse_move(on_mouse_move)
canvas.on_mouse_up(on_mouse_up)
picker = ColorPicker(description="Color:", value="#ff0000") #red
# picker = ColorPicker(description="Color:", value="#0000ff") #blue
# picker = ColorPicker(description="Color:", value="#00ff00") #green

link((picker, "value"), (canvas, "stroke_style"))
link((picker, "value"), (canvas, "fill_style"))
link((slidealpha, "value"), (resultdisplay, "global_alpha"))

HBox((Mcanvas,picker))
# HBox((Mcanvas,)) #picker

Out [92]:

original shape:  (750, 1916)
diyplay shape :  (749, 1915)  at:  (0, 0)

HBox(children=(MultiCanvas(height=749, width=1915), ColorPicker(value='#ff0000', description='Color:')))

In [81]:

# crude method to adjust brightness and contrast
tfs.plot_im_histogram(im8)
# im8 = TS.current_im8
# im8 = tfs.adjust_image_contrast(im8,20,200)

inspect labels and training progress

In [93]:

fig, axes = plt.subplots(1,6, figsize=(20,10))
axes[0].imshow(TS.current_result, 'gray')
axes[1].imshow(TS.current_im8, 'gray')

# TS.current_diff_im = TS.current_im-TS.current_first_im
# TS.current_diff_im = TS.current_diff_im/TS.current_diff_im.max()*255
axes[2].imshow(-TS.current_diff_im)#,vmin=6e4)
# axes[3].imshow(im8old, 'gray')
axes[3].imshow(TS.current_first_im, 'gray')
axes[4].imshow(TS.current_truth)
if TS.current_computed:
    axes[5].imshow(TS.current_feat_stack[:,:,-10])
else:
    axes[5].imshow(TS.current_result, 'gray')

for ax in axes:
    ax.set_xticks([])
    ax.set_yticks([])

update training set if labels are ok

In [84]:

label_set = canvas.get_image_data()

TS.current_truth[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]][label_set[:,:,0]>0] = 1
TS.current_truth[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]][label_set[:,:,1]>0] = 2
TS.current_truth[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]][label_set[:,:,2]>0] = 4

imageio.imsave(TS.current_truthpath, TS.current_truth)

train!

In [87]:

# TODO: see, if training gets slow for many label sets, currently stored in training_dict and read as loop. or if it is just the larger amount of data
TS.train_slice()

now actually calculating the features

2023-03-29 10:43:42,902 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-03-29 10:46:50,886 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-03-29 10:47:32,421 - distributed.worker.memory - WARNING - gc.collect() took 1.027s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-03-29 10:49:51,644 - distributed.worker.memory - WARNING - gc.collect() took 1.240s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-03-29 10:49:52,590 - distributed.worker.memory - WARNING - gc.collect() took 1.523s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-03-29 10:50:09,993 - distributed.worker.memory - WARNING - gc.collect() took 1.379s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-03-29 10:52:55,347 - tornado.application - ERROR - Uncaught exception GET /status/ws (::1)
HTTPServerRequest(protocol='http', host='127.0.0.1:35000', method='GET', uri='/status/ws', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/tornado/websocket.py", line 942, in _accept_connection
    open_result = handler.open(*handler.open_args, **handler.open_kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/tornado/web.py", line 3208, in wrapper
    return method(self, *args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/bokeh/server/views/ws.py", line 149, in open
    raise ProtocolError("Token is expired.")
bokeh.protocol.exceptions.ProtocolError: Token is expired.
ERROR:tornado.application:Uncaught exception GET /status/ws (::1)
HTTPServerRequest(protocol='http', host='127.0.0.1:35000', method='GET', uri='/status/ws', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/tornado/websocket.py", line 942, in _accept_connection
    open_result = handler.open(*handler.open_args, **handler.open_kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/tornado/web.py", line 3208, in wrapper
    return method(self, *args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/bokeh/server/views/ws.py", line 149, in open
    raise ProtocolError("Token is expired.")
bokeh.protocol.exceptions.ProtocolError: Token is expired.

feat_stack is not a numpy array! check why
training and classifying

In [421]:

# # load exisiting classifier and segment one slice
# clf = pickle.load(open(os.path.join(training_path, 'classifier.p'), 'rb'))
# feat = TS.current_feat_stack.compute()
# shp = feat[...,0].shape
# num_feat = feat.shape[-1]
# feat = feat.reshape(-1,num_feat)
# seg = clf.predict(feat)
# seg = seg.reshape(shp).astype(np.uint8)
# plt.figure(figsize=(16,9))
# plt.imshow(seg, cmap='gray_r')

In [422]:

# plt.figure(figsize=(16,9))
# plt.imshow(im8, cmap='gray')

go back until happy

check on training progress by plausible feature importance

In [94]:

plt.figure(figsize=(16,9))
plt.stem(TS.combined_feature_names, TS.clf.feature_importances_,'x')
plt.xticks(rotation=90)
plt.ylabel('importance') 
# plt.xticks(rotation = 60)

Out [94]:

Text(0, 0.5, 'importance')

when done, maybe save the classifier and optional the training dict (avoids recalculating the training sets, but might be large)

In [43]:

# TS.pickle_classifier()
pickle.dump(TS.training_dict, open(os.path.join(TS.training_path, pytrain_git_sha+'_training_dict.p'),'wb'))
pickle.dump(TS.combined_feature_names, open(os.path.join(TS.training_path, pytrain_git_sha+'_feature_names.p'),'wb'))

In [46]:

TS.training_dict_full

Out [46]:

{'label_image_y_255_time_40_.tif': (array([[-1.46428571e+02, -4.71285714e+02,  1.71676120e-01, ...,
           1.11674225e+04,  1.06050000e+04,  1.70411936e-01],
         [ 1.54714286e+02, -1.69714286e+02,  1.72399418e-01, ...,
           1.13587042e+04,  1.08410000e+04,  1.71066865e-01],
         [-1.65000000e+02, -3.60142857e+02,  1.71660520e-01, ...,
           1.10823803e+04,  1.03730000e+04,  1.70895247e-01],
         ...,
         [ 1.56428571e+02,  2.81285714e+02,  1.78094529e-01, ...,
           1.19940141e+04,  1.14850000e+04,  1.78078066e-01],
         [ 1.63285714e+02,  1.47000000e+02,  1.77728446e-01, ...,
           1.19303803e+04,  1.13000000e+04,  1.77728446e-01],
         [-2.08571429e+02, -3.50142857e+02,  1.76900536e-01, ...,
           1.15639014e+04,  1.10180000e+04,  1.76805548e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_z_307_time_25_.tif': (array([[-2.46142857e+02, -3.74142857e+02,  1.76275724e-01, ...,
           1.18232958e+04,  1.10610000e+04,  1.74925798e-01],
         [-3.42000000e+02, -4.85857143e+02,  1.76316241e-01, ...,
           1.18405352e+04,  1.11300000e+04,  1.75319188e-01],
         [-1.90428571e+02, -3.74857143e+02,  1.77512776e-01, ...,
           1.21114930e+04,  1.14260000e+04,  1.76005940e-01],
         ...,
         [-1.81714286e+02,  4.80285714e+02,  1.76016286e-01, ...,
           1.10960563e+04,  1.01840000e+04,  1.74601599e-01],
         [ 9.84285714e+01,  6.54285714e+02,  1.75308977e-01, ...,
           1.09472676e+04,  1.03020000e+04,  1.73986060e-01],
         [ 3.74714286e+02,  1.56428571e+02,  1.76564722e-01, ...,
           1.16763662e+04,  1.06990000e+04,  1.75170552e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_x_157_time_5_.tif': (array([[ 6.12714286e+02,  4.12000000e+02,  1.81691079e-01, ...,
           1.18927465e+04,  1.12010000e+04,  1.80019835e-01],
         [ 2.26857143e+02, -1.45142857e+02,  1.81563116e-01, ...,
           1.18323099e+04,  1.09740000e+04,  1.79961364e-01],
         [ 4.12428571e+02,  2.41142857e+02,  1.81434953e-01, ...,
           1.18325352e+04,  1.08200000e+04,  1.79912975e-01],
         ...,
         [ 4.61285714e+02,  1.17900000e+03,  1.82986851e-01, ...,
           1.16421831e+04,  1.06770000e+04,  1.76633232e-01],
         [ 2.95285714e+02,  1.14028571e+03,  1.82246264e-01, ...,
           1.15427042e+04,  1.07200000e+04,  1.76239893e-01],
         [ 2.36571429e+02,  1.05700000e+03,  1.81376676e-01, ...,
           1.14711549e+04,  1.06250000e+04,  1.76194815e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_y_71_time_27_.tif': (array([[-1.35857143e+02,  1.11428571e+01,  1.93519196e-01, ...,
           1.29737324e+04,  1.19270000e+04,  1.91659363e-01],
         [ 1.04714286e+02,  1.25000000e+02,  1.93022896e-01, ...,
           1.29130563e+04,  1.20690000e+04,  1.91258373e-01],
         [ 2.80000000e+01,  1.83000000e+02,  1.92580109e-01, ...,
           1.28927042e+04,  1.18410000e+04,  1.90972621e-01],
         ...,
         [ 1.77571429e+02,  9.50000000e+01,  1.82714666e-01, ...,
           1.17465211e+04,  1.13000000e+04,  1.81494558e-01],
         [ 1.76714286e+02,  2.65714286e+02,  1.83214647e-01, ...,
           1.17702676e+04,  1.11810000e+04,  1.81948427e-01],
         [ 9.18571429e+01,  2.36428571e+02,  1.83764860e-01, ...,
           1.17896901e+04,  1.12550000e+04,  1.82611815e-01]]),
  array([1., 1., 1., ..., 2., 2., 2.])),
 'label_image_x_270_time_50_.tif': (array([[ 7.62285714e+02,  3.66285714e+02,  1.77352465e-01, ...,
           1.17248873e+04,  1.08400000e+04,  1.76199144e-01],
         [ 5.79571429e+02,  1.56857143e+02,  1.77569550e-01, ...,
           1.21831408e+04,  1.15700000e+04,  1.76699324e-01],
         [ 1.67714286e+02, -1.69857143e+02,  1.77088113e-01, ...,
           1.21293662e+04,  1.14650000e+04,  1.76558669e-01],
         ...,
         [ 1.76571429e+02,  3.25571429e+02,  1.78726329e-01, ...,
           1.17209718e+04,  1.11370000e+04,  1.78671522e-01],
         [-1.60285714e+02,  5.28571429e+01,  1.78158173e-01, ...,
           1.14837465e+04,  1.09890000e+04,  1.78069821e-01],
         [-4.00000000e+02, -9.31428571e+01,  1.77880470e-01, ...,
           1.14104507e+04,  1.08650000e+04,  1.77633669e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_x_456_time_0_.tif': (array([[2.07142857e+02, 1.14642857e+03, 1.90178932e-01, ...,
          1.22672817e+04, 1.08070000e+04, 1.77169381e-01],
         [2.49714286e+02, 1.03714286e+03, 1.91092602e-01, ...,
          1.24148451e+04, 1.11670000e+04, 1.79968294e-01],
         [2.52714286e+02, 8.14428571e+02, 1.92194164e-01, ...,
          1.25856901e+04, 1.17740000e+04, 1.83731002e-01],
         ...,
         [2.74285714e+02, 1.28857143e+02, 1.75245020e-01, ...,
          1.16030423e+04, 1.10330000e+04, 1.74652647e-01],
         [1.13857143e+02, 2.27142857e+02, 1.75125835e-01, ...,
          1.12481831e+04, 1.06070000e+04, 1.74509806e-01],
         [3.50428571e+02, 4.15428571e+02, 1.75175942e-01, ...,
          1.12752817e+04, 1.07240000e+04, 1.74449912e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_z_320_time_10_.tif': (array([[8.92857143e+01, 1.87571429e+02, 1.82469480e-01, ...,
          1.22107042e+04, 1.11800000e+04, 1.81068155e-01],
         [6.86000000e+02, 4.32857143e+02, 1.80689966e-01, ...,
          1.20154648e+04, 1.09530000e+04, 1.79250545e-01],
         [3.05285714e+02, 1.21428571e+01, 1.79607390e-01, ...,
          1.19377746e+04, 1.04670000e+04, 1.77957822e-01],
         ...,
         [1.13571429e+02, 2.35428571e+02, 1.76514598e-01, ...,
          1.14960563e+04, 1.07940000e+04, 1.74584281e-01],
         [2.81000000e+02, 4.33000000e+02, 1.76417716e-01, ...,
          1.15155493e+04, 1.07560000e+04, 1.74400619e-01],
         [1.56428571e+02, 1.06857143e+02, 1.76227649e-01, ...,
          1.15197183e+04, 1.08520000e+04, 1.74042724e-01]]),
  array([0., 0., 0., ..., 1., 1., 1.])),
 'label_image_z_564_time_0_.tif': (array([[ 3.13285714e+02,  2.64285714e+02,  1.96006687e-01, ...,
           1.27615915e+04,  1.21920000e+04,  1.94477154e-01],
         [-8.00000000e+01, -1.59285714e+02,  1.95622355e-01, ...,
           1.27900704e+04,  1.22220000e+04,  1.94649750e-01],
         [-9.24285714e+01, -1.13000000e+02,  1.95298843e-01, ...,
           1.27974789e+04,  1.22600000e+04,  1.94626234e-01],
         ...,
         [-4.71428571e+00,  2.92857143e+02,  1.79363554e-01, ...,
           1.12649577e+04,  1.05480000e+04,  1.73660473e-01],
         [ 4.69857143e+02,  7.61000000e+02,  1.79049298e-01, ...,
           1.11457746e+04,  1.03290000e+04,  1.72011410e-01],
         [ 4.55142857e+02,  7.02285714e+02,  1.78584209e-01, ...,
           1.10580845e+04,  1.04030000e+04,  1.70629070e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_z_320_time_30_.tif': (array([[ 2.14857143e+02,  1.52285714e+02,  2.01065715e-01, ...,
           1.33168028e+04,  1.19130000e+04,  1.96774246e-01],
         [-3.23714286e+02,  1.71428571e+00,  2.01161255e-01, ...,
           1.33518310e+04,  1.18630000e+04,  1.96969844e-01],
         [-3.62714286e+02, -9.71428571e+00,  2.01125502e-01, ...,
           1.33800563e+04,  1.20740000e+04,  1.96864119e-01],
         ...,
         [-1.48485714e+03, -6.54714286e+02,  1.77340966e-01, ...,
           1.15403099e+04,  1.06520000e+04,  1.76559372e-01],
         [-1.58242857e+03, -6.48857143e+02,  1.79606184e-01, ...,
           1.18334648e+04,  1.09120000e+04,  1.78930439e-01],
         [-7.85000000e+02,  1.34000000e+02,  1.82066069e-01, ...,
           1.23317606e+04,  1.13080000e+04,  1.81227066e-01]]),
  array([0., 0., 0., ..., 1., 1., 1.])),
 'label_image_x_264_time_29_.tif': (array([[-8.55714286e+01,  4.90000000e+01,  1.95446785e-01, ...,
           1.27986338e+04,  1.23040000e+04,  1.93879462e-01],
         [ 1.22857143e+02,  2.16428571e+02,  1.95580021e-01, ...,
           1.27709296e+04,  1.20220000e+04,  1.93749050e-01],
         [ 1.40000000e+01,  2.14285714e+01,  1.95790805e-01, ...,
           1.28141268e+04,  1.21830000e+04,  1.93688263e-01],
         ...,
         [-4.59857143e+02, -3.76000000e+02,  1.76360268e-01, ...,
           1.13471831e+04,  1.02530000e+04,  1.70212563e-01],
         [-2.67285714e+02, -5.01000000e+02,  1.76233976e-01, ...,
           1.13495915e+04,  1.02230000e+04,  1.70287334e-01],
         [ 9.31428571e+01, -1.57714286e+02,  1.75952877e-01, ...,
           1.13085915e+04,  1.02450000e+04,  1.70298715e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_y_280_time_57_.tif': (array([[-8.58571429e+01,  7.68571429e+01,  1.82389721e-01, ...,
           1.20554085e+04,  1.14720000e+04,  1.81354376e-01],
         [-2.18000000e+02,  1.72000000e+02,  1.82640049e-01, ...,
           1.22886620e+04,  1.16720000e+04,  1.81573052e-01],
         [ 3.24000000e+02,  3.61428571e+02,  1.82303870e-01, ...,
           1.19934366e+04,  1.13380000e+04,  1.81405977e-01],
         ...,
         [ 9.55714286e+01,  2.29714286e+02,  1.71066023e-01, ...,
           1.12507606e+04,  9.87400000e+03,  1.69633661e-01],
         [ 1.79000000e+02,  3.07428571e+02,  1.71033091e-01, ...,
           1.12309014e+04,  1.01770000e+04,  1.69707987e-01],
         [ 2.13714286e+02,  1.01428571e+01,  1.70781424e-01, ...,
           1.11918310e+04,  9.93100000e+03,  1.69562468e-01]]),
  array([0., 0., 0., ..., 1., 1., 1.])),
 'label_image_y_250_time_23_.tif': (array([[ 6.29571429e+02,  2.80571429e+02,  1.83566668e-01, ...,
           1.19565634e+04,  1.11340000e+04,  1.81957534e-01],
         [ 2.94000000e+02,  9.17142857e+01,  1.83483545e-01, ...,
           1.19102113e+04,  1.11850000e+04,  1.81933184e-01],
         [-2.17428571e+02, -4.80142857e+02,  1.83503721e-01, ...,
           1.19503380e+04,  1.11650000e+04,  1.81941278e-01],
         ...,
         [-5.99142857e+02, -3.23000000e+02,  1.75209452e-01, ...,
           1.13515493e+04,  1.06300000e+04,  1.73828147e-01],
         [-2.53428571e+02,  3.61428571e+01,  1.76540683e-01, ...,
           1.16698169e+04,  1.10110000e+04,  1.76169142e-01],
         [ 3.13714286e+02,  4.45285714e+02,  1.77460691e-01, ...,
           1.18957606e+04,  1.12630000e+04,  1.76957676e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.]))}

In [103]:

TS.training_path

Out [103]:

'/mpc/homes/fische_r/NAS/DASCOELY/processing/05_water_GDL_ML/1'

In [50]:

pytrain_git_sha

Out [50]:

'ec4415d'

Segmentation of full data set

segmentation has not yet be checked for functionality after dask update! probably needs major restructuring but should work eventually

In [37]:

from segmentation import segmentation

In [38]:

classifier_path=os.path.join(training_path, 'classifier.p')
SM = segmentation(training_path = training_path, classifier_path=classifier_path)

In [39]:

# SM.import_lazy_feature_data(IF.result)
# SM.import_classifier(TS.clf)
SM.clf = pickle.load(open(os.path.join(training_path, 'classifier.p'), 'rb'))

In [40]:

clf = SM.clf
clf.n_jobs = 64

if host == 'mpc2053.psi.ch':
    clf.n_jobs = 20

In [41]:

#TODO create result as a stream for every feature set of chunks, i.e stack of 67 feature chunks
# clf = TS.clf
# clf.n_jobs = 32

In [42]:

# loc_feats = [-4, -5, -6]
# ids = np.ones(72, dtype=bool)
# for f in loc_feats:
#     ids[f] = False

In [43]:

# ids

merge time-independent features

In [44]:

test = dask.array.stack([TS.feat_data['feature_stack_time_independent'][:,:,:,0,:]]*da.shape[-1], axis=-2)

2023-04-28 08:57:28,625 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-04-28 08:57:33,885 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

In [45]:

test

Out [45]:

	Array	Chunk
Bytes	543.37 GiB	256.00 kiB
Shape	(750, 130, 1700, 88, 5)	(32, 32, 32, 1, 1)
Dask graph	4049760 chunks in 3 graph layers
Data type	float64 numpy.ndarray

In [46]:

feat = dask.array.concatenate([TS.feat_data['feature_stack'], test], axis=-1)

2023-04-28 08:57:55,416 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 08:58:41,121 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 08:59:32,401 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:00:31,800 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:01:44,643 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:03:17,745 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

In [47]:

# feat = feat[...,ids]

In [48]:

feat

Out [48]:

	Array	Chunk
Bytes	7.32 TiB	256.00 kiB
Shape	(750, 130, 1700, 88, 69)	(32, 32, 32, 1, 1)
Dask graph	59019840 chunks in 450 graph layers
Data type	float64 numpy.ndarray

check aligment of chunks and select suitable "super-chunk" shape to split calculation in parts, example below

In [49]:

750*2.3

Out [49]:

1724.9999999999998

In [50]:

350/50

Out [50]:

7.0

In [51]:

i = 12
j = i
# j = 3
dim1 = 64#better use multiple of chunk size !?
dim2 = int(2.3*dim1)
feat[i*dim1:(i+1)*dim1,:,j*dim2:(j+1)*dim2,:,:] #select all features and all time steps, but you are free in space; als large as possible and as small as necessary. limit is the available RAM to collect the dask result

Out [51]:

	Array	Chunk
Bytes	0 B	0 B
Shape	(0, 130, 0, 88, 69)	(0, 32, 0, 1, 1)
Dask graph	36432 chunks in 451 graph layers
Data type	float64 numpy.ndarray

In [52]:

# segs = np.zeros(feat.shape[:4], dtype=np.uint8)
segs = pickle.load(open(os.path.join(training_path,'segs.p'), 'rb'))

2023-04-28 09:06:48,538 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:07:43,656 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-04-28 09:07:45,739 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-04-28 09:07:48,023 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-04-28 09:07:50,311 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-04-28 09:07:52,682 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-04-28 09:07:55,129 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-04-28 09:07:57,693 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-04-28 09:08:00,064 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-04-28 09:08:02,809 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)

In [53]:

# import gc
client.run(gc.collect)
gc.collect()

Out [53]:

2023-04-28 09:08:05,439 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-04-28 09:08:25,753 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)

In [54]:

import ctypes
def trim_memory() -> int:
    libc = ctypes.CDLL("libc.so.6")
    return libc.malloc_trim(0)

In [55]:

limit = 12
# TODO: Workaround: write intermediates to disk and restart dask client/scheduler/workers to get rid of unmanaged memory
# possible solution: do data handling on mpc2053 and calculations on mpc2959 -> leaves some RAM --> gc.collect() not necessary anymore
# appearantly starts to become critical after 12 iteratons
# from i=7, j=10 already done
# TODO write i and j to file to track progress even when closing jupyter
for i in range(1,limit):
    pickle.dump(segs, open(os.path.join(training_path,'segs_temp.p'), 'wb'))
    
    print(str(i+1)+'/'+str(limit))
    # plt.figure()
    # plt.imshow(segs[:,100,:, 50])
    # plt.savefig(os.path.join(training_path, sample+'_'+str(i)+'_progress.png'))
    # client.run(gc.collect)
    # client.run(trim_memory)
    start = 0
    if i == 1:
        start = 8
    for j in range(start,limit):
        gc.collect() 
        client.run(gc.collect)
        client.run(trim_memory)
        print(j)
        #with joblib.parallel_backend('dask'):
        part = feat[i*dim1:(i+1)*dim1,:,j*dim2:(j+1)*dim2,:,:] #.persist() #compute() may blow up the memory ?! https://stackoverflow.com/questions/73770527/dask-compute-uses-twice-the-expected-memory
        if 0 in part.shape:
            print('hit the edge (one dimension 0), ignore')
            continue
        part = part.compute()
        # try to release old unmanaged memory
        client.run(gc.collect)
        client.run(trim_memory)

        shp = part.shape
        num_feat = part.shape[-1]  
        part = part.reshape(-1,num_feat)

        psplit = int(part.shape[0]/2)

        print('create part 1')
        part1 = part[:psplit,:]
        print('create part 2')
        part2 = part[psplit:,:]
        print('segmenting 1')

        # with joblib.parallel_backend('dask'):
        seg1 = clf.predict(part1).astype(np.uint8)
        del part1
        print('segmenting 2')
        # with joblib.parallel_backend('dask'):
        seg2 = clf.predict(part2).astype(np.uint8)
        print('wrap results')
        del part2
        del part
        # gc.collect()

        seg = np.concatenate([seg1,seg2])
        print(seg.shape)
        del seg1
        del seg2

        seg = seg.reshape(shp[:4])  #this step needs a lot of RAM ?! appearantly not
    
    # not sure if this switch cases are necessary
        if i < limit-1 and j < limit-1:
            segs[i*dim1:(i+1)*dim1,:,j*dim2:(j+1)*dim2,:] = seg
        elif not i < limit-1 and j < limit-1:
            segs[i*dim1:,:,j*dim2:(j+1)*dim2,:] =  seg
        elif not j < limit-1 and i < limit-1:
            segs[i*dim1:(i+1)*dim1,:,j*dim2:,:] =  seg
        else:
            segs[i*dim1:,:,j*dim2:,:] = seg
            
        del seg

2023-04-28 09:08:27,911 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-04-28 09:08:32,501 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 09:08:36,089 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 09:08:39,271 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 09:08:43,344 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 09:08:47,329 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 09:08:51,357 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 09:08:56,011 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 09:09:01,183 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 09:09:06,433 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)

1/12

2023-04-28 09:09:25,644 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:09:26,978 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 09:11:01,156 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:12:35,274 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:12:41,381 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:13:15,229 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:13:54,218 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:14:01,552 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:14:09,541 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:14:17,976 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:14:26,864 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:14:35,784 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:14:45,427 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:14:55,782 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:15:07,106 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:15:18,367 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:15:30,881 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:15:43,739 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:15:57,649 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:16:13,692 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:16:30,197 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:16:45,526 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:17:02,614 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:17:21,621 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:17:43,388 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:18:04,014 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:18:26,656 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:18:54,293 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:19:22,554 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:19:50,806 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:20:17,625 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:20:44,917 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:21:15,715 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:21:50,743 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:22:29,074 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:23:10,350 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:23:49,683 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:24:34,795 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:25:14,922 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:25:59,410 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:26:56,132 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:30:19,717 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 09:33:50,338 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:33:54,598 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 09:37:46,191 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 09:38:27,164 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:38:54,606 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:39:25,908 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:39:58,196 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:40:33,729 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:41:10,943 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:41:50,088 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:42:30,832 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:43:14,191 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:43:58,595 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:44:43,626 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:46:00,337 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 09:48:51,714 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 09:52:24,671 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:52:29,122 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 09:56:18,982 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 09:57:00,346 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:57:28,002 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:58:00,087 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:58:33,375 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:59:09,327 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 09:59:45,641 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:00:25,569 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:01:05,448 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:01:49,007 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:02:33,796 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:03:20,597 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:05:16,109 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:07:28,457 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 10:11:00,336 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:11:04,810 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 10:14:56,290 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:15:38,548 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:16:05,579 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:16:36,660 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:17:09,639 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:17:45,001 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:18:22,414 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:19:03,250 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:19:43,380 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:20:26,470 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:21:12,471 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:22:00,256 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:23:44,218 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:26:08,975 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 10:29:48,016 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:29:52,217 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)

2023-04-28 10:33:41,743 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:34:22,697 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 10:34:51,289 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 10:35:19,666 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 10:35:53,907 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:36:29,200 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:37:07,892 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:37:46,968 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:38:26,660 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:39:10,050 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:39:54,893 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:40:42,002 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:42:35,835 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:44:41,386 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 10:48:11,129 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:48:15,871 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 10:52:05,326 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:52:47,845 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:53:15,818 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:53:50,242 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:54:24,344 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:54:59,402 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:55:37,198 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:56:17,524 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 10:56:59,300 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:57:42,657 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:58:27,726 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 10:59:14,335 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:00:35,482 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:03:26,363 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 11:07:03,483 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:07:07,776 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 11:10:58,040 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:11:39,632 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:12:06,874 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:12:39,453 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:13:13,249 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:13:49,158 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:14:27,064 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:15:07,297 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:15:48,265 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:16:31,585 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:17:15,741 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:18:03,977 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:19:36,288 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:22:16,177 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 11:25:48,682 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 11:25:53,441 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 11:29:44,829 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 11:30:26,363 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:30:53,563 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:31:26,302 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:31:59,741 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 11:32:35,852 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:33:15,321 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:33:55,385 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:34:37,626 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:35:21,658 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:36:07,559 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:36:54,741 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:38:49,685 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:41:03,867 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 11:44:40,915 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 11:44:45,410 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 11:48:37,955 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 11:49:19,274 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:49:46,124 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:50:18,363 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 11:50:51,032 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:51:26,612 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:52:04,024 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:52:43,596 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:53:25,026 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:54:08,284 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:54:53,228 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:55:41,312 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:56:49,790 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 11:59:53,855 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 12:03:31,389 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 12:03:36,150 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 12:07:28,107 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 12:08:10,790 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 12:08:39,711 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 12:09:09,877 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:09:45,295 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:10:25,262 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:11:02,139 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:11:40,774 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:12:21,747 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:13:04,126 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:13:50,461 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:14:38,215 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:16:32,854 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:18:36,025 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 12:22:03,472 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 12:22:08,496 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

2023-04-28 12:26:01,489 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 12:26:46,874 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:27:15,977 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:27:48,595 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:28:21,703 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:28:58,852 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:29:36,198 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:30:15,999 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:30:58,120 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:31:41,151 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 12:32:27,322 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:33:16,405 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:34:23,925 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:37:30,309 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 12:41:10,472 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 12:41:14,911 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 12:45:03,101 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 12:45:44,089 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 12:46:14,342 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 12:46:45,671 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:47:19,015 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:47:52,533 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:48:30,022 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:49:09,080 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:49:52,034 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 12:50:35,136 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:07:20,522 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 13:08:07,391 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 13:10:50,284 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-04-28 13:11:33,187 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 13:12:21,331 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 13:15:58,072 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 13:16:02,683 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 13:19:59,444 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 13:20:41,630 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 13:21:10,911 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 13:21:39,342 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 13:22:14,396 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 13:22:51,258 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:23:30,507 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 13:24:10,381 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:24:52,140 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:25:37,076 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:26:24,691 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:27:12,810 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:28:02,144 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:31:36,336 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 13:35:13,229 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 13:35:17,909 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

2023-04-28 13:39:14,703 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 13:39:57,621 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 13:40:26,990 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 13:40:56,744 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:41:34,049 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:42:11,988 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:42:50,804 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:43:31,230 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:44:14,245 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:44:57,120 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:45:42,661 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:46:31,464 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:47:29,831 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:50:53,316 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 13:54:35,278 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 13:54:40,081 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

2023-04-28 13:58:34,454 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 13:59:17,889 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 13:59:47,860 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:00:18,009 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:00:54,121 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:01:31,134 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:02:09,636 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:02:49,887 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:03:30,509 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:04:14,728 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 14:05:04,603 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 14:05:53,088 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 14:06:43,267 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 14:10:13,672 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 14:13:35,523 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 14:13:40,258 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)

2023-04-28 14:17:35,143 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 14:18:17,955 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 14:18:46,786 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 14:19:15,680 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-04-28 14:19:51,514 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:20:25,755 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:21:03,917 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:21:43,329 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:22:24,718 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:23:08,127 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:23:54,127 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:24:42,352 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:25:40,825 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:29:02,860 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 14:32:29,962 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 14:32:34,324 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

2023-04-28 14:36:30,886 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 14:37:13,969 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:37:41,245 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:38:13,130 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:38:46,967 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:39:23,051 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:40:01,342 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:40:40,535 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 14:41:20,840 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:42:04,244 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:42:50,823 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:43:37,942 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:44:27,596 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:47:53,847 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 14:51:25,857 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 14:51:30,790 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 14:55:29,831 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 14:56:13,924 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 14:56:43,053 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:57:12,522 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:57:46,826 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:58:22,673 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:59:01,253 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 14:59:40,928 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:00:20,946 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:01:05,100 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:01:49,891 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:02:39,776 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:03:32,132 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:07:10,902 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 15:10:36,955 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 15:10:41,352 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

2023-04-28 15:14:34,270 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 15:15:15,894 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 15:15:44,486 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:16:13,034 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:16:49,559 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:17:24,824 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:18:03,753 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:18:43,737 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:19:24,557 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:20:08,061 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 15:20:53,711 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 15:21:41,409 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 15:22:41,104 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-04-28 15:26:02,808 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

create part 1
create part 2
segmenting 1
segmenting 2
wrap results
(107627520,)

2023-04-28 15:29:26,309 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-04-28 15:29:30,723 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)

2023-04-28 15:33:25,978 - distributed.core - ERROR - Timed out during handshake while connecting to tcp://127.0.0.1:36551 after 30 s
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/base_events.py", line 1868, in _run_once
    event_list = self._selector.select(timeout)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/selectors.py", line 469, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/core.py", line 328, in connect
    handshake = await asyncio.wait_for(comm.read(), time_left())
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last)2023-04-28 15:33:28,390 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
:
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
    return await func(*args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 1566, in close
    await r.close_gracefully(reason=reason)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1224, in send_recv_from_rpc
    comm = await self.pool.connect(self.addr)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1468, in connect
    return await connect_attempt
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1389, in _connect
    comm = await connect(
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/core.py", line 333, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://127.0.0.1:36551 after 30 s

[0;31m---------------------------------------------------------------------------[0m
[0;31mKeyboardInterrupt[0m                         Traceback (most recent call last)
Cell [0;32mIn[55], line 29[0m
[1;32m     27[0m     [38;5;28mprint[39m([38;5;124m'[39m[38;5;124mhit the edge (one dimension 0), ignore[39m[38;5;124m'[39m)
[1;32m     28[0m     [38;5;28;01mcontinue[39;00m
[0;32m---> 29[0m part [38;5;241m=[39m [43mpart[49m[38;5;241;43m.[39;49m[43mcompute[49m[43m([49m[43m)[49m
[1;32m     30[0m [38;5;66;03m# try to release old unmanaged memory[39;00m
[1;32m     31[0m client[38;5;241m.[39mrun(gc[38;5;241m.[39mcollect)

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/base.py:314[0m, in [0;36mDaskMethodsMixin.compute[0;34m(self, **kwargs)[0m
[1;32m    290[0m [38;5;28;01mdef[39;00m [38;5;21mcompute[39m([38;5;28mself[39m, [38;5;241m*[39m[38;5;241m*[39mkwargs):
[1;32m    291[0m [38;5;250m    [39m[38;5;124;03m"""Compute this dask collection[39;00m
[1;32m    292[0m 
[1;32m    293[0m [38;5;124;03m    This turns a lazy Dask collection into its in-memory equivalent.[39;00m
[0;32m   (...)[0m
[1;32m    312[0m [38;5;124;03m    dask.base.compute[39;00m
[1;32m    313[0m [38;5;124;03m    """[39;00m
[0;32m--> 314[0m     (result,) [38;5;241m=[39m [43mcompute[49m[43m([49m[38;5;28;43mself[39;49m[43m,[49m[43m [49m[43mtraverse[49m[38;5;241;43m=[39;49m[38;5;28;43;01mFalse[39;49;00m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m[43m)[49m
[1;32m    315[0m     [38;5;28;01mreturn[39;00m result

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/base.py:599[0m, in [0;36mcompute[0;34m(traverse, optimize_graph, scheduler, get, *args, **kwargs)[0m
[1;32m    596[0m     keys[38;5;241m.[39mappend(x[38;5;241m.[39m__dask_keys__())
[1;32m    597[0m     postcomputes[38;5;241m.[39mappend(x[38;5;241m.[39m__dask_postcompute__())
[0;32m--> 599[0m results [38;5;241m=[39m [43mschedule[49m[43m([49m[43mdsk[49m[43m,[49m[43m [49m[43mkeys[49m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m[43m)[49m
[1;32m    600[0m [38;5;28;01mreturn[39;00m repack([f(r, [38;5;241m*[39ma) [38;5;28;01mfor[39;00m r, (f, a) [38;5;129;01min[39;00m [38;5;28mzip[39m(results, postcomputes)])

File [0;32m~/miniconda3/lib/python3.10/site-packages/distributed/client.py:3136[0m, in [0;36mClient.get[0;34m(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)[0m
[1;32m   3134[0m         should_rejoin [38;5;241m=[39m [38;5;28;01mFalse[39;00m
[1;32m   3135[0m [38;5;28;01mtry[39;00m:
[0;32m-> 3136[0m     results [38;5;241m=[39m [38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43mgather[49m[43m([49m[43mpacked[49m[43m,[49m[43m [49m[43masynchronous[49m[38;5;241;43m=[39;49m[43masynchronous[49m[43m,[49m[43m [49m[43mdirect[49m[38;5;241;43m=[39;49m[43mdirect[49m[43m)[49m
[1;32m   3137[0m [38;5;28;01mfinally[39;00m:
[1;32m   3138[0m     [38;5;28;01mfor[39;00m f [38;5;129;01min[39;00m futures[38;5;241m.[39mvalues():

File [0;32m~/miniconda3/lib/python3.10/site-packages/distributed/client.py:2305[0m, in [0;36mClient.gather[0;34m(self, futures, errors, direct, asynchronous)[0m
[1;32m   2303[0m [38;5;28;01melse[39;00m:
[1;32m   2304[0m     local_worker [38;5;241m=[39m [38;5;28;01mNone[39;00m
[0;32m-> 2305[0m [38;5;28;01mreturn[39;00m [38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43msync[49m[43m([49m
[1;32m   2306[0m [43m    [49m[38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43m_gather[49m[43m,[49m
[1;32m   2307[0m [43m    [49m[43mfutures[49m[43m,[49m
[1;32m   2308[0m [43m    [49m[43merrors[49m[38;5;241;43m=[39;49m[43merrors[49m[43m,[49m
[1;32m   2309[0m [43m    [49m[43mdirect[49m[38;5;241;43m=[39;49m[43mdirect[49m[43m,[49m
[1;32m   2310[0m [43m    [49m[43mlocal_worker[49m[38;5;241;43m=[39;49m[43mlocal_worker[49m[43m,[49m
[1;32m   2311[0m [43m    [49m[43masynchronous[49m[38;5;241;43m=[39;49m[43masynchronous[49m[43m,[49m
[1;32m   2312[0m [43m[49m[43m)[49m

File [0;32m~/miniconda3/lib/python3.10/site-packages/distributed/utils.py:338[0m, in [0;36mSyncMethodMixin.sync[0;34m(self, func, asynchronous, callback_timeout, *args, **kwargs)[0m
[1;32m    336[0m     [38;5;28;01mreturn[39;00m future
[1;32m    337[0m [38;5;28;01melse[39;00m:
[0;32m--> 338[0m     [38;5;28;01mreturn[39;00m [43msync[49m[43m([49m
[1;32m    339[0m [43m        [49m[38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43mloop[49m[43m,[49m[43m [49m[43mfunc[49m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[43margs[49m[43m,[49m[43m [49m[43mcallback_timeout[49m[38;5;241;43m=[39;49m[43mcallback_timeout[49m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m
[1;32m    340[0m [43m    [49m[43m)[49m

File [0;32m~/miniconda3/lib/python3.10/site-packages/distributed/utils.py:401[0m, in [0;36msync[0;34m(loop, func, callback_timeout, *args, **kwargs)[0m
[1;32m    399[0m [38;5;28;01melse[39;00m:
[1;32m    400[0m     [38;5;28;01mwhile[39;00m [38;5;129;01mnot[39;00m e[38;5;241m.[39mis_set():
[0;32m--> 401[0m         [43mwait[49m[43m([49m[38;5;241;43m10[39;49m[43m)[49m
[1;32m    403[0m [38;5;28;01mif[39;00m error:
[1;32m    404[0m     typ, exc, tb [38;5;241m=[39m error

File [0;32m~/miniconda3/lib/python3.10/site-packages/distributed/utils.py:390[0m, in [0;36msync.<locals>.wait[0;34m(timeout)[0m
[1;32m    388[0m [38;5;28;01mdef[39;00m [38;5;21mwait[39m(timeout):
[1;32m    389[0m     [38;5;28;01mtry[39;00m:
[0;32m--> 390[0m         [38;5;28;01mreturn[39;00m [43me[49m[38;5;241;43m.[39;49m[43mwait[49m[43m([49m[43mtimeout[49m[43m)[49m
[1;32m    391[0m     [38;5;28;01mexcept[39;00m [38;5;167;01mKeyboardInterrupt[39;00m:
[1;32m    392[0m         loop[38;5;241m.[39madd_callback(cancel)

File [0;32m~/miniconda3/lib/python3.10/threading.py:607[0m, in [0;36mEvent.wait[0;34m(self, timeout)[0m
[1;32m    605[0m signaled [38;5;241m=[39m [38;5;28mself[39m[38;5;241m.[39m_flag
[1;32m    606[0m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m signaled:
[0;32m--> 607[0m     signaled [38;5;241m=[39m [38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43m_cond[49m[38;5;241;43m.[39;49m[43mwait[49m[43m([49m[43mtimeout[49m[43m)[49m
[1;32m    608[0m [38;5;28;01mreturn[39;00m signaled

File [0;32m~/miniconda3/lib/python3.10/threading.py:324[0m, in [0;36mCondition.wait[0;34m(self, timeout)[0m
[1;32m    322[0m [38;5;28;01melse[39;00m:
[1;32m    323[0m     [38;5;28;01mif[39;00m timeout [38;5;241m>[39m [38;5;241m0[39m:
[0;32m--> 324[0m         gotit [38;5;241m=[39m [43mwaiter[49m[38;5;241;43m.[39;49m[43macquire[49m[43m([49m[38;5;28;43;01mTrue[39;49;00m[43m,[49m[43m [49m[43mtimeout[49m[43m)[49m
[1;32m    325[0m     [38;5;28;01melse[39;00m:
[1;32m    326[0m         gotit [38;5;241m=[39m waiter[38;5;241m.[39macquire([38;5;28;01mFalse[39;00m)

[0;31mKeyboardInterrupt[0m:

In [58]:

Out [58]:

In [59]:

Out [59]:

In [57]:

plt.imshow(segs[:,50,:, 20])

Out [57]:

<matplotlib.image.AxesImage at 0x7f61a8872200>

In [56]:

pickle.dump(segs, open(os.path.join(training_path,'segs.p'), 'wb'))

2023-04-28 15:34:27,314 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing

Warning:

Output truncated. This notebook contains too many cells to display efficiently.

1.2 MiB Raw Blame History

Segment water in GDL to quantify influence of liquid water on electrochemistry

functionalities for interactive training

fire up dask

Test ROI selection

Data preparation

create dask array

define border to crop to GDL

let dask load the data

get data into image filter class

prepare features

Training

set up objects

interactive training

check for existing training sets

re-train with existing label sets. clear the training dictionary if necessary (training_dict)

import training dict of other samples

suggest a new training coordinate

calculate the feature stack of the current slice for training

canvas for labeling

inspect labels and training progress

update training set if labels are ok

train!

go back until happy

check on training progress by plausible feature importance

when done, maybe save the classifier and optional the training dict (avoids recalculating the training sets, but might be large)

Segmentation of full data set

merge time-independent features

check aligment of chunks and select suitable "super-chunk" shape to split calculation in parts, example below

1.2 MiB

Raw Blame History