LEC-5422-Public/pyTrainSeg

Fork 0

Files

T

fische_r a78c9e802d created readme

2023-06-05 16:23:14 +02:00

1.2 MiB

Raw Blame History

Segment water in GDL to quantify influence of liquid water on electrochemistry

TODO:

select only the data containing the GDL and channels
set up ML and train on one sample
apply classfier to test sample and segment all
- align samples with new cropping --> can be independent of membrane segmentation
allow (de-)selection of features during training
allow to add features after most others have been already calculated

In [1]:

# modules
import os
import xarray as xr
import matplotlib.pyplot as plt
import numpy as np
import dask
import dask.array
from scipy import ndimage
from skimage import filters, feature, io
from skimage.morphology import disk,ball
import sys
from itertools import combinations_with_replacement
import pickle
import imageio
import json
from dask.distributed import Client, LocalCluster
import socket
import subprocess
import gc

# import joblib
# 4 year old tutorial https://www.youtube.com/watch?v=5Zf6DQaf7jk&t=88s does this instead of importing joblib: 
# import dask_ml.joblib
# from sklearn.externals import joblib


from dask import config as cfg
cfg.set({'distributed.scheduler.worker-ttl': None, # Workaround so that dask does not kill workers while they are busy fetching data: https://dask.discourse.group/t/dask-workers-killed-because-of-heartbeat-fail/856, maybe this helps: https://www.youtube.com/watch?v=vF2VItVU5zg?
        'distributed.scheduler.transition-log-length': 100, #potential workaround for ballooning scheduler memory https://baumgartner.io/posts/how-to-reduce-memory-usage-of-dask-scheduler/
         'distributed.scheduler.events-log-length': 100
        })

# get the ML functions, TODO: make a library once it works/is in a stable state
pytrainpath = '/mpc/homes/fische_r/lib/pytrainseg'
cwd = os.getcwd()
os.chdir(pytrainpath)
from filter_functions import image_filter
import training_functions as tfs
from training_functions import train_segmentation
from segmentation import segmentation
pytrain_git_sha = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip()
os.chdir(cwd)

#paths
host = socket.gethostname()
if host == 'mpc2959.psi.ch':
    gitpath = '/mpc/homes/fische_r/lib/co2ely-tomcat'
    toppath = '/mpc/homes/fische_r/NAS/DASCOELY'
    toppathSSD = '/mnt/SSD/fische_r/COELY'
    temppath = '/mnt/SSD/fische_r/tmp'
    temppath_2 = '/mpc/homes/fische_r/NAS/tmp'
    training_path = '/mpc/homes/fische_r/NAS/DASCOELY/processing/05_water_GDL_ML/'
    memlim = '700GB'
elif host == 'mpc2053.psi.ch':
    gitpath = '/mpc/homes/fische_r/lib/co2ely-tomcat'
    toppath = '/mpc/homes/fische_r/NAS/DASCOELY'
    toppathSSD = os.path.join(toppath, 'processing')
    temppath = '/mnt/SSD_2TB_nvme0n1/Robert/tmp/'
    temppath_2 = '/mpc/homes/fische_r/NAS/tmp'
    training_path = '/mpc/homes/fische_r/NAS/DASCOELY/processing/05_water_GDL_ML/'
    memlim = '360GB'
else:
    print('host '+host+' currently not supported')
    
path_02_4D = os.path.join(toppathSSD, '02_registered_3p1D') #h5 with registered data

# fetch githash
cwd = os.getcwd()
os.chdir(gitpath)
git_sha = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip()
githash = subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip()
os.chdir(cwd)

functionalities for interactive training

In [2]:

from ipywidgets import Image
from ipywidgets import ColorPicker, IntSlider, link, AppLayout, HBox
from ipycanvas import  hold_canvas,  MultiCanvas #RoughCanvas,Canvas,

def on_mouse_down(x, y):
    global drawing
    global position
    global shape
    drawing = True
    position = (x, y)
    shape = [position]

def on_mouse_move(x, y):
    global drawing
    global position
    global shape
    if not drawing:
        return
    with hold_canvas():
        canvas.stroke_line(position[0], position[1], x, y)
        position = (x, y)
    shape.append(position)

def on_mouse_up(x, y):
    global drawing
    global positiondu
    global shape
    drawing = False
    with hold_canvas():
        canvas.stroke_line(position[0], position[1], x, y)
        canvas.fill_polygon(shape)
    shape = []
    
def display_feature(i, TS):
    print('selected '+TS.feature_names[i])
    im = TS.current_feat_stack[:,:,i]
    im8 = im-im.min()
    im8 = im8/im8.max()*255
    return im8

fire up dask

In [3]:

tempfolder = temppath  #a big SSD is a major adavantage to allow spill to disk and still be efficient. large dataset might crash with too small SSD or be slow with normal HDD
# tempfolder = temppath_2
dask.config.config['temporary-directory'] = tempfolder
# dask.config.config['distributed']['worker']['memory']['recent-to-old-time'] = '200000s'

# here you have the option to use a virtual cluster or even slurm on ra (not attempted yet)
cluster = LocalCluster(dashboard_address=':35000', memory_limit = memlim, n_workers=1) #settings optimised for mpc2959, play around if needed, if you know nothing else is using RAM then you can almost go to the limit
# maybe less workers with more threads makes better use of shared memory 

# scheduler_port = 'tcp://129.129.188.222:8786' #<-- if scheduler on mpc2959; scheduler on mpc2053 -> 'tcp://129.129.188.248:8786'
# cluster = scheduler_port

client = Client(cluster)
client.amm.start()
print('Dashboard at '+client.dashboard_link)

2023-05-16 09:28:37,202 - distributed.diskutils - INFO - Found stale lock file and directory '/mnt/SSD/fische_r/tmp/dask-worker-space/worker-aqx4i1f3', purging
/mpc/homes/fische_r/miniconda3/lib/python3.10/contextlib.py:142: UserWarning: Creating scratch directories is taking a surprisingly long time. (3.69s) This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)

Dashboard at http://127.0.0.1:35000/status

Test ROI selection

In [4]:

# sample = '4'

In [5]:

# # file = '01_'+sample+'_rotated_3p1D.nc'
# file = '02_'+sample+'_registered_3p1D.nc'
# imagepath = os.path.join(path_02_4D, file)
# data = xr.open_dataset(imagepath)
# images = [im for im in data.keys() if im[3:7] == 'imag']
# images.sort()

In [6]:

# im300 = data[images[4]][:,:,300].data

In [7]:

# plt.imshow(im300[50:-50,50:370], cmap='gray', vmin=10000, vmax=15000)

In [8]:

# data.close()

In [9]:

## individual cropping appears to be require for a good ROI for all samples, use same with and height!!

In [10]:

training_path

Out [10]:

'/mpc/homes/fische_r/NAS/DASCOELY/processing/05_water_GDL_ML/'

In [11]:

# height = 750
# width = 340
# a = 0
# b = a+height
# c = 30
# d = c + width

# crop_dict = {}
# crop_dict['1'] = (120, 120+height, 10, 10+width) # sample 1 should be aligned by rotation
# crop_dict['3II'] = (5, 5+height, 0, width)
# crop_dict['3III'] = (80, 80+height, 20, 20+width)
# crop_dict['4'] = (40, 40+height, 40, 40+width)
# crop_dict['4II'] = (0, 0+height, 10, 10+width)
# crop_dict['5II'] = (25, 25+height, 15, 15+width)
# crop_dict['5'] = (57, 57+height, 10, 10+width)
# crop_dict['6'] = (105, 105+height, 30, 30+width)
# crop_dict['7x'] = (70, 70+height, 15, 15+width)
# crop_dict['8x'] = (0, 0+height, 25, 25+width)

# samplesall = ['1', '3II', '3III', '4', '4II', '5II', '5', '6', '7x', '8x']

# if not os.path.exists(training_path):
#     os.mkdir(training_path)

# jsonpath = os.path.join(training_path, 'cathode_cropping_and_aligning.json')
# with open(jsonpath, "w") as outfile:
#     json.dump(crop_dict, outfile,indent = 4)

# # samplesall = samplesall[6:10]

# for sample in samplesall:
#     (a1,b1,c1,d1) = (a,b,c,d)
#     if sample in crop_dict.keys():
#         (a1,b1,c1,d1) = crop_dict[sample]
#     file = '02_'+sample+'_registered_3p1D.nc'
#     imagepath = os.path.join(path_02_4D, file)
#     data = xr.open_dataset(imagepath)
#     images = [im for im in data.keys() if im[3:7] == 'imag']
#     images.sort()
#     im300 = data[images[6]][:,:,300].data
#     im0 = data[images[6]][:,:,50].data
#     imfin = data[images[6]][:,:,-50].data
#     data.close()
#     fig, axs = plt.subplots(1,3)
#     axs[0].imshow(im300[a1:b1,c1:d1], cmap='gray', vmin=10000, vmax=15000)
#     axs[1].imshow(im0[a1:b1,c1:d1], cmap='gray', vmin=10000, vmax=15000)
#     axs[2].imshow(imfin[a1:b1,c1:d1], cmap='gray', vmin=10000, vmax=15000)
#     print(imfin.shape, imfin[a1:b1,c1:d1].shape)
    
#     for ax in axs:
#         ax.vlines(20,0,height-1,'w', linestyle='--')
#         ax.hlines(40,0,width-1,'w', linestyle='--')
    
#     plt.title(sample +' '+ images[4])

Data preparation

create dask array

In [12]:

sample = '4'
file = '02_'+sample+'_registered_3p1D.nc'
imagepath = os.path.join(path_02_4D, file)
data = xr.open_dataset(imagepath)
images = [im for im in data.keys() if im[3:7] == 'imag']
images.sort()

In [13]:

t_utc = data['t_utc'].data
time = data['time'].data

In [14]:

# with obtained cropping data
jsonpath = os.path.join(training_path, 'cathode_cropping_and_aligning.json')
crop_dict = json.load(open(jsonpath, 'r'))

(a,b,c,d) = crop_dict[sample]
(e,f) = (50,-50)

#corrections to crop coordinates
# f = e+1750
# e = e+100

define border to crop to GDL

In [15]:

test_im1 = data[images[2]][a:b, c:d, e:f].data
test_im2 = data[images[-5]][a:b, c:d, e:f].data

In [16]:

GDL_crop = 0
plt.figure(figsize=(16,9))
plt.imshow(test_im1[600,GDL_crop:, :], vmin =10000, vmax=15000, cmap='gray')

Out [16]:

<matplotlib.image.AxesImage at 0x7f5fa9009b70>

In [17]:

shp = data[images[4]][a:b, c:d, e:f].shape
# shp = data[images[4]][a:b, c+GDL_crop:d, e:f].shape
shp = shp + (len(images),)
print(shp, test_im1[:,GDL_crop:, :].shape)

(750, 340, 1916, 71) (750, 340, 1916)

In [18]:

im = np.zeros(shp, dtype = np.uint16)

In [19]:

# with obtained cropping data
for i in range(shp[-1]):
    if i%10==0:
        print(i)
    # im[...,i] = data[images[i]][a:b, c:d, e:f].data
    im[...,i] = data[images[i]][a:b, c+GDL_crop:d, e:f].data

In [20]:

chunk_space = 36
da = dask.array.from_array(im, chunks = (chunk_space,chunk_space,chunk_space,len(images)))

In [21]:

da

Out [21]:

	Array	Chunk
Bytes	64.61 GiB	6.32 MiB
Shape	(750, 340, 1916, 71)	(36, 36, 36, 71)
Dask graph	11340 chunks in 1 graph layer
Data type	uint16 numpy.ndarray

In [22]:

del im
gc.collect()

Out [22]:

In [23]:

data.close()

get data into image filter class

In [24]:

# TODO: include this routine into pytrainseg

IF = image_filter(sigmas = [2,4,6])# , sigma_0_derivatives=True) #old default settings
IF.data = da
shp = da.shape
coords = {'x': np.arange(shp[0]), 'y': np.arange(shp[1]), 'z': np.arange(shp[2]), 'time': np.arange(shp[3])}
IF.original_dataset = xr.Dataset({'tomo': (['x','y','z','time'], da)},
                                 coords = coords
                                )
# IF.data = IF.data.rechunk('auto')

prepare features

In [25]:

IF.prepare()

In [26]:

IF.stack_features()

In [27]:

IF.compute_time_independent_features()

2023-05-16 09:44:20,911 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)

In [28]:

IF.feature_stack

Out [28]:

	Array	Chunk
Bytes	16.15 TiB	25.27 MiB
Shape	(750, 340, 1916, 71, 64)	(36, 36, 36, 71, 1)
Dask graph	903168 chunks in 465 graph layers
Data type	float64 numpy.ndarray

In [29]:

IF.make_xarray_nc()

Training

set up objects

In [30]:

# quick fix: copy full feature set and cast selection into TS.training_dict
# TODO: select for training without copying

def features_to_keep(feat_length, features_to_remove):
    ids = np.ones(feat_length, dtype=bool)
    for feat in features_to_remove:
        ids[feat] = False
    return ids

def ignore_feat_per_slice(entry, ids):
    truth = entry[1]
    feats = entry[0]
    feats = feats[:,ids]
    return (feats, truth)

def ignore_features(TS, features_to_remove = []):
    if TS.training_dict_full is None:
        TS.combined_feature_names_full = TS.combined_feature_names.copy()
        TS.training_dict_full =  TS.training_dict.copy()
        
        
    ids = features_to_keep(len(TS.combined_feature_names_full), features_to_remove)
    temp_dict = {}
    for key in TS.training_dict_full:
        entry = TS.training_dict_full[key]
        temp_dict[key] = ignore_feat_per_slice(entry, ids)
    TS.training_dict = temp_dict
    TS.combined_feature_names = np.array(TS.combined_feature_names_full)[ids]
    
    return TS, ids

In [31]:

training_path_sample = os.path.join(training_path, sample)
if not os.path.exists(training_path_sample):
    os.mkdir(training_path_sample)

In [32]:

TS = train_segmentation(training_path=training_path_sample)

In [33]:

# TS.training_dict_full = {}

In [34]:

TS.import_lazy_feature_data(IF.result, IF.original_dataset)

In [35]:

IF.combined_feature_names = list(IF.feature_names) + list(IF.feature_names_time_independent)

In [36]:

TS.combined_feature_names = IF.combined_feature_names

In [37]:

TS.combined_feature_names

Out [37]:

['diff_to_first_',
 'diff_to_last_',
 'Gaussian_4D_Blur_2.0',
 'Gaussian_4D_Blur_4.0',
 'Gaussian_4D_Blur_6.0',
 'diff_of_gauss_4D_4.0_2.0',
 'diff_of_gauss_4D_6.0_2.0',
 'diff_of_gauss_4D_6.0_4.0',
 'Gradient_sigma_2.0_0',
 'Gradient_sigma_2.0_1',
 'Gradient_sigma_2.0_2',
 'Gradient_sigma_2.0_3',
 'hessian_sigma_2.0_00',
 'hessian_sigma_2.0_01',
 'hessian_sigma_2.0_02',
 'hessian_sigma_2.0_03',
 'hessian_sigma_2.0_11',
 'hessian_sigma_2.0_12',
 'hessian_sigma_2.0_13',
 'hessian_sigma_2.0_22',
 'hessian_sigma_2.0_23',
 'hessian_sigma_2.0_33',
 'Gradient_sigma_4.0_0',
 'Gradient_sigma_4.0_1',
 'Gradient_sigma_4.0_2',
 'Gradient_sigma_4.0_3',
 'hessian_sigma_4.0_00',
 'hessian_sigma_4.0_01',
 'hessian_sigma_4.0_02',
 'hessian_sigma_4.0_03',
 'hessian_sigma_4.0_11',
 'hessian_sigma_4.0_12',
 'hessian_sigma_4.0_13',
 'hessian_sigma_4.0_22',
 'hessian_sigma_4.0_23',
 'hessian_sigma_4.0_33',
 'Gradient_sigma_6.0_0',
 'Gradient_sigma_6.0_1',
 'Gradient_sigma_6.0_2',
 'Gradient_sigma_6.0_3',
 'hessian_sigma_6.0_00',
 'hessian_sigma_6.0_01',
 'hessian_sigma_6.0_02',
 'hessian_sigma_6.0_03',
 'hessian_sigma_6.0_11',
 'hessian_sigma_6.0_12',
 'hessian_sigma_6.0_13',
 'hessian_sigma_6.0_22',
 'hessian_sigma_6.0_23',
 'hessian_sigma_6.0_33',
 'Gaussian_time_2.0',
 'Gaussian_time_4.0',
 'Gaussian_time_6.0',
 'diff_of_gauss_time_4.0_2.0',
 'diff_of_gauss_time_6.0_2.0',
 'diff_of_gauss_time_6.0_4.0',
 'Gaussian_space_2.0',
 'Gaussian_space_4.0',
 'Gaussian_space_6.0',
 'diff_of_gauss_space_4.0_2.0',
 'diff_of_gauss_space_6.0_2.0',
 'diff_of_gauss_space_6.0_4.0',
 'diff_to_min_',
 'diff_temp_min_Gauss_2.0',
 'first_',
 'last_',
 'full_temp_mean_',
 'full_temp_min_',
 'full_temp_min_Gauss_2.0']

interactive training

check for existing training sets

In [38]:

existing_sets = os.listdir(os.path.join(training_path_sample, 'label_images'))
existing_sets.sort()
existing_sets

Out [38]:

['label_image_x_157_time_5_.tif',
 'label_image_x_172_time_17_.tif',
 'label_image_x_172_time_2_.tif',
 'label_image_x_252_time_0_.tif',
 'label_image_x_264_time_29_.tif',
 'label_image_x_270_time_50_.tif',
 'label_image_x_307_time_25_.tif',
 'label_image_x_368_time_45_.tif',
 'label_image_x_456_time_0_.tif',
 'label_image_x_503_time_15_.tif',
 'label_image_x_531_time_50_.tif',
 'label_image_y_241_time_0_.tif',
 'label_image_y_245_time_8_.tif',
 'label_image_y_250_time_23_.tif',
 'label_image_y_250_time_33_.tif',
 'label_image_y_255_time_20_.tif',
 'label_image_y_255_time_40_.tif',
 'label_image_y_255_time_64_.tif',
 'label_image_y_270_time_47_.tif',
 'label_image_y_280_time_57_.tif',
 'label_image_y_71_time_27_.tif',
 'label_image_z_307_time_0_.tif',
 'label_image_z_307_time_10_.tif',
 'label_image_z_307_time_25_.tif',
 'label_image_z_320_time_10_.tif',
 'label_image_z_320_time_30_.tif',
 'label_image_z_320_time_45_.tif',
 'label_image_z_468_time_11_.tif',
 'label_image_z_546_time_0_.tif',
 'label_image_z_564_time_0_.tif']

In [39]:

training_path

Out [39]:

'/mpc/homes/fische_r/NAS/DASCOELY/processing/05_water_GDL_ML/'

In [40]:

# you can load a compatible pickled training dict, check feature names
TS.training_dict = pickle.load(open(os.path.join(TS.training_path, pytrain_git_sha+'_training_dict.p'),'rb'))

In [ ]:

TS.training_dict

re-train with existing label sets. clear the training dictionary if necessary (training_dict)

In [41]:

TS.train()

training with existing label images
label_image_z_546_time_0_.tif
label_image_y_255_time_40_.tif already done
label_image_z_307_time_25_.tif already done
label_image_x_157_time_5_.tif already done
label_image_y_71_time_27_.tif already done
label_image_x_270_time_50_.tif already done
label_image_x_456_time_0_.tif already done
label_image_z_320_time_10_.tif already done
label_image_x_307_time_25_.tif
label_image_z_564_time_0_.tif already done
label_image_z_320_time_30_.tif already done
label_image_x_264_time_29_.tif already done
label_image_y_280_time_57_.tif already done
label_image_y_250_time_23_.tif already done
label_image_x_172_time_17_.tif

2023-05-16 09:45:00,371 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 09:45:14,621 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-05-16 09:45:35,188 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:46:04,314 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-05-16 09:49:22,459 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:50:54,694 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:52:31,246 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:53:45,077 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 09:54:17,198 - distributed.worker - ERROR - Timed out during handshake while connecting to tcp://127.0.0.1:40027 after 30 s
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/core.py", line 328, in connect
    handshake = await asyncio.wait_for(comm.read(), time_left())
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
    return await func(*args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 1566, in close
    await r.close_gracefully(reason=reason)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1224, in send_recv_from_rpc
    comm = await self.pool.connect(self.addr)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1468, in connect
    return await connect_attempt
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 1389, in _connect
    comm = await connect(
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/comm/core.py", line 333, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://127.0.0.1:40027 after 30 s

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
    return await func(*args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 1518, in close
    await self.finished()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 494, in finished
    await self._event_finished.wait()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError
2023-05-16 09:54:47,223 - distributed.worker - CRITICAL - Error trying close worker in response to broken internal state. Forcibly exiting worker NOW
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
    return await func(*args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 1518, in close
    await self.finished()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/core.py", line 494, in finished
    await self._event_finished.wait()
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/distributed/worker.py", line 230, in _force_close
    await asyncio.wait_for(
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
2023-05-16 09:54:47,590 - distributed.nanny - WARNING - Restarting worker
2023-05-16 09:55:24,784 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-05-16 09:55:27,257 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:30,068 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:33,322 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:36,585 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:39,938 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:43,304 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:46,738 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:51,772 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:55:56,134 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:00,644 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:06,342 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:11,303 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:17,016 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:22,878 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:29,303 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:36,880 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:43,433 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:50,846 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:56:59,606 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:07,562 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:16,888 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:26,843 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:37,066 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:47,981 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:57:59,347 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:58:11,759 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:58:25,270 - distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently (threshold: 10%)
2023-05-16 09:58:40,363 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-05-16 09:58:54,131 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-05-16 09:59:11,643 - distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)
2023-05-16 09:59:30,384 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-05-16 09:59:49,345 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-05-16 10:00:11,604 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-05-16 10:00:50,460 - distributed.utils_perf - WARNING - full garbage collections took 15% CPU time recently (threshold: 10%)
2023-05-16 10:01:38,505 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 10:02:19,995 - distributed.utils_perf - WARNING - full garbage collections took 14% CPU time recently (threshold: 10%)
2023-05-16 10:03:03,981 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-05-16 10:03:44,879 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-05-16 10:04:16,568 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-05-16 10:05:04,492 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-05-16 10:06:05,845 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:06:44,248 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:07:48,311 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:08:33,565 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:09:17,781 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:10:02,956 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:10:56,186 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:11:48,945 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:13:03,665 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:14:37,086 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-05-16 10:16:28,883 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 10:17:57,100 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 10:20:21,831 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 10:40:24,306 - distributed.worker.memory - WARNING - gc.collect() took 11.618s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
/mpc/homes/fische_r/lib/pytrainseg/training_functions.py:375: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if not X == 'no labels':

label_image_x_503_time_15_.tif

2023-05-16 11:00:29,991 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-05-16 11:02:55,018 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-05-16 11:04:08,520 - distributed.utils_perf - WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)
2023-05-16 11:21:07,596 - distributed.worker.memory - WARNING - gc.collect() took 11.310s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 11:34:20,746 - distributed.worker.memory - WARNING - gc.collect() took 12.740s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.

label_image_y_255_time_20_.tif

2023-05-16 11:58:42,997 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 12:00:56,045 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 12:03:19,904 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 12:05:08,338 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-05-16 12:40:23,700 - distributed.worker.memory - WARNING - gc.collect() took 15.942s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 12:54:44,026 - distributed.worker.memory - WARNING - gc.collect() took 16.627s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 13:06:40,104 - distributed.worker.memory - WARNING - gc.collect() took 18.170s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 13:23:04,136 - distributed.worker.memory - WARNING - gc.collect() took 19.639s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-05-16 13:39:07,655 - distributed.worker.memory - WARNING - gc.collect() took 25.502s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.

label_image_y_250_time_33_.tif

[0;31m---------------------------------------------------------------------------[0m
[0;31mKeyboardInterrupt[0m                         Traceback (most recent call last)
Cell [0;32mIn[41], line 1[0m
[0;32m----> 1[0m [43mTS[49m[38;5;241;43m.[39;49m[43mtrain[49m[43m([49m[43m)[49m

File [0;32m~/lib/pytrainseg/training_functions.py:374[0m, in [0;36mtrain_segmentation.train[0;34m(self, clear_dict, redo)[0m
[1;32m    372[0m     [38;5;28;01mcontinue[39;00m
[1;32m    373[0m [38;5;28mprint[39m(label_name)
[0;32m--> 374[0m X, y [38;5;241m=[39m [43mtraining_set_per_image[49m[43m([49m[43mlabel_name[49m[43m,[49m[43m [49m[43mpath[49m[43m,[49m[43m [49m[43mfeat_data[49m[43m,[49m[43m [49m[38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43mlazy[49m[43m)[49m
[1;32m    375[0m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m X [38;5;241m==[39m [38;5;124m'[39m[38;5;124mno labels[39m[38;5;124m'[39m:
[1;32m    376[0m     [38;5;28mself[39m[38;5;241m.[39mtraining_dict[label_name] [38;5;241m=[39m X,y

File [0;32m~/lib/pytrainseg/training_functions.py:138[0m, in [0;36mtraining_set_per_image[0;34m(label_name, trainingpath, feat_data, lazy)[0m
[1;32m    130[0m [38;5;66;03m# if lazy:[39;00m
[1;32m    131[0m [38;5;66;03m#     print('Need to actually calculate the features for each slice, seems inefficient')[39;00m
[1;32m    132[0m [38;5;66;03m# #   not sure how efficient this is[39;00m
[0;32m   (...)[0m
[1;32m    135[0m [38;5;66;03m#     feat_stack = feat_stack.compute()[39;00m
[1;32m    136[0m [38;5;66;03m# else:[39;00m
[1;32m    137[0m [38;5;28;01mif[39;00m [38;5;28mtype[39m(feat_stack) [38;5;129;01mis[39;00m [38;5;129;01mnot[39;00m np[38;5;241m.[39mndarray:
[0;32m--> 138[0m         feat_stack [38;5;241m=[39m [43mfeat_stack[49m[38;5;241;43m.[39;49m[43mcompute[49m[43m([49m[43m)[49m
[1;32m    139[0m [38;5;28;01mif[39;00m [38;5;28mtype[39m(feat_stack_t_idp) [38;5;129;01mis[39;00m [38;5;129;01mnot[39;00m np[38;5;241m.[39mndarray:
[1;32m    140[0m         feat_stack_t_idp [38;5;241m=[39m feat_stack_t_idp[38;5;241m.[39mcompute()

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/base.py:314[0m, in [0;36mDaskMethodsMixin.compute[0;34m(self, **kwargs)[0m
[1;32m    290[0m [38;5;28;01mdef[39;00m [38;5;21mcompute[39m([38;5;28mself[39m, [38;5;241m*[39m[38;5;241m*[39mkwargs):
[1;32m    291[0m [38;5;250m    [39m[38;5;124;03m"""Compute this dask collection[39;00m
[1;32m    292[0m 
[1;32m    293[0m [38;5;124;03m    This turns a lazy Dask collection into its in-memory equivalent.[39;00m
[0;32m   (...)[0m
[1;32m    312[0m [38;5;124;03m    dask.base.compute[39;00m
[1;32m    313[0m [38;5;124;03m    """[39;00m
[0;32m--> 314[0m     (result,) [38;5;241m=[39m [43mcompute[49m[43m([49m[38;5;28;43mself[39;49m[43m,[49m[43m [49m[43mtraverse[49m[38;5;241;43m=[39;49m[38;5;28;43;01mFalse[39;49;00m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m[43m)[49m
[1;32m    315[0m     [38;5;28;01mreturn[39;00m result

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/base.py:593[0m, in [0;36mcompute[0;34m(traverse, optimize_graph, scheduler, get, *args, **kwargs)[0m
[1;32m    585[0m     [38;5;28;01mreturn[39;00m args
[1;32m    587[0m schedule [38;5;241m=[39m get_scheduler(
[1;32m    588[0m     scheduler[38;5;241m=[39mscheduler,
[1;32m    589[0m     collections[38;5;241m=[39mcollections,
[1;32m    590[0m     get[38;5;241m=[39mget,
[1;32m    591[0m )
[0;32m--> 593[0m dsk [38;5;241m=[39m [43mcollections_to_dsk[49m[43m([49m[43mcollections[49m[43m,[49m[43m [49m[43moptimize_graph[49m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m[43m)[49m
[1;32m    594[0m keys, postcomputes [38;5;241m=[39m [], []
[1;32m    595[0m [38;5;28;01mfor[39;00m x [38;5;129;01min[39;00m collections:

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/base.py:366[0m, in [0;36mcollections_to_dsk[0;34m(collections, optimize_graph, optimizations, **kwargs)[0m
[1;32m    364[0m [38;5;28;01mfor[39;00m opt, val [38;5;129;01min[39;00m groups[38;5;241m.[39mitems():
[1;32m    365[0m     dsk, keys [38;5;241m=[39m _extract_graph_and_keys(val)
[0;32m--> 366[0m     dsk [38;5;241m=[39m [43mopt[49m[43m([49m[43mdsk[49m[43m,[49m[43m [49m[43mkeys[49m[43m,[49m[43m [49m[38;5;241;43m*[39;49m[38;5;241;43m*[39;49m[43mkwargs[49m[43m)[49m
[1;32m    368[0m     [38;5;28;01mfor[39;00m opt_inner [38;5;129;01min[39;00m optimizations:
[1;32m    369[0m         dsk [38;5;241m=[39m opt_inner(dsk, keys, [38;5;241m*[39m[38;5;241m*[39mkwargs)

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/array/optimization.py:57[0m, in [0;36moptimize[0;34m(dsk, keys, fuse_keys, fast_functions, inline_functions_fast_functions, rename_fused_keys, **kwargs)[0m
[1;32m     54[0m [38;5;28;01mif[39;00m config[38;5;241m.[39mget([38;5;124m"[39m[38;5;124moptimization.fuse.active[39m[38;5;124m"[39m) [38;5;129;01mis[39;00m [38;5;28;01mFalse[39;00m:
[1;32m     55[0m     [38;5;28;01mreturn[39;00m dsk
[0;32m---> 57[0m dependencies [38;5;241m=[39m [43mdsk[49m[38;5;241;43m.[39;49m[43mget_all_dependencies[49m[43m([49m[43m)[49m
[1;32m     58[0m dsk [38;5;241m=[39m ensure_dict(dsk)
[1;32m     60[0m [38;5;66;03m# Low level task optimizations[39;00m

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/highlevelgraph.py:813[0m, in [0;36mHighLevelGraph.get_all_dependencies[0;34m(self)[0m
[1;32m    811[0m [38;5;28;01mif[39;00m missing_keys:
[1;32m    812[0m     [38;5;28;01mfor[39;00m layer [38;5;129;01min[39;00m [38;5;28mself[39m[38;5;241m.[39mlayers[38;5;241m.[39mvalues():
[0;32m--> 813[0m         [38;5;28;01mfor[39;00m k [38;5;129;01min[39;00m [43mmissing_keys[49m[43m [49m[38;5;241;43m&[39;49m[43m [49m[43mlayer[49m[38;5;241;43m.[39;49m[43mkeys[49m[43m([49m[43m)[49m:
[1;32m    814[0m             [38;5;28mself[39m[38;5;241m.[39mkey_dependencies[k] [38;5;241m=[39m layer[38;5;241m.[39mget_dependencies(k, all_keys)
[1;32m    815[0m [38;5;28;01mreturn[39;00m [38;5;28mself[39m[38;5;241m.[39mkey_dependencies

File [0;32m~/miniconda3/lib/python3.10/_collections_abc.py:638[0m, in [0;36mSet.__and__[0;34m(self, other)[0m
[1;32m    636[0m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m [38;5;28misinstance[39m(other, Iterable):
[1;32m    637[0m     [38;5;28;01mreturn[39;00m [38;5;28mNotImplemented[39m
[0;32m--> 638[0m [38;5;28;01mreturn[39;00m [38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43m_from_iterable[49m[43m([49m[43mvalue[49m[43m [49m[38;5;28;43;01mfor[39;49;00m[43m [49m[43mvalue[49m[43m [49m[38;5;129;43;01min[39;49;00m[43m [49m[43mother[49m[43m [49m[38;5;28;43;01mif[39;49;00m[43m [49m[43mvalue[49m[43m [49m[38;5;129;43;01min[39;49;00m[43m [49m[38;5;28;43mself[39;49m[43m)[49m

File [0;32m~/miniconda3/lib/python3.10/_collections_abc.py:880[0m, in [0;36mKeysView._from_iterable[0;34m(cls, it)[0m
[1;32m    878[0m [38;5;129m@classmethod[39m
[1;32m    879[0m [38;5;28;01mdef[39;00m [38;5;21m_from_iterable[39m([38;5;28mcls[39m, it):
[0;32m--> 880[0m     [38;5;28;01mreturn[39;00m [38;5;28;43mset[39;49m[43m([49m[43mit[49m[43m)[49m

File [0;32m~/miniconda3/lib/python3.10/_collections_abc.py:638[0m, in [0;36m<genexpr>[0;34m(.0)[0m
[1;32m    636[0m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m [38;5;28misinstance[39m(other, Iterable):
[1;32m    637[0m     [38;5;28;01mreturn[39;00m [38;5;28mNotImplemented[39m
[0;32m--> 638[0m [38;5;28;01mreturn[39;00m [38;5;28mself[39m[38;5;241m.[39m_from_iterable(value [38;5;28;01mfor[39;00m value [38;5;129;01min[39;00m other [38;5;28;01mif[39;00m [43mvalue[49m[43m [49m[38;5;129;43;01min[39;49;00m[43m [49m[38;5;28;43mself[39;49m)

File [0;32m~/miniconda3/lib/python3.10/_collections_abc.py:883[0m, in [0;36mKeysView.__contains__[0;34m(self, key)[0m
[1;32m    882[0m [38;5;28;01mdef[39;00m [38;5;21m__contains__[39m([38;5;28mself[39m, key):
[0;32m--> 883[0m     [38;5;28;01mreturn[39;00m [43mkey[49m[43m [49m[38;5;129;43;01min[39;49;00m[43m [49m[38;5;28;43mself[39;49m[38;5;241;43m.[39;49m[43m_mapping[49m

File [0;32m~/miniconda3/lib/python3.10/site-packages/dask/highlevelgraph.py:540[0m, in [0;36mMaterializedLayer.__contains__[0;34m(self, k)[0m
[1;32m    537[0m     [38;5;28msuper[39m()[38;5;241m.[39m[38;5;21m__init__[39m(annotations[38;5;241m=[39mannotations)
[1;32m    538[0m     [38;5;28mself[39m[38;5;241m.[39mmapping [38;5;241m=[39m mapping
[0;32m--> 540[0m [38;5;28;01mdef[39;00m [38;5;21m__contains__[39m([38;5;28mself[39m, k):
[1;32m    541[0m     [38;5;28;01mreturn[39;00m k [38;5;129;01min[39;00m [38;5;28mself[39m[38;5;241m.[39mmapping
[1;32m    543[0m [38;5;28;01mdef[39;00m [38;5;21m__getitem__[39m([38;5;28mself[39m, k):

[0;31mKeyboardInterrupt[0m:

import training dict of other samples

(replace sample name and repeat for multiple samples), if necessary check features for overlap

In [39]:

oldsample = '4'
oldgitsha = 'ec4415d'
if oldsample == '4':
    training_dict_old = pickle.load(open(os.path.join(toppathSSD, '05_water_GDL_ML', '4', 'ec4415d_training_dict_without_loc_feat.p'), 'rb'))
else:
    training_dict_old = pickle.load(open(os.path.join(training_path, oldsample,  oldgitsha+'_training_dict.p'),'rb'))
oldfeatures = pickle.load(open(os.path.join(training_path, oldsample,  oldgitsha+'_feature_names.p'),'rb'))
    
    # pickle.dump(TS.training_dict, open(os.path.join(TS.training_path, pytrain_git_sha+'_training_dict.p'),'wb'))
# pickle.dump(TS.feature_names, open(os.path.join(TS.training_path, pytrain_git_sha+'_feature_names.p'),'wb'))

for key in training_dict_old.keys():
    TS.training_dict[oldsample+key] = training_dict_old[key]

suggest a new training coordinate

currently retraining with new feature stack not properly implemented. Workaround: choose from the exiting training sets and train with them (additional labeling optional)

In [42]:

TS.suggest_training_set()

You could try  x = 55  and  z = 1058
However, please sort it like the original xyztimetime_0feature

In [70]:

c1 = 'y'
p1 = 240
c2 = 'time'
p2 = 14

In [71]:

TS.load_training_set(c1, p1, c2, p2)
im8 = TS.current_im8

In [72]:

# TS.get_slice_feat_stack()

In [73]:

feat_data = TS.feat_data
[c1,p1,c2,p2] = TS.current_coordinates
newslice = True

if c1 == 'x' and c2 == 'time':
    feat_stack = feat_data['feature_stack'].sel(x = p1, time = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(x = p1, time_0 = 0).data
elif c1 == 'x' and c2 == 'y':
    feat_stack = feat_data['feature_stack'].sel(x = p1, y = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(x = p1, y = p2).data
elif c1 == 'x' and c2 == 'z':
    feat_stack = feat_data['feature_stack'].sel(x = p1, z = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(x = p1, z = p2).data
elif c1 == 'y' and c2 == 'z':
    feat_stack = feat_data['feature_stack'].sel(y = p1, z = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(y = p1, z = p2).data
elif c1 == 'y' and c2 == 'time':
    feat_stack = feat_data['feature_stack'].sel(y = p1, time = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(y = p1, time_0 = 0).data
elif c1 == 'z' and c2 == 'time':
    feat_stack = feat_data['feature_stack'].sel(z = p1, time = p2).data
    feat_stack_t_idp = feat_data['feature_stack_time_independent'].sel(z = p1, time_0 = 0).data


TS.current_feat_stack = dask.array.concatenate([feat_stack, feat_stack_t_idp], axis = 2)

if type(TS.current_feat_stack) is not np.ndarray:
    TS.current_computed = False

canvas for labeling

In [74]:

# imm = mean[:,p1,:].copy()
# im8m= imm-imm.min()
# im8m = im8m/im8m.max()*255
# im8 = im8m

In [92]:

alpha = 0.15
zoom1 = (-500,-1)
zoom2 = (600,1400)

zoom1 = (0, -1)
zoom2 = (0, -1)

# im8 = TS.current_im8
#trick: use gaussian_time_4_0 to label static phases ()
# im8 = display_feature(-2, TS)
# im8 = display_feature(-20, TS)
# print(IF.combined_feature_names[-20])
print('original shape: ',im8.shape)
im8_display = im8.copy()[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]]
print('diyplay shape : ',im8_display.shape,' at: ', (zoom1[0], zoom2[0]))

resultim = TS.current_result.copy()

resultim_display = resultim[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]]


width = im8_display.shape[1]
height = im8_display.shape[0]
Mcanvas = MultiCanvas(4, width=width, height=height)
background = Mcanvas[0]
resultdisplay = Mcanvas[2]
truthdisplay = Mcanvas[1]
canvas = Mcanvas[3]
canvas.sync_image_data = True
drawing = False
position = None
shape = []
image_data = np.stack((im8_display, im8_display, im8_display), axis=2)
background.put_image_data(image_data, 0, 0)
slidealpha = IntSlider(description="Result overlay", value=0.15)
resultdisplay.global_alpha = alpha #slidealpha.value
if np.any(resultim>0):
    result_data = np.stack((255*(resultim_display==0), 255*(resultim_display==1), 255*(resultim_display==2)), axis=2)
else:
    result_data = np.stack((0*resultim, 0*resultim, 0*resultim), axis=2)
resultdisplay.put_image_data(result_data, 0, 0)
canvas.on_mouse_down(on_mouse_down)
canvas.on_mouse_move(on_mouse_move)
canvas.on_mouse_up(on_mouse_up)
picker = ColorPicker(description="Color:", value="#ff0000") #red
# picker = ColorPicker(description="Color:", value="#0000ff") #blue
# picker = ColorPicker(description="Color:", value="#00ff00") #green

link((picker, "value"), (canvas, "stroke_style"))
link((picker, "value"), (canvas, "fill_style"))
link((slidealpha, "value"), (resultdisplay, "global_alpha"))

HBox((Mcanvas,picker))
# HBox((Mcanvas,)) #picker

Out [92]:

original shape:  (750, 1916)
diyplay shape :  (749, 1915)  at:  (0, 0)

HBox(children=(MultiCanvas(height=749, width=1915), ColorPicker(value='#ff0000', description='Color:')))

In [81]:

tfs.plot_im_histogram(im8)
# im8 = TS.current_im8
# im8 = tfs.adjust_image_contrast(im8,20,200)

inspect labels and training progress

In [93]:

fig, axes = plt.subplots(1,6, figsize=(20,10))
axes[0].imshow(TS.current_result, 'gray')
axes[1].imshow(TS.current_im8, 'gray')

# TS.current_diff_im = TS.current_im-TS.current_first_im
# TS.current_diff_im = TS.current_diff_im/TS.current_diff_im.max()*255
axes[2].imshow(-TS.current_diff_im)#,vmin=6e4)
# axes[3].imshow(im8old, 'gray')
axes[3].imshow(TS.current_first_im, 'gray')
axes[4].imshow(TS.current_truth)
if TS.current_computed:
    axes[5].imshow(TS.current_feat_stack[:,:,-10])
else:
    axes[5].imshow(TS.current_result, 'gray')

for ax in axes:
    ax.set_xticks([])
    ax.set_yticks([])

update training set if labels are ok

In [84]:

label_set = canvas.get_image_data()

TS.current_truth[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]][label_set[:,:,0]>0] = 1
TS.current_truth[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]][label_set[:,:,1]>0] = 2
TS.current_truth[zoom1[0]:zoom1[1], zoom2[0]:zoom2[1]][label_set[:,:,2]>0] = 4

imageio.imsave(TS.current_truthpath, TS.current_truth)

(already removed from filter functions, not necessary to remove anymore) ignore pixel location as feature as this can be dangerous

try ignoring full temp mean as clf relies too much on it

In [85]:

# TODO: find feature index by name
# loc_feats = [-4, -5, -6]

In [86]:

# ignore features in training dict
# TS, ids = ignore_features(TS, loc_feats)
# ignore features in current feature set
#  risky workaround for now
# currfeatpath = '/mnt/SSD/fische_r/tmp/temp_curr_feat.p'
# pickle.dump(TS.current_feat_stack, open(currfeatpath, 'wb'))
# if newslice:
#     TS.current_feat_stack = TS.current_feat_stack[:,:,ids]
#     newslice = False

train!

In [87]:

# TODO: see, if training gets slow for many label sets, currently stored in training_dict and read as loop. or if it is just the larger amount of data
TS.train_slice()

now actually calculating the features

2023-03-29 10:43:42,902 - distributed.utils_perf - WARNING - full garbage collections took 12% CPU time recently (threshold: 10%)
2023-03-29 10:46:50,886 - distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
2023-03-29 10:47:32,421 - distributed.worker.memory - WARNING - gc.collect() took 1.027s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-03-29 10:49:51,644 - distributed.worker.memory - WARNING - gc.collect() took 1.240s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-03-29 10:49:52,590 - distributed.worker.memory - WARNING - gc.collect() took 1.523s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-03-29 10:50:09,993 - distributed.worker.memory - WARNING - gc.collect() took 1.379s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help.
2023-03-29 10:52:55,347 - tornado.application - ERROR - Uncaught exception GET /status/ws (::1)
HTTPServerRequest(protocol='http', host='127.0.0.1:35000', method='GET', uri='/status/ws', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/tornado/websocket.py", line 942, in _accept_connection
    open_result = handler.open(*handler.open_args, **handler.open_kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/tornado/web.py", line 3208, in wrapper
    return method(self, *args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/bokeh/server/views/ws.py", line 149, in open
    raise ProtocolError("Token is expired.")
bokeh.protocol.exceptions.ProtocolError: Token is expired.
ERROR:tornado.application:Uncaught exception GET /status/ws (::1)
HTTPServerRequest(protocol='http', host='127.0.0.1:35000', method='GET', uri='/status/ws', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/tornado/websocket.py", line 942, in _accept_connection
    open_result = handler.open(*handler.open_args, **handler.open_kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/tornado/web.py", line 3208, in wrapper
    return method(self, *args, **kwargs)
  File "/mpc/homes/fische_r/miniconda3/lib/python3.10/site-packages/bokeh/server/views/ws.py", line 149, in open
    raise ProtocolError("Token is expired.")
bokeh.protocol.exceptions.ProtocolError: Token is expired.

feat_stack is not a numpy array! check why
training and classifying

In [421]:

# # load exisiting classifier and segment one slice
# clf = pickle.load(open(os.path.join(training_path, 'classifier.p'), 'rb'))
# feat = TS.current_feat_stack.compute()
# shp = feat[...,0].shape
# num_feat = feat.shape[-1]
# feat = feat.reshape(-1,num_feat)
# seg = clf.predict(feat)
# seg = seg.reshape(shp).astype(np.uint8)
# plt.figure(figsize=(16,9))
# plt.imshow(seg, cmap='gray_r')

In [422]:

# plt.figure(figsize=(16,9))
# plt.imshow(im8, cmap='gray')

go back until happy

check on training progress by plausible feature importance

In [94]:

plt.figure(figsize=(16,9))
plt.stem(TS.combined_feature_names, TS.clf.feature_importances_,'x')
plt.xticks(rotation=90)
plt.ylabel('importance') 
# plt.xticks(rotation = 60)

Out [94]:

Text(0, 0.5, 'importance')

when done, maybe save the classifier and optional the training dict (avoids recalculating the training sets, but might be large)

In [43]:

# TS.pickle_classifier()
pickle.dump(TS.training_dict, open(os.path.join(TS.training_path, pytrain_git_sha+'_training_dict.p'),'wb'))
pickle.dump(TS.combined_feature_names, open(os.path.join(TS.training_path, pytrain_git_sha+'_feature_names.p'),'wb'))

In [46]:

TS.training_dict_full

Out [46]:

{'label_image_y_255_time_40_.tif': (array([[-1.46428571e+02, -4.71285714e+02,  1.71676120e-01, ...,
           1.11674225e+04,  1.06050000e+04,  1.70411936e-01],
         [ 1.54714286e+02, -1.69714286e+02,  1.72399418e-01, ...,
           1.13587042e+04,  1.08410000e+04,  1.71066865e-01],
         [-1.65000000e+02, -3.60142857e+02,  1.71660520e-01, ...,
           1.10823803e+04,  1.03730000e+04,  1.70895247e-01],
         ...,
         [ 1.56428571e+02,  2.81285714e+02,  1.78094529e-01, ...,
           1.19940141e+04,  1.14850000e+04,  1.78078066e-01],
         [ 1.63285714e+02,  1.47000000e+02,  1.77728446e-01, ...,
           1.19303803e+04,  1.13000000e+04,  1.77728446e-01],
         [-2.08571429e+02, -3.50142857e+02,  1.76900536e-01, ...,
           1.15639014e+04,  1.10180000e+04,  1.76805548e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_z_307_time_25_.tif': (array([[-2.46142857e+02, -3.74142857e+02,  1.76275724e-01, ...,
           1.18232958e+04,  1.10610000e+04,  1.74925798e-01],
         [-3.42000000e+02, -4.85857143e+02,  1.76316241e-01, ...,
           1.18405352e+04,  1.11300000e+04,  1.75319188e-01],
         [-1.90428571e+02, -3.74857143e+02,  1.77512776e-01, ...,
           1.21114930e+04,  1.14260000e+04,  1.76005940e-01],
         ...,
         [-1.81714286e+02,  4.80285714e+02,  1.76016286e-01, ...,
           1.10960563e+04,  1.01840000e+04,  1.74601599e-01],
         [ 9.84285714e+01,  6.54285714e+02,  1.75308977e-01, ...,
           1.09472676e+04,  1.03020000e+04,  1.73986060e-01],
         [ 3.74714286e+02,  1.56428571e+02,  1.76564722e-01, ...,
           1.16763662e+04,  1.06990000e+04,  1.75170552e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_x_157_time_5_.tif': (array([[ 6.12714286e+02,  4.12000000e+02,  1.81691079e-01, ...,
           1.18927465e+04,  1.12010000e+04,  1.80019835e-01],
         [ 2.26857143e+02, -1.45142857e+02,  1.81563116e-01, ...,
           1.18323099e+04,  1.09740000e+04,  1.79961364e-01],
         [ 4.12428571e+02,  2.41142857e+02,  1.81434953e-01, ...,
           1.18325352e+04,  1.08200000e+04,  1.79912975e-01],
         ...,
         [ 4.61285714e+02,  1.17900000e+03,  1.82986851e-01, ...,
           1.16421831e+04,  1.06770000e+04,  1.76633232e-01],
         [ 2.95285714e+02,  1.14028571e+03,  1.82246264e-01, ...,
           1.15427042e+04,  1.07200000e+04,  1.76239893e-01],
         [ 2.36571429e+02,  1.05700000e+03,  1.81376676e-01, ...,
           1.14711549e+04,  1.06250000e+04,  1.76194815e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_y_71_time_27_.tif': (array([[-1.35857143e+02,  1.11428571e+01,  1.93519196e-01, ...,
           1.29737324e+04,  1.19270000e+04,  1.91659363e-01],
         [ 1.04714286e+02,  1.25000000e+02,  1.93022896e-01, ...,
           1.29130563e+04,  1.20690000e+04,  1.91258373e-01],
         [ 2.80000000e+01,  1.83000000e+02,  1.92580109e-01, ...,
           1.28927042e+04,  1.18410000e+04,  1.90972621e-01],
         ...,
         [ 1.77571429e+02,  9.50000000e+01,  1.82714666e-01, ...,
           1.17465211e+04,  1.13000000e+04,  1.81494558e-01],
         [ 1.76714286e+02,  2.65714286e+02,  1.83214647e-01, ...,
           1.17702676e+04,  1.11810000e+04,  1.81948427e-01],
         [ 9.18571429e+01,  2.36428571e+02,  1.83764860e-01, ...,
           1.17896901e+04,  1.12550000e+04,  1.82611815e-01]]),
  array([1., 1., 1., ..., 2., 2., 2.])),
 'label_image_x_270_time_50_.tif': (array([[ 7.62285714e+02,  3.66285714e+02,  1.77352465e-01, ...,
           1.17248873e+04,  1.08400000e+04,  1.76199144e-01],
         [ 5.79571429e+02,  1.56857143e+02,  1.77569550e-01, ...,
           1.21831408e+04,  1.15700000e+04,  1.76699324e-01],
         [ 1.67714286e+02, -1.69857143e+02,  1.77088113e-01, ...,
           1.21293662e+04,  1.14650000e+04,  1.76558669e-01],
         ...,
         [ 1.76571429e+02,  3.25571429e+02,  1.78726329e-01, ...,
           1.17209718e+04,  1.11370000e+04,  1.78671522e-01],
         [-1.60285714e+02,  5.28571429e+01,  1.78158173e-01, ...,
           1.14837465e+04,  1.09890000e+04,  1.78069821e-01],
         [-4.00000000e+02, -9.31428571e+01,  1.77880470e-01, ...,
           1.14104507e+04,  1.08650000e+04,  1.77633669e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_x_456_time_0_.tif': (array([[2.07142857e+02, 1.14642857e+03, 1.90178932e-01, ...,
          1.22672817e+04, 1.08070000e+04, 1.77169381e-01],
         [2.49714286e+02, 1.03714286e+03, 1.91092602e-01, ...,
          1.24148451e+04, 1.11670000e+04, 1.79968294e-01],
         [2.52714286e+02, 8.14428571e+02, 1.92194164e-01, ...,
          1.25856901e+04, 1.17740000e+04, 1.83731002e-01],
         ...,
         [2.74285714e+02, 1.28857143e+02, 1.75245020e-01, ...,
          1.16030423e+04, 1.10330000e+04, 1.74652647e-01],
         [1.13857143e+02, 2.27142857e+02, 1.75125835e-01, ...,
          1.12481831e+04, 1.06070000e+04, 1.74509806e-01],
         [3.50428571e+02, 4.15428571e+02, 1.75175942e-01, ...,
          1.12752817e+04, 1.07240000e+04, 1.74449912e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_z_320_time_10_.tif': (array([[8.92857143e+01, 1.87571429e+02, 1.82469480e-01, ...,
          1.22107042e+04, 1.11800000e+04, 1.81068155e-01],
         [6.86000000e+02, 4.32857143e+02, 1.80689966e-01, ...,
          1.20154648e+04, 1.09530000e+04, 1.79250545e-01],
         [3.05285714e+02, 1.21428571e+01, 1.79607390e-01, ...,
          1.19377746e+04, 1.04670000e+04, 1.77957822e-01],
         ...,
         [1.13571429e+02, 2.35428571e+02, 1.76514598e-01, ...,
          1.14960563e+04, 1.07940000e+04, 1.74584281e-01],
         [2.81000000e+02, 4.33000000e+02, 1.76417716e-01, ...,
          1.15155493e+04, 1.07560000e+04, 1.74400619e-01],
         [1.56428571e+02, 1.06857143e+02, 1.76227649e-01, ...,
          1.15197183e+04, 1.08520000e+04, 1.74042724e-01]]),
  array([0., 0., 0., ..., 1., 1., 1.])),
 'label_image_z_564_time_0_.tif': (array([[ 3.13285714e+02,  2.64285714e+02,  1.96006687e-01, ...,
           1.27615915e+04,  1.21920000e+04,  1.94477154e-01],
         [-8.00000000e+01, -1.59285714e+02,  1.95622355e-01, ...,
           1.27900704e+04,  1.22220000e+04,  1.94649750e-01],
         [-9.24285714e+01, -1.13000000e+02,  1.95298843e-01, ...,
           1.27974789e+04,  1.22600000e+04,  1.94626234e-01],
         ...,
         [-4.71428571e+00,  2.92857143e+02,  1.79363554e-01, ...,
           1.12649577e+04,  1.05480000e+04,  1.73660473e-01],
         [ 4.69857143e+02,  7.61000000e+02,  1.79049298e-01, ...,
           1.11457746e+04,  1.03290000e+04,  1.72011410e-01],
         [ 4.55142857e+02,  7.02285714e+02,  1.78584209e-01, ...,
           1.10580845e+04,  1.04030000e+04,  1.70629070e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_z_320_time_30_.tif': (array([[ 2.14857143e+02,  1.52285714e+02,  2.01065715e-01, ...,
           1.33168028e+04,  1.19130000e+04,  1.96774246e-01],
         [-3.23714286e+02,  1.71428571e+00,  2.01161255e-01, ...,
           1.33518310e+04,  1.18630000e+04,  1.96969844e-01],
         [-3.62714286e+02, -9.71428571e+00,  2.01125502e-01, ...,
           1.33800563e+04,  1.20740000e+04,  1.96864119e-01],
         ...,
         [-1.48485714e+03, -6.54714286e+02,  1.77340966e-01, ...,
           1.15403099e+04,  1.06520000e+04,  1.76559372e-01],
         [-1.58242857e+03, -6.48857143e+02,  1.79606184e-01, ...,
           1.18334648e+04,  1.09120000e+04,  1.78930439e-01],
         [-7.85000000e+02,  1.34000000e+02,  1.82066069e-01, ...,
           1.23317606e+04,  1.13080000e+04,  1.81227066e-01]]),
  array([0., 0., 0., ..., 1., 1., 1.])),
 'label_image_x_264_time_29_.tif': (array([[-8.55714286e+01,  4.90000000e+01,  1.95446785e-01, ...,
           1.27986338e+04,  1.23040000e+04,  1.93879462e-01],
         [ 1.22857143e+02,  2.16428571e+02,  1.95580021e-01, ...,
           1.27709296e+04,  1.20220000e+04,  1.93749050e-01],
         [ 1.40000000e+01,  2.14285714e+01,  1.95790805e-01, ...,
           1.28141268e+04,  1.21830000e+04,  1.93688263e-01],
         ...,
         [-4.59857143e+02, -3.76000000e+02,  1.76360268e-01, ...,
           1.13471831e+04,  1.02530000e+04,  1.70212563e-01],
         [-2.67285714e+02, -5.01000000e+02,  1.76233976e-01, ...,
           1.13495915e+04,  1.02230000e+04,  1.70287334e-01],
         [ 9.31428571e+01, -1.57714286e+02,  1.75952877e-01, ...,
           1.13085915e+04,  1.02450000e+04,  1.70298715e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.])),
 'label_image_y_280_time_57_.tif': (array([[-8.58571429e+01,  7.68571429e+01,  1.82389721e-01, ...,
           1.20554085e+04,  1.14720000e+04,  1.81354376e-01],
         [-2.18000000e+02,  1.72000000e+02,  1.82640049e-01, ...,
           1.22886620e+04,  1.16720000e+04,  1.81573052e-01],
         [ 3.24000000e+02,  3.61428571e+02,  1.82303870e-01, ...,
           1.19934366e+04,  1.13380000e+04,  1.81405977e-01],
         ...,
         [ 9.55714286e+01,  2.29714286e+02,  1.71066023e-01, ...,
           1.12507606e+04,  9.87400000e+03,  1.69633661e-01],
         [ 1.79000000e+02,  3.07428571e+02,  1.71033091e-01, ...,
           1.12309014e+04,  1.01770000e+04,  1.69707987e-01],
         [ 2.13714286e+02,  1.01428571e+01,  1.70781424e-01, ...,
           1.11918310e+04,  9.93100000e+03,  1.69562468e-01]]),
  array([0., 0., 0., ..., 1., 1., 1.])),
 'label_image_y_250_time_23_.tif': (array([[ 6.29571429e+02,  2.80571429e+02,  1.83566668e-01, ...,
           1.19565634e+04,  1.11340000e+04,  1.81957534e-01],
         [ 2.94000000e+02,  9.17142857e+01,  1.83483545e-01, ...,
           1.19102113e+04,  1.11850000e+04,  1.81933184e-01],
         [-2.17428571e+02, -4.80142857e+02,  1.83503721e-01, ...,
           1.19503380e+04,  1.11650000e+04,  1.81941278e-01],
         ...,
         [-5.99142857e+02, -3.23000000e+02,  1.75209452e-01, ...,
           1.13515493e+04,  1.06300000e+04,  1.73828147e-01],
         [-2.53428571e+02,  3.61428571e+01,  1.76540683e-01, ...,
           1.16698169e+04,  1.10110000e+04,  1.76169142e-01],
         [ 3.13714286e+02,  4.45285714e+02,  1.77460691e-01, ...,
           1.18957606e+04,  1.12630000e+04,  1.76957676e-01]]),
  array([0., 0., 0., ..., 2., 2., 2.]))}

In [103]:

TS.training_path

Out [103]:

'/mpc/homes/fische_r/NAS/DASCOELY/processing/05_water_GDL_ML/1'

In [50]:

pytrain_git_sha

Out [50]:

'ec4415d'

Segmentation of full data set

remember that you removed features to train the clf

In [37]:

from segmentation import segmentation

In [38]:

classifier_path=os.path.join(training_path, 'classifier.p')
SM = segmentation(training_path = training_path, classifier_path=classifier_path)

In [39]:

# SM.import_lazy_feature_data(IF.result)
# SM.import_classifier(TS.clf)
SM.clf = pickle.load(open(os.path.join(training_path, 'classifier.p'), 'rb'))

In [40]:

clf = SM.clf
clf.n_jobs = 64

if host == 'mpc2053.psi.ch':
    clf.n_jobs = 20

In [41]:

#TODO create result as a stream for every feature set of chunks, i.e stack of 67 feature chunks
# clf = TS.clf
# clf.n_jobs = 32

In [42]:

# loc_feats = [-4, -5, -6]
# ids = np.ones(72, dtype=bool)
# for f in loc_feats:
#     ids[f] = False

In [43]:

# ids

merge time-independent features

In [44]:

test = dask.array.stack([TS.feat_data['feature_stack_time_independent'][:,:,:,0,:]]*da.shape[-1], axis=-2)

2023-04-28 08:57:28,625 - distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
2023-04-28 08:57:33,885 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

In [45]:

test

Out [45]:

	Array	Chunk
Bytes	543.37 GiB	256.00 kiB
Shape	(750, 130, 1700, 88, 5)	(32, 32, 32, 1, 1)
Dask graph	4049760 chunks in 3 graph layers
Data type	float64 numpy.ndarray

In [46]:

feat = dask.array.concatenate([TS.feat_data['feature_stack'], test], axis=-1)

2023-04-28 08:57:55,416 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 08:58:41,121 - distributed.utils_perf - WARNING - full garbage collections took 17% CPU time recently (threshold: 10%)
2023-04-28 08:59:32,401 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:00:31,800 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:01:44,643 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)
2023-04-28 09:03:17,745 - distributed.utils_perf - WARNING - full garbage collections took 16% CPU time recently (threshold: 10%)

Warning:

Output truncated. This notebook contains too many cells to display efficiently.

1.2 MiB Raw Blame History

Segment water in GDL to quantify influence of liquid water on electrochemistry

functionalities for interactive training

fire up dask

Test ROI selection

Data preparation

create dask array

define border to crop to GDL

get data into image filter class

prepare features

Training

set up objects

interactive training

check for existing training sets

re-train with existing label sets. clear the training dictionary if necessary (training_dict)

import training dict of other samples

suggest a new training coordinate

canvas for labeling

inspect labels and training progress

update training set if labels are ok

(already removed from filter functions, not necessary to remove anymore) ignore pixel location as feature as this can be dangerous

train!

go back until happy

check on training progress by plausible feature importance

when done, maybe save the classifier and optional the training dict (avoids recalculating the training sets, but might be large)

Segmentation of full data set

merge time-independent features

1.2 MiB

Raw Blame History