slsDetectorPackage/pyctbgui/pyctbgui/utils/numpyWriter/usage.md

Using numpyWriter module

concept

numpyWriter is used to write and load huge numpy arrays that can't be fully loaded in RAM. It is designed to write frames of a constant shape (defined by user) and incrementally add to .npy and .npz files without accessing all of its contents

NumpyFileManager

class to handle writing in .npy files frame by frame. its positional parameter file can be of type: str,pathlib.Path, zipfile.ZipExtFile. This way we can use NumpyFileManager to open files by getting their path or in read mode it can receiver file-like objects to read their data.

the complexity of initializing from file-like objects is added to be able to read from .npz files which are simply a zip of .npy files. Furthermore now we can save our files .npz files and read from them (even when compressed (⊙_⊙) ) without loading the whole .npy or .npz in memory.

NpzFileWriter

class used to handle .npz file functionalities. it can zip existing .npy files, write a whole array in an .npz file without loading the whole .npz in memory, and read frames from .npy files inside the .npz file

Usage

# create .npy file
npw = NumpyFileManager('file.npy', 'w', (400, 400), np.int32)
npw.addFrame(np.ones([400, 400], dtype=np.int32))
npw.close()

# read frames from existing .npy file
npw = NumpyFileManager('file.npy')
# if arr is stored in the .npy file this statement will return arr[50:100]
npw.readFrames(50, 100)

# Numpy like interface
# NumpyFileManager is also subscriptable
npw[50:100] # returns the array arr[50:100]
npw[0] # returns the array arr[0]

# File like interface
# the npw class's cursors is initialized on the first frame
npw.read(5) # reads five frames and updates the cursor
npw.seek(99) # updates the cursor to point it to the 99-th frame

# to ensure that files are written to disk
npw.flush()

# zip existing .npy files (stored on disk) 
# filePaths: the paths to .npy files
# keys: name of the arrays inside of the .npz file
NpzFileWriter.zipNpyFiles('file.npz', filePaths, keys, compressed=True)

# add numpy arrays incrementally to a .npz file
with NpzFileWriter('tmp.npz', 'w', compress_file=True) as npz:
    npz.writeArray('adc', arr1)
    npz.writeArray('tx', arr2)

# read frames from adc.npy inside of tmp.npz
with NpzFileWriter('tmp.npz', 'r') as npz:
    frames = npz.readFrames('adc', 5, 8)

# NpzFileWriter is also subscriptable and returns a NumpyFileManager initialized 
# to open the the file with the given key inside the .npz file
npz = NpzFileWriter('tmp.npz', 'r')
npz.writeArray('adc', arr1)


npz['adc'] # returns a NumpyFileManager
npz['adc'][50:100] # returns the array from 50 to 100 
# note once a NumpyFileManager instance is created internally NpzFileWriter stores it
# this is done to avoid opening and closing the same file 
# also file-like interface can be used 
npz['adc'].read(5) # returns arr[:5]
npz['adc'].seek(100) # updates the cursor
npz['adc'].read(2) # returns arr[100:2]