Dev/pyctbgui merge (#960)

* added empty c extension

* added rotation to the decoding

* added color map, options and findex

* minor

* move checks to before acquisition

* added pixel map based decoder

* cleanup

* no thread creation for single thread processing

* added rotation and test to compare

* allow high and low water mark for zmq (also buffer size) for fast readouts

* removed roatation during decoding

* added Transpose to image and invert Y False to invert it

* retains the zoomed state after the first image of gui, catch and display exception if no detector connected

* moved start frame to dockable widget, removed extra frame number label, moved current measurement also to dockable widget, hide frame plot entirely when showing patternviewer

* first image dependin on which plot

* remember settings of main window size and position, dockewidget if docked, its size and posisiotn as well, then update it next time the gui is opened

* change in comment

* using c decoder for moench 04 and matterhorn

* catch exception from invalid image from decoder

* clean up

* update row and col when choosing image type, neeeded to show values

* fix for previous PR

* disable resetting colormap values
keep the range selected for every new acquisition

* fix typos + tested on virtual matterhorn

* minor print

* refactored Slow ADCs Tab

* refactored DAC tab

* refactored power supplies

* refactored signals tab

* refactored transceiver tab

* fix typo

* fix typo2

* remove commented code

* delete commented code

* delete commented code

* delete commented signals code

* remove commented code for transceiver tab

* refactor adc tab

* refactor Pattern Tab

* Refactor transceivers tab (PR#5) (#118)

* refactored transceiver tab

* remove commented code for transceiver tab

---------

Co-authored-by: Erik Frojdh <erik.frojdh@gmail.com>

* refactor adc tab (PR#6) (#119)


* refactor adc tab

* refactored Plot and Acquisition Tabs

* fix the regression issue

* restructure project files

* applying singleton and renaming tabs to services

* working install using pip

* applies singleton to tab classes and remove CI erros

* added pyzmq and pillow

* remove the singleton implementation and keep changes

* fix merge errors in mainWindow

* moved misplaced init file

* rename service to tab

* reorganize imports

* iterate over tabs

* reorder tabs

* add slowadc to the list

* saving changes (buggy)
power supply ui not showing in the gui

* split power supply tab

* fixed tests

* add hardcoded values to defines file

* fix error

* separate power supply

* fix errors for powerSuppliesTab

* split dacs

* split slow adcs

* split signals tab

* added tests for bit_utils

* add slowAdc class to defines

* split transceiver ui file

* split adc.ui

* split pattern ui file

* split plot and acquisition ui file

* added basic test for parsing bit names

* removed redundant code in read_alias_file

* fix dacs ui position

* testing for correct exception

* cmd and args at split

* group radio buttons

* fix comments from PR#1

* show legend

* added python version and dev requirements to setup.py

* fix dac issue

* moved _decoder into pkg

* added inplace build

* removed clear

* fixed dependencies

* make tests run without slsdet

* updated name of action

* define colcount

* fixed loading of alias file

* add yapf and ruff

* apply formatting

* fix E and F rules

* add more ruff rules

* change variable name

* squashing gh debugging commits and add pre-commit

* update label values to mv units

* add hook for yapf

* reconfigure yapf precommit hook

* add format and check_format to makefile

* change gh actions

* update readme

* added check_format

* WIP

* added linting in github action

* updated readme]

* add more control for color choice

* remove useless file

* fix un-updated line after refactoring Defines
BIT0_31_MASK is not found in Defines.signals

* visually improve the interface

* fix last commit

* add only selected plots for legend

* add hide legend button

* change hide legend to show legend
checkbox show legend is checked by default

* add support for saving in numpy

* solve conversations

* fix acq index offset

* fix browse button in pattern error

* fix other browse button errors

* finish tests and add usage.md

* remove buffer

* add file,numpy-like interface and tests

* remove useless .npy files

* subscriptible npz files

* remove useless files

* remove repetetive tests

* save changes

* add mode r+, add with support,remove logging

* remove offset of acqIndex between raw and numpy saving

* fix only saving last frame

* save signals of multiple devices

* add comments and move condition for clearer code

* fix bug when vieweing pattern file

* iterate over enabled and plotted plots

* add padestal substraction to transceiver and analog data

* init pedestal frames to detector.frames

* restore old exception

* add pedestal substraction for digital signals

* remove frames spinbox from plotTab

* remove comments and use str instead of Path

* avoid saving all frames

* correct exception and log error's trace

* add gui tests

* add waveform test

* add pedestal test

* refactor by using fixtures

* add tests for moench analog and pattern

* add pytest-qt to dependencies

* add save and load gui parameters

* remove nohup file

* fix old bug IndexError

* save plot type

* a

* handle canceling load, loading matterhorn pedestal for moench

* remove comparing .png files for pattern test

* save plot type

* red error on status bar when shape mismatch for loaded pedestal

* fix makefile and docstrings

* fix PRs conversation

* move code into different function

* fix wrong function names for power supply

* removed old ctbgui

* removed unnecessary files

---------

Co-authored-by: Erik Frojdh <erik.frojdh@gmail.com>
Co-authored-by: Braham Bechir <braham_b@pc11979.psi.ch>
Co-authored-by: Bechir <bechir.braham@psi.ch>
Co-authored-by: Bechir <bechir.brahem420@gmail.com>
This commit is contained in:
2024-09-10 16:00:04 +02:00
committed by GitHub
parent c6477e0ed6
commit 5b61ff24bb
93 changed files with 25399 additions and 7292 deletions

View File

@@ -0,0 +1,129 @@
BIT0 gHG
BIT1 DGS6 1
BIT2 pulse
BIT3 DGS7 1
BIT4 clearSSR
BIT5 DGS8 1
BIT6 inSSR
BIT7 DGS9 1
BIT8 clkSSR
BIT9 clk_fb
BIT10 prechargeConnect
BIT11 clk
BIT12 clear
BIT13 shr_out_15
BIT14 bypassCDS
BIT15 Sto2
BIT16 ENprechPre
BIT17 Sto0_bot
BIT18 pulseOFF
BIT19 Sto1
BIT20 BYPASS
BIT21 connCDS
BIT22 Dsg3
BIT23 OutSSRbot
BIT24 resCDS
BIT25 res
BIT26 shr_out_31
BIT27 Dsg1
BIT28 READ
BIT29 Sto0_top
BIT30 resDGS
BIT31 shr_in
BIT48 DGS23 1
BIT49 DGS31 1
BIT50 DGS22 1
BIT51 DGS30 1
BIT52 DGS21 1
BIT53 DGS29 1
BIT54 DGS20 1
BIT55 DGS28 1
BIT56 DGS19 1
BIT57 DGS27 1
BIT58 DGS18 1
BIT59 DGS26 1
BIT60 DGS17 1
BIT61 DGS25 1
BIT62 DGS16 1
BIT63 DGS24 1
#BIT62 dbit_ena
#BIT63 adc_ena
SENSE3 T_boa.(C)
SENSE1 Va+
SENSE6 Vdd_ps
SENSE4 Vsh
SENSE2 Vcc_int
SENSE0 Vcc_iochip
SENSE7 Vcc1.8A
SENSE5 Vcc1.8D
DAC0 VprechPre
DAC1 prechargeV
DAC2 vb_sda
DAC3 vbp_colbuf
DAC4 vcasc_SFP
DAC5 ibias_SFP
DAC6 vIpreCDS
DAC7 vin_com_bot
DAC8 vIpre
DAC9 s2dVinTest
DAC10 VPH
DAC11 vIpreDGS
DAC12 vrefDGS
DAC13 empty13
DAC14 vin_com_top
DAC15 VPL
DAC16 vout_com
DAC17 empty17
ADC0 SCOL9 1
ADC1 SCOL8 1
ADC2 SCOL11 1
ADC3 SCOL10 1
ADC4 SCOL13 1
ADC5 SCOL12 1
ADC6 SCOL15 1
ADC7 SCOL14 1
ADC8 SCOL1 1
ADC9 SCOL0 1
ADC10 SCOL3 1
ADC11 SCOL2 1
ADC12 SCOL5 1
ADC13 SCOL4 1
ADC14 SCOL7 1
ADC15 SCOL6 1
ADC16 SCOL23 1
ADC17 SCOL22 1
ADC18 SCOL21 1
ADC19 SCOL20 1
ADC20 SCOL19 1
ADC21 SCOL18 1
ADC22 SCOL17 1
ADC23 SCOL16 1
ADC24 SCOL31 1
ADC25 SCOL30 1
ADC26 SCOL29 1
ADC27 SCOL28 1
ADC28 SCOL27 1
ADC29 SCOL26 1
ADC30 SCOL25 1
ADC31 SCOL24 1
VA VddA
VB VddPre
VIO VddD
#location of the pattern compiler
#PATCOMPILER /afs/psi.ch/project/sls_det_software/slsDetectorsPackage/slsDetectorSoftware/patternGenerator/generate.sh
PATCOMPILER /afs/psi.ch/project/sls_det_software/sabina/slsDetectorPackage/ctbGui/patternGenerator/generate.sh
#location of the pattern to run
#PATFILE /afs/psi.ch/user/d/dinapoli/jctb/moench02_rob/patterns_newctb/moench02_pulsing_marco.p
#PATFILE /afs/psi.ch/project/sls_det_software/sabina/moench04/patterns/moench04_readout_g4.p
#PATFILE /afs/psi.ch/user/m/maliakal_d/setupfiles/ctb_julian.pat
#PATFILE /tmp/bla.p
#PATFILE /afs/psi.ch/project/mythen/Moench04/moench04_pulsing_readout_g4.p

View File

@@ -0,0 +1,63 @@
import pytest
from pathlib import Path
from pyctbgui import alias_utility as at
def test_read_non_existing_file_throws():
with pytest.raises(FileNotFoundError):
at.read_alias_file('saijvcaiewjrvijaerijvaeoirvjveiojroiajgv')
def test_parse_sample_file():
fpath = Path(__file__).parent / 'data/moench04_start_trigger.alias'
bit_names, bit_plots, bit_colors, adc_names, adc_plots, adc_colors, dac_names, sense_names, power_names, \
pat_file_name = at.read_alias_file(fpath)
assert len(bit_names) == 64
true_names = [
'gHG', 'DGS6', 'pulse', 'DGS7', 'clearSSR', 'DGS8', 'inSSR', 'DGS9', 'clkSSR', 'clk_fb', 'prechargeConnect',
'clk', 'clear', 'shr_out_15', 'bypassCDS', 'Sto2', 'ENprechPre', 'Sto0_bot', 'pulseOFF', 'Sto1', 'BYPASS',
'connCDS', 'Dsg3', 'OutSSRbot', 'resCDS', 'res', 'shr_out_31', 'Dsg1', 'READ', 'Sto0_top', 'resDGS', 'shr_in',
None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, 'DGS23',
'DGS31', 'DGS22', 'DGS30', 'DGS21', 'DGS29', 'DGS20', 'DGS28', 'DGS19', 'DGS27', 'DGS18', 'DGS26', 'DGS17',
'DGS25', 'DGS16', 'DGS24'
]
true_plot = [
None, True, None, True, None, True, None, True, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None, None, True, True, True, True, True, True,
True, True, True, True, True, True, True, True, True, True
]
# Make sure we didn't mess up anything in this file
assert len(true_names) == len(bit_names)
for value, reference in zip(bit_names, true_names):
assert value == reference
assert len(true_plot) == 64
for value, reference in zip(bit_plots, true_plot):
assert value == reference
# sense names
true_sese = ['Vcc_iochip', 'Va+', 'Vcc_int', 'T_boa.(C)', 'Vsh', 'Vcc1.8D', 'Vdd_ps', 'Vcc1.8A']
assert len(true_sese) == 8
assert true_sese == sense_names
# ADC
true_adc = [
'SCOL9', 'SCOL8', 'SCOL11', 'SCOL10', 'SCOL13', 'SCOL12', 'SCOL15', 'SCOL14', 'SCOL1', 'SCOL0', 'SCOL3',
'SCOL2', 'SCOL5', 'SCOL4', 'SCOL7', 'SCOL6', 'SCOL23', 'SCOL22', 'SCOL21', 'SCOL20', 'SCOL19', 'SCOL18',
'SCOL17', 'SCOL16', 'SCOL31', 'SCOL30', 'SCOL29', 'SCOL28', 'SCOL27', 'SCOL26', 'SCOL25', 'SCOL24'
]
assert len(true_adc) == 32
assert true_adc == adc_names
def test_single_value_parse_gives_exception():
# Check that we get the correct exception
with pytest.raises(Exception, match=r'Alias file parsing failed'):
at.parse_alias_lines(['hej'])

View File

@@ -0,0 +1,27 @@
import numpy as np
from pyctbgui import bit_utils as bt
def test_set_bit():
num = np.int32(0)
num = bt.set_bit(num, 5)
assert num == 2**5
def test_remove_bit():
num = np.int32(2**5)
num = bt.remove_bit(num, 5)
assert num == 0
def test_manipulate_bit():
num = np.int32(0)
num = bt.manipulate_bit(True, num, 5) # True means setting bit
assert num == 2**5
def test_manipulate_bit2():
num = np.int32(2**8)
num = bt.manipulate_bit(False, num, 8) # False means clearing the bit
assert num == 0

View File

@@ -0,0 +1,42 @@
import numpy as np
import sys
from pyctbgui.utils import decoder
from pyctbgui.utils.pixelmap import moench04_analog, matterhorn_transceiver
def test_simple_decode():
pixel_map = np.zeros((2, 2), dtype=np.uint32)
pixel_map.flat = [0, 1, 2, 3]
raw_data = np.zeros(4, dtype=np.uint16)
raw_data.flat = [8, 9, 10, 11]
data = decoder.decode(raw_data, pixel_map)
assert data[0, 0] == 8
assert data[0, 1] == 9
assert data[1, 0] == 10
assert data[1, 1] == 11
# Make sure we didn't mess up the reference counts
assert sys.getrefcount(data) == 2
assert sys.getrefcount(raw_data) == 2
assert sys.getrefcount(pixel_map) == 2
def test_compare_python_and_c_decoding():
"""
Check that the python and C implementation give the same results
provides a regression test in case we change the C implementation
"""
pixel_map = moench04_analog()
raw_data = np.arange(400 * 400, dtype=np.uint16)
c_data = decoder.decode(raw_data, pixel_map)
py_data = decoder.moench04(raw_data)
assert (c_data == py_data).all()
pixel_map = matterhorn_transceiver()
raw_data = np.arange(48 * 48, dtype=np.uint16)
c_data = decoder.decode(raw_data, pixel_map)
py_data = decoder.matterhorn(raw_data)
assert (c_data == py_data).all()

View File

@@ -0,0 +1,160 @@
import filecmp
from pathlib import Path
import pytest
from pyctbgui.utils.numpyWriter.npy_writer import NumpyFileManager
import numpy as np
def test_create_new_file(tmp_path):
npw = NumpyFileManager(tmp_path / 'tmp.npy', 'w', (400, 400), np.int32)
npw.writeOneFrame(np.ones([400, 400], dtype=np.int32))
npw.writeOneFrame(np.ones([400, 400], dtype=np.int32))
npw.writeOneFrame(np.ones([400, 400], dtype=np.int32))
npw.writeOneFrame(np.ones([400, 400], dtype=np.int32))
npw.close()
arr = np.load(tmp_path / 'tmp.npy')
assert arr.dtype == np.int32
assert arr.shape == (4, 400, 400)
assert np.array_equal(arr, np.ones([4, 400, 400], dtype=np.int32))
np.save(tmp_path / 'tmp2.npy', np.ones([4, 400, 400], dtype=np.int32))
assert filecmp.cmp(tmp_path / 'tmp.npy', tmp_path / 'tmp2.npy')
def test_open_old_file(tmp_path):
npw = NumpyFileManager(tmp_path / 'tmp.npy', 'w', (4000, ), np.float32)
npw.writeOneFrame(np.ones(4000, dtype=np.float32))
npw.writeOneFrame(np.ones(4000, dtype=np.float32))
npw.close()
npw2 = NumpyFileManager(tmp_path / 'tmp.npy', 'r+')
assert npw2.frameCount == 2
assert npw2.frameShape == (4000, )
assert npw2.dtype == np.float32
npw2.writeOneFrame(np.ones(4000, dtype=np.float32))
del npw2
np.save(tmp_path / 'tmp2.npy', np.ones([3, 4000], dtype=np.float32))
assert filecmp.cmp(tmp_path / 'tmp.npy', tmp_path / 'tmp2.npy')
@pytest.mark.parametrize('mode', ['w', 'x'])
def test_init_parameters2(mode, tmp_path):
# test opening files with missing parameters for write
with pytest.raises(AssertionError):
NumpyFileManager(tmp_path / 'abaababababa.npyx', mode)
with pytest.raises(AssertionError):
NumpyFileManager(tmp_path / 'abaababababa.npyx2', mode, frameShape=(12, 34))
with pytest.raises(AssertionError):
NumpyFileManager(tmp_path / 'abaababababa.npyx3', mode, dtype=np.int64)
# opening new file with required parameters (this should work)
NumpyFileManager(tmp_path / 'abaababababa.npyx3', mode, dtype=np.int64, frameShape=(6, 6))
assert Path.is_file(tmp_path / 'abaababababa.npyx3')
def test_init_parameters(tmp_path):
with pytest.raises(TypeError):
NumpyFileManager()
# test opening file that does not exist
with pytest.raises(FileNotFoundError):
NumpyFileManager(tmp_path / 'abaababababa.npyx')
with pytest.raises(FileNotFoundError):
NumpyFileManager(tmp_path / 'abaababababa.npyx2', frameShape=(12, 34))
with pytest.raises(FileNotFoundError):
NumpyFileManager(tmp_path / 'abaababababa.npyx3', dtype=np.int64)
with pytest.raises(FileNotFoundError):
NumpyFileManager(tmp_path / 'abaababababa.npyx3', dtype=np.int64, frameShape=(6, 6))
# re-opening the same file
NumpyFileManager(tmp_path / 'abaababababa.npyx3', 'w', dtype=np.int64, frameShape=(6, 6))
NumpyFileManager(tmp_path / 'abaababababa.npyx3')
# re-opening the file with wrong parameters
with pytest.raises(AssertionError):
NumpyFileManager(tmp_path / 'abaababababa.npyx3', frameShape=(6, 2))
with pytest.raises(AssertionError):
NumpyFileManager(tmp_path / 'abaababababa.npyx3', dtype=np.int32)
with pytest.raises(AssertionError):
NumpyFileManager(tmp_path / 'abaababababa.npyx3', dtype=np.float32, frameShape=(5, 5))
# test resetting an existing file
npw = NumpyFileManager(tmp_path / 'tmp4.npy', 'w', dtype=np.float32, frameShape=(5, 5))
npw.writeOneFrame(np.ones((5, 5), dtype=np.float32))
npw.close()
assert np.load(tmp_path / 'tmp4.npy').shape == (1, 5, 5)
npw = NumpyFileManager(tmp_path / 'tmp4.npy', 'w', dtype=np.int64, frameShape=(7, 7))
npw.flush()
assert np.load(tmp_path / 'tmp4.npy').shape == (0, 7, 7)
# test adding frames with the wrong shape to an existing file
with pytest.raises(ValueError, match=r'frame shape given \(9, 4, 4\) '):
npw.writeOneFrame(np.ones((9, 4, 4)))
def test_get_item(tmp_path):
rng = np.random.default_rng(seed=42)
arr = rng.random((1000, 20, 20))
npw = NumpyFileManager(tmp_path / 'tmp.npy', 'w', frameShape=(20, 20), dtype=arr.dtype)
for frame in arr:
npw.writeOneFrame(frame)
assert np.array_equal(npw[50:100], arr[50:100])
assert np.array_equal(npw[0:1], arr[0:1])
assert np.array_equal(npw[0:10000], arr)
assert np.array_equal(npw[999:1000], arr[999:1000])
assert np.array_equal(npw[999:1005], arr[999:1000])
assert np.array_equal(npw[49:300], arr[49:300])
assert np.array_equal(npw[88:88], arr[88:88])
assert np.array_equal(npw[0:0], arr[0:0])
assert np.array_equal(npw[0:77], arr[0:77])
assert np.array_equal(npw[77:0], arr[77:0])
with pytest.raises(NotImplementedError):
npw[-1:-3]
with pytest.raises(NotImplementedError):
npw[10:20:2]
with pytest.raises(NotImplementedError):
npw[-5:-87:5]
with pytest.raises(NotImplementedError):
npw[-5:-87:-5]
with pytest.raises(NotImplementedError):
npw.readFrames(-1, -4)
with pytest.raises(NotImplementedError):
npw.readFrames(0, -77)
with pytest.raises(NotImplementedError):
npw.readFrames(-5, -5)
def test_file_functions(tmp_path):
rng = np.random.default_rng(seed=42)
arr = rng.random((1000, 20, 20))
npw = NumpyFileManager(tmp_path / 'tmp.npy', 'w', frameShape=(20, 20), dtype=arr.dtype)
for frame in arr:
npw.writeOneFrame(frame)
assert np.array_equal(npw.read(10), arr[:10])
assert np.array_equal(npw.read(10), arr[10:20])
assert np.array_equal(npw.read(10), arr[20:30])
npw.readFrames(500, 600)
assert np.array_equal(npw.read(10), arr[30:40])
npw.readFrames(0, 2)
assert np.array_equal(npw.read(100), arr[40:140])
npw.writeOneFrame(arr[700])
assert np.array_equal(npw.read(5), arr[140:145])
npw.seek(900)
assert np.array_equal(npw.read(20), arr[900:920])
npw.seek(5)
npw.readFrames(500, 600)
assert np.array_equal(npw.read(10), arr[5:15])
def test_with_statement(tmp_path):
arr = np.ones((5, 5))
with NumpyFileManager(tmp_path / 'tmp.npy', 'w', (5, 5), arr.dtype) as npw:
npw.writeOneFrame(arr)
np.save(tmp_path / 'tmp2.npy', np.expand_dims(arr, 0))
assert filecmp.cmp(tmp_path / 'tmp2.npy', tmp_path / 'tmp.npy')

View File

@@ -0,0 +1,194 @@
import filecmp
from pathlib import Path
import pytest
from pyctbgui.utils.numpyWriter.npy_writer import NumpyFileManager
import numpy as np
from pyctbgui.utils.numpyWriter.npz_writer import NpzFileWriter
@pytest.mark.parametrize('compressed', [True, False])
def test_incremental_npz(compressed, tmp_path):
arr1 = np.ones((10, 5, 5))
arr2 = np.zeros((10, 5, 5), dtype=np.int32)
arr3 = np.ones((10, 5, 5), dtype=np.float32)
with NpzFileWriter(tmp_path / 'tmp.npz', 'w', compress_file=compressed) as npz:
npz.writeArray('adc', arr1)
npz.writeArray('tx', arr2)
npz.writeArray('signal', arr3)
npzFile = np.load(tmp_path / 'tmp.npz')
assert sorted(npzFile.files) == sorted(('adc', 'tx', 'signal'))
assert np.array_equal(npzFile['adc'], np.ones((10, 5, 5)))
assert np.array_equal(npzFile['tx'], np.zeros((10, 5, 5), dtype=np.int32))
assert np.array_equal(npzFile['signal'], np.ones((10, 5, 5), dtype=np.float32))
if compressed:
np.savez_compressed(tmp_path / 'tmp2.npz', adc=arr1, tx=arr2, signal=arr3)
else:
np.savez(tmp_path / 'tmp2.npz', adc=arr1, tx=arr2, signal=arr3)
assert filecmp.cmp(tmp_path / 'tmp2.npz', tmp_path / 'tmp.npz')
@pytest.mark.parametrize('compressed', [True, False])
def test_zipping_npy_files(compressed, tmp_path):
data = {
'arr1': np.ones((10, 5, 5)),
'arr2': np.zeros((10, 5, 5), dtype=np.int32),
'arr3': np.ones((10, 5, 5), dtype=np.float32)
}
filePaths = [tmp_path / (file + '.npy') for file in data.keys()]
for file in data:
np.save(tmp_path / (file + '.npy'), data[file])
NpzFileWriter.zipNpyFiles(tmp_path / 'file.npz', filePaths, list(data.keys()), compressed=compressed)
npz = np.load(tmp_path / 'file.npz')
assert npz.files == list(data.keys())
for file in data:
assert np.array_equal(npz[file], data[file])
if compressed:
np.savez_compressed(tmp_path / 'numpy.npz', **data)
else:
np.savez(tmp_path / 'numpy.npz', **data)
numpyz = np.load(tmp_path / 'numpy.npz')
for file in data:
assert np.array_equal(numpyz[file], npz[file])
assert npz.files == numpyz.files
# different files :(
# for some reason numpy savez (most likely a trick with zipfile library) doesn't write the time
# of last modification
# assert filecmp.cmp(prefix / 'numpy.npz', prefix / 'file.npz')
@pytest.mark.parametrize('compressed', [True, False])
def test_npy_writer_with_zipfile_in_init(compressed, tmp_path):
npz = NpzFileWriter(tmp_path / 'tmp.npz', 'w', compress_file=compressed)
rng = np.random.default_rng(42)
arr = rng.random((8000, 5, 5))
npz.writeArray('adc', arr)
with npz.file.open('adc.npy', mode='r') as outfile:
npw = NumpyFileManager(outfile)
assert np.array_equal(npw.readFrames(720, 7999), arr[720:7999])
def test_compression(tmp_path):
arr = np.zeros((1000, 5, 5), dtype=np.int32)
with NpzFileWriter(tmp_path / 'tmp.npz', 'w', compress_file=True) as npz:
npz.writeArray('adc', arr)
np.savez(tmp_path / 'tmp2.npz', adc=arr)
assert Path(tmp_path / 'tmp2.npz').stat().st_size > Path(tmp_path / 'tmp.npz').stat().st_size
@pytest.mark.parametrize('compressed', [True, False])
@pytest.mark.parametrize('isPath', [True, False])
@pytest.mark.parametrize('deleteOriginals', [True, False])
def test_delete_files(compressed, isPath, deleteOriginals, tmp_path):
data = {
'arr1': np.ones((10, 5, 5)),
'arr2': np.zeros((10, 5, 5), dtype=np.int32),
'arr3': np.ones((10, 5, 5), dtype=np.float32)
}
filePaths = [tmp_path / (file + '.npy') for file in data.keys()]
for file in data:
np.save(tmp_path / (file + '.npy'), data[file])
path = tmp_path / 'file.npz'
path = str(path) if isPath else path
NpzFileWriter.zipNpyFiles(path,
filePaths,
list(data.keys()),
deleteOriginals=deleteOriginals,
compressed=compressed)
if deleteOriginals:
for file in filePaths:
assert not Path.exists(file)
else:
for file in filePaths:
assert Path.exists(file)
def test_npz_read_frames(tmp_path):
rng = np.random.default_rng(seed=42)
arr1 = rng.random((10, 5, 5))
with NpzFileWriter(tmp_path / 'tmp.npz', 'w') as npz:
npz.writeArray('adc', arr1)
with NpzFileWriter(tmp_path / 'tmp.npz', 'r') as npz:
frames = npz.readFrames('adc', 5, 8)
assert np.array_equal(frames, arr1[5:8])
def test_file_modes(tmp_path):
rng = np.random.default_rng(seed=42)
arr1 = rng.random((10, 5, 5))
# check reopening with mode w
with NpzFileWriter(tmp_path / 'tmp.npz', 'w') as npz:
npz.writeArray('adc', arr1)
with NpzFileWriter(tmp_path / 'tmp.npz', 'w') as npz:
assert npz.file.namelist() == []
# check reopening with mode x
with NpzFileWriter(tmp_path / 'tmp.npz', 'w') as npz:
npz.writeArray('adc', arr1)
with pytest.raises(FileExistsError):
with NpzFileWriter(tmp_path / 'tmp.npz', 'x'):
pass
# check reopening with mode r
with NpzFileWriter(tmp_path / 'tmp.npz', 'r') as npz:
assert np.array_equal(npz.readFrames('adc', 4, 6), arr1[4:6])
with pytest.raises(ValueError, match=r'write\(\) requires mode \'w\'\, \'x\'\, or \'a\''):
npz.writeArray('adc2', arr1)
@pytest.mark.filterwarnings('ignore::UserWarning')
def test_file_mode_a(tmp_path):
rng = np.random.default_rng(seed=42)
arr1 = rng.random((10, 5, 5))
# check reopening with mode a
with NpzFileWriter(tmp_path / 'tmp.npz', 'w') as npz:
npz.writeArray('adc', arr1)
with NpzFileWriter(tmp_path / 'tmp.npz', 'a') as npz:
npz.writeArray('adc2', arr1)
npz.writeArray('adc', arr1)
@pytest.mark.parametrize('compressed', [True, False])
def test_get_item(compressed, tmp_path):
rng = np.random.default_rng(seed=42)
arr1 = rng.random((10, 5, 5))
arr2 = rng.random((3, 2, 2))
# check reopening with mode a
npz = NpzFileWriter(tmp_path / 'tmp.npz', 'w', compress_file=compressed)
npz.writeArray('adc1', arr1)
npz.writeArray('adc2', arr2)
npz.writeArray('adc3', arr1)
assert np.array_equal(npz['adc1'].read(3), arr1[:3])
assert np.array_equal(npz['adc2'].read(1), arr2[:1])
assert np.array_equal(npz['adc2'].read(1), arr2[1:2])
assert np.array_equal(npz['adc2'].read(1), arr2[2:3])
assert np.array_equal(npz['adc1'].read(3), arr1[3:6])
assert np.array_equal(npz['adc1'].read(3), arr1[6:9])
@pytest.mark.parametrize('compressed', [True, False])
def test_namelist(compressed, tmp_path):
rng = np.random.default_rng(seed=42)
arr1 = rng.random((10, 5, 5))
arr2 = rng.random((3, 2, 2))
# check reopening with mode a
npz = NpzFileWriter(tmp_path / 'tmp.npz', 'w', compress_file=compressed)
npz.writeArray('adc1', arr1)
npz.writeArray('adc2', arr2)
npz.writeArray('adc3', arr1)
assert npz.namelist() == ['adc1', 'adc2', 'adc3']