The library is used as the low-level interface between C++ host application and Jungfraujoch FPGA card. It provides wrapper over kernel driver ioctl calls. Preferred way to use Jungfraujoch is via the full jfjoch_broker application, however for more tailored solutions - one can use directly API described below.

Configure the detector

To use the API, one needs to configure the detector via slsDetectorPackage for data acquisition. Besides the usual configuration used for JUNGFRAU, the following settings are necessary:

Detector frame numbers must be restarted to start always from 1 using the following command-line interface command (or C++/Python equivalent):

sls_detector_put nextframenumber 1

Detector has to be configured to measure 6 frames more than needed for data acquisition, for the acquisition to stop automatically
Set destination MAC and IPv4 address to be consistent with ones for the card. At the time being, all the ethernet interfaces of the card (100G and 4x10G) share the same MAC/IPv4 address, though this might change in the future. UDP port is not used.
Each module has to have its sequential number assigned through column field in the detector header, which has to be set to 2 * module number. This is currently done by writing module number * 2 * 65536 to register 0x7C of the module. IMPORTANT: Module numbering for each FPGA card is independent and has to start for 1. With 4 FPGA cards and 16 modules, there will be four modules with number 1, four with number 2, etc.
The card support both 1 and 2 network interfaces per JUNGFRAU module.

Designing with the API

For each FPGA card, one needs to instantiate JungfraujochDevice object, using device name of format /dev/jfjoch<number> and read/write parameter. Only one process can open the device with write access, while multiple processes might open the device for read in parallel.

Configuring network

The first step for using the card is configuring network. To use network, one needs to select data source with JungfraujochDevice::SetDataSource() function - the options are no data source (default), 100G, second 100G and internal generator. Next, addresses need to be configured for the network stack.

For 100G only one MAC is enabled by default. Second one has to be enabled with function JungfraujochDevice::EnableSecond100GMAC(). However, as both 100 Gbit/s cannot operate simultaneously, one needs to choose one with the JungfraujochDevice::SetDataSource() function.

The card will receive MAC address automatically based on Xilinx assigned number, but IPv4 address has to be configured with JungfraujochDevice::SetIPv4Address() function. IPv4 address can be set independently for each network interface.

Mapping kernel buffers

Next, kernel buffers need to be mapped to the user space. These buffers are allocated with memory physically continuous, simplifying operation of the card and the driver. Count of these buffers can be checked with JungfraujochDevice::GetBufferCount() function. Buffers can be mapped with JungfraujochDevice::MapKernelBuffer() function and deallocated with JungfraujochDevice::UnmapKernelBuffer() functions. Structure of the kernel buffer is DeviceOutput and described in pcie_driver/jfjoch_fpga.h header file.

Uploading calibration

Uploading calibration goes with two steps:

Copy the calibration data into one of kernel buffers mapped with JungfraujochDevice::MapKernelBuffer(). It requires to cast DeviceOutput::pixels element into a particular type (float or uint16_t).
Execute JungfraujochDevice::LoadCalibration() function for the card to upload the data. This function is synchronous and will return when the calibration is uploaded.

The following can be uploaded:

destination	Bit-width	Data type (per pixel)	Description	Allowed value
LOAD_CALIBRATION_DEST_GAIN_G0	32	float	1/(gain parameter)	0 - 0.125 (0 = mask pixel)
LOAD_CALIBRATION_DEST_GAIN_G1	32	float	1/(gain parameter)	-16.0 - 0.0 (0 = mask pixel)
LOAD_CALIBRATION_DEST_GAIN_G2	32	float	1/(gain parameter)	-64.0 - 0.0 (0 = mask pixel)
LOAD_CALIBRATION_DEST_PEDESTAL_G0	16	uint16_t	pedestal, no fractional part	0 - 16384 (16384 = mask pixel)
LOAD_CALIBRATION_DEST_PEDESTAL_G1	16	uint16_t	pedestal, no fractional part	0 - 16384 (16384 = mask pixel)
LOAD_CALIBRATION_DEST_PEDESTAL_G2	16	uint16_t	pedestal, no fractional part	0 - 16384 (16384 = mask pixel)
LOAD_CALIBRATION_DEST_INTEGRATION_MAP	16	uint16_t	integration bin	0 - 1023 (higher values = ignore pixel)
LOAD_CALIBRATION_DEST_INTEGRATION_WEIGHTS	32	float	weight for the pixel	0.0 - 2.0
LOAD_CALIBRATION_DEST_FRAME_GEN	16	uint16_t	raw pixel values for generated frames	0 - 65534

Preparing data collection

Before any operation one needs to check if card is idle (not running data collection) with JungfraujochDevice::IsIdle() function. Most configuration parameters cannot be changed, when card is in not-idle state.

The card can be then configured with JungfraujochDevice::SetConfig() function. Details of the configuration data structure are given in pcie_driver/jfjoch_fpga.h header file.

Data collection

Then one can start the card with JungfraujochDevice::Start() function. Final step is to wait for first completion (with value HANDLE_START defined in pcie_driver/jfjoch_fpga.h as buffer number) using JungfraujochDevice::ReadWorkCompletion().

Standard operation of the card requires exchange of buffer ownership between the host application and FPGA card. At the beginning all buffers are owned by host application and should be "given" to the card with JungfraujochDevice::SendWorkRequest() function. Then card will wait for the detector to send data. After full module is collected, data are written via Direct Memory Access to host memory and kernel driver is informed with an interrupt that data are ready. Host application can "learn" what was collected by the card by running JungfraujochDevice::ReadWorkCompletion() function. Buffer returned by the function is owned by the host application and is safe to process. After processing the buffer has to be given back to card via JungfraujochDevice::SendWorkRequest(). If the card doesn't receive enough work requests (open buffers) it won't be able to receive data, resulting in lost packets.

Some important points to mention:

Both functions mentioned in the above paragraph may fail due to work request queue being full and completion queue being empty respectively. Please always check that return value is true.
Both functions JungfraujochDevice::SendWorkRequest() and JungfraujochDevice::ReadWorkCompletion() are thread-safe and can be executed in parallel context. All other functions in the library that change configuration or state of the card are NOT thread-safe, anyway running them in parallel would give nondeterministic result.
Reading work completion will wait up to 1 second before returning.
JungfraujochDevice::ReadWorkCompletion() adds data collection ID as the highest 16-bit (16-31) - this allows to avoid mixing previous and current data collection.
Work requests sent before HANDLE_START was received by host application will be likely discarded.

The card will end acquisition in two situations:

Frame with number provided in the configuration is received
JungfraujochDevice::Cancel() function is called by host application

The host application will know that the data collection finished by receiving completion with value of HANDLE_END as buffer number. After data collection is finished one should call JungfraujochDevice::End() to finalize.

Internal generator

When detector is not installed, or one would like to check the Jungfraujoch card without running a detector, it is possible to generate detector packets internally. Internal generator makes packets with all the headers (Ethernet, UDP/IP, SLS Detector) and is entering data stream in similar location to Ethernet Media Access Control (MAC) cores.

Before starting data collection, it is necessary to load content of module to card FPGA memory. At this moment, for each module a different content can be provided, but all frames for particular model will be the same. It will hopefully change in the future. To load the data, one needs to place content of each module (in 16-bit) into respective kernel buffer (allocated with JungfraujochDevice::MapKernelBuffer()) - first module to buffer 0, second module to buffer 1, etc. Then one needs to call JungfraujochDevice::LoadInternalGeneratorFrame() with specified number of modules.

One also needs to switch data source by executing JungfraujochDevice::SetDataSource() with respective value.

The next step is to do all the preparations to start data collection, up to JungfraujochDevice::Start() and completion handshake. Then one can run JungfraujochDevice::RunFrameGenerator() function, with parameters described in the pcie_driver/jfjoch_fpga.h header file. The function is asynchronous, and will start generation, but doesn't wait for the end. Though one can assume that frame generator is done, when data collection is finished.

Spot finding parameters

Spot finding parameters can be updated with function JungfraujochDevice::SetSpotFinderParameters(). Contrary to other configuration functions, this one is safe to execute during data collection.