eiger manual

manual/manual-client/Eiger_short.tex

@@ -6,6 +6,7 @@
 \usepackage{verbatim}
 \usepackage{xspace}
 \usepackage{hyperref}
+\usepackage{xcolor}
 \newcommand{\E}{EIGER\xspace}  
 \newcommand{\p}{sls\_detector\_put}
 \newcommand{\g}{sls\_detector\_get}
@@ -24,7 +25,7 @@ Figure ~\ref{boards} show the readout board basic components on an Eiger half mo
 \begin{center}
 \includegraphics[width=1\textwidth]{Boards}
 \end{center}
-\caption{Picture with most relevant components of the EIGER readout system. The readout system starts with the Front End Boards (FEB) which performs data descrambling (also converts the packets from 12 $\to$ 16 bits) and rate correction. The BackEndBoard (BEB) has 2x2GB DDR2 memories and can perform data buffering (storing images on board) and data summation (16 bit $\to$ 32 bits). The controls to the detector are passed through the 1Gb, while in most installations, the data are sent out through the 10GB ethernet connection.} 
+\caption{Picture with most relevant components of the EIGER readout system. The readout system starts with the Front End Boards (FEB) which performs data descrambling (also converts the packets from 12 $\to$ 16 bits) and rate correction. The BackEndBoard (BEB) has 2x2GB DDR2 memories and can perform data buffering (storing images on board) and data summation (16 bit $\to$ 32 bits). The controls to the detector are passed through the 1Gb, while in most installations, the data are sent out through the 10~GB ethernet connection.} 
 \label{boards}
 \end{figure}
 
@@ -156,6 +157,14 @@ outdir /sls/X12SA/data/x12saop/Data10/Eiger0.5M
 threaded 1
 \end{verbatim}
 
+The geometry on af an EIGER module, showing the quadrants corresponding to the udp ports and the 2 receivers is shown in figure~\ref{fig:eigerports}.
+\begin{figure}[t]
+\begin{center}
+\includegraphics[width=0.9\textwidth]{Eiger-Chips}
+\end{center}
+\caption{Geometry of UDP ports and receivers in a singel module.}
+\label{fig:eigerports}
+\end{figure}
 
 In the case you are developing your own receiver, then you need to remove the 1Gb receiver hostname {\tt{rx\_hostname}} and substitute it with the mac address of the device:
 \begin{verbatim}
@@ -163,7 +172,6 @@ configuremac 0
 rx_udpmac xx:xx:...
 \end{verbatim}
 
-
 One can configure all the detector settings in a parameter file {\tt{setup.det}}, which is loaded by doing:
 \begin{verbatim}
 sls_detector_put 0-parameters setup.det
@@ -219,6 +227,7 @@ We have added a special command, {\tt{thresholdnotb}}, which allows to scan the
 \begin{verbatim}
 sls_detector_put 0-thresholdnotb energy_in_eV
 \end{verbatim}
+See section~\ref{sec:fastthresholdscan}.
 
 \section{Standard acquisition}
 
@@ -317,31 +326,49 @@ The size of the gap pixels between modules to insert is
    GapPixelsBetweenModules_y = 36
 \end{verbatim}
 where the {\tt{GapPixelsBetweenModules\_x}} are the one on the short side of the module, while {\tt{GapPixelsBetweenModules\_y}} are the ones on the long side of the module (where the wirebonds take physical space).
+  \section{QUAD special geometry}
+Starting from release 4.1.0, we support a special geometry with 2x2 pixels. This is for a Quad, where a single half module reads out 4 chips but in a quad shape. For now this hardware is only available as a PEEM detector at SIM. 
+The {\tt{detsizechan 1024 512}} needs to remain set like this for a half module. However, thanks to the command: 
+ \begin{verbatim}
+./sls_detector_put quad 1
+\end{verbatim}
+
+a 512x512 geomtry will be read out if {\tt{gappixels 0}} and 514x514 will be readout if {\tt{gappixels 1}}. Note that as above, {\tt{gappixels 1}} is not supported for {\tt{dr 4}}.   
+If  {\tt{gappixels 0}}, in the master.raw file you will read: 
+\begin{verbatim}
+row                        : 512 pixels
+col                        : 256 pixels
+\end{verbatim}
+else if {\tt{gappixels 1}}, in the master.raw file you will read: 
+\begin{verbatim}
+row                        : 514 pixels
+col                        : 257 pixels
+\end{verbatim}
 
 \section{Readout timing- maximum frame rate}\label{timing}
 IMPORTANT: to have faster readout and smaller dead time, one can configure {\tt{clkdivider}}, i.e. the speed at which the data are read, i.e. 200/100/50~MHz for {\tt{clkdivider 0/1/2}} and the dead time between frames through {\tt{flags parallel}}, i.e. acquire and read at the same time or acquire and then read out.
 The configuration of this timing variables allows to achieve different frame rates. NOTE THAT IN EIGER, WHATEVER YOU DO, THE FRAME RATE LIMITATIONS COME FROM THE NETWORK BOTTLENECK AS THE HARDWARE GOES FASTER THAN THE DATA OUT.
 
-In the case of REAL CONTINUOUS readout, i.e. continuous acquire and readout from the boards (independent on how the chip is set), the continuous frame rates are listed in table~\ref{tcont}.
+In the case of REAL CONTINUOUS readout, i.e. continuous acquire and readout from the boards (independent on how the chip is set), the continuous frame rates are listed in table~\ref{tcont}. The time to send out the frame out of the board 
 \begin{table} 
-\begin{tabular}{|c|c|c|c|}
+\begin{tabular}{|c|c|c|c|c|}
 \hline
-GbE & dynamic range & continuos maximum frame rate(Hz) & minimum period ($\mu$s)\\
+\tiny{GbE} & \tiny{dynamic range} & \tiny{continuos maximum frame rate(Hz)} & \tiny{minimum period ($\mu$s)}& \tiny{time to send out data ($\mu$s)}\\
 \hline
-1 & 16 &  \textbf{256} & 3901\\ 
+1 & 16 &  \textbf{256} & 3901 & \\ 
 \hline 
-1 &  32 &  \textbf{128} & 7820\\
+1 &  32 &  \textbf{128} & 7820 & \\
 \hline 
-10 & 16 & \textbf{2560} & 391\\  
+10 & 4 & \textbf{10240} & 98 & 100\\
 \hline 
-10 & 32 & \textbf{1280}& 782\\  
+10 & 8 & \textbf{5120} & 196 & 200\\
 \hline
-10 & 8 & \textbf{5120} & 196\\
+10 & 16 & \textbf{2560} & 391 & 400\\  
 \hline 
-10 & 4 & \textbf{10240} & 98\\
+10 & 32 & \textbf{1280} & 782 & 800\\  
 \hline 
 \end{tabular}
-\caption{Frame rate limits for the CONTINUOS streaming out of images, i.e. the data rate out is just below 1Gb/s or 10Gb/s.} 
+\caption{Frame rate limits for the CONTINUOS streaming out of images, i.e. the data rate out is just below 1Gb/s or 10Gb/s. 1280~Hz for 32-bit, 10GbE is obtained from the 10GbE limitation. The maximum achievable frame rate is 977~Hz.} 
 \label{tcont}\end{table}
  Note that in the {\tt{continuous}} flag mode, some buffering is still done on the memories, so a higher frame rate than the proper real continuous one can be achieved. Still, this extra buffering is possible till the memories are not saturated. The number of images that can be stored on the DDR2 on board memories are listed in table~\ref{timgs}.
 \begin{table}
@@ -411,18 +438,9 @@ where the 'minimum time between frames' and the minimum period will be discussed
 \hline
 4 & 0 & \tiny {parallel} & 3.4 & 22  & 44 & 44.01 & 30k/50k\\
 \hline
-4 & 1 & \tiny {parallel} & 6 & 10.5 & 92 & 92.02 & 30k/100k\\
-\hline
-4 & 2 & \tiny {parallel} & 11.2 & 5.4 & 197& 197.01 & infinite\\
-\hline
 \hline
 8 & 0 & \tiny {parallel} & 3.4 & 11.1 & 89 & 89.01 & 15k/24k\\
 \hline
-8 & 1 & \tiny {parallel} & 6.1 & 5.7 & 181 & 181.01 & 15k/52k\\
-\hline
-8 & 2 & \tiny {parallel} & 11.2 &  2.9 &  342 & 342.01 & infinite\\
-\hline
-\hline
 16 & 0 & \tiny {parallel} & 3.4 & 6.1 & (126+38)* =164 & 164.02 & 8k/12k\\
 \hline
 16 & 0 & \tiny {nonparallel} &  127 & 5.6 & (126+52)*= 179 & 179.01& 8k/23k\\
@@ -431,14 +449,6 @@ where the 'minimum time between frames' and the minimum period will be discussed
 \hline
 16 & 1 & \tiny {nonparallel} & 255 & 3.3 &  303 & 303.01 & infinite\\
 \hline
-16 & 2 & \tiny {parallel} &  11.2 & 1.9 & 526 & 526.2 & infinite \\
-\hline
-16 & 2 & \tiny {nonparallel} & 505 & 1.8  & 555 & 555.01& infinite\\
-\hline
-%32 & 2 & parallel &  11 & 2&  & &\\
-%\hline
-%32 & 2 & nonparallel & 504 & $<2$&  & &\\
-%\hline
 \end{tabular}
 \caption{Readout settings. The {\tiny{min exptime}} possible is 5$-$10~$\mu$s. This is due to the time to pass the pixel enable signal in the whole chip. The time between frames ($\Delta$t) has been measured with the oscilloscope and the maximum frames rate (max FR) has been tested with an external gating from a pulse generator at known frequency. The minimum period is obtained as 1/$\textrm{max frame rate}$.}
 \label{tframes}
@@ -449,7 +459,6 @@ where the 'minimum time between frames' and the minimum period will be discussed
 \textbf{From software version 4.0.0, there is a very useful function {\tt{sls\_detector\_get measuredperiod}} which return the measured period AFTER the acquisition. This is important to check that the settings to obtain the targeted frame rate was correct.} 
 
 \textbf{If you run too fast, the detector could become noisier (see problem shooting), it is important to match the detector settings to your frame rate. This can be done having more parameters files and load the one suitable with your experiment.} We experienced that with low energy settings could not reach 6~kHz and no noise.
-
 In 16 bit mode, it could make sense, in case of noise and low threshold to either reduce the frame rate:
 \begin{equation} 
 \textrm{new period}  = \textrm{exptime} + \textrm{minimum time between frames} + (\textrm{10$-$20 }\mu \textrm{s})
@@ -459,7 +468,56 @@ to let the signal settle  or,  if the frame rate is important, leave the {\tt{pe
 \textrm{new exptime}  = \textrm{old exptime} - (\textrm{10$-$20 }\mu \textrm{s})
 \end{equation}
 
-In general, choose first the desired dead time: this will tell you if you want to run in parallel or non parallel mode, although most likely it is parallel mode. Then, choose the maximum frame rate you want to aim, not exceeding what you aim for not to increase the noise. In 4 and 8 bit modes it makes no sense to run nonparallel as the exposure time is too small compared to the readout time. 
+In general, choose the maximum frame rate you want to aim, not exceeding what you aim for not to increase the noise. In 4 and 8 bit modes it makes no sense to run nonparallel as the exposure time is too small compared to the readout time. 
+
+Here below are the final tables for settting the detcetor correctly:
+
+\begin{itemize}
+\item CONTINUOUS redout (imagesnot stored on board memories,  frames can be achieved. {\tt{flags parallel}}, {\tt{clkdivider 0}} are always set. In 32-bit no extra {\tt{subdeadtime}} is assumed. The difference between {\tt{exptime}} and {\tt{period}} has to be $\approx$5 $\mu$s:
+\begin{center}
+  \begin{tabular}{ |c| c| } 
+    \hline
+    max frame rate & settings\\
+    \hline
+    \textcolor{red}{189~Hz (977~Hz max)} &  \textcolor{red}{32-bit} \\
+    & Nframes=infinite\\
+    \hline
+      \textcolor{red}{2.56 kHz} &  \textcolor{red}{16-bit}\\
+     & Nframes=infinite\\
+     \hline
+      \textcolor{red}{5.1 kHz} &  \textcolor{red}{8-bit}\\
+     & Nframes=infinite\\
+     \hline
+      \textcolor{red}{10.2 kHz} &  \textcolor{red}{4-bit}\\
+     & Nframes=infinite\\
+     \hline
+  \end{tabular}
+\end{center}
+BE CAREFUL that if you have the transmission delays setup (see sec.~\ref{network}), this will slow down the sending of the data and you risk to fill up the memories of the boards on eiger (30000 images in 4 bit mode) and you will get corrupted data (parts of the memory on the boads will be overwritten).
+
+\item BUFFERED readout (images stored on board memories, such that the maximum frame rate can be achieved for a limited amount of frames. {\tt{flags parallel}}, {\tt{clkdivider 0}} are always set. In 32-bit no extra {\tt{subdeadtime}} is assumed. The difference between {\tt{exptime}} and {\tt{period}} has to be $\approx$5 $\mu$s:
+\begin{center}
+  \begin{tabular}{ |c| c| } 
+    \hline
+    max frame rate & settings\\
+    \hline
+    \textcolor{red}{189~Hz (977~Hz)} &  \textcolor{red}{32-bit} \\
+    & Nframes=infinite\\
+    \hline
+     \textcolor{red}{6.1 kHz} &  \textcolor{red}{16-bit}\\
+    & Nframes=7600\\
+    \hline
+     \textcolor{red}{11.1 kHz} &  \textcolor{red}{8-bit}\\
+    & Nframes=15000\\
+    \hline
+     \textcolor{red}{22 kHz} &  \textcolor{red}{4-bit}\\
+    & Nframes=30000\\
+    \hline
+  \end{tabular}
+\end{center}
+\end{itemize}
+
+
 
 \subsubsection{4 and 8 bit mode}
 In {\tt{parallel}} mode, the minimum time between frames is due to the time required to latch the values of the counter with capacitors. These values are determined in firmware and they can be estimated as:
@@ -497,16 +555,16 @@ The time between 12-bit subframes are listed in table~\ref{t32bitframe}.
 \begin{tiny}
 \begin{table}
 \begin{flushleft}
-\begin{tabular}{|c|c|c|c|c|c|}
+\begin{tabular}{|c|c|c|c|c|c|c|}
 \hline
-\tiny{dr} & \tiny{clkdivider} & \tiny{flags} & \tiny{t difference between subframes($\mu$s)} & \tiny{max internal subframe rate (kHz)} & \tiny{maximum frame rate (Hz)}\\
+\tiny{dr} & \tiny{clkdivider} & \tiny{flags} & \tiny{subexptime (s)} & \tiny{t difference between subframes($\mu$s)} & \tiny{max internal subframe rate (kHz)} & \tiny{maximum frame rate (Hz)}\\
 \hline
-32 & 2 & parallel & 12 & 2 & 170\\
+32 & 2 & parallel & 0.00262144 & 12 & 380 & 189\\
 \hline
-32 & 2 & nonparallel & 504 & $<2$ & 160\\
+32 & 2 & parallel & 0.000490 & 12 & 2 & 997\\
 \hline
 \end{tabular}
-\caption{Timing for the 32bit case. The maximum frame rate has been computed assuming 2 subframes of default {\tt{subexptime}} of 2.62144 ms.}
+\caption{Timing for the 32bit case. The maximum frame rate has been computed assuming 2 subframes of default {\tt{subexptime}} of 2.62144 ms, which is the default value. By setting up {\tt{subexptime}} to 490~$\mu$s one can achieve a maximum frame rate. Note that one has to leave 490$\mu$s extra between a frame and the following.}
 \label{t32bitframe}
 \end{flushleft}
 \end{table}
@@ -561,7 +619,10 @@ Here are the implemented options so far:
 \item {\tt{auto}} is the software controlled acquisition (does not use triggers), where {\tt{exptime}} and {\tt{period}} have to be set. Set number of cycles (i.e. triggers) to 1 using {\tt{cycles}}. Set number of frames using {\tt{frames}}.
 \item {\tt{trigger}} 1 frame taken for 1 trigger. Your {\tt{frames}} needs to be  1 always, {\tt{cycles}} can be changed and defines how many triggers are considered.  {\tt{exptime}} needs to be set. In the GUI this is called trigger exposure series.  
 \item {\tt{burst\_trigger}} gets only 1 trigger, but allows to take many frames. With {\tt{frames}} one can change the number of frames. {\tt{cycles}} needs to be 1.  {\tt{exptime}} and {\tt{period}} have to be set. In the gui it is called trigger readout.
-\item{\tt{gating}} allows to get a frame only when the trigger pulse is gating. Note that in this case the exp time and period only depend on the gating signal. {\tt{cycles}} allows to select how many gates to consider.  Set number of frames to 1 using {\tt{frames}}. ATTENTION: if you are in 16 bit mode and you are applying online rate corrections, as now the exptime is generated by the trigger, you might not have correct rate corrections. If you know what the exposure time is in the gating signal, then you can set the {\tt{exptime}} once and the rate corrections will be correct. If the exposure time is unknow, it is recommended that you switch off the rate corrections. In 32 bit mode, it does not matter as the rate corrections depends on the {\tt{subexptime}} which is software set independently from the gate exptime.
+\item{\tt{gating}} allows to get a frame only when the trigger pulse is gating. Note that in this case the exp time and period only depend on the gating signal. {\tt{cycles}} allows to select how many gates to consider.  Set number of frames to 1 using {\tt{frames}}. IMPORTANT: Up to firmware 23, the last subframe is oblige to finish being taken, despite the gate signal going down. This will be configurable from later fw and software version. Also, in gating mode, due to timimg of the state machine, you need to leave 500~$\mu$s deadtime between the end on an acquisition and the next. This is as the state machine is unable to check for changes in the status in the first 500~$\mu$s. ATTENTION: if you are in 16 bit mode and you are applying online rate corrections, as now the exptime is generated by the trigger, you might not have correct rate corrections. If you know what the exposure time is in the gating signal, then you can set the {\tt{exptime}} once and the rate corrections will be correct. In 32 bit mode, it does not matter as the rate corrections depends on the {\tt{subexptime}} which is software set independently from the gate exptime.
+
+When using 32-bit mode, by default the acquisition ends the last complete subframe that was started when still the acquisition time was valid. This has been chosen as many people wants to know the exact acquisition time for when the detector was taking data and also, if {\tt{ratecorr}} are active, the last subframe will be correctly corrected, while otherwise it will be corrected with a wrong subdeadtime. 
+However, from 4.1.0, in gating mode, an option to immediately terminate the subframe when the gate signal goes down it is implemented to stop the acquisition at the same time.  This option is {\tt{./sls\_detector\_put interruptsubframe 1}} while the default option is {\tt{./sls\_detector\_put interruptsubframe 0}}.  
 
 \end{itemize}
 
@@ -576,7 +637,7 @@ sls_detector_put 0-frames x
 sls_detector_put 0-cycles y
 sls_detector_status trigger
 \end{verbatim}
- Note that this functionality is very (!) useful if you need to do something between and acquisition and the next. This can be used to do a fast threshold scan for example. See section~\ref{Sec:fastthresholdscan}.
+ Note that this functionality is very (!) useful if you need to do something between and acquisition and the next. This can be used to do a fast threshold scan for example. See section~\ref{sec:fastthresholdscan}.
 
 
 \section{Autosumming and rate corrections} \label{advanced}
@@ -616,6 +677,19 @@ sls_detector_put 0-ratecorr -1
 
 Every time either the rate corrections are activated, $\tau$ is changed or the subframe length is changed, then a new correction table is evaluated. Note that computing the correction table is time consuming. 
 
+Here in figure~\ref{rateplots} you can find typical values of $\tau$  and typical values of the rates for which we have small non linerities (10\% non linearity) as a function of the beam energy.
+\begin{figure}[t]
+\begin{center}
+\includegraphics[width=.7\textwidth]{tauvsE}
+\includegraphics[width=.7\textwidth]{Ratecapabilityflux}
+\end{center}
+\caption{Typical values of the dead time for the detector and safe fluxes as a function of the beam energy}.
+\label{rateplots}
+\end{figure}
+
+
+
+
 \section{Dependent parameters and limits}
 Here is a list of dependent parameters:
 \begin{enumerate}
@@ -664,7 +738,7 @@ LEDs on the backpanel board at the back of each half module signal:
 
 
 
-\subsection{Delays in sending for 1Gb/s, 10Gb/s, 10Gb flow control, receiver fifo}
+\subsection{Delays in sending for 1Gb/s, 10Gb/s, 10Gb flow control, receiver fifo}\label{network}
 
 Extremely advanced options allow to:
 \begin{itemize}
@@ -693,6 +767,7 @@ for X in $(seq 0 4); do ./sls_detector_put $X:txndelay_left $((X*100000)); done
 ./sls_detector_put txndelay_frame zzzz
 \end{verbatim}
 In the example before, it would be: {\tt{zzzz}}=4*100000+ 100000
+To decide the size of the transmission delays, look at subsection~\ref{delays}. 
 
 \item Readjust the size of the fifo of the receiver between listening and writing (useful when writing is limited)
 \begin{verbatim}
@@ -711,16 +786,37 @@ To activate back a module, do:
 \end{verbatim}
 
 \end{itemize}
-\subsection{Setting up 10Gb correctly: experience so far}\label{10g}
+
+\section{Choose correct transmission delays}\label{delays}
+
+Transmission delays should be chosen only to accomodate the writing speed of the disk. The 10Gb network should be optimised independently to allow no packet loss situation. This can be done by turning off the writing of the files and check only for missing frames. 
+
+Table~\ref{tcont} gives the times that are needed to transfer 1 images out of the 10~Gb Ethernet connection. This reflects the CONTINUOS frame rate achieavable. The disk speed can be monitored with {\tt{dstat}}. One you have worked out this, you can calculated the {\tt{txndelay\_frame}} delay as:
+
+\begin{equation}
+ {\tt{txndelay\_frame}}=-tsending+dr \cdot \frac{4*256*256*N\_half\_modules}{1024 \cdot 1000 \cdot disk\_speed [MB/s]}
+\end{equation}
+In 4-bit mode, for a disk seed of 320MB/s, the  {\tt{txndelay\_frame}} is 300~$\mu$s (30000 to be set up to the detector). The sending time is 100~$\mu$s, such that an total ackievable writing sped of 1/400~$\mu$s (2.5~kHz) in achieved.  
+
+\section{Setting up the PC settings for 10Gb}\label{10g}
 
 For configuring well the 10Gb card not to loose packets, 
 \begin{itemize}
 \item MTU must be set up to 9000 (jumbo frames) on all the involved sides: detector, switch, server NIC
 \item you should set up static MAC address tables with separated VLANs
 \end{itemize}
-As root, also do:
+As root, also first check your ethtool settings (-small letter arguments), then change the settings (-capital letter arguments):
 \begin{verbatim}
- ethtool -G xth1 rx 4096, ethtool -C xth1 rx-usecs 100 
+ethtool -g xth1 
+ethtool -c xth1 
+ethtool -a xth1
+\end{verbatim}
+
+To change settings:
+\begin{verbatim}
+ethtool -G xth1 rx 4096 #or wheterver is the max number for your pc
+ethtool -C xth1 rx-usecs 100 
+ethtool -A xth1 rx on
 \end{verbatim}
 where {\tt{xth1}} can be replaced with the correct 10Gb device. To minimise loosing packets, priorities are set better as root user, so have the receiver as root.
 To try to bypass being root, we trued something like this:
@@ -733,13 +829,74 @@ Very important is to activate the flow control in 10Gb (in 1Gb it is on by defau
 \begin{verbatim}
 ./sls_detector_put flowcontrol_10g 1
 \end{verbatim}
-Set the transmission delays as explained in the manual.
+You ned to check that flow control is setup on the reeceiving interfaces. Check with:
+\begin{verbatim}
+ethtool -a eth1
+\end{verbatim}
+.RX should be ON. Set the transmission delays as explained in the manual if necessary. These transmission delays should help matching teh writing speed performance of your disk. You can check how fast you are writing using the {\tt{dstat}} command.  
+
+Now setup the computer server propery:
+Check the size of: 
+\begin{verbatim}
+sysctl -a | grep backlog
+sysctl -a | grep rmem
+\end{verbatim}
+Set it correctly with: 
+\begin{verbatim}
+sysctl net.core.netdev_max_backlog=250000
+sysctl net.core.rmem_default=$((100*1024*1024))
+sysctl net.core.rmem_max=$((100*1024*1024))
+\end{verbatim}
+
+Other way to setup the same,  increase the socket receiving buffer size and the maximum length of the input queue:
+\begin{verbatim}
+echo $((100*1024*1024)) > /proc/sys/net/core/rmem_max
+echo 250000 > /proc/sys/net/core/netdev_max_backlog
+\end{verbatim}
+to make the settings permanent, edit /etc/sysctl.conf:
+ \begin{verbatim}
+# 100MiB
+net.core.rmem_max = 104857600
+net.core.netdev_max_backlog = 250000
+\end{verbatim}
+and run \textbf{sysctl -p}.
+
+Last, you can disable power saving in the CPU frequency (chose the appropriate command for your system):
+\begin{verbatim}
+cpupower frequency-info
+cpupower frequency-set -g performance 
+\end{verbatim}
+or
+\begin{verbatim}
+cpufreq-info
+for i in `seq 0 7`; do cpufreq-set -c $i -g performance; done
+\end{verbatim}
 
 It can help to increase the fifo size of the receiver to {\tt{rx\_fifodepth}} to 1000 images
 \begin{verbatim}
 ./sls_detector_put rx_fifodepth 1000
 \end{verbatim}
-One needs to keep into account that in 16 bit mode for 1 image we expect each slsReceiver  to allocate 0.5MB. So for 1000 images, we expect  500MB memory for each receiver. This can be monitored in Linux with "top" or "free -m".
+One needs to keep into account that in 16 bit mode for 1 image we expect each slsReceiver  to allocate 0.5MB. So for 1000 images, we expect  500MB memory for each receiver. This can be monitored in Linux with "top" or "free -m". To receive the max number of images possible on the detector, a minimum of 8~GB of memories are required.
+For very high frame rate, very long measurements, we had to increase the {\tt{rx\_fifodepth}} to 10000 images but this only in 4, 8, 16 bit mode. In 32 bit mode it will assign 40~GB of memory, which is more than what normal PC would have. So make sure you do not require too much memory.  
+
+Last, it is very important that not too many files are created. There is high possibility to loose packets in the time to close and open files for the writer. IN 3.1.x, the default number of images written per file, in Eiger is 2000. This is defined by the line:
+\begin{verbatim}
+#define EIGER_MAX_FRAMES_PER_FILE	2000
+\end{verbatim}
+in {\tt{slsDetectorsPackage/slsReceiverSoftware/include/sls\_receiver\_defs.h}}. In 4.0.x, this is interactively defined using the command: {\tt{r\_framesperfile}}. By default it is 10000.  
+
+If you do not have a large disk, you can write to memory if your pc is not fast enough:
+\begin{verbatim}
+mount -t tmpfs none /ramdisk_folder
+\end{verbatim}
+or 
+\begin{verbatim}
+mount -t tmpfs none /mnt/ramdisk -o size=10G
+\end{verbatim}
+check how many GB memory you can allocate, to avoid swapping otherwise
+
+
+
 
 \section{Offline processing and monitoring}
 
@@ -819,9 +976,14 @@ Checkout the {\tt{developer}} branch if in a 3.1.X release or the {\tt{v4.0.0}}
 
 Three possible conversions are possible: into \textbf{cbf}, \textbf{hdf5} and \textbf{root} format. The detector writes 4 raw files per receiver. An offline image reconstruction executable has been written to collate the possible files together and produce output files. By default an interpolation between the values of the large pixels is performed. Gap pixels between modules are also inserted.
 
+Note that the number of images per file is hardcoded and needs to match whatever you are using in {\tt{slsDetectorsPackage/slsReceiverSoftware/include/sls\_receiver\_defs.h}}:
+\begin{verbatim}
+#define EIGER_MAX_FRAMES_PER_FILE	2000
+\end{verbatim}
+The default is 2000.
 
 \subsubsection{cbf}
-The cbf executable executable uses the CBFlib-0.9.5 library (downloaded from the web as it download some architecture dependent packages at installation).Edit the Makefile to correclty point at it.\\
+The cbf executable executable uses the CBFlib-0.9.5 library (downloaded from the web as it downloads architecture dependent packages at installation).Edit the Makefile to correclty point at it.\\
 \underline{At cSAXS, the CBFlib-0.9.5 has been compiled -such that the required packages are}\\\underline{ downloaded in /sls/X12SA/data/x12saop/EigerPackage/CBFlib-0.9.5.}\\ 
 
 To use it for a single module:
@@ -832,7 +994,7 @@ eg.
 {\tt{cbfMaker /scratch/run\_63\_d1\_f000000000000\_3.raw}}\\
 
 To use it any geometry:\\
-{\tt{cbfMaker [filename] [pixels x, def=1024] [pixels y, def=512] [singlemodulelongside\_x, def=1] [fillgaps, def=Interpolate Big Pixels] [hdf5datasetname, def="Eiger"] [start det,def=0]}}\\
+{\tt{cbfMaker [filename] [outdir, def=same as filename] [pixels x, def=1024] [pixels y, def=512] [singlemodulelongside\_x, def=1] [fillgaps, def=Interpolate Big Pixels] [hdf5datasetname, def="Eiger"] [start det,def=0]}}\\
 eg.
 {\tt cbfMaker /scratch/run\_63\_d0\_f000000000000\_3.raw  3072 512 1 2 ``Eiger'' 0}.\\
 
@@ -876,7 +1038,7 @@ make cbfMaker; make cbfMakerOMNY;
 \end{verbatim}
 
 \subsubsection{hdf5}
-In case of HDF5 output file, we rely on having the HDF5 1.10.1 library installed. Edit the Makefile to correclty point at it. Different compression methods are being tried so different external filters might be to be installed. This work is not finished yet.
+In case of HDF5 output file, we rely on having the HDF5 1.10.1 library installed and {\tt{HDF5-External-Filter-Plugins}} installed. With HDF5 1.10.3, no need of the external plugings package is needed, but a small modification to the file is needed. Edit the Makefile to correclty point at it. Different compression methods are being tried so different external filters might be to be installed. This work is not finished yet.
 
 To choose HDF5, with ZLIB implementation, open {\tt{slsImageReconstruction/src/main\_csaxs.cpp}} and make sure that 
 \begin{verbatim}
@@ -888,7 +1050,7 @@ are not commented out. All other options need to be commented out. Copile the co
 make hdf5Maker; make hdf5MakerOMNY;
 \end{verbatim}
 
-If you are at cSAXS. all images collected will be written in a single file. If you are not at CSAXS, most likely you want to have all the images written in a single raw file into an HDF5 file. The multiple HDF5 files are then linked in a master file, with many subdatasets (can be read by albula) or by a virtual file with a single dataset. If you want a mster o virtual file, uncomment this option:
+If you are at cSAXS. all images collected will be written in a single file. If you are not at CSAXS, most likely you want to have all the images written in a single raw file into an HDF5 file. The multiple HDF5 files are then linked in a master file, with many subdatasets (can be read by albula) or by a virtual file with a single dataset. If you want a master o virtual file, uncomment this option:
 \begin{verbatim}
 #define MASTERVIRTUAL
 \end{verbatim}
@@ -910,7 +1072,7 @@ The data will be written as TH2D in root format. Edit the {\tt{Makefile}} to poi
 \begin{verbatim}
 make image 
 \end{verbatim}
-There is no program other executable that alredy keeps into account the geometry for it.
+There is no program other executable that already keeps into account the geometry for it.
 To use it any geometry:\\
 {\tt{image [filename] [pixels x, def=1024] [pixels y, def=512] [singlemodulelongside\_x, def=1] [fillgaps, def=Interpolate Big Pixels] [hdf5datasetname, def="Eiger"] [start det,def=0]}}\\
 eg.
@@ -978,9 +1140,9 @@ ssh root@$i sync; done
 
 \section{Loading firmware bitfiles}
 
-\textbf{As a new procedure, the first thing to do is to kill the server on the boards, copy the new one there without starting it.} Note taht failure to do this step before may cause the linux on the baords to crash and not being able to ping it (this if the registers between the old and new firmware change).
+\textbf{As a new procedure, the first thing to do is to kill the server on the boards, copy the new one there without starting it.} Note that failure to do this step before may cause the linux on the boards to crash and not being able to ping it (this if the registers between the old and new firmware change).
 
-This is teh procedure from a terminal;
+This is the procedure from a terminal;
 \begin{verbatim}
 for i in beb111 beb070; 
 do ssh root@$i killall eigerDetectorServer; 
@@ -1058,12 +1220,40 @@ To load the special noise file look at {\tt{settingsdir/eiger/standard/eigernois
 \begin{verbatim}
 sls_detector_put trimbits ../settingsdir/eiger/standard/eigernoise
 \end{verbatim}
-To exit from this pattern noise, just set the theshold to something known. 
+To exit from this pattern noise, just set the threshold to something known. 
 \begin{verbatim}
 \item sls_detector_put threshold 50000 standard
 \end{verbatim}
 where 5000 would be a value in eV and {/tt{standard}} is important in this case.
 
+\section{(Fast) threshold scans during beam operation}\label{sec:fastthresholdscan}
+Occasionally you might need to do threshold scans during your data taking (for example for Laue experiments or to get any spectral information). Setting the threshold in this case would be not optimal as you would change trimbits at every energy and this could give you a ``step'' behaviour. What you could do is to use the 
+\begin{verbatim}
+\item sls_detector_put thresholdnotb 50000 
+\end{verbatim}
+ which set the threshold to an energy but does not change trimbits. We suggest that before using this function you load the {\tt{threshold}} at an energy in the middle of your scan range and then change {\tt{thresholdnotb}}.
+We have also been requested if we could speed up the threshold scan. At the moment no specific function has been integrated in firmware, but one could use the software trigger option to perform what you need:
+\begin{verbatim}
+./sls_detector_put exptime 0.01
+./sls_detector_put timing trigger
+./sls_detector_put enablefwrite 0
+./sls_detector_put resetframescaught 0 
+./sls_detector_put index 0
+./sls_detector_put cycles 21
+./sls_detector_put receiver start
+./sls_detector_put status start 
+for i in $(seq 0 20); 
+ do
+   #./sls_detector_put thresholdnotb 5000 ##this is still too slow as it loads the module 
+ ./sls_detector_put 0:vrf 3199 #need to know the appropriate vrf at every energy 
+ ./sls_detector_put 1:vrf 3199 #need to know the appropriate vrf at every energy 
+ ./sls_detector_put status trigger
+    #sleep 0.005
+done
+./sls_detector_put  receiver stop
+./sls_detector_put  resetframescaught 0 
+./sls_detector_put  timing auto
+\end{verbatim} 
 
 \section{Troubleshooting}
 \subsection{Cannot successfully finish an acquisition}
@@ -1215,7 +1405,7 @@ Scroll up in the terminal till you find:\\
 *************** MASTER/SLAVE ***************\\
 *************** NORMAL/SPECIAL ***************\\
 
-There is also an easier way, that is that only the master module will reaturn the real value of the HV. If you have more than 1 detector system, then you will have more than 1 physical master, as the HV needs to be applied to all the systems.
+There is also an easier way, that is that only the master module will return the real value of the HV. If you have more than 1 detector system, then you will have more than 1 physical master, as the HV needs to be applied to all the systems.
 
 \begin{verbatim}
 for i in $(seq 0 36); do sls_detector_put $i:vhighvoltage; done
@@ -1225,10 +1415,14 @@ for i in $(seq 0 36); do sls_detector_put $i:vhighvoltage; done
 \subsection{'Cannot connect to socket'}
 This error is typically due to the detector server not running. For why, see section~\ref{servernot}.
 
+\subsection{Running at low frame rate, the communication to receiver stops}
+If running in 32-bit mode (or even in 16-bit mode), if more memory than what your machine can handle is asked for, the receiver process could be terminated by the kernel of your machine. It would loook like you had executed a clean control-c on the receiver executable. This has been the case, when setting up by mistake
+\tt{rx\_fifodepth} to 10000 images in 32 bit mode. In 32 bit mode it will assign 40~GB of memory, which is more than what normal PC would have. So make sure you do not require too much memory. The same is tru also for smaller values of {\tt{rx\_fifodepth}} if your  machine has not much memory.  
+
 \subsection{Detector server is not running}\label{servernot}
 The detector server could not be running: either the detector was powered off, or it powered off itself due to too high temperature or, in the case of the 9M, if the waterflow sensor detected no flux and powered it off (the chiller stops occasionally as cSAXS).
 
-If the powering and the temperature are OK, instead, it can be that the firmware version is incompatible to the server version and/or the client software version. So check the consistency of firmware/software/server versions.
+If the powering and the temperature are OK, instead, it can be that the firmware version is incompatible to the server version and/or the client software version. In software packages 3.x.y, the eigerDetectorServer was killed automatically. So check the consistency of firmware/software/server versions if using this version of the software. From 4.x.y onwards, the server, if associated to a wrong firmware, does not kill itself.   
 
 \subsection{'Acquire has already started' error message}
 If you see the client returning the following error message:\\ 
@@ -1239,10 +1433,10 @@ If you see the client returning the following error message:\\
 \end{verbatim}
 
 \subsection{There is noise running the detector in 32-bit}
-If you are running the detector in 32-bit (autosumming), there might be some noise, particularly at lower thereshold energies. This is due to the fact that the analog part of the chips require some latency time to settle which is larger than the readout time. It is possible to run the detector only in {\tt{parallel}} or {\tt{nonparallel}} mode, respectively with readout times between frames of 12~$\mu$s and 504~$\mu$s. If you switch {\tt{flags}} to non {\tt{nonparallel}} mode you will give enough time for the signals to settle. From release 4.0.0, there is a configurable delay that can be set through the {\tt{subdeadtime}} variable, such that you can remain with the {\tt{parallel}} flag, but can obtain a configurable dead time between frames. Ask the SLS detector group for an appropriate dead time for your detector, but typically a dead time of 20-50~$\mu$s should be enough. Note that this {\tt{subdeadtime}} need to include the 12~$\mu$s minimum readout time, so it has to be larger than 12~$\mu$s to do anything.  
+If you are running the detector in 32-bit (autosumming), there might be some noise, particularly at lower threshold energies. This is due to the fact that the analog part of the chips require some latency time to settle which is larger than the readout time. It is possible to run the detector only in {\tt{parallel}} or {\tt{nonparallel}} mode, respectively with readout times between frames of 12~$\mu$s and 504~$\mu$s. If you switch {\tt{flags}} to non {\tt{nonparallel}} mode you will give enough time for the signals to settle. From release 4.0.0, there is a configurable delay that can be set through the {\tt{subdeadtime}} variable, such that you can remain with the {\tt{parallel}} flag, but can obtain a configurable dead time between frames. Ask the SLS detector group for an appropriate dead time for your detector, but typically a dead time of 20-50~$\mu$s should be enough. Note that this {\tt{subdeadtime}} need to include the 12~$\mu$s minimum readout time, so it has to be larger than 12~$\mu$s to do anything.  
  
 \subsection{There is noise running the detector at high frame rate(4,8,16 bit)}
-If are running in {\tt{parallel}} mode, in particular at low thereshold energies, you might encounter some noise. The reason is that the analog part of the chips require some latency time to settle which is larger than the readout time. 
+If are running in {\tt{parallel}} mode, in particular at low threshold energies, you might encounter some noise. The reason is that the analog part of the chips require some latency time to settle which is larger than the readout time. 
 \begin{enumerate}
 \item You can lower the frame rate and relax requirements on period: 
 At low frame rate, you normally leave enough time between the end of the acquisition and the starting of the next, so you should not see this effect. In any case setting a {\tt{period}}={\tt{exptime}}+readout time from Table~\ref{tchipro} +extra 20$\mu$s cures the problem. The 20$\mu$s could also be 10~$\mu$s, they are very hardware dependent.