sics/doc/master.tex

\documentclass[12pt,a4paper]{article}
%%\usepackage[dvips]{graphics}
%%\usepackage{epsf}
\setlength{\textheight}{24cm}
\setlength{\textwidth}{16cm}
\setlength{\headheight}{0cm}
\setlength{\headsep}{0cm}
\setlength{\topmargin}{0cm}
\setlength{\oddsidemargin}{0cm}
\setlength{\evensidemargin}{0cm}
\setlength{\hoffset}{0cm}
\setlength{\marginparwidth}{0cm}
\setlength{\parindent}{0pt}
\setlength{\parskip}{0.5cm}
\newtheorem{error}{Error}

\begin{document}

\begin{center}
{\huge Sinq Systems Overview and Documentation Guide}
\\
Revision: \today \\
Author: Mark K\"onnecke
\end{center}
\tableofcontents
\clearpage
\section{Introduction}
This document gives an overview about all the hardware and software systems 
involved in SINQ instrument control. It describes the relationships between 
the various pieces and points to additional documentation, if available. 
A copy of all documents referred in this paper is stored at:
/afs/psi.ch/project/sinq/commmon/share/sicsdoc.
 If they exist in electronic form, this is. The SICS user 
 and manager information is available on the WWW starting from
 http://lns00.psi.ch/. 


\section{Hardware Overview}
For each SINQ instrument the following computing hardware is available:
\begin{itemize}
\item A dedicated instrument workstation. Most of them are Intel x86
  machines running Linux. Three instruments use alpha workstations
  running Tru64Unix. 
\item A TCP/IP terminal server providing access to serial ports.
\item Optionally, there are 1-3 histogram memory computers installed for those
 instruments which have area detectors. These histogram memory computers are 
 Motorolla 6800 VME board processors running the vxWorks realtime 
 operating system. Two types of boards are in use: the older Motorolla
 MVME1603 boards with 16 MB of memory and MEN-A12 boards with 512MB
 memory.  
\item Optionally there are National Instruments ENET-100 GPIB/TCPIP
  converters in use in order to access GPIB devices, especially those
  which came with the Risoe instruments.
\end{itemize}
Most instrument hardware is accessed through RS--232 interfaces and
 the terminal server. Histogram memories are accessed through the
 TCP/IP network . Generally ethernet is used as the main instrument bus. 

In addition to the computers at the instrument the following systems are 
 available:
\begin{itemize}
\item A True64Unix laboratory server (lnsa15) for data management and 
 primary data anlalysis.
\item True64Unix PSI server systems for data processing.
\item A dual processor linux science server (lnsl15).
\item The PSI Linux Login cluster (llc)
\item A WWW--server currently installed on lns00.
\end{itemize}

\section{Software Overview}
\subsection{Software at the Instrument}
The main software component at the instrument is the Sinq Instrument Control 
  Software (SICS). This is a client server system controlling the operation
 of the instrument. A central server running on the instrument workstation 
 implements all the functionality of instrument control. Clients to this
 server implement the user interface to the instrument. The protocoll between
 clients and the SICS server is a ASCII command protocoll through either 
 telnet or plain sockets. 

 RS--232 devices are not directly accessed through the terminal server but 
  through a special server program, the SerPortServer program. The reasons
 for this are both historical and practical as this system permits access
 to a given serial port for several clients which is a valuable aid for
 debugging. Clients to this SerPortServer such as the SICS server communicate
 with the server through a special ASCII protocoll transported through TCP/IP. 
 The only documentation 
 for both the SerPortServer and the ASCII protocoll is the source
  code. And David Maden. 

There exists another support program, the FileSync server, on any instrument
computer. This is a little Java server which listens at a UDP port for a
 message from SICS. If such a message is recognized or any two hours, this
 server starts  a shell script which copies new data files to the laboratory 
 server. This program is again documented only in its configuration file and
 the Java source code.

Then there is the TecsServer. This is a server which maintains and configures 
 Lakeshore temperature controllers used as standard temperature controllers 
 at SINQ. The only documentation for this program is Markus Zolliker. 

On many instruments there are histogram memory computers. These
usually run the following programs: 
\begin{description}
\item[bootUtil] a utility running when the histogram memory is booted which 
 starts all the other tasks and configures the histogram memory from its
 command line parameters.
\item[sinqHMfiller] this task reads data from the electronics and stores it
 into the right histogram. 
\item[SinqHMserver] This process handles the TCP/IP communication with 
 clients of the histogram memory. It spawns child processes for doing so.
\end{description}
 The histogram memory software, its setup and the TCP/IP protocoll for its
 control is described in great detail in the documents titled:
\newline\centerline{Software Support Notes for the SINQ Histogram Memory}
and \newline
\centerline{The Sinq histogram memory, SinqHM}
both authored by David Maden. The sources for these documents are distributed 
 with the source code for the histogram memory.

The SICS software is described in a variety of documentation:
\begin{itemize}
\item There is user documentation for various instruments.
\item The installation and setup of SICS on an instrument computer is
 described in the "SICS Manager Manual".
\item The programming concepts of SICS are discussed in the "SICS Programmers 
 Reference".
\item An 97\% complete description of SICS source code modules is available 
 as "SICS Reference Manual".
\end{itemize}

The RITA--2 instrument from Ris\o{ } runs the TASCOM software.
This software is quite well documented in various documents which can be
 obtained at WHGA/112 or directly from the Ris\o{ } team and Per Skarup.

\subsection{Central Facilities}
\subsubsection{Central Data Repository}
Under the /afs/psi.ch/project/sinqdata directory there exists a
central  storage area for
data files measured at the instruments. Newly measured data files are
 automatically mirrored to this area once a measurement has finished. 
There are directories for each year of SINQ operation. These contain
subdirectories for the various instruments. 

\subsubsection{The SINQ File Database}
Right early on it was decided to separate the file archival function from the
 file search facility. For this decision there are two reasons:
\begin{itemize}
\item If more search criteria are required, only the database will need to be
 rebuilt.
\item Most SINQ data files are binary. A special tool was needed anyway in 
 order to extract the searchable information from the data files.
\end{itemize}
The SINQ File database is described in more detail in the document:\newline
\centerline{The SINQ File Database}
which is part of the source distribution for the system. Here only an 
 overview of the SINQ database system is given.

The SINQ File Database consists of the following components:
\begin{itemize}
\item The database itself. The database is installed on the central
  database server PSIP0 provided by central computing. The database
  software is oracle.
\item Any night a database update program is started as a cron
  job. This programs runs on the linux server lnsl15. The program
  itself is a tcl script which uses oratcl for accessing the Oracle
  database server and nxinter for accessing NeXus files. 
\item A querying system. Curently the only query interface is a WWW--interface
 installed at the WWW--server lns00.
\end{itemize}


\subsubsection{The NeXus Data Server}
In order to provide access to SINQ data file to various clients on the 
 network a NeXus Data Server was installed. This piece of software 
 is required in order to support the "Load Old File" facility in the
 Java status display clients for various instruments. 

\subsubsection{The CVS Repository}
CVS is a shortcut for Concurrent Version control System. This is a system for
 keeping track of various versions of software. As a lot of source code has
 to be maintained at SINQ and we aspire to have several people working at
 it such a control system became necessary. The LNS setup is described in
 the document:\newline
\centerline{The LNS CVS Repository}
This document also explains how to get hold the source code for most
 of the software used at SINQ.

\subsubsection{Backup}
Most user files and the raw data is stored within the AFS network file
system. These files are backed up by central computing. Data files of
older years are additionally archived in the PSI archive system.

The instrument and guest  accounts on the instrument computers should
be  backed up
too because they sometimes contain valuable scripts. These are local
accounts, thus they are not covered through the AFS backup. Data from
these accounts is synchronized nightly onto a special account on the
linux server lnsl15.
\begin{itemize}
\item There is a script which does the copying. It is {\bf backupinst}
in /afs/.psi.ch/project/sinq/linux/bin. This script is called with {\bf
backupinst INST} where INST must be replaced by the instrument name in
capitals.  This script essentially calls rsync with appropriate
arguments to synchronize the instrument account minus the data
directory (which is automatically copied anyway). 
\item backupinst is triggered by a cron job. For this to work each
instrument computer must have an entry in crontab reading like:
\begin{verbatim}
	0 3 * * * /data/lnslib/bin/backupinst TOPSI >/tmp/topsi.log 2>&1 
\end{verbatim}
Please replace paths and instrument names to match the instrument. 
\item The backup account on lnsl15 is named SINQINST, password:
SINQINSTLNS. It contains a directory for each supported
instruments. For backupinst to work properly there must be a line in
the .rhosts file of this account for each instrument computer which
reads:
\begin{verbatim}
	instcomputer.psi.ch root
\end{verbatim} Please note that this has to be the pcXXXX name for
linux instrument computers, not the DNS alias.
\end{itemize}  


\subsubsection{Motor Parameter Backup}
At SINQ the PSI EL--734 motor controllers are used. These motor controllers 
 hold a set of parameters for each motor. As a safeguard against instrument
 scientists gone wild or a loss of parameters due some electronic or 
 electrical incident these motor parameters should be saved
 regularly. To this purpose there exits motor subdirectories in each
 instrument account. Motors are either saved into this directory
 through a script in this directory or from SICS (TASP,
 MORPHEUS). Motor parameter backup is a responsability of the
 instrument scientists and must be triggered manually. 

\subsubsection{License Server}
Unfortunately some people use the proprietary IDL programming system. This
 system requires a license server to run on some computer. This license 
 server is also installed together with the actual IDL distribution on the
 laboratory server. LNS has bought a number of network licenses of IDL.
 This means at any time only a certain number of people can use IDL on the
 network. When IDL starts up it requests a license from the license server 
 and gives it back when it finishes. This only works if the license server 
 still has enough spare licenses. This implies that any IDL user should
 close his program when she is done and not leave it running all weekend.
 The installation of the license server is described in the IDL installation
 instructions. License servers are black magic and should be handled by
 wizards only. There are only two things which can be done easily:
\begin{enumerate}
\item Check if it is running. the command: \begin{verbatim}
 ps -A | grep lmgrd
\end{verbatim} should  yield at least two entries.
 yield at least two entries. 
\item Start it if it is not running:
\begin{enumerate}
\item Become root
\item Run: /usr/opt/idl/idl/bin/idlstart
\end{enumerate}
\end{enumerate}


\subsection{Software at the WWW--server}
The WWW--server running at lns00 does not provide static information
only but also  active content calculated at runtime. The following
services are implemented:
\begin{itemize}
\item Keyword search in the SICS documentation.
\item Searching the SINQ file database for files matching various criteria.
\item Display of the current instrument status for most instruments.
\item An experimental WWW control system for powder diffractometers. 
\end{itemize}
Most of these services are provided through java servlets. In order to do this 
 a servlet engine was needed. The combination Apache WWW-server and Apache
 Jakarta Tomcat  was choosen. 

The database search servlets are described in more detail in the document:
\newline \centerline{The SINQ File Database} 
the status display and control servlets in the document:\newline
\centerline{The SICS WWW--servlet System}
. Both documents are part of their respective source code packages. 

The SICS documentation searching system is built on top of the program SWISH-E which is available
 from http://sunsite.berkeley.edu/SWISH-E. 
The actual search is performed through a TCL--cgi script which calls the swishe
 application with appropriate parameters. This script can be found as 
 swishsearch.cgi in the cgi-bin directory of the WWW hierarchy on lns00.
 Prerequisite for such a search is the existence of an index file. This must
 be created offline after any major change to the documentation. In order to
 create this index, cd into the sics directory of the WWW hierarchy and run:
 \begin{verbatim}
 d:\bin\swishe sics.config
 \end{verbatim}
 This must be done on the lns00 computer. The placement of the index file, 
 swishe executable and the root for serving documents are configured in
 sics/sicssearch.html and swishsearch.cgi. For more information about 
 swish-e see the homepage mentioned.


\section{Access Patterns}
In order to understand the system better it is useful to look at the flow
 of information and the collaboration of the individual components in more
 detail.
\subsection{Data Files}
\begin{enumerate}
\item Data Files are generated by the instrument control programs at the
 instrument computer.
\item Data Files are automatically mirrored to the central AFS 
repository. This happens through the FileSync server, 
a shell script and last not least the unix rsync utility. All this is
installed at the instrument computer.
\item Nightly new files are scanned for relevant information and their
 characteristics are entered into the database system from lnsl15. 
\end{enumerate}


\subsection{SICS servers}
SICS instrument control servers usually communicate with the terminal servers
 through the SerPortServer program, with temperature controllers through the
 TecsServer and directly via TCP/IP with the histogram memory computers. 
\subsection{SICS Java Clients}
The SICS Java clients access the SICS servers on the selected instruments 
 workstation. Most Java clients can display data from previously measured
 files. Such data is accessed through the NeXus Data Server running on the 
 laboratory server lnsa15. 
\subsection{WWW Services}
The file search service on the WWW--server lns00 accesses the database 
 server PSIP0.

The instrument status display servlets on the WWW--server lns00 access the
 SICS instrument control servers on the appropriate instrument computers 
 for data. 

\section{Maintainance Tasks}
\subsection{Yearly Maintainance}

Whith each new year a couple of things must be done in order to keep the
 system ship shape:
\begin{enumerate}
\item On each instrument computer:
 \begin{itemize}
 \item Create a new subdirectory for data files for the new year.
 \item In the new subdirectory create a DataNumber file with the data file
 number set to 0.
 \item Edit the path names in the data file mirroring script to point to the
 new years subdirectory. This script is usually called {\tt instsync} with
 inst replaced by the name of the instrument.
 \item Edit the instrument configuration file and adapt the path names for
 the data directory and the DataNumber file for the new year.
 \end{itemize}
\item On the laboratory server lnsa15:
\begin{itemize}
  \item Create a new year hierarchy for the new year.
  \item Store the old years data into the PSI archive. 
  \item Move the old years data into the non backed up AFS area.
  \item Make the new year the backed up data on AFS. 
\end{itemize}
\item On the WWW--server lns00:
\begin{itemize}
\item Add the new year into the html--pages for the data file search.
\item Check the configuration file for the status servlets, sics.conf, 
 for necessary updates caused by computer changes.
\end{itemize}
\end{enumerate}
Do these things quite early on in the year. People start measuring background
 quite early!

\subsection{Exchange of Instrument Computers}
In order to keep up with ageing 1-2 computers have to be exchanged any year.
 A instrument computer change requires the following adaptions:
\begin{enumerate}
\item Start with a PC with the PSI Linux distribution installed.
\item Create two local user accounts: INST and instlnsg. 
\item Copy the SICS software to the instrument computer.
\item Edit the instrument configuration file and change all occurences of
 the old computers name to the new name.
\item Install the cron job for backing up the local instrument
  accounts.
\item Make an entry in the .rhosts file of the SINQINST account on lnsl15. 
\item For the Java clients edit lnet/SINQConfig.java to point to the
 new computer and rebuild all clients. Redistribute them as well.
\item For the WWW status, adapt the sics.conf file on lns00 and restart 
 the WWW-server.
\end{enumerate}
\end{document}