SICS Trouble Shooting

There is no such thing as bug free software. There are always bugs, nasty behaviour etc. This document shall help to solve these problems. The usual symptom will be that a client cannot connect to the server or the server is not responding. Or error messages show up. This section helps to solve such problems.

Looking at Log Files

The first thing to do, especially when confronted with confusing statements from either users or instrument scientists, is to look at the SICS servers log files. The last 1000 lines of the instrument log are accessible from any SICS client or through the WWW interface. The SICS commands:

commandlog tail: shows the last 20 lines of the log.
commandlog tail n: shows the last n lines of the log.

will show you the information available. In order to see more, log in to the instrument account. There the following unix commands might help:

sicstail shows the last 20 lines of the current log file and its name
sicstail n shows the last n lines of the current log file.

In order to see some more, cd into the log directory of the instrument account. In there are files with names like:

auto2001-08-08@00-01-01.log

This means the log file has been started at August, 8, 2001 at 00:01:01. There is a new log file daily. Load appropriate files into the editor and look what really happened.

Another good ideas is to use the unix command grep on assorted log files. A grep for the strings ERROR or WARNING will more ofteh then not give an indication for the nature of the problem.

The log files show you all commands given and all the responses of the system. Additionally there are hourly time stamps in the file which allow to narrow in when the problem started. Things to watch out for are:

MOTOR ALARM: This message means that the motor failed to reach his position for a couple of times. This is caused by either a concrete shielding element blocking the movement of the instrument, badly adjusted motor parameters, mechanical failures or the air cushions not operating properly.
EL734__BAD_EMERG_STOP: Somebody has pushed the emergency stop button. This must be released before the instrument can move again. Moreover the motor controller will not respond to further commands in this mode. Thus restarting SICS on this error message will make SICS fail to initialize the motors affected!
EL***__BAD_PIPE, BAD_RECV, BAD_ILLG, BAD_TMO, BAD_SEND: Network communication problems. Can generaly be solved by restarting SICS.
EL737__BAD_BSY: A counting operation was aborted while the beam was off. Unfortunately, the counter box does not respond to commands in this state and ignores the stop command sent to it during the abort operation. This can be safely ignored, SICS fixes this condition.

Starting SICS

An essential prerequisite of SICS is that the server is up and running. The system is configured to restart the SICServer whenever it fails. Only after a reboot or when the keepalive processes were killed (see below) the SICServer must be restarted. This is done for all instruments by typing:

startsics

at the command prompt. startsics actually starts two programs: one is the replicator application which is responsible for the automatic copying of data files to the laboratory server. The other is the SICS server. Both programs are started by means of a shell script called keepalive. keepalive is basically an endless loop which calls the program again and again and thus ensures that the program will never stop running.

When the SICS server hangs, or you want to enforce an reinitialization of everything the server process must be killed. This can be accomplished either manually or through a shell script.

Stopping SICS

All SICS processes can be stopped through the command:

killsics

given at the unix command line. You must be the instrument user (for example DMC) on the instrument computer for this to work properly.

Restart Everything

If nothing seems to work any more, no connections can be obtained etc, then the next guess is to restart everything. This is especially necessary if mechanics or electronics people were closer to the instrument then 400 meters.

Reboot the histogram memory. It has a tiny button labelled RST. That' s the one. Can be operated with a hairpin, a ball point pen or the like.
Wait 5 minutes.
Restart the SICServer. Watch for any messages about things not being connected or configured.
Restart and reconnect the client programs.

If this fails (even after a second) time there may be a network problem which can not be resolved by simple means.

Checking SICS Startup

Sometimes it happens that the SICServer hangs while starting up or hardware components are not properly initialized. In such cases it is useful to look at the SICS servers startup messages. In order to do so, both the SICServer and its keepalive process must be killed first. On the instrument acount issue the command:

ps -A | grep SICS

A message like this will be printed:

  23644 ??       I        0:00.00 ksh keepalive SICServer focus.tcl
 23672 ??       R       59:24.05 SICServer focus.tcl
  7119 ttyp6    S  +     0:00.00 grep SICS

Remember the numbers in the first columns (the PID's) and kill both programs by issuing the command:

kill -9 pid pid

Example:

kill -9 23644 23672

Note, the numbers are those displayed with the ps -A command. Then cd into the bin directory of the instrument account and issue the unix command:

SICServer inst.tcl | more

Replace inst.tcl with the name of the appropriate instrument initialisation file. This allows to page through SICS startup messages and will help to identify the troublesome component. The proceed to check the component and the connections to it.

Getting New SICS Software

Sometimes you might want to be sure that you have the latest SICS software. This is how to get it:

Login to the instrument account.
If you are no there type cd to get into the home directory.
Type killsics at the unix prompt in order to stop the SICS server.
Type sicsinstall exe at the unix prompt for copying new SICS software from the general distribution area.
Type startsics to restart the SICS software.

Hot Fixes

When there is trouble with SICS you may be asked by one of the SICS programmers to copy the most recent development reason of the SICS server to your machine. This is done as follows:

Login to the instrument account.
cd into the bin directory, for example: /home/DMC/bin.
Type killsics at the unix prompt in order to stop the SICS server.
Type cp /data/koenneck/src/sics/SICServer . at the unix prompt.
Type startsics to restart the SICS software.

!!!!!! WARNING !!!!!!!. Do this only when advised to do so by a competent SICS programmer. Otherwise you might be copying a SICS server in an instable experimental state!

HELP debugging!!!!

The SICS server hanging or crashing should not happen. In order to sort such problems out it is very helpful if any available debugging information is saved and presented to the programmers. Information available are the log files as written continously by the SICS server and posssible core files lying around. They have just this name: core. In order to save them create a new directory (for example dump2077) and copy the stuff in there. This looks like:

/home/DMC> mkdir dump2077
/home/DMC> cp log/*.log dump2077
/home/DMC> cp core dump2077

The /home/DMC> is just the command prompt. Please note, that core files are only available after crashes of the server. These few commands will help to analyse the cause of the problem and to eventually resolve it.