
MUD Data Format

Overview:

The muon data (MUD) format used at TRIUMF is an efficent framework
for storing and retrieving muSR data.  Its existing structure is
optimized for muSR data, but the basic framework is universal and
the definition is extensible, so it can grow to meet new demands.

The MUD format is characterized by the organization of data
into Sections.  The instance of each Section is defined by two
long integers: the Type identification (secID) and a secondary
number specifying the instance of the type (instanceID).
The format has been designed to allow for quick implementation of
new Section types or modifications of old types, high flexibility
while maintaining a standard, and ease of use for the application
programmer.  Type identifiers can be assigned to new Section types
used for applications local to a lab, but the ideal is to have
shared data type specifications, with generally-accepted ID codes.
To this end, ranges of secID numbers are reserved for individual
laboratories, to prevent conflicts of local definitions.
Additionally, there is a range for generic secIDs whose
definitions can be collected (at TRIUMF) and distributed with
the MUD program libraries.

File Contents:

A MUD data format file consists of sequential contiguous Sections
in a stream file. Common to all types of MUD Section is a small
Core structure giving Section type and instance, size, and a
pointer to the next Section.  This design, with the relative file
position of the next Section encoded into the Core part of each
Section, allows for traversal of the file and for the modification
of the contents of a Section type while maintaining backwards
compatibility with MUD file readers.

Sections of any combination of types may be organized into Groups.
The Group is simply a class of MUD Section which indicates that a
number of following Sections are to be grouped together. The Group
provides an index of (relative file position) pointers to those
ensuing Sections, which may be thought of as being contained within
the Group Section.  The whole MUD file is a particular case
of a MUD group.

The most important MUD Sections are those that hold data, and
the secID implies how to read that data. Not only the content
and the organization, but also the encoding are specified by the
definition of the Section type. Encoding includes byte order
and floating-point format (although floating-point should be
avoided). The MUD library includes standard routines to read
the file's encoding into the computer's native format.

Copying MUD files:

MUD files are pure binary streams, without any record format.
This is attractive for moving files between operating systems, 
some of which have no concept of file records.  On VMS systems,
with a rich repertoire of record types, MUD files are typically
called "stream LF" because that is what C programs automatically
produce, even though the correct type should be "stream".  FTP
transfer between VMS and Unix systems does not work well, because
it usually assumes a particular record format (fixed 512 byte
records) on the VMS end.  To use FTP one should first "zip" the
files, and transfer the zip archive.  NFS works well.  

Software Library/Applications:

It is intended that the data be accessed via the set of supplied
routines for reading and writing the MUD format.  There are both
high-level (API) routines for accessing particular components of
existing MUD Sections, and low-level routines used to implement
the API or to read and write MUD files directly.  If such
low-level access is desired, inspection of the (C language) source
code (which is not extensive) should reveal the necessary
information.

Defining New Types:

The specification of a Section type includes the following four
steps:

    1.  Definition of a structure in C, and optionally a
        corresponding structure in Fortran (see Example 1)

    2.  Writing one C subroutine that handles the specifics of
        the I/O, etc., in a brief and well-defined manner
        (Example 2).

    3.  Adding an entry to the C subroutine that dynamically
        creates instances of each Section.

    4.  Reserving the unique 32-bit integer identifier(s) for
	the new type.

and optionally,

    5.  Adding the corresponding "friendly" API functions.

The Section definition and its ID should be contributed to the
centralized library (maintained at TRIUMF), allowing all MUD-aware
applications to understand them.

Applications may be written in C or Fortran and linked to the MUD
library, although the natural language is C for low-level access.
In C, the Sections may be written from a linked list of
structures.  Routines are available for the creation of Sections and
maintaining the list.  In both languages, the entire file may be
read into a linked list, and then search routines are used to access
specific Sections of the list (see Example 3).  Alternatively, the
I/O of each Section may be done separately, also in both languages
(see Example 4).  Access to individual Sections in the data file
may be sequential or pseudo-direct.  The pseudo-direct access
involves the call to a routine with the request for a Section of a
certain ID; the routine then searches the file from the current
position for the requested Section and positions the file pointer
to the beginning of this Section.


------------------------------------------------------------------------------

Example 1.  Sample MUD format structure (in C; from mud.h)

typedef struct {
    MUD_CORE	core;

    UINT32  ID;
    UINT32  prevReplyID;
    UINT32  nextReplyID;
    TIME    time;
    char*   author;
    char*   title;
    char*   comment;
} MUD_SEC_CMT;


The same structure for (VAX) Fortran (from mud.finc)

        structure /MUD_SEC_CMT/
	    record /MUD_CORE/ core

	    integer*4  ID
	    integer*4  prevReplyID
	    integer*4  nextReplyID
	    integer*4  time
	    integer*4  pcsAuthor
	    integer*4  pcsTitle
	    integer*4  pcsComment
        end structure
        
The same structure for Fortran 90/95 (from mud.f90)

        type MUD_SEC_CMT
            sequence
	    type(MUD_CORE) core

	    integer*4  ID
	    integer*4  prevReplyID
	    integer*4  nextReplyID
	    integer*4  time
	    integer*4  pcsAuthor
	    integer*4  pcsTitle
	    integer*4  pcsComment
        end type

Note that the Fortran structures are identical to the C structure,
including the use of pointers to strings.  Subroutines are provided
for conversion between these pointers and ordinary Fortran character
variables (see Example 4, fMUD_ctofString).

-----------------------------------------------------------------------------

Example 2.  Subroutine to handle the type in Example 1:


int
MUD_SEC_CMT_proc( op, pBuf, pMUD )
    MUD_OPT op;
    BUF* pBuf;
    MUD_SEC_CMT* pMUD;
{
    int size;
    char tempStr1[32];

    switch( op )
    {
	case MUD_FREE:
	    _free( pMUD->author );
	    _free( pMUD->title );
	    _free( pMUD->comment );
	    break;
	case MUD_DECODE:
	    decode_4( pBuf, &pMUD->ID );
	    decode_4( pBuf, &pMUD->prevReplyID );
	    decode_4( pBuf, &pMUD->nextReplyID );
	    decode_4( pBuf, &pMUD->time );
	    decode_str( pBuf, &pMUD->author );
	    decode_str( pBuf, &pMUD->title );
	    decode_str( pBuf, &pMUD->comment );
	    break;
	case MUD_ENCODE:
	    encode_4( pBuf, &pMUD->ID );
	    encode_4( pBuf, &pMUD->prevReplyID );
	    encode_4( pBuf, &pMUD->nextReplyID );
	    encode_4( pBuf, &pMUD->time );
	    encode_str( pBuf, &pMUD->author );
	    encode_str( pBuf, &pMUD->title );
	    encode_str( pBuf, &pMUD->comment );
	    break;
	case MUD_GET_SIZE:
	    size = 3*sizeof( UINT32 );
	    size += 1*sizeof( TIME );
	    size += sizeof( MUD_STR_LEN_TYPE ) + _strlen( pMUD->author );
	    size += sizeof( MUD_STR_LEN_TYPE ) + _strlen( pMUD->title );
	    size += sizeof( MUD_STR_LEN_TYPE ) + _strlen( pMUD->comment );
	    return( size );
	case MUD_SHOW:
	    printf( "  MUD_SEC_CMT: \n" );
	    printf( "    number:[%ld],  prevReply:[%ld],  nextReply:[%ld]\n", 
			pMUD->ID, pMUD->prevReplyID, pMUD->nextReplyID );
	    strcpy( tempStr1, ctime( (time_t*)&pMUD->time ) );
	    tempStr1[strlen(tempStr1)-1] = '\0';
	    printf( "    time:[%s]\n", tempStr1 );
	    if( pMUD->author ) printf( "    author:\"%s\"\n", pMUD->author );
	    if( pMUD->title ) printf( "    title:\"%s\"\n", pMUD->title );
	    if( pMUD->comment ) printf( "    comment:\"%s\"\n", pMUD->comment );
	    break;
	case MUD_HEADS:
            printf( "Comment number %ld.     ", pMUD->ID );
            if( pMUD->prevReplyID > 0 )
              printf("  Re: #%ld.    ", pMUD->prevReplyID );
            if( pMUD->nextReplyID > 0 )
              printf("  Next: #%ld.", pMUD->nextReplyID );
            printf( "\n" );
	    strcpy( tempStr1, ctime( (time_t*)&pMUD->time ) );
	    tempStr1[strlen(tempStr1)-1] = '\0';
	    if( pMUD->author ) printf( "    author: %s,     time: %s\n", pMUD->author, tempStr1 );
	    if( pMUD->title ) printf( "    title: %s\n", pMUD->title );
	    if( pMUD->comment ) printf( "%s\n", pMUD->comment );
	    break;
    }
    return( 1 );
}



-----------------------------------------------------------------------------

Example 3.  Sample C application:

/*
 *  mud_test.c
 */

#include 'mud.h'

int
main( void )
{
    FILE* fin;
    FILE* fout;
    MUD_SEC* pMUD_head = NULL;
    MUD_SEC_GEN_RUN_DESC* pMUD_desc;
    MUD_SEC_GEN_HIST* pMUD_hist;
    char* filename = "006663.mud";

    /*
     *  Read an MUD format file into a linked list
     */
    fin = MUD_openInput( filename );
    if( fin == NULL ) exit( 0 );

    MUD_readFile( fin, &pMUD_head );

    fclose( fin );

    /*
     *  Access the (header for) the third histogram in the TD histogram group
     */
    pMUD_hist = MUD_search( pMUD_head, MUD_SEC_GROUP_ID, MUD_GRP_TRI_TD_HIST_ID,
				       MUD_SEC_GEN_HIST_HDR_ID, 3, 
				       0 );
    printf( "Number of bins: %d\n", pMUD_hist->nBins )

    /*
     *  Add a run description section (#2) to the list
     */
    pMUD_desc = MUD_new( MUD_SEC_GEN_RUN_DESC_ID, 2 );
    MUD_add( &pMUD_head, pMUD_desc );

    /*
     .
     .
     .

     */

    /*
     *  Write an MUD format file
     */
    fout = MUD_openOutput( filename );
    if( fout == NULL ) exit( 0 );

    MUD_writeFile( fout, pMUD_head, MUD_FMT_ALL_ID );

    fclose( fout );

    /*
     *  Free the linked list
     */
    MUD_free( pMUD_head );
}

/*
 *  end mud_test.c
 */


-------------------------------------------------------------------------------

Example 4.  Sample Fortran applications:

Modern Fortran:
~~~~~~~~~~~~~~~
	program mud_test_fortran
	implicit none

	include 'mud.f90'

	integer, parameter :: i4 = selected_int_kind(9) ! integer*4

	integer(kind=i4) status
	integer(kind=i4) i
	character(len=32) filename
	integer(kind=i4) fileHandle
	character(len=20) title

	type(MUD_SEC_GEN_HIST_HDR) MUD_hist_hdr(8)


	!
	!  Open an MUD format file
	!
	filename = '001234.msr'

	fileHandle = fMUD_openInput( filename )
        if (fileHandle .eq. 0) then
           write (*,*) 'Could not open file ',filename,
     +          '  (',fileHandle,')'
           stop
        endif
        write (*,*) 'Opened file ', filename

	!
	!  Position the file before the first histogram of the 
	!  TD histogram group
	!
	status = fMUD_fseek( fileHandle, 
     +			     MUD_SEC_GRP_ID, MUD_GRP_TRI_TD_HIST_ID,
     +			     0, 0 )
        if( status .eq. -1 ) then
           write (*,*) 'Failed to find histogram group!  status=',status
           goto 999
        endif
	!
	!  Read the histogram headers
	!

	do i=1,8  !  we dimensioned MUD_hist_hdr(8)

	    status = fMUD_fseek( fileHandle, 
     +			   MUD_SEC_GEN_HIST_HDR_ID, i, 
     +                     0)
            !
	    !  If no more histograms, then we are finished:
            !
	    if (status .eq. -1 ) exit

	    status = fMUD_read( fileHandle, MUD_hist_hdr(i) )

	    !
	    !  Access the histogram title
	    !
            if (status.eq.1) then
               call fMUD_ctofString( title, MUD_hist_hdr(i)%pcsTitle )
               write (*,*) 'histogram ',i,'  title = "',trim(title),'"'
            else
               write (*,*) 'Failed to read header for histogram',i
            endif

	end do

 999	continue

	call fMUD_close( fileHandle )

	end program mud_test_fortran



Sample for VAX Fortran:
~~~~~~~~~~~~~~~~~~~~~~~
	program mud_test_fortran
	implicit none

	include 'mud.finc'

	integer*4 status
	integer*4 i
	character*32 filename
	integer*4 fileHandle
	character*20 title

        record /MUD_SEC_GEN_HIST_HDR/ MUD_hist_hdr(8)


	!
	!  Open an MUD format file
	!
	filename = '001234.msr'

	fileHandle = fMUD_openInput( filename )
        if (fileHandle .eq. 0) then
           write (*,*) 'Could not open file ',filename,
     >          '  (',fileHandle,')'
           stop
        endif
        write (*,*) 'Opened file ', filename

	!
	!  Position the file before the first histogram of the 
	!  TD histogram group
	!
	status = fMUD_fseek( fileHandle, 
     +			     MUD_SEC_GRP_ID, MUD_GRP_TRI_TD_HIST_ID,
     +			     0, 0 )
        if( status .eq. -1 ) then
           write (*,*) 'Failed to find histogram group!  status=',status
           goto 999
        endif
	!
	!  Read the histogram headers
	!

	do i=1,8  !  we dimensioned MUD_hist_hdr(8)

	    status = fMUD_fseek( fileHandle, 
     +			   MUD_SEC_GEN_HIST_HDR_ID, i, 
     +                     0)
            !
	    !  If no more histograms, then we are finished:
            !
	    if (status .eq. -1 ) goto 999

	    status = fMUD_read( fileHandle, MUD_hist_hdr(i) )

	    !
	    !  Access the histogram title
	    !
            if (status.eq.1) then
               call fMUD_ctofString( title, MUD_hist_hdr(i).pcsTitle )
               write (*,*) 'histogram ',i,'  title = "',title,'"'
            else
               write (*,*) 'Failed to read header for histogram',i
            endif

	end do

 999	continue

	call fMUD_close( fileHandle )

	end ! program mud_test_fortran


Sample for old (but extended) Fortran77 (g77):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	program mud_test_fortran

	include 'mud.f77'

	integer*4 status
	integer*4 i
	character*32 filename
	integer*4 fileHandle
	character*20 title

	integer*4 MUD_hist_hdr

*  Hist header structure, implemented as a common block.
*  All hist header elements are named hh_... The core elements
*  are named hh_c_...

            integer*4	hh_c_pNext		!* pointer to next section *
            integer*4	hh_c_size
            integer*4	hh_c_secID		!* Ident of section type *
            integer*4	hh_c_instanceID
            integer*4	hh_c_sizeof
            integer*4	hh_c_proc
!         
	    integer*4	hh_histType
	    integer*4	hh_nBytes
	    integer*4	hh_nBins
	    integer*4	hh_bytesPerBin
	    integer*4	hh_fsPerBin
	    integer*4	hh_t0_ps
	    integer*4	hh_t0_bin
	    integer*4	hh_goodBin1
	    integer*4	hh_goodBin2
	    integer*4	hh_bkgd1
	    integer*4	hh_bkgd2
	    integer*4	hh_nEvents
	    integer*4	hh_pcsTitle


        common /cmn_hdr/ 
     +   hh_c_pNext, hh_c_size, hh_c_secID, hh_c_instanceID, 
     +   hh_c_sizeof, hh_c_proc,
     +   hh_histType, hh_nBytes, hh_nBins, hh_bytesPerBin, 
     +   hh_fsPerBin, hh_t0_ps, hh_t0_bin, hh_goodBin1, hh_goodBin2,
     +   hh_bkgd1, hh_bkgd2, hh_nEvents, hh_pcsTitle

	equivalence(hh_c_pNext,MUD_hist_hdr)

	!
	!  Open an MUD format file
	!
	filename = '001234.msr'

	fileHandle = fMUD_openInput( filename )
        if (fileHandle .eq. 0) then
           write (*,*) 'Could not open file ',filename,
     >          '  (',fileHandle,')'
           stop
        endif
        write (*,*) 'Opened file ', filename

	!
	!  Position the file before the first histogram of the 
	!  TD histogram group
	!
	status = fMUD_fseek( fileHandle, 
     +			     MUD_SEC_GRP_ID, MUD_GRP_TRI_TD_HIST_ID,
     +			     0 )
        if( status .eq. -1 ) then
           write (*,*) 'Failed to find histogram group!  status=',status
           goto 999
        endif
	!
	!  Read the histogram headers
	!
	do i=1,16

	    status = fMUD_fseek( fileHandle, 
     +			   MUD_SEC_GEN_HIST_HDR_ID, i, 
     +                     0 )

	    if (status .eq. -1 ) goto 999

	    status = fMUD_read( fileHandle, MUD_hist_hdr )

	    !
	    !  Access the histogram title
	    !
            if (status.eq.1) then
               call fMUD_ctofString( title, hh_pcsTitle )
               write (*,*) ' histogram title = <', title, '>'
            else
               write (*,*) ' Failed to read header for histogram',i
            endif

	end do

 999	continue
	call fMUD_close( fileHandle )
	stop
	end


------------------------------------------------------------------------------
