updated NEWS with details about 1.6 changes; changed hsize_t type in H5PartSetViewIndices; fixes to the build system; fixed bugs that were preventing the use of H5PartSetView during writes; added open object checks to regression test

NEWS

@@ -1,3 +1,129 @@
+#### H5PART 1.6 ##############################################################
+
+Updated Documentation
+---------------------
+
+The Fortran API has been added to the doxygen documentation. Most entries have
+a reference to their respective C API call.
+
+Regression Test Suite
+---------------------
+
+The test subdirectory has been reorganized to include a systematic series
+of regression tests of common use cases for the API. The goal is to have both
+complete coverage of the API (every call is exercised) and testing of some
+unlikely or unsuspected values and cases.
+
+Currently, the C test covers the entire H5Part API and the Fortran test is
+still in development.
+
+Benchmark Utilities
+-------------------
+
+Two benchmarking utilities, H5PartBench and H5BlockBench, have been added to
+the tools collection. Both require the parallel library and have functionality
+and syntax similar to the IOR benchmark:
+
+  http://sourceforge.net/projects/ior-sio/
+
+File Mode Flags
+---------------
+
+Previously, the only flags used were H5PART_READ, H5PART_WRITE, and
+H5PART_APPEND, and these were mutually exclusive. Additional flags have been
+added to select the underlying virtual file driver (VFD) that HDF5 uses in
+parallel mode and to accommodate the lustre filesystem:
+
+H5PART_VFD_MPIPOSIX (use the MPI-POSIX driver, which bypasses MPI-IO)
+H5PART_VFD_MPIIO_IND (use MPI-IO in independent mode)
+H5PART_FS_LUSTRE (activate H5P tunings for the lustre stripe size)
+
+These flags can be specified using the typical bitwise OR method, e.g.
+
+  char flag = H5PART_WRITE | H5PART_VFD_MPIPOSIX | H5PART_FS_LUSTRE;
+
+will select write mode with these two additional options.
+
+The parallel library now defaults to using the MPI-IO collective mode driver
+(previously it used independent mode). The collective buffering algorithm in
+the Cray XT4/5 environment has been substantially improved in the last year,
+and defaulting to collective mode allows us to take advantage of these
+improvements automatically.
+
+There are also new open calls H5PartOpenAlign and H5PartOpenParallelAlign that
+take an additional 'alignment' value. This value is passed to HDF5 and used to
+pad out objects so that they align to filesystem boundaries. For instance, it
+would make sense to use the stripe size as the alignment value when writing to
+a lustre filesystem.
+
+Unified 'View' Model
+--------------------
+
+Views are now supported in both read and write mode (previously only supported
+in read mode). It is possible to specify non-sensical views: you could for
+instance specify a read view that is larger than the dataset on disk. Or you
+could create a write view that is larger than the data in memory. In those
+cases, you will encounter an error when you try to perform a read or write
+operation on a dataset.
+
+There are two new methods for selecting views. H5PartSetNumParticlesStrided
+selects a view with $n$ particles per processor, but such that the data in
+memory is expected to have a 'stride' factor. For instance, if you have
+particle data with fields $x$ and $y$ and a single array with entries
+$x1,y1,x2,y2...$, then the stride factor is 2. Striding works for both reads
+and writes and only affects the view of memory: individual fields are still
+stored as individual arrays on disk.
+
+The second new method is H5PartSetViewIndices, which allows for point
+selections of datasets. You can pass a list of indices for the points you
+want to select. This is useful, for instance, when interfacing with FastBit
+to perform queries that select a small subset of non-contiguous particles.
+
+Because views are now supported on write, it is possible to write a dataset
+using multiple 'passes' or to leave some values unwritten.
+
+Internal Handling of Dataspaces
+-------------------------------
+
+H5PartSetNumParticles and H5PartSetNumParticlesStrided now share the same
+HDF5 dataspace state in the H5PartFile struct with the calls that modify
+the view (H5PartSetView etc.). Previously, the memory and disk dataspaces
+were regenerated from the view on every read access, which was unnecessary.
+
+Autogeneration of API Calls
+---------------------------
+
+Python scripts have been added to automatically generate read/write call
+variants for different data types.
+
+Renamed Fortran Include
+-----------------------
+
+The automake system no longer greps the *F90.inc files to generate an include
+called 'H5Part.inc'. Instead, the *F90.inc files have been renamed to *.f90,
+and awk is used to generate an 'H5PartF.h' file (the same naming convention
+as in MPI's 'mpif.h').
+
+The change to *.f90 was necessary to facilitate integration of the Fortran
+API into doxygen. Note that the *.f90 files will not compile, nor can they
+be included directly. Simply use
+
+  include 'H5PartF.h'
+
+in your Fortran code to include all the definitions for Fortran H5Part/H5Block
+calls.
+
+64 Char Limit on Dataset Names
+------------------------------
+
+Previously, a user could overrun internal buffers for dataset names. Now, a
+fixed limit of 64 chars is imposed. Dataset names that are longer than this are
+truncated and a warning is printed. We expect that most users are using
+short canoncical names like x, px, id, etc.
+
+
+#### H5PART 1.4 ##############################################################
+
 Error Handling
 --------------
  Now all functions, in which an error could occure, are returning a

configure.ac

@@ -390,6 +390,7 @@ if test "X$USE_PARALLEL" = "Xyes"; then
         fi
     fi
 
+    H5P_LIB_NAME="-lpH5Part"
     MTARGET="libpH5Part.a"
     TTARGET="test"
 
@@ -417,6 +418,7 @@ if test "X$USE_PARALLEL" = "Xyes"; then
 	    FC=${MPIFC}
 	fi
  
+        H5P_LIB_NAME="-lpH5PartF"
         MTARGET="${MTARGET} libpH5PartF.a"
         TTARGET="${TTARGET} testf"
     fi
@@ -424,10 +426,12 @@ if test "X$USE_PARALLEL" = "Xyes"; then
 else  # --enable-parallel=no
     AC_MSG_RESULT([no])
 
+    H5P_LIB_NAME="-lH5Part"
     MTARGET="libH5Part.a"
     TTARGET="test"
 
     if test "X$USE_FORTRAN" = "Xyes"; then
+        H5P_LIB_NAME="-lH5PartF"
         MTARGET="${MTARGET} libH5PartF.a"
         TTARGET="${TTARGET} testf"
     fi
@@ -596,6 +600,7 @@ AC_SUBST(CFLAGS)
 AC_SUBST(FFLAGS)
 AC_SUBST(STDCXX)
 AC_SUBST(H5P_LIB_LOC)
+AC_SUBST(H5P_LIB_NAME)
 AC_SUBST(UNDERSCORE_H)
 AC_SUBST(BUILD_TOOLS)
 AC_SUBST(LDFLAGS)

src/H5Part.c

@@ -32,7 +32,7 @@ Papers:
 </LI>
 
 
-<LI> A. Adelmann, R.D. Ryne, J. Shalf, C. Siegerist,"H5Part: A Portable High Performance Parallel Data Interface for Particle Simulations," <i>
+<LI> A. Adelmann, R.D. Ryne, J. Shalf, C. Siegerist, "H5Part: A Portable High Performance Parallel Data Interface for Particle Simulations," <i>
 <a href="http://www.sns.gov/pac05">Particle Accelerator Conference (PAC05)</a></i>, Knoxville TN., May 16-20, 2005.
 <a href="http://vis.lbl.gov/Publications/2005/FPAT083.pdf">FPAT083.pdf</a>
 </LI>
@@ -440,7 +440,6 @@ H5PartOpenFileParallelAlign (
 
   \return	File handle or \c NULL
  */
-
 H5PartFile*
 H5PartOpenFile (
 	const char *filename,	/*!< [in] The name of the data file to open. */
@@ -497,23 +496,12 @@ _H5Part_file_is_valid (
 		return H5PART_ERR_BADFD;
 }
 
-/*!
-  \ingroup h5part_open
-
-  Closes an open file.
-
-  \return	\c H5PART_SUCCESS or error code
-*/
 h5part_int64_t
-H5PartCloseFile (
+_H5Part_close_hdf_ids (
 	H5PartFile *f		/*!< [in] filehandle of the file to close */
 	) {
 
-	SET_FNAME ( "H5PartCloseFile" );
 	herr_t r = 0;
-	_h5part_errno = H5PART_SUCCESS;
-
-	CHECK_FILEHANDLE ( f );
 
 	if ( f->block && f->close_block ) {
 		(*f->close_block) ( f );
@@ -564,11 +552,38 @@ H5PartCloseFile (
 		if ( r < 0 ) HANDLE_H5P_CLOSE_ERR ( "f->create_prop" );
 		f->create_prop = H5P_DEFAULT;
 	}
+
+	return H5PART_SUCCESS;
+}
+
+/*!
+  \ingroup h5part_open
+
+  Closes an open file.
+
+  \return	\c H5PART_SUCCESS or error code
+*/
+h5part_int64_t
+H5PartCloseFile (
+	H5PartFile *f		/*!< [in] filehandle of the file to close */
+	) {
+
+	SET_FNAME ( "H5PartCloseFile" );
+	herr_t r = 0;
+	_h5part_errno = H5PART_SUCCESS;
+
+	CHECK_FILEHANDLE ( f );
+
+	r = _H5Part_close_hdf_ids ( f );
+	if ( r < 0 ) return r;
+
 	if ( f->file ) {
 		r = H5Fclose( f->file );
 		if ( r < 0 ) HANDLE_H5F_CLOSE_ERR;
 		f->file = 0;
 	}
+
+	/* free memory from H5PartFile struct */
 	if( f->pnparticles ) {
 		free( f->pnparticles );
 	}
@@ -688,6 +703,12 @@ _set_num_particles (
 	herr = _reset_view ( f );
 	if ( herr < 0 ) return herr;
 
+	if ( f->shape != H5S_ALL ) {
+		herr = H5Sclose ( f->shape );
+		if ( herr < 0 ) return HANDLE_H5S_CLOSE_ERR;
+		f->shape = H5S_ALL;
+	}
+
 	f->nparticles = (hsize_t) nparticles;
 
 	/* declare local memory datasize with striding */
@@ -781,12 +802,8 @@ _set_num_particles (
 		&stride,
 		&count, NULL );
 	if ( herr < 0 ) return HANDLE_H5S_SELECT_HYPERSLAB_ERR;
-
-	if ( f->timegroup < 0 ) {
-		herr = _H5Part_set_step ( f, 0 );
-		if ( herr < 0 ) return herr;
-	}
 #endif
+
 	return H5PART_SUCCESS;
 }
 
@@ -797,7 +814,7 @@ _set_num_particles (
   After you call this subroutine, all subsequent 
   operations will assume this number of particles will be written.
 
-  For the parallel library, the \a nparticles value is the number of
+  For the parallel library, the \c nparticles value is the number of
   particles that the \e individual task will write. You can use
   a different value on different tasks.
   This function uses an \c MPI_Allgather
@@ -808,11 +825,11 @@ _set_num_particles (
 
   This function assumes that your particles' data fields are in stored in
   contiguous 1D arrays.
-  For instance, the fields $x$ and $y$ for your particles are stored
+  For instance, the fields \e x and \e y for your particles are stored
   in separate arrays \c x[] and \c y[].
   
   If instead you store your particles as tuples, so that the values
-  are arranged $x_1,y_1,x_2,y_2$... than you need to setup striding
+  are arranged \f$ x_1,y_1,x_2,y_2\f$... than you need to setup striding
   (in this case with value 2) using \ref H5PartSetNumParticlesStrided.
 
   \return	\c H5PART_SUCCESS or error code
@@ -842,7 +859,7 @@ H5PartSetNumParticles (
   After you call this subroutine, all subsequent 
   operations will assume this number of particles will be written.
 
-  For the parallel library, the \a nparticles value is the number of
+  For the parallel library, the \c nparticles value is the number of
   particles that the \e individual task will write. You can use
   a different value on different tasks.
   This function uses an \c MPI_Allgather
@@ -852,8 +869,8 @@ H5PartSetNumParticles (
   few times as possible when running at large concurrency.
 
   This function assumes that your particles' data fields are
-  stored tuples. For instance, the fields $x$ and $y$ of your
-  particles are arranged $x_1,y_1,x_2,y_2$... in a single data
+  stored tuples. For instance, the fields \e x and \e y of your
+  particles are arranged \f$x_1,y_1,x_2,y_2\f$... in a single data
   array. In this example, the stride value would be 2.
   
   If you instead have a separate array for each fields,
@@ -861,12 +878,12 @@ H5PartSetNumParticles (
   use \ref H5PartSetNumParticles.
 
   \return	\c H5PART_SUCCESS or error code
- */
+*/
 h5part_int64_t
 H5PartSetNumParticlesStrided (
 	H5PartFile *f,				/*!< [in] Handle to open file */
 	const h5part_int64_t nparticles,	/*!< [in] Number of particles */
-	const h5part_int64_t stride		/*!< [in] Stride (e.g. number of fields in the particle array) */
+	const h5part_int64_t stride		/*!< [in] Stride value (e.g. number of fields in the particle array) */
 	) {
 
 	SET_FNAME ( "H5PartSetNumParticlesStrided" );
@@ -880,6 +897,24 @@ H5PartSetNumParticlesStrided (
 	return H5PART_SUCCESS;
 }
 
+static void
+_normalize_dataset_name (
+	const char *name,
+	char *name2
+	) {
+
+	if ( strlen(name) > H5PART_DATANAME_LEN ) {
+		strncpy ( name2, name, H5PART_DATANAME_LEN - 1 );
+		name2[H5PART_DATANAME_LEN-1] = '\0';
+		_H5Part_print_warn (
+			"Dataset name '%s' is longer than maximum %d chars. "
+			"Truncated to: '%s'",
+			name, H5PART_DATANAME_LEN, name2 );
+	} else {
+		strcpy ( name2, name );
+	}
+}
+
 static h5part_int64_t
 _write_data (
 	H5PartFile *f,		/*!< IN: Handle to open file */
@@ -887,12 +922,17 @@ _write_data (
 	const void *array,	/*!< IN: Array to commit to disk */
 	const hid_t type	/*!< IN: Type of data */
 	) {
+
 	herr_t herr;
 	hid_t dataset_id;
 
-	_H5Part_print_debug ( "Create a dataset[%s] mounted on the "
+	char name2[H5PART_DATANAME_LEN];
+	_normalize_dataset_name ( name, name2 );
+
+	_H5Part_print_debug (
+			"Create a dataset[%s] mounted on "
 			"timestep %lld",
-			name, (long long)f->timestep );
+			name2, (long long)f->timestep );
 
 	if ( f->shape == H5S_ALL ) {
 		_H5Part_print_warn (
@@ -901,28 +941,39 @@ _write_data (
 		return HANDLE_H5PART_BAD_VIEW_ERR ( f->viewstart, f->viewend );
 	}
 
+	H5E_BEGIN_TRY
+	dataset_id = H5Dopen ( f->timegroup, name2
 #ifndef H5_USE_16_API
-	htri_t exists = H5Lexists ( f->timegroup, name, H5P_DEFAULT );
-	if ( exists > 0 ) return HANDLE_H5D_EXISTS_ERR ( name, f->timestep );
-
-	dataset_id = H5Dcreate2 ( 
-		f->timegroup,
-		name,
-		type,
-		f->shape,
-		H5P_DEFAULT,
-		H5P_DEFAULT,
-		H5P_DEFAULT );
-#else
-	dataset_id = H5Dcreate ( 
-		f->timegroup,
-		name,
-		type,
-		f->shape,
-		H5P_DEFAULT );
+		, H5P_DEFAULT
 #endif
-	if ( dataset_id < 0 )
-		return HANDLE_H5D_CREATE_ERR ( name, f->timestep );
+		);
+	H5E_END_TRY
+
+	if ( dataset_id > 0 ) {
+		_H5Part_print_warn (
+			"Dataset[%s] at timestep %lld "
+			"already exists", name2, (long long)f->timestep );
+	} else {
+#ifndef H5_USE_16_API
+		dataset_id = H5Dcreate2 ( 
+			f->timegroup,
+			name2,
+			type,
+			f->shape,
+			H5P_DEFAULT,
+			H5P_DEFAULT,
+			H5P_DEFAULT );
+#else
+		dataset_id = H5Dcreate ( 
+			f->timegroup,
+			name2,
+			type,
+			f->shape,
+			H5P_DEFAULT );
+#endif
+		if ( dataset_id < 0 )
+			return HANDLE_H5D_CREATE_ERR ( name2, f->timestep );
+	}
 
 #ifdef PARALLEL_IO
 	herr = _H5Part_start_throttle ( f );
@@ -942,7 +993,7 @@ _write_data (
 	if ( herr < 0 ) return herr;
 #endif
 
-	if ( herr < 0 ) return HANDLE_H5D_WRITE_ERR ( name, f->timestep );
+	if ( herr < 0 ) return HANDLE_H5D_WRITE_ERR ( name2, f->timestep );
 
 	herr = H5Dclose ( dataset_id );
 	if ( herr < 0 ) return HANDLE_H5D_CLOSE_ERR;
@@ -1825,7 +1876,7 @@ _H5Part_set_step (
 #endif
 					  );
 		if ( f->timegroup < 0 )
-			return HANDLE_H5G_OPEN_ERR( stepname );
+			return HANDLE_H5G_OPEN_ERR ( stepname );
 	}
 	else {
 		_H5Part_print_debug (
@@ -1834,14 +1885,11 @@ _H5Part_set_step (
 			(long long)step,
 			(long long)(size_t) f );
 
-		f->timegroup = H5Gcreate( f->file,
-					  stepname,
-					  0
+		f->timegroup = H5Gcreate( f->file, stepname, 0
 #ifndef H5_USE_16_API
 					  , H5P_DEFAULT, H5P_DEFAULT
 #endif
 					  );
-
 		if ( f->timegroup < 0 )
 			return HANDLE_H5G_CREATE_ERR ( stepname );
 	}
@@ -1893,7 +1941,10 @@ _H5Part_have_group (
 #ifndef H5_USE_16_API
 	return (H5Lexists( id, name, H5P_DEFAULT ) ? 1 : 0);
 #else
-	return (H5Gget_objinfo( id, name, 1, NULL ) >= 0 ? 1 : 0);
+	H5E_BEGIN_TRY
+	herr_t exists = H5Gget_objinfo( id, name, 1, NULL );
+	H5E_END_TRY
+	return (exists >= 0 ? 1 : 0);
 #endif
 }
 
@@ -2118,11 +2169,11 @@ _H5Part_get_num_objects_matching_pattern (
 	herr = H5Giterate ( group_id, group_name, &idx,
 			_H5Part_iteration_operator, &data );
 #endif
-
 	if ( herr < 0 ) return herr;
 
 #ifndef H5_USE_16_API
-	H5Gclose( child_id );
+	herr = H5Gclose ( child_id );
+	if ( herr < 0 ) return HANDLE_H5G_CLOSE_ERR;
 #endif
 
 	return data.count;
@@ -2166,13 +2217,14 @@ _H5Part_get_object_name (
 	if ( herr < 0 ) {
 		return HANDLE_H5L_ITERATE_ERR;
 	}
-	else if ( herr == 0 ) return 0;
 
 #ifndef H5_USE_16_API
-	herr = H5Gclose ( child_id );
-	if ( herr < 0 ) return HANDLE_H5G_CLOSE_ERR;
+	herr_t herr2 = H5Gclose ( child_id );
+	if ( herr2 < 0 ) return HANDLE_H5G_CLOSE_ERR;
 #endif
 
+	if ( herr == 0 ) return 0;
+
 	return 1;
 }
 
@@ -2393,11 +2445,8 @@ _H5Part_get_num_particles (
 	) {
 
 	h5part_int64_t herr;
-	hid_t space_id;
-	hid_t dataset_id;
-	char dataset_name[128];
-	hsize_t nparticles;
-
+	h5part_int64_t nparticles;
+	char dataset_name[H5PART_DATANAME_LEN];
 	char stepname[H5PART_STEPNAME_LEN];
 	_H5Part_get_step_name(f, f->timestep, stepname);
 
@@ -2406,38 +2455,70 @@ _H5Part_get_num_particles (
 		stepname,
 		H5G_DATASET,
 		0,
-		dataset_name, sizeof (dataset_name) );
+		dataset_name, H5PART_DATANAME_LEN );
 	if ( herr < 0 ) return herr;
-	else if ( herr == 0 ) {
-		_H5Part_print_warn ( 
-			"There are no datasets in timestep %s: "
-			"reporting 0 particles.", stepname );
-		return 0;
+	/* returns 0 if there are no datasets on disk */
+	else if ( herr == 0 )
+	{
+		/* try to recover number of particles from a previous
+		 * H5PartSetNumParticles call. */
+#ifdef PARALLEL_IO
+		nparticles = 0;
+		int i;
+		for (i=0; i < f->nprocs; i++) {
+			nparticles += f->pnparticles[i];
+		}
+#else
+		nparticles = f->nparticles;
+#endif
+
+		if ( nparticles > 0 ) {
+			_H5Part_print_debug (
+				"Using existing view to report "
+				"nparticles = %lld", (long long)nparticles );
+			return nparticles;
+		}
+		else {
+			_H5Part_print_warn ( 
+				"There are no datasets in timestep %s "
+				"or existing views: "
+				"reporting 0 particles.", stepname );
+			return 0;
+		}
 	}
 
-	dataset_id = H5Dopen ( f->timegroup, dataset_name
+	/* if a view exists, use its size as the number of particles */
+	if ( _H5Part_has_view ( f ) )
+	{
+		nparticles = H5Sget_select_npoints ( f->diskshape );
+		if ( nparticles < 0 ) return HANDLE_H5S_GET_SELECT_NPOINTS_ERR;
+
+		/* double check that the size of the diskshape agrees with
+		 * the size of the view */
+		if ( nparticles != f->viewend - f->viewstart + 1 ) {
+			_H5Part_print_warn (
+				"Number of particles (%lld) does not agree "
+				"with view range.", (long long)nparticles );
+			return HANDLE_H5PART_BAD_VIEW_ERR (
+					f->viewstart, f->viewend);
+		}
+	}
+	/* otherwise, report all particles on disk in the first dataset
+	 * for this timestep */
+	else
+	{
+		hid_t space_id;
+		hid_t dataset_id;
+
+		dataset_id = H5Dopen ( f->timegroup, dataset_name
 #ifndef H5_USE_16_API
 				, H5P_DEFAULT
 #endif
 				 );
 
-	if ( dataset_id < 0 ) 
-		return HANDLE_H5D_OPEN_ERR ( dataset_name );
+		if ( dataset_id < 0 ) 
+			return HANDLE_H5D_OPEN_ERR ( dataset_name );
 
-	/* if a view exists, use its size as the number of particles */
-	if ( _H5Part_has_view ( f ) ) {
-		nparticles = H5Sget_select_npoints ( f->diskshape );
-		if ( nparticles < 0 ) return HANDLE_H5S_GET_SELECT_NPOINTS_ERR;
-		if ( nparticles != f->viewend - f->viewstart + 1 )
-			_H5Part_print_warn (
-				"Number of particles (%lld) does not agree "
-				"with view range.", nparticles );
-			return HANDLE_H5PART_BAD_VIEW_ERR (
-					f->viewstart, f->viewend);
-	}
-	/* otherwise, report all particles on disk in the first dataset
-	 * for this timestep */
-	else {
 		space_id = H5Dget_space ( dataset_id );
 		if ( space_id < 0 ) return HANDLE_H5D_GET_SPACE_ERR;
 
@@ -2447,12 +2528,12 @@ _H5Part_get_num_particles (
 
 		herr = H5Sclose ( space_id );
 		if ( herr < 0 ) return HANDLE_H5S_CLOSE_ERR;
+
+		herr = H5Dclose ( dataset_id );
+		if ( herr < 0 ) return HANDLE_H5D_CLOSE_ERR;
 	}
 
-	herr = H5Dclose ( dataset_id );
-	if ( herr < 0 ) return HANDLE_H5D_CLOSE_ERR;
-
-	return (h5part_int64_t) nparticles;
+	return nparticles;
 }
 
 /*!
@@ -2499,12 +2580,6 @@ _reset_view (
 	f->viewstart = -1;
 	f->viewend = -1;
 
-	if ( f->shape != H5S_ALL ) {
-		herr = H5Sclose ( f->shape );
-		if ( herr < 0 ) return HANDLE_H5S_CLOSE_ERR;
-		f->shape = H5S_ALL;
-	}
-
 	if ( f->diskshape != H5S_ALL ) {
 		herr = H5Sclose ( f->diskshape );
 		if ( herr < 0 ) return HANDLE_H5S_CLOSE_ERR;
@@ -2580,11 +2655,15 @@ _set_view (
 	  end==-1 to mean end of file
 	*/
 	total = (hsize_t) _H5Part_get_num_particles ( f );
-	if ( total < 0 ) return HANDLE_H5PART_GET_NUM_PARTICLES_ERR ( total );
-
-	_H5Part_print_debug ( "Total nparticles=%lld", (long long)total );
-
-	if ( total == 0 ) return H5PART_SUCCESS;
+	if ( total < 0 ) {
+		return HANDLE_H5PART_GET_NUM_PARTICLES_ERR ( total );
+	}
+	else if ( total == 0 ) {
+		/* No datasets have been created yet and no veiws are set.
+		 * We have to leave the view empty because we don't know how
+		 * many particles there should be! */
+		return H5PART_SUCCESS;
+	}
 
 	if ( start == -1 ) start = 0;
 	if ( end == -1 )   end = total - 1; // range is *inclusive*
@@ -2604,10 +2683,7 @@ _set_view (
 	f->viewend =    end;
 	f->nparticles = end - start + 1;
 	
-	/* declare overall datasize */
-	f->shape = H5Screate_simple ( 1, &total, NULL );
-	if ( f->shape < 0 )
-		return HANDLE_H5S_CREATE_SIMPLE_ERR ( total );
+	_H5Part_print_debug ( "nparticles=%lld", (long long)f->nparticles );
 
 	/* declare overall data size but then select a subset */
 	f->diskshape = H5Screate_simple ( 1, &total, NULL );
@@ -2636,7 +2712,7 @@ _set_view (
 static h5part_int64_t
 _set_view_indices (
 	H5PartFile *f,			/*!< [in]  Handle to open file */
-	const hsize_t *indices,		/*!< [in]  List of indices */
+	const h5part_int64_t *indices,	/*!< [in]  List of indices */
 	h5part_int64_t nelems		/*!< [in]  Size of list */
 	) {
 
@@ -2661,7 +2737,15 @@ _set_view_indices (
 	  end==-1 to mean end of file
 	*/
 	total = (hsize_t) _H5Part_get_num_particles ( f );
-	if ( total < 0 ) return HANDLE_H5PART_GET_NUM_PARTICLES_ERR ( total );
+	if ( total < 0 ) {
+		return HANDLE_H5PART_GET_NUM_PARTICLES_ERR ( total );
+	}
+	else if ( total == 0 ) {
+		/* No datasets have been created yet and no veiws are set.
+		 * We have to leave the view empty because we don't know how
+		 * many particles there should be! */
+		return H5PART_SUCCESS;
+	}
 
 	_H5Part_print_debug ( "Total nparticles=%lld", (long long)total );
 
@@ -2677,11 +2761,6 @@ _set_view_indices (
 		f->nparticles = (hsize_t) nelems;
 	}
 
-	/* declare overall datasize */
-	f->shape = H5Screate_simple ( 1, &total, NULL );
-	if ( f->shape < 0 )
-		return HANDLE_H5S_CREATE_SIMPLE_ERR ( total );
-
 	/* declare overall data size  but then will select a subset */
 	f->diskshape = H5Screate_simple ( 1, &total, NULL );
 	if ( f->diskshape < 0 )
@@ -2698,7 +2777,7 @@ _set_view_indices (
 			f->diskshape,
 			H5S_SELECT_SET,
 			nelems,
-			indices );
+			(hsize_t*)indices );
 	} else {
 		herr = H5Sselect_none ( f->diskshape );
 	}
@@ -2768,7 +2847,7 @@ H5PartSetView (
 h5part_int64_t
 H5PartSetViewIndices (
 	H5PartFile *f,			/*!< [in]  Handle to open file */
-	const hsize_t *indices,		/*!< [in]  List of indices */
+	const h5part_int64_t *indices,	/*!< [in]  List of indices */
 	h5part_int64_t nelems		/*!< [in]  Size of list */
 	) {
 
@@ -2930,17 +3009,20 @@ _read_data (
 	hid_t memspace_id;
 
 	if ( f->timegroup < 0 ) {
-		h5part_int64_t h5err = _H5Part_set_step ( f, f->timestep );
-		if ( h5err < 0 ) return h5err;
+		herr = _H5Part_set_step ( f, f->timestep );
+		if ( herr < 0 ) return herr;
 	}
 
-	dataset_id = H5Dopen ( f->timegroup, name
+	char name2[H5PART_DATANAME_LEN];
+	_normalize_dataset_name ( name, name2 );
+
+	dataset_id = H5Dopen ( f->timegroup, name2
 #ifndef H5_USE_16_API
 				, H5P_DEFAULT
 #endif
 				);
 
-	if ( dataset_id < 0 ) return HANDLE_H5D_OPEN_ERR ( name );
+	if ( dataset_id < 0 ) return HANDLE_H5D_OPEN_ERR ( name2 );
 
 	/* default spaces, if not using a view selection */
 	memspace_id = H5S_ALL;
@@ -2971,7 +3053,7 @@ _read_data (
 				"Ignoring view: dataset[%s] has fewer "
 				"elements on disk (%lld) than are selected "
 				"(%lld).",
-				name, (long long)ndisk, (long long)nread );
+				name2, (long long)ndisk, (long long)nread );
 			nread = ndisk;
 		}
 	}
@@ -2997,7 +3079,7 @@ _read_data (
 				"Ignoring view: dataset[%s] has more "
 				"elements selected (%lld) than are available "
 				"in memory (%lld).",
-				name, (long long)nread, (long long)nmem );
+				name2, (long long)nread, (long long)nmem );
 			memspace_id == H5S_ALL;
 		}
 	}
@@ -3016,7 +3098,7 @@ _read_data (
 					   (get hyperslab if needed) */
 		f->xfer_prop,		/* ignore... its for parallel reads */
 		array );
-	if ( herr < 0 ) return HANDLE_H5D_READ_ERR ( name, f->timestep );
+	if ( herr < 0 ) return HANDLE_H5D_READ_ERR ( name2, f->timestep );
 
 #ifdef PARALLEL_IO
 	herr = _H5Part_end_throttle ( f );

src/H5Part.h

@@ -205,7 +205,7 @@ H5PartSetView (
 h5part_int64_t
 H5PartSetViewIndices (
 	H5PartFile *f,			/*!< [in]  Handle to open file */
-	const hsize_t *indices,		/*!< [in]  List of indices */
+	const h5part_int64_t *indices,	/*!< [in]  List of indices */
 	h5part_int64_t nelems		/*!< [in]  Size of list */
 	);
 

src/H5PartTypes.h

@@ -24,6 +24,7 @@ typedef unsigned long		MPI_Comm;
 #endif
 
 #define H5PART_STEPNAME_LEN	64
+#define H5PART_DATANAME_LEN	64
 
 struct H5BlockFile;
 

src/Makefile.am

@@ -4,6 +4,11 @@ OBJEXT=o
 
 INCLUDES = -I@HDF5ROOT@/include @MPIINC@
 
+EXTRA_HEADERS = 	H5PartPrivate.h \
+			H5BlockPrivate.h \
+			H5MultiBlockPrivate.h \
+			H5PartAttrib.h
+
 # Extra files that I wish to include in the dist tar ball.
 EXTRA_DIST =		TestUnderscoreC.c \
 			TestUnderscore.f \
@@ -13,7 +18,8 @@ EXTRA_DIST =		TestUnderscoreC.c \
 			H5Part.f90 \
 			H5PartAttrib.f90 \
 			H5Block.f90 \
-			H5BlockReadWrite.f90
+			H5BlockReadWrite.f90 \
+			$(EXTRA_HEADERS)
 
 # Files that I don't want to include in the dist tar ball
 nodist_include_HEADERS = H5PartF.h @UNDERSCORE_H@
@@ -40,12 +46,6 @@ include_HEADERS = 	H5Part.h \
 			H5PartF.h \
 			@UNDERSCORE_H@
 
-# Listing of all possible headers that I may include
-EXTRA_HEADERS =		H5PartPrivate.h \
-			H5BlockPrivate.h \
-			H5MultiBlockPrivate.h \
-			H5PartAttrib.h
-
 # Listing of sources
 libH5Part_a_SOURCES =	H5Part.c \
 			H5PartAttrib.c \
@@ -54,7 +54,8 @@ libH5Part_a_SOURCES =	H5Part.c \
 			H5MultiBlock.c \
 			H5MultiBlockReadWrite.c
 
-libH5PartF_a_SOURCES =	H5PartF.c \
+libH5PartF_a_SOURCES =	$(libH5Part_a_SOURCES) \
+			H5PartF.c \
 			H5PartAttribF.c \
 			H5BlockF.c \
 			H5BlockReadWriteF.c

test/Makefile.am

@@ -3,7 +3,7 @@
 OBJEXT=o
 
 HDFLIB = -L@HDF5ROOT@/lib -lhdf5 -lz @SZLIB@
-LIBS = -L@H5P_LIB_LOC@ -lH5PartF -lH5Part $(HDFLIB) -lm
+LIBS = -L@H5P_LIB_LOC@ @H5P_LIB_NAME@ $(HDFLIB) -lm
 
 INCLUDES = -I@H5P_LIB_LOC@ -I@HDF5ROOT@/include @MPIINC@
 
@@ -18,7 +18,8 @@ test_SOURCES =	test.c \
 		testframe.c \
 		write.c \
 		read.c \
-		testframe.h
+		testframe.h \
+		params.h
 
 testf_SOURCES = testf.F90
 

test/params.h

@@ -2,7 +2,7 @@
 #define _H5PART_TEST_PARAMS_H_
 
 #define FILENAME "test.h5"
-#define LONGSTEPNAME "thisisaverylongstepnamethatshouldexceedthelimitof64charcausingawarningtoprint"
+#define LONGNAME "thisisaverylongnamethatshouldexceedthelimitof64charcausingawarningtoprint"
 #define NTIMESTEPS 10
 
 /* do not decrease this value below 99, or it will break assumptions

test/test.c

@@ -61,10 +61,10 @@ int main(int argc, char **argv)
 	//if (GetTestCleanup() && !getenv("HDF5_NOCLEANUP"))
         //    TestCleanup();
 
-	//return GetTestNumErrs();
-
 #ifdef PARALLEL_IO
 	return MPI_Finalize();
+#else
+	return GetTestNumErrs();
 #endif
 }
 

test/testframe.c

@@ -649,3 +649,38 @@ test_is_valid(H5PartFile *file)
 	RETURN(status, H5PART_SUCCESS, "H5PartFileIsValid");
 }
 
+void
+test_open_objects(H5PartFile *file, int max_objects)
+{
+	ssize_t nopen = H5Fget_obj_count(file->file, H5F_OBJ_ALL);
+	if (nopen > max_objects)
+	{
+		TestErrPrintf(	"*** TOO MANY OBJECTS OPEN: %d > %d "
+				"at line %4d in %s\n", nopen, max_objects,
+				(int)__LINE__, __FILE__ );
+
+		hid_t *list = malloc(sizeof(hid_t)*nopen);
+		H5Fget_obj_ids(file->file, H5F_OBJ_ALL, nopen, list);
+
+                H5O_info_t info;
+		int i;
+		for (i=0; i<nopen; i++) {
+			H5Oget_info(list[i], &info);
+                        switch (info.type) {
+                        case H5O_TYPE_GROUP: 
+			TestErrPrintf("obj%d has type GROUP\n", i);
+                        break;
+                        case H5O_TYPE_DATASET: 
+			TestErrPrintf("obj%d has type DATASET\n", i);
+                        break;
+                        case H5O_TYPE_NAMED_DATATYPE: 
+			TestErrPrintf("obj%d has type NAMED_DATATYPE\n", i);
+                        break;
+                        default:
+			TestErrPrintf("obj%d has unknown type\n", i);
+                        }
+		}
+		free(list);
+	}
+}
+

test/testframe.h

@@ -184,5 +184,8 @@ get_attr_name(char *name, char *tag, int id);
 void
 test_is_valid(H5PartFile *file);
 
+void
+test_open_objects(H5PartFile *file, int max_objects);
+
 #endif
 

test/write.c

@@ -83,6 +83,10 @@ test_write_data64(H5PartFile *file, int nparticles, int step)
 	status = H5PartSetNumParticlesStrided(file, nparticles, -1);
 	RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticlesStrided");
 
+	/* invalid nparticles will produce an error */
+	status = H5PartSetNumParticlesStrided(file, -1, 2);
+	RETURN(status, H5PART_ERR_INVAL, "H5PartSetNumParticlesStrided");
+
 #if PARALLEL_IO
 	TEST("Setting throttle");
 	status = H5PartSetThrottle(file, 2);
@@ -195,6 +199,7 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
 {
 	int i,t;
 	h5part_int32_t status;
+	int rank, nprocs;
 
 	float *x,*y,*z;
 	float *px,*py,*pz;
@@ -216,6 +221,12 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
 	TEST("Setting throttle");
 	status = H5PartSetThrottle(file, 2);
 	RETURN(status, H5PART_SUCCESS, "H5PartSetThrottle");
+
+	MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+	MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
+#else
+	rank = 0;
+	nprocs = 1;
 #endif
 
 	TEST("Writing 32-bit data");
@@ -236,8 +247,11 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
 		status = H5PartSetStep(file, t);
 		RETURN(status, H5PART_SUCCESS, "H5PartSetStep");
 
-		status = H5PartSetNumParticles(file, nparticles);
-		RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticles");
+		/* test a two-part write using views */
+		status = H5PartSetView(file,
+			rank*nparticles,
+			rank*nparticles + 31);
+		RETURN(status, H5PART_SUCCESS, "H5PartSetView");
 
 		status = H5PartWriteDataFloat32(file, "x", x);
 		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
@@ -259,7 +273,34 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
 		status = H5PartWriteDataFloat32(file, "pz", pz);
 		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
 		
-		status = H5PartWriteDataInt32(file, "id", id);
+		status = H5PartWriteDataInt32(file, LONGNAME, id);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataInt32");
+
+		/* the second write phase... */
+		status = H5PartSetView(file,
+			rank*nparticles + 32,
+			rank*nparticles + nparticles - 1);
+		RETURN(status, H5PART_SUCCESS, "H5PartSetView");
+		/* offset the input arrays */
+		status = H5PartWriteDataFloat32(file, "x", x+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+
+		status = H5PartWriteDataFloat32(file, "y", y+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+
+		status = H5PartWriteDataFloat32(file, "z", z+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+		
+		status = H5PartWriteDataFloat32(file, "px", px+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+
+		status = H5PartWriteDataFloat32(file, "py", py+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+
+		status = H5PartWriteDataFloat32(file, "pz", pz+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+		
+		status = H5PartWriteDataInt32(file, LONGNAME, id+32);
 		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataInt32");
 	}
 }
@@ -274,9 +315,6 @@ test_write_strided_data32(H5PartFile *file, int nparticles, int step)
 
 	data=(float*)malloc(6*nparticles*sizeof(float));
 
-	//status = H5PartSetNumParticlesStrided(file, nparticles, 6);
-	//RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticlesStrided");
-
 	TEST("Writing 32-bit strided data");
 
 	for (t=step; t<step+NTIMESTEPS; t++)
@@ -332,6 +370,8 @@ void test_write1(void)
 	test_write_data32(file1, NPARTICLES, 1);
 	test_write_file_attribs(file1, 0);
 
+	_H5Part_close_hdf_ids(file1);
+	test_open_objects(file1, 1);
 	status = H5PartCloseFile(file1);
 	RETURN(status, H5PART_SUCCESS, "H5PartCloseFile");
 }
@@ -372,7 +412,7 @@ void test_write3(void)
 	test_is_valid(file1);
 
 	TEST("Redefining step name");
-	status = H5PartDefineStepName(file1, LONGSTEPNAME, 16);
+	status = H5PartDefineStepName(file1, LONGNAME, 16);
 	RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
 
 	test_write_strided_data64(file1, NPARTICLES, 0);
@@ -401,10 +441,10 @@ void test_write4(void)
 	test_is_valid(file2);
 
 	TEST("Redefining step name");
-	status = H5PartDefineStepName(file1, LONGSTEPNAME, 16);
+	status = H5PartDefineStepName(file1, LONGNAME, 16);
 	RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
 
-	status = H5PartDefineStepName(file2, LONGSTEPNAME, 16);
+	status = H5PartDefineStepName(file2, LONGNAME, 16);
 	RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
 
 	test_write_data64(file1, NPARTICLES, NTIMESTEPS);

tools/Makefile.am

@@ -3,7 +3,7 @@
 OBJEXT=o
 
 HDFLIB = -L@HDF5ROOT@/lib -lhdf5 -lz @SZLIB@
-LIBS = -L@H5P_LIB_LOC@ -lH5Part $(HDFLIB) -lm @STDCXX@
+LIBS = -L@H5P_LIB_LOC@ @H5P_LIB_NAME@ $(HDFLIB) -lm @STDCXX@
 
 INCLUDES = -I@H5P_LIB_LOC@ -I@HDF5ROOT@/include @MPIINC@