i NCSA HDF Calling Interfaces and Utilities Introduction i National Center for Supercomputing Applications March 1993 Introduction Overview This introduction provides an very brief overview of NCSA Hierarchical Data Format (HDF), a description of the differences between HDF 3.2 and HDF 3.1, and a description of the organization and use of this manual and notational conventions. What Is HDF? Hierarchical Data Format (HDF) is a multi-object file format for the transfer of graphical and numerical data between machines. The design of this format allows self-definition of data content, and easy extensibility for future enhancements or compatibility with other standard formats. Features of NCSA HDF Version 3.2 include the following: * Facilities for sharing data across machines and systems. The systems now currently supported by NCSA include: Machine Operating Platform Type System1 Sun-4 SUN SunOS Cray-2 UNICOS UNICOS Cray-Y/MP UNICOS UNICOS Convex (IEEE format) CONVEX ConvexOS Convex (native format) CONVEXNATIVE ConvexOS SGI Iris 4D IRIS4 IRIX IBM RS/6000 IBM6000 AIX DECstation 3100/5000 MIPSEL Ultrix Vax VMS VMS HP 9000 HP9000 HPUX Macintosh MAC MacOS IBM PC & compatibles PC MS Windows/MSDOS NeXT NeXT NeXTStep Unofficial support for other platforms is available, thanks to HDF users' contributions. Contributed ports of HDF can be found in the "contrib" directory on NCSA's anonymous ftp server. * FORTRAN and C calling interfaces for storing and retrieving 8- and 24-bit raster images, palettes, scientific data and accompanying annotations * Utilities for editing and displaying HDF files, and converting raw data files to HDF files and vice versa * General purpose routines for creating HDF files Differences Between HDF 3.1 and HDF 3.2 HDF 3.2 represents a major increase in the functionality over HDF 3.1. In addition the addition of several new supported platforms (see list above), some of the interfaces have been expanded substantially. SDS Interface The Scientific Data Set (SDS) interface has undergone the most changes that affect typical users. Some of them may necessitate changes to your code. The three biggest changes are (1) the addition of support for new number types in SDS, (2) changes to SDS so that the code no longer transposes arrays, and (3) the addition of a tag for storing calibration information. These changes have necessitated some changes in the SDS interface. The following routines are no longer supported: DFSDsettype (dsstype) DFSDsetmaxmin (dssmaxm) DFSDgetmaxmin (dsgmaxm) The following new routines have been added: DFSDsetNT (dssnt) tells which number type is to be used for next DFSDadddata or DFSDputdata; replaces DFSDsettype1 DFSDgetNT (dsgnt) determines number type of current SDS DFSDsetrange (dssrang) specifies max and min values of next SDS to be written; replaces DFSDsetmaxmin DFSDgetrange (dsgrang) determines max and min values of current SDS; replaces DFSDgetmaxmin DFSDpre32sdg (dsp32sd) tests if the current SDS was written by HDF3.2 or a previous release. These new routines are described in detail in Chapter 4. Annotations Interface A number of annotations routines require programmers to explicitly handle the opening and closing of files. In the past, this was accomplished by calling DFopen and DFclose, which use a structure called a DF. In HDF 3.2, DFopen and DFclose have been replaced by two new routines, Hopen and Hclose. Hopen takes the same arguments as DFopen, but returns an int32 file handle, which is suitable for use with the newly modified annotations routines and Hclose. Therefore programmers who previously used DFopen and DFclose in conjunction with the annotations interface should now use Hopen and Hclose to do file opening and closing. Vset Interface Previously, the Vset calling interface had its own library. It was not included as part of HDF 3.1. In HDF 3.2, the Vset calling interface is contained in the same library as the other HDF interfaces. In previous releases, the set of number types that Vdatas supported was different from the set supported in the SDS interface. In HDF 3.2, the two interfaces support the same number types. Although the Vset interface is part of the HDF 3.2 library, Vset documentation is not included in this document. The Vset documentation is currently being rewritten, and will eventually be merged with the documentation for the other interfaces. New General Purpose Interface The lower layer of HDF has been completely redesigned and reimplemented, and all application interfaces, such as RIS8 and SDS, have been reimplemented on this layer. The new lower layer incorporates the following improvements: * More consistent data and function types * More meaningful and extensive reporting of errors. * Simplification of key lower level functions * Improved techniques for facilitating portability * Support for alternate forms of physical storage, such as linked blocks storage, and storage of part of an object in an external file. * A version tag indicating which version of the HDF library last changed an HDF file. * Hooks to support simultaneous access to multiple files Since users do not normally access the general purpose layer of HDF, most changes will not affect users. However, a few changes should be noted: 1. Hopen should be used instead of DFopen 2. Hclose should now be used instead of DFclose 3. File handles are now of type int32, rather than pointers. Thus, Hopen returns a value of type int32, and all routines that previously had arguments of type "DF *" now take arguments of type int32 instead. The routines that this is most likely to affect for most users are DFfindnextref and the annotation routines that involve file labels and descriptions (see "Annotations Interface"). 4. There is no FORTRAN version of this interface. 5. Hnumber should now be used instead of DFnumber. 6. Hishdf should now be used instead of DFishdf. Details of the new interface are contained in the manual "HDF Specifications," January 1993, currently in draft form and available on the NCSA anonymous ftp server. Old General Purpose Interface Although the previous general purpose interface has been replaced by a new general purpose interface, backward compatibility is maintained by a set of routines that emulate the old routines. All of the old routines that begin with the letters "DF" (DFopen, DFclose, DFgetelement, etc.) have been rewritten on top of the new "H" layer (Hopen, Hclose, etc.). Users who currently use the "DF" routines should be able to continue to use them for some time. However, users are encouraged to switch to using the new "H" routines as soon as they reasonably can, since corresponding "DF" routines will be phased out in some future HDF release. Utilities HDF 3.2 contains the following new utilities, which are described in detail in the chapter "NCSA HDF Command Line Utilities." ristsds converts a series RIS8 HDF files into a single 3D SDS HDF file hdfcomp compresses 8-bit raster images from an HDF file, storing them in a new HDF file. hdf24hdf8 quantizes a HDF RGB 24 bit "pixel" image into a 8 bit image with RGB palette and stores it as a HDF 8-bit raster image file hdfpack condenses an HDF file by reading in all the objects in an HDF file and writing them out to a new file. New Test Suite The HDF 3.2 package distributed on NCSA's anonymous ftp server now contains a test suite that tests the basic functionality of all of the HDF interfaces. Future plans As of this writing (February 1993), substantial new enhancements to HDF are in the pipeline, including: * Support for the netCDF data model and interface within HDF, enabling programs to treat netCDF variables and SDSs interchangeably, where appropriate. * Support for JPEG compression of images, and other forms of compression for SDS storage. * Support for random hyperslab writing to SDS arrays, lifting the current restriction that slices must be written to SDSs in the order in which they are stored. * Support for simultaneous multiple HDF file access with high level interfaces (SDS, RIS, etc.), lifting the current restriction that only one HDF file can be accessed at a time. * New documentation, including a manual "Getting Started with HDF", an "HDF User's Guide," an "HDF Reference Manual," and a rewritten "HDF Specification." Use of This Manual This manual is designed for users who are working on an application that involves the use of HDF files. If you are completely unfamiliar with HDF, you should read or scan Chapter 1, "HDF Basics." The most important information about a particular interface is covered in the chapter that deals with that interface. Each chapter contains a description of the type of data with which the interface deals, descriptions of the routines that constitute the interface, and examples. Manual Contents The manual is organized into the following chapters: Chapter 1, "NCSA HDF Basics," describes HDF, explains the reasons for its creation, and gives a brief overview of available applications software. Chapter 2, "Storing Raster Images," describes the routines that are available for storing and retrieving raster images in HDF format, with or without compression, and with or without palettes. Chapter 3, "Storing Palettes," describes the routines that are available for storing and retrieving 8-bit palettes. Chapter 4, "Storing Rectangular Gridded Arrays of Scientific Data," describes the routines that are available for storing and retrieving scientific datasets. Chapter 5, "Annotating Data Objects and Files," describes the routines that are available for storing and retrieving data and file annotations. Chapter 6, "NCSA HDF Command Line Utilities," lists currently available utility routines and command line utilities that are available for working with HDF files. Appendix A, "NCSA HDF Tags," presents a table of brief descriptions of most of the tags assigned at NCSA for general use. Appendix B, "Routine Lists," lists all the current routines available for use with NCSA HDF. Form of Presentation The material in this manual is presented in text, screen displays, or command line notation. Text In explaining various features and commands, this manual often presents a word within a paragraph in italics to indicate that the word is defined within the paragraph, or that it is a significant term that should be noted and/or is being mentioned for the first time. So that they are more easily identifiable within this manuscript, utilities, variables, integer arguments, tags, etc. have been presented in courier style. Portions of this manual refer to other portions of the manual where the other portions explain related topics. These cross references usually mention the title of sections or chapters enclosed in quotation marks, such as, See Chapter 1, "NCSA HDF Basics." Command Line Format Notation Throughout this manual, many explanations instruct you to make entries by typing on the keyboard. These entry instructions are printed in courier bold type and appear within a paragraph or on a separate line. The command lines in this manual are normally shown in lowercase, except in rare instances where uppercase is required. When it is necessary for you to press a key that is labeled with more than one character (such as the RETURN key), this manual identifies the key with all capital (uppercase) letters. Keys to be entered are printed in bold type. Keys that are pressed simultaneously or in succession are linked with a hyphen. For example, press CONTROL-A. The meaning of each special notation applied to format lines is listed in Table 1. Table I.1 Meaning of Entry Format Notations Appearance Example Entry Method On separate line; dothis Enter the keys for each character. lowercase, courier bold type Within a text line; RETURN Press the single key indicated. uppercase Within a text line; uppercase; CONTROL-A While holding down the first one or two key(s) hyphens between key names indicated, press the last key indicated. On separate line or within a filename This notation is a variable, which text line; italic, lowercase, represents a certain kind of entry, but may courier bold type consist of different characters every time you make the entry. Square brackets; bold type [ ] Do not enter the square brackets. Material presented between brackets is optional and should be entered only in certain cases. Ellipses; bold type . . . Do not enter ellipses. Ellipses indicate you may enter more material similar to the material preceding the ellipses. Figure 1 shows you how to read and enter a command line. ED. NOTE: Figures are not available in this plain text version of the specification. Figure I.1 Reading and Entering a Command Line Further Reading More detailed information about the basic HDF structure and how it works can be found in NCSA HDF Specifications, which you may download via FTP (refer to "How to Get HDF" in Chapter 1) or may request by writing NCSA at the addresses listed on the Bugs and Suggestions report form at the back of this manual. 1Since operating system release numbers often change, we have not listed them here. In general, HDF is supported on the most recent release of an operating system. 1DFSDsettype also was used to set the storage order of SDS data. This function is no longer needed, as explained in the section "Backward Compatibility" in Chapter 4.