C API

The GSD C API consists of a single header and source file. Developers can drop the implementation into any package that needs it.

Functions

int gsd_create(const char *fname, const char *application, const char *schema, uint32_t schema_version)

Create an empty gsd file in a file of the given name. Overwrite any existing file at that location. The generated gsd file is not opened. Call gsd_open() to open it for writing.

Parameters
  • fname – File name

  • application – Generating application name (truncated to 63 chars)

  • schema – Schema name for data to be written in this GSD file (truncated to 63 chars)

  • schema_version – Version of the scheme data to be written (make with gsd_make_version())

Returns

0 on success, -1 on a file IO failure - see errno for details

int gsd_open(struct gsd_handle* handle, const char *fname, const gsd_open_flag flags)

Open a GSD file and populates the handle for use by later API calls.

Parameters
  • handle – Handle to open.

  • fname – File name to open.

  • flags – Either GSD_OPEN_READWRITE, GSD_OPEN_READONLY, or GSD_OPEN_APPEND.

Prefer the modes GSD_OPEN_APPEND for writing and GSD_OPEN_READONLY for reading. These modes are optimized to only load as much of the index as needed. GSD_OPEN_READWRITE needs to store the entire index in memory: in files with millions of chunks, this can add up to GiB.

Returns

0 on success. Negative value on failure:

  • -1: IO error (check errno)

  • -2: Not a GSD file

  • -3: Invalid GSD file version

  • -4: Corrupt file

  • -5: Unable to allocate memory

int gsd_create_and_open(struct gsd_handle* handle, const char *fname, const char *application, const char *schema, uint32_t schema_version, const gsd_open_flag flags, int exclusive_create)

Create an empty gsd file in a file of the given name. Overwrite any existing file at that location. Open the generated gsd file in handle.

Parameters
  • handle – Handle to open

  • fname – File name

  • application – Generating application name (truncated to 63 chars)

  • schema – Schema name for data to be written in this GSD file (truncated to 63 chars)

  • schema_version – Version of the scheme data to be written (make with gsd_make_version())

  • flags – Either GSD_OPEN_READWRITE, or GSD_OPEN_APPEND

  • exclusive_create – Set to non-zero to force exclusive creation of the file

Returns

0 on success. Negative value on failure:

  • -1: IO error (check errno)

  • -2: Not a GSD file

  • -3: Invalid GSD file version

  • -4: Corrupt file

  • -5: Unable to allocate memory

  • -6: Invalid argument

int gsd_truncate(gsd_handle* handle)

Truncate a GSD file opened by gsd_open().

After truncating, a file will have no frames and no data chunks. The file size will be that of a newly created gsd file. The application, schema, and schema version metadata will be kept. Truncate does not close and reopen the file, so it is suitable for writing restart files on Lustre file systems without any metadata access.

Parameters
  • handle – Handle to truncate.

Returns

0 on success. Negative value on failure:

  • -1: IO error (check errno)

  • -2: Not a GSD file

  • -3: Invalid GSD file version

  • -4: Corrupt file

  • -5: Unable to allocate memory

int gsd_close(gsd_handle* handle)

Close a GSD file opened by gsd_open(). Call gsd_end_frame() after the last call to gsd_write_chunk() before closing the file.

Parameters
  • handle – Handle to close.

Warning

Do not write chunks to the file with gsd_write_chunk() and then immediately close the file with gsd_close(). This will result in data loss. Data chunks written by gsd_write_chunk() are not updated in the index until gsd_end_frame() is called. This is by design to prevent partial frames in files.

Returns

0 on success, -1 on a file IO failure - see errno for details, and -2 on invalid input

int gsd_end_frame(gsd_handle* handle)

Move on to the next frame after writing 1 or more chunks with gsd_write_chunk(). Increase the frame counter by 1 and flush the cached index to disk.

Parameters
  • handle – Handle to an open GSD file.

Returns

0 on success, -1 on a file IO failure - see errno for details, and -2 on invalid input

int gsd_write_chunk(struct gsd_handle* handle, const char *name, gsd_type type, uint64_t N, uint32_t M, uint8_t flags, const void *data)

Write a data chunk to the current frame. The chunk name must be unique within each frame. The given data chunk is written to the end of the file and its location is updated in the in-memory index. The data pointer must be allocated and contain at least contains at least N * M * gsd_sizeof_type(type) bytes.

Parameters
  • handle – Handle to an open GSD file.

  • name – Name of the data chunk (truncated to 63 chars).

  • type – type ID that identifies the type of data in data.

  • N – Number of rows in the data.

  • M – Number of columns in the data.

  • flags – Unused, set to 0

  • data – Data buffer.

Returns

0 on success, -1 on a file IO failure - see errno for details, and -2 on invalid input

const struct gsd_index_entry_t* gsd_find_chunk(struct gsd_handle* handle, uint64_t frame, const char *name)

Find a chunk in the GSD file. The found entry contains size and type metadata and can be passed to gsd_read_chunk() to read the data.

Parameters
  • handle – Handle to an open GSD file

  • frame – Frame to look for chunk

  • name – Name of the chunk to find

Returns

A pointer to the found chunk, or NULL if not found.

int gsd_read_chunk(gsd_handle* handle, void* data, const gsd_index_entry_t* chunk)

Read a chunk from the GSD file. The index entry must first be found by gsd_find_chunk(). data must point to an allocated buffer with at least N * M * gsd_sizeof_type(type) bytes.

Parameters
  • handle – Handle to an open GSD file

  • data – Data buffer to read into

  • chunk – Chunk to read

Returns

0 on success

  • -1 on a file IO failure - see errno for details

  • -2 on invalid input

  • -3 on invalid file data

uint64_t gsd_get_nframes(gsd_handle* handle)

Get the number of frames in the GSD file.

Parameters
  • handle – Handle to an open GSD file.

Returns

The number of frames in the file, or 0 on error.

size_t gsd_sizeof_type(gsd_type type)

Query size of a GSD type ID.

Parameters
  • type – Type ID to query

Returns

Size of the given type, or 1 for an unknown type ID.

uint32_t gsd_make_version(unsigned int major, unsigned int minor)

Specify a version number.

Parameters
  • major – major version.

  • minor – minor version.

Returns

a packed version number aaaa.bbbb suitable for storing in a gsd file version entry.

const char *gsd_find_matching_chunk_name(struct gsd_handle* handle, const char* match, const char *prev)

Search for chunk names in a gsd file

Parameters
  • handle – Handle to an open GSD file

  • match – String to match

  • prev – Search starting point

To find the first matching chunk name, pass NULL for prev. Pass in the previous found string to find the next after that, and so on. Chunk names match if they begin with the string in match. Chunk names returned by this function may be present in at least one frame.

Returns

Pointer to a string, NULL if no more matching chunks are found found, or NULL if prev is invalid.

Constants

Data types

gsd_type GSD_TYPE_UINT8

Type ID: 8-bit unsigned integer.

gsd_type GSD_TYPE_UINT16

Type ID: 16-bit unsigned integer.

gsd_type GSD_TYPE_UINT32

Type ID: 32-bit unsigned integer.

gsd_type GSD_TYPE_UINT64

Type ID: 64-bit unsigned integer.

gsd_type GSD_TYPE_INT8

Type ID: 8-bit signed integer.

gsd_type GSD_TYPE_INT16

Type ID: 16-bit signed integer.

gsd_type GSD_TYPE_INT32

Type ID: 32-bit signed integer.

gsd_type GSD_TYPE_INT64

Type ID: 64-bit signed integer.

gsd_type GSD_TYPE_FLOAT

Type ID: 32-bit single precision floating point.

gsd_type GSD_TYPE_DOUBLE

Type ID: 64-bit double precision floating point.

Open flags

gsd_open_flag GSD_OPEN_READWRITE

Open file in read/write mode.

gsd_open_flag GSD_OPEN_READONLY

Open file in read only mode.

gsd_open_flag GSD_OPEN_APPEND

Open file in append only mode.

Data structures

gsd_handle

Handle to an open GSD file. All members are read-only. Only public members are documented here.

gsd_header_t header

File header. Use this field to access the header of the GSD file.

int64_t file_size

Size of the open file in bytes.

gsd_open_flag open_flags

Flags used to open the file.

gsd_header_t

GSD file header. Access version, application, and schema information.

uint32_t gsd_version

File format version: 0xaaaabbbb => aaaa.bbbb

char application[64]

Name of the application that wrote the file.

char schema[64]

Name of schema defining the stored data.

uint32_t schema_version

Schema version: 0xaaaabbbb => aaaa.bbbb

gsd_index_entry_t

Entry for a single data chunk in the GSD file.

uint64_t frame

Frame index of the chunk.

uint64_t N

Number of rows in the chunk data.

uint8_t M

Number of columns in the chunk.

uint8_t type

Data type of the chunk. See Data types.

gsd_open_flag

Enum defining the file open flag. Vaild values are GSD_OPEN_READWRITE, GSD_OPEN_READONLY, and GSD_OPEN_APPEND.

gsd_type

Enum defining the file type of the GSD data chunk.

uint8_t

8-bit unsigned integer (defined by C compiler)

uint32_t

32-bit unsigned integer (defined by C compiler)

uint64_t

64-bit unsigned integer (defined by C compiler)

int64_t

64-bit signed integer (defined by C compiler)