tpl User Guide

Tpl is a library for serializing C data. The data is stored in its natural binary form. The API is small and tries to stay "out of the way". Tpl can serialize many C data types, including structures.

Tpl makes a convenient file format. For example, suppose a program needs to store a list of user names and ids. This can be expressed using the format string A(si). If the program needs two such lists (say, one for regular users and one for administrators) this could be expressed as A(si)A(si). It is easy to read and write this kind of structured data using tpl.

Tpl can also be used as an IPC message format. It handles byte order issues and deframing individual messages off of a stream automatically.

The "data type" of a tpl is explicitly stated as a format string. There is never any ambiguity about the type of data stored in a tpl. Some examples:

A tpl image is the serialized form of a tpl, stored in a memory buffer or file, or written to a file descriptor.

There is no need to understand the internal structure of the tpl image. But for the curious, the image is a strictly defined binary buffer having two sections, a header and the data. The header encodes the length of the image, its format string, endian order and other flags. The data section contains the packed data.

A property of the tpl image is that consecutive images can be written to a stream without requiring any delimiter between them. The reader making use of tpl_gather (or tpl_load in TPL_FD mode) will obtain exactly one tpl image at a time. Therefore tpl images can be used as an IPC message format without any higher-level framing protocol.

A tpl image generated on one kind of CPU will generally be portable to other CPU types when tpl is used properly. This may be a surprise considering that tpl is a binary format. But tpl has been carefully designed to make this work. Each format character has an associated explicitly-sized type. For integer and floating point types, whose "endian" or byte-order convention varies from one CPU to another, tpl automatically and transparently corrects the endian order (if needed) during the unpacking process. Floating point numbers present their own special difficulties. No guarantees are made with regard to floating point portability. That said, because many modern CPU’s use IEEE 754 floating point representation, data is likely to be portable among them.

Note: The tplxml utility and the Perl module are currently unsupported in tpl 1.5.

While a tpl image is a binary entity, you can view any tpl image in XML format using the included tplxml utility, located in the lang/perl directory.

The utility is bidirectional, as shown. The file extension is not important; tplxml inspects its input to see if it’s tpl or XML. You can also pipe data into it instead of giving it a filename. The tplxml utility is slow. Its purpose is two-fold: debugging (manual inspection of the data in a tpl), and interoperability with XML-based programs. The resulting XML is often ten times the size of the original binary tpl image.

There is a Perl module in lang/perl/Tpl.pm. The Perl API is convenient for writing Perl scripts that interoperate with C programs, and need to pass structured data back and forth. It is written in pure Perl.

The tpl software was developed for POSIX systems and has been tested on 32- and 64-bit platforms including:

This software is made available under the revised BSD license. It is free and open source.

Please follow the link to download on the tpl website.

If you need help, you are welcome to email the author at thanson@users.sourceforge.net.

Tpl has no dependencies on libraries other than the system C library. You can simply copy the tpl source into your project, so you have no dependencies. Alternatively, you can build tpl as a library and link it to your program.

The simplest way to use tpl is to copy the source files tpl.h and tpl.c (from the src/ directory) right into your project, and build them with the rest of your source files. No special compiler flags are required.

Alternatively, to build tpl as a library, from the top-level directory, run:

This installs a static library libtpl.a and a shared library (e.g., libtpl.so), if your system supports them, in standard places. The installation directory can be customized using ./configure --prefix=/some/directory. Run configure --help for further options.

You can compile and run the built-in test suite by running:

On the tpl home page, a Visual Studio 2008 solution package is available for download. This zip file contains pre-built 32- and 64-bit versions of tpl as a DLL. If you like, you can build the DLL yourself using VS2008 or VS2010 (the free Express Edition is sufficient) by opening the solution file and choosing Build Solution.

Alternatively, tpl can be used directly (instead of as a DLL) by compiling the tpl sources right into your program. To do this, add tpl.c, tpl.h, win/mman.h and win/mmap.c to your program’s source and header files and add the preprocessor definition TPL_NOLIB.

Prior to tpl release 1.5, using tpl on Windows required building it with MinGW or Cygwin. This is no longer necessary. If you want to build it that way anyway, use the non-Windows (i.e. tar.bz2) tpl download and follow the "configure; make; make install" approach.

To use tpl, you need to know the order in which to call the API functions, and the background concepts of format string, arrays and index numbers.

Creating a tpl is always the first step, and freeing it is the last step. In between, you either pack and dump the tpl (if you’re serializing data) or you load a tpl image and unpack it (if you’re deserializing data).

When a tpl is created using tpl_map(), its data type is expressed as a format string. Each character in the format string has an associated argument of a specific type. For example, this is how a format string and its arguments are passed in to tpl_map:

The sizes of data types such as long and double vary by platform. This must be kept in mind because most tpl format characters require a pointer argument to a specific-sized type, listed above. You can use explicit-sized types such as int32_t (defined in inttypes.h) in your program if you find this helpful.

Unfortunately there are no standard explicit-sized floating-point types-- no float64_t, for example. If you plan to serialize double on your platform using tpl’s f format character, first be sure that your double is 64 bits. Second, if you plan to deserialize it on a different kind of CPU, be sure that both CPU’s use the same floating-point representation such as IEEE 754.

Arrays come in two kinds: fixed-length and variable-length arrays. Intuitively, they can be thought of like conventional C arrays and linked lists. In general, use fixed-length arrays if possible, and variable-length arrays if necessary. The variable-length arrays support more complex data types, and give or receive the elements to your program one by one.

Before explaining all the concepts, it’s illustrative to see how both kinds of arrays are used. Let’s pack the integers 0 through 9 both ways.

Note that the length of the fixed-length array (10) was passed as an argument to tpl_map(). The corresponding unpacking example is listed further below. Now let’s see how we would pack 0-9 as a variable-length array:

Notice how we called tpl_pack in a loop, once for each element 0-9. Again, there is a corresponding unpacking example shown later in the guide. You might also notice that this time, we passed 1 as the final argument to tpl_pack. This is an index number designating which variable-length array we’re packing. In this case, there is only one.

Index numbers identify a particular variable-length array in the format string. Each A(…) in a format string has its own index number. The index numbers are assigned left-to-right starting from 1. Examples:

The special index number 0 designates all the format characters that are not inside an A(…). Examples of what index 0 does (and does not) designate:

An index number is passed to tpl_pack and tpl_unpack to specify which variable-length array (or non-array, in the case of index number 0) to act upon.

The array examples above demonstrated how integers could be packed. We’ll show some further examples here of unpacking integers and dealing with multi-dimensional arrays. The same program could be used to demonstrate working with byte, 16-bit shorts, 32-bit or 64-bit signed and unsigned integers with only a change to the data type and the format character.

For completeness, let’s also see how to unpack a variable-length integer array.

A multi-dimensional matrix of integers can be packed and unpacked the same way as any fixed-length array.

This single call to tpl_pack packs the entire matrix.

Tpl can serialize C strings. A different format is used for char* vs. char[ ] as described below. Let’s look at char* first:

The char* must point to a null-terminated string or be a NULL pointer.

When deserializing (unpacking) a C string, space for it will be allocated automatically, but you are responsible for freeing it (unless it is NULL):

The s format character is only for use with char* types. In the example above, s is a char*. If it had been a char s[14], we would use the format characters c# to pack or unpack it, as a fixed-length character array. (This unpacks the characters "in-place", instead of into a dynamically allocated buffer). Also, a fixed-length buffer described by c# need not be null-terminated.

You can use fixed- or variable-length arrays of strings in tpl. An example of packing a fixed-length two-dimensional array of strings is shown here.

Later, when unpacking these strings, the programmer must remember to free them one by one, after they are no longer needed.

Packing an arbitrary-length binary buffer (tpl format character B) makes use of the tpl_bin structure. You must declare this structure and populate it with the address and length of the binary buffer to be packed.

When you unpack a binary buffer, tpl will automatically allocate it, and will populate your tpl_bin structure with its address and length. You are responsible for eventually freeing the buffer.

You can use tpl to pack and unpack structures, and arrays of structures.

As shown, omit the individual arguments for the format characters inside the parenthesis. The exception is for fixed-length arrays; when S(…) contains a # character, its length argument is required: tpl_map("S(f#i)", &s, 10);

When using the S(…) format, the only characters allowed inside the parentheses are iujvcsfIU#$().

Arrays of structures are the same as simple arrays. Fixed- or variable- length arrays are supported.

The differences between fixed- and variable-length arrays are explained in the Arrays section.

When dealing with nested structures, the outermost structure uses the S format character, and the inner nested structures use the $ format. Only the outermost structure’s address is given to tpl_map.

Structures can nest to any level. Currently tpl does not support fixed-length array suffixes on inner structures. However the outermost structure can have a length suffix even if it contains some nested structures.

While tpl has no specific data type for a linked list, the technique for packing them is illustrated here. First describe your list element as a format string and then surround it with A(…) to describe it as variable-length array. Then, using a temporary variable, iterate over each list element, copying it to the temporary variable and packing it.

Unpacking is similar. The for loop is just replaced with:

As you can see, tpl does not reinstate the whole list at once-- just one element at a time. You need to link the elements manually. A future release of tpl may support pointer swizzling to make this easier.

The only way to create a tpl is to call tpl_map(). The first argument is the format string. This is followed by a list of arguments as required by the particular characters in the format string. E.g,

The function creates a mapping between the items in the format string and the C program variables whose addresses are given. Later, the C variables will be read or written as the tpl is packed or unpacked.

This function returns a tpl_node* on success, or NULL on failure.

The function tpl_pack() packs data into a tpl. The arguments to tpl_pack() are a tpl_node* and an index number.

It is necessary to pack index number 0 only if the format string contains characters that are not inside an A(…), such as the i in the format string iA(c).

To add elements to a variable-length array, call tpl_pack() repeatedly. Each call adds another element to the array.

It’s perfectly acceptable to pack nothing into a variable-length array, resulting in a zero-length array.

In a format string containing a nested, variable-length array, such as A(A(s)), the inner, child array should be packed prior to the parent array.

When you pack a parent array, a "snapshot" of the current child array is placed into the parent’s new element. Packing a parent array also empties the child array. This way, you can pack new data into the child, then pack the parent again. This creates distinct parent elements which each contain distinct child arrays.

The example below creates a tpl having the format string A(A(c)).

This creates a nested array in which the parent has two elements: the first element is the two-element nested array a, b; and the second element is the three-element nested array 1, 2, 3. The nested unpacking example shows how this tpl is unpacked.

After packing a tpl, tpl_dump() is used to write the tpl image to a file, memory buffer or file descriptor. The corresponding modes are shown below. A final mode is for querying the output size without actually performing the dump.

The first argument is the tpl_node* and the second is one of these constants:

The return value is 0 on success, or -1 on error.

The tpl_dump() function does not free the tpl. Use tpl_free() to release the tpl’s resources when done.

This API function reads a previously-dumped tpl image from a file, memory buffer or file descriptor, and prepares it for subsequent unpacking. The format string specified in the preceding call to tpl_map() will be cross-checked for equality with the format string stored in the tpl image.

The first argument to tpl_load() is the tpl_node*. The second argument is one of the constants:

During loading, the tpl image will be extensively checked for internal validity.

This function returns 0 on success or -1 on error.

When reading a tpl image from a file or memory (but not from a file descriptor) the size of the file or memory buffer must exactly equal that of the tpl image stored therein. In other words, no excess trailing data beyond the tpl image is permitted. The bit flag TPL_EXCESS_OK can be OR’d with TPL_MEM or TPL_FILE to relax this requirement.

A situation where this flag can be useful is in conjunction with tpl_dump in the TPL_MEM|TPL_PREALLOCD mode. In this example, the program does not concern itself with the actual tpl size as long as LEN is sufficiently large.

The tpl_unpack() function unpacks data from the tpl. When data is unpacked, it is copied to the C program variables originally specified in tpl_map(). The first argument to tpl_unpack is the tpl_node* for the tpl and the second argument is an index number.

It is necessary to unpack index number 0 only if the format string contains characters that are not inside an A(…), such as the i in the format string iA(c).

For variable-length arrays, each call to tpl_unpack() unpacks another element. The return value can be used to tell when you’re done: if it’s positive, an element was unpacked; if it’s 0, nothing was unpacked because there are no more elements. A negative retun value indicates an error (e.g. invalid index number). In this document, we usually unpack variable-length arrays using a while loop:

When unpacking a variable-length array, it may be convenient to know ahead of time how many elements will need to be unpacked. You can use tpl_Alen() to get this number.

In a format string containing a nested variable-length array such as A(A(s)), unpack the outer, parent array before unpacking the child array.

When you unpack a parent array, it prepares the child array for unpacking. After unpacking the elements of the child array, the program can repeat the process by unpacking another parent element, then the child elements, and so on. The example below unpacks a tpl having the format string A(A(c)).

The file test40.tpl is from the nested packing example. When run, this program prints:

The final step for any tpl is to release it using tpl_free(). Its only argument is the the tpl_node* to free.

This function does not return a value (it is void).

This function takes a tpl_node* and an index number and returns an int specifying the number of elements in the variable-length array.

This is mainly useful for programs that unpack data and need to know ahead of time the number of elements that will need to be unpacked. (It returns the current number of elements; it will decrease as elements are unpacked).

This function peeks into a file or a memory buffer containing a tpl image and and returns a copy of its format string. It can also peek at the lengths of any fixed-length arrays in the format string, or it can also peek into the data stored in the tpl.

The format string can be obtained like this:

On success, a copy of the format string is returned. The caller must eventually free it. On error, such as a non-existent file, or an invalid tpl image, it returns NULL.

The lengths of all fixed-length arrays in the format string can be queried using the TPL_FXLENS mode. It provides the number of such fixed-length arrays and their lengths. If the former is non-zero, the caller must free the latter array when finished. The format string itself must also be freed.

The TPL_FXLENS mode is mutually exclusive with TPL_DATAPEEK.

To peek into the data, additional arguments are used. This is a quick alternative to mapping, loading and unpacking the tpl, but peeking is limited to the data in index 0. In other words, no peeking into A(…) types. Suppose the tpl image in file.tpl has the format string siA(i). Then the index 0 format characters are si. This is how to peek at their content:

Now s, i, and fmt have been populated with data. The caller must eventually free fmt and s because they are allocated strings. Of course, it works with TPL_MEM as well as TPL_FILE. Notice that TPL_DATAPEEK was OR’d with the mode. You can also specify any leading portion of the index 0 format if you don’t want to peek at the whole thing:

The TPL_DATAPEEK mode is mutually exclusive with TPL_FXLENS.

Lastly you can peek into S(…) structures in index 0, but omit the surrounding S(…) in the format, and specify an argument to receive each structure member individually. You can specify any leading portion of the structure format. For example if struct.tpl has the format string S(si), you can peek at its data in these ways:

This is a quick shortcut for generating a tpl. It can be used instead of the usual "map, pack, dump, and free" lifecycle. With tpl_jot all those steps are handled for you. It only works for simple formats-- namely, those without A(…) in their format string. Here is how it is used:

It supports the three standard modes, TPL_FILE, TPL_FD and TPL_MEM. It returns -1 on failure (such as a bad format string or error writing the file) or 0 on success.

Most users will just leave these hooks at their default values. You can change these hook values if you want to modify tpl’s internal memory management and error reporting behavior.

A global structure called tpl_hook encapsulates the hooks. A program can reconfigure any hook by specifying an alternative function whose prototype matches the default. For example:

The oops has the same prototype as printf. The built-in default oops handling function writes the error message to stderr.

The fatal hook is invoked when a tpl function cannot continue because of an out- of-memory condition or some other usage violation or inconsistency. It has this prototype:

The fatal hook must not return. It must either exit, or if the program needs to handle the failure and keep executing, setjmp and longjmp can be used. The default behavior is to exit(-1).

This example is included in tests/test123.c. When run, this program prints:

The prototype for this function is:

The mode argument is one of three constants listed below, which must be followed by the mode-specific required arguments:

In this mode, tpl_gather blocks while reading file descriptor fd until one complete tpl image is read. No bytes past the end of the tpl image will be read. The address of the buffer containing the image is returned in img and its size is placed in sz. The caller is responsible for eventually freeing the buffer. The function returns 1 on success, 0 on end-of-file, or a negative number on error.

This mode is for non-blocking, event-driven programs that implement their own file descriptor readability testing using select() or the like. In this mode, tpl images are gathered in chunks as data becomes readable. Whenever a full tpl image has been gathered, it invokes a caller-specified callback to do something with the image. The arguments are the file descriptor fd which the caller has determined to be readable and which must be in non-blocking mode, a pointer to a file-descriptor-specific handle which the caller has declared (explained below); a callback to invoke when a tpl image has been read; and an opaque pointer that will passed to the callback.

For each file descriptor on which tpl_gather will be used, the caller must declare a tpl_gather_t* and initialize it to NULL. Thereafter it will be used internally by tpl_gather whenever data is readable on the descriptor.

The callback will only be invoked whenever tpl_gather() has accumulated one complete tpl image. It must have this prototype:

The callback can do anything with the tpl image but it must not free it. It can be copied if it needs to survive past the callback’s return. The callback should return 0 under normal circumstances, or a negative number to abort; that is, returning a negative number causes tpl_gather itself to discard any remaining full or partial tpl images that have been read, and to return a negative number (-4 in particular) to signal its caller to close the file descriptor.

The return value of tpl_gather() is negative if an error occured or 0 if a normal EOF was encountered-- both cases require that the caller close the file descriptor (and stop monitoring it for readability, obviously). If the return value is positive, the function succeeded in gathering whatever data was currently readable, which may have been a partial tpl image, or one or more complete images.

The program will have established a file descriptor in non-blocking mode and be monitoring it for readability, using select(). Whenever it’s readable, the program calls tpl_gather(). In skeletal terms:

This mode is identical to TPL_GATHER_NONBLOCKING except that it gathers from a memory buffer instead of from a file descriptor. In other words, if some other layer of code-- say, a decryption function (that is decrypting fixed-size blocks) produces tpl fragments one-by-one, this mode can be used to reconstitute the tpl images and invoke the callback for each one. Its parameters are the same as for the TPL_GATHER_NONBLOCKING mode except that instead of a file descriptor, it takes a buffer address and size. The return values are also the same as for TPL_GATHER_NONBLOCKING noting of course there is no file descriptor to close on a non-positive return value.