Serialized Data and the Event Format String

Any data associated with an event as its payload must be a linearly contiguous block of memory. In order for clients receiving the event to decode the event data memory block back into structured content a key must be provided. The format string that is provided as part of the event is this decoding key.

The format string describes how the individual bytes of event data are to be grouped together as specific data types. For example the Lua script plugin can use the format string to convert the event data memory block into Lua variables that conform to Lua's type system. Once converted, the symbolic name for the data, provided as part of the format string, can be used to reference that particular information. Other clients, such as C or C++ programs, may not need to interpret the data symbolically but may use a language specific mechanism to convert the memory block.

The format string is relatively straightforward to create and is a series of entries formatted as [numbytes][signed/unsigned][numelements][ ][name]. For the standard C data types the number formatting would look like:

C/C++ TypeFormat StringData Size
int8_t1s11 byte
uint8_t1u11 byte
int16_t2s12 bytes
uint16_t2u12 bytes
int32_t4s14 bytes
uint32_t4u14 bytes
int64_t8s18 bytes
uint64_t8u18 bytes
float (IEEE754 float)4f14 bytes
char *1s0Length of string including nul terminator

So, if you were transmitting the following C/C++ structure you would presume that the bytewise memory layout would be:

You would use a format string of 4s1 a 2u1 b to describe the event.  

The symbolic field descriptions a and b are optional but highly recommended. They are used to give the data symbolic representation for clients that can't access the memory bytes directly (such as Lua). These symbolic field descriptions do not need to match the names of the structure member variables so an equally valid format string for the above structure might have been 4s1 angle 2u1 magnitude if angle and magnitude were better symbolic names for what the data represents.

The format string provided describes the linear memory layout of the event data. Consequently it is important that the format string take into consideration any alignment or padding inserted when the memory block is created. Consider changing the order of the members in the sample structure:

without any additional guidance to tell it otherwise the C/C++ compiler is going to create storage for the structure such that members are aligned to boundaries that match their data types (ie 4 byte types are aligned on 4 byte boundaries). This can create holes in the memory layout.

Here the 32 bit/4 byte member of the structure a comes after b but there are two additional bytes of padding inserted to ensure a starts on a 4 byte memory boundary. Since the format string must describe the linear memory layout for clients, we would have to change the format string to accomodate the extra padding inserted for alignment and the format string would be 2u1 b 2u1 pad 4s1 a. It is always good practice to avoid wasting extra bytes on padding alignment, but Storyboard does not perform any sort of interpretation. In fact providing a format string that mis-aligns data can result in unpredictable behaviour.

Event data frequently will contain string information. Strings are simply an array of one byte values with a nul terminating character, often represented as a pointer to this memory (i.echar *). All text in Storyboard is encoded using UTF-8 so this statement applies regardless of the text values being represented. If an event's data payload is composed of a single string, then the bytes of that string can be used directly as the block of memory:

char * event_data = "Crank";

The event_data variable, as a pointer to memory, can be used directly and the format string used to represent it would be 1s0 msg, where msg can be whatever symbolic name makes sense.

It is not possible to send C/C++ structures that contain strings as members if those variables are declared as pointers because the memory of the structure (including the string) is not linear and the event data must be a linearly contiguous block of memory.

However it is possible to include strings within structures by either fixing their size which will force their storage to be included as part of a structure block, i.e char msg[20], or if only a single string is being sent then the C/C++ idom of overallocating the size of a structure can be used to force a linear memory layout:

The C/C++ code technique for using this would look something like:

struct event_data {
    int      a;  //Assume 32 bit integers
    char     b[1];

struct event_data *ed;

//Allocate the memory for the base structure and the string to follow it
ed = malloc(sizeof(*ed) + strlen("Crank"));

//Assign the values to the allocated structure
ed->a = 2018;
strcpy(ed->b, "Crank");          //nul character is accounted for by b[1]

In this case the data can now be described with the format string 4s1 a 1s0 b where the 1s0 is shorthand for nul terminated strings and would be equivalent to saying 1s6 where 6 is the number of bytes in the string "Crank" plus the nul terminating character.