FARGOS Development Protocol Description Utilities

Part of the FARGOS Development, LLC Utility Library

FARGOS Development Protocol Definition Utilities

Interprocess communication often requires the transmission of nontrivial data between processing entities. Quick solutions tend to be both inefficient and difficult to upgrade. Imposing a formal structure of protocol data units allows generic handling of many desirable features, such as pretty-printing, caching, retransmission of missed packets, etc. The FARGOS Development protocol definition utilities make provision for concisely describing protocol elements and automatically generating code to implement those features.

Key Features

There are several key features realized by the FARGOS Development protocol utilities:

Fixed- and variable-length fields are supported.
Standard formatting to allow traversal of packets and PDUs by generic utilities.
Hetergeneous architectures supported with receiver-makes-right processing to minimize overhead.
Automatic generation of pretty-printers for debugging.
Fields maintained with natural alignment.
Standard receive-and-process mechanisms. This is a lengthy topic, as the facilities support high-availability multicast transmission with redundant feeds, periodic refresh of state, per-client subscriptions, acknowledged delivery, toleration of the loss of several back-to-back packets, rate-limited transmission (potentially on a per-client basis), caching of last received value of named elements and more.

A core collection of PDU elements are considered "standard" and library facilities exist to exploit their characteristics to implement capabilities such as reliable multicast, replay of previously transmitted state to synchronize newly-started applications, subscriptions, consumption of potentially compressed archived data, etc.

Performance is the paramount feature of the generated PDU infrastructure, but provision is made for evolving formats without requiring simultaneous updating of all producers and consumers.

Encapsulation Hierarchy

There is a hierchy of encapsulation realized by the PDU and network feed utilities. From lowest-layer to highest:

Level	Usage
Field	A primitive field, such as an integer or string.
Substructure	A named grouping of fields.
PDU	An identifiable grouping of fields and substructures, some of which may be optional.
Packet	An arbitrary collection of PDUs suitable for transmission.
Packet capture	A `tcpdump`/`wireshark` packet capture. Supported by the `StripPCAP_HeaderAndParse` class of the FeedStream library.
Compression block	A block of compressed packets; `gzip` and `LZ4` streams are supported by the `GUnzipFileStreamAndForward` and `LZ4DecompressFileStreamAndForward` classes of the FeedStream library..

Field Types

The fundamental native types are supported along with a few types that have special semantics, such as posix_nanoseconds and zero.

Type Name	Usage
int8_t	Signed 8-bit integer value. `char` is an alias.
int16_t	Signed 16-bit integer value
int32_t	Signed 32-bit integer value
int64_t	Signed 64-bit integer value
uint8_t	Unsigned 8-bit integer value. `unsigned char` is an alias.
uint16_t	Unsigned 16-bit integer value
uint32_t	Unsigned 32-bit integer value
uint64_t	Unsigned 64-bit integer value
char	Single text character
text	Text string; displayed within quotes by pretty-printer.
binary	Block of binary data; displayed as hexadecimal string by pretty-printer.
zero	Zero-filled field
posix_nanoseconds	64-bit value intepreted as nanoseconds since January 1, 1970; displayed as timestamp by pretty-printer. `nanoseconds` is an alias.
float	32-bit floating point value
double	64-bit floating point value

Field Descriptions

Each field within a PDU must have a distinct name. Some fields are used simply for padding and should not be displayed by a pretty-printer; others might contain sensitive information that should not be casually displayed in a log file. Any field that should not be displayed by the automatically-generated pretty-printer should be named with a leading underscore ("_").

A field has a type and may either be a single fundamental type, a previously described grouping of elements or an array of identically-typed elments. An array of elements may be either fixed-length or variable-length. Fixed-length fields are declared using an integer subscript, but variable-length fields are declared using the name of a previously-declared integer field as a subscript.

There are some special-case keywords that can be used as a substitute for a constant integer subscript:

Subscript Name	Description
pad8	Generate sufficient length to yield 8-byte alignment at end of field.
pad8.4	Generate sufficient length to yield alignment 4 bytes short of an 8-byte end-of-field alignment.

Protocol Specifications

On-the-wire and in-memory descriptions of Protocol Data Units can be generated by the emitPDUs utility. The original implementation of the backend was an AWK script that consumed a CSV spreadsheet and output parsers in C++, C# and Python. The historical input format appeared similar to the example below.

PacketHeader,Header used to encapsulate block of PDUs in a physical packet or within a stream.,,,
Name,Offset,Length,Type,Description
protocolVersion,0,4,binary[],Useable for identification in packet capture utilities like wireshark.
byteOrder,4,4,uint32_t,0x11223344 in sender's native byte order.
messageType,8,4,uint32_t,Don't use ids with values that are identical under different byte-order schemes; this allows message type to be used to directly select decode routines.
headerLength,12,2,uint16_t,Total size of the packet header; offset of first PDU.
packetLength,14,2,uint16_t,Total size of packet.  2 bytes restricts size to under 64 Kbytes.  Provided for use on stream transports.
serviceName,16,15,text[],Name of service; fill with nulls on right if not full length of field
serviceInstance,31,1,unsigned char,"Normally 1; redundant services use lower 6 bits for id (1-63), upper 2 bits for total replicas known"
sourceTime,32,8,posix_nanoseconds,Nanoseconds since 1970
sourceId,40,4,uint32_t,Hash of service name or other administratively-assigned value
sourceGeneration,44,4,uint32_t,"Monotonically increasing generation id, typically generated by time() or equivalent."
sourceSequence,48,4,uint32_t,"Monotonically increasing packet sequence, starting from 0 in each new generation"
_pad,52,4,zero[],reserved
END,56,,,

In contrast to the CSV-based format illustrated above, the current version described here uses a formal grammar that is easier to read and has less redundant content.

/*< \brief Standard packet header definition
*/  
standard packet PacketHeader { 
    binary                  protocolVersion[4];     /*!< Character sequence useable for identification in packet capture utilities like wireshark. */
    uint32_t                byteOrder;              /*!< 0x11223344 in sender's native byte order. */
    uint32_t                messageType;            /*!< Don't use ids with values that are identical under different byte-order schemes; this allows message type to be used to directly select decode routines. */
    uint16_t                headerLength;           /*!< Total size of the packet header; offset of first PDU. */
    uint16_t                packetLength;           /*!< Total size of packet.  2 bytes restricts size to under 64 Kbytes.  Provided for use on stream transports. */
    text                    serviceName[15];        /*!< Name of service; fill with nulls on right if not full length of field */
    uint8_t                 serviceInstance;        /*!< Normally 1; redundant services use lower 6 bits for id (1-63), upper 2 bits for total replicas known */
    posix_nanoseconds       sourceTime;             /*!< Nanoseconds since 1970 */
    uint32_t                sourceId;               /*!< Hash of service name or other administratively-assigned value */
    uint32_t                sourceGeneration;       /*!< Monotonically increasing generation id, typically generated by time() or equivalent. */
    uint32_t                sourceSequence;         /*!< Monotonically increasing packet sequence, starting from 0 in each new generation */
    zero                    _pad[4];                /*!< reserved */
}

/*! Definition of ordering constraints */
standard enum enPDUorder {
        enPDUorder_NOT_IDEMPOTENT=1, /*! Indicates repeats not OK */
        enPDUorder_NO_DUPLICATES=1, /*! Alias for NOT_IDEMPOTENT */
        enPDUorder_GAPS_PERMITTED=2, /*! Indicates if packet loss permitted */
        enPDUorder_ORDER_SIGNIFICANT=4, /*! Indicates sequence 1,2 is different than 2,1 */
        enPDUorder_SKIP_OLD=6, /*! In order, duplicates and gaps permitted */
        enPDUorder_IN_STRICT_SEQUENCE=5, /*! In order, no duplicates, no gaps */
        enPDUorder_IN_SEQUENCE_SKIP_MISSING=7, /*! In order, no duplicates, but skip gaps */
        enPDUorder_FEC_RESEND=128 /*! PDU resent for forward error correction */
}

/*! Mandatory header for every PDU */
standard struct PDUheader {
    uint16_t                pduLength;              /*!< Length of Protocol Data Unit */
    unsigned char           pduOrderingFlag;        /*!< See enPDUorder enum bitmask; indicates idempotent, order mandatory, gaps permitted, optional */
    unsigned char           pduHeaderLength;        /*!< PDUheader length (limits expansion to 255 bytes) */
    uint16_t                pduType;                /*!< PDU type, administratively assigned to be unique */
    uint16_t                pduVersion;             /*!< PDU version, start numbering from 1 */
    uint32_t                checkSum;               /*!< checksum of PDU contents after this field to just before checkSumCopy */
    uint32_t                pduSequence;            /*!< Update sequence */
    posix_nanoseconds       updateTime;             /*!< Nanoseconds since 1970 */
    uint32_t                indexId;                /*!< If non-zero, normally a (sourceGeneration,sourceId)-specific identifier, assigned contiguously.  Special cases exist for StartSequenceOfPDUs and EndSequenceOfPDUs, where it identifies the pduType of the sequence. */
    uint32_t                afterSourceID;          /*!< Process after (sourceId,generationId,sequenceId); all 0 if not applicable */
    uint32_t                afterGeneration;
    uint32_t                afterSequence;
}

/*! Used to encapsulate PDUs in a packet so that they are only
processed by the named receipients and ignored by others (or the inverse).
*/
standard PDU IntendedForPDU {
    PDUheader               pduHdr;
    unsigned char           pduScope;               /*!< PDUs affected; 0 implies all remaining in packet */
    unsigned char           defaultAction;          /*!< ignore/accept (see enAction enum) */
    uint16_t                targetCount;            /*!< Not required to be non-zero, but uninteresting if zero */
    uint32_t                targetList[targetCount]; /*!< implied field based on targetCount elements */
    zero                    _pad[pad8.4];
    uint32_t                checkSumCopy;
}

The IntendedForPDU definition illustrates the use of a variable-length field, targetList, whose size is indicated by the value of targetCount and a zero-filled padding field _pad that is size to be 4 bytes short of the next multiple-of-8 boundary. By convention, the checkSumCopy field always appears as the last 4 bytes of a PDU and holds the value of checkSum from the PDUheader. It is exploited by non-blocking threads that grab a copy of a PDU image and need to verify it was not updated by a different thread during the copy operation.

The fragments above yield automatically-generated source similar to the following:

/*!  Used to encapsulate PDUs in a packet so that they are only^M
processed by the named receipients and ignored by others (or the inverse).
*/
struct IntendedForPDU {
    PDUheader                       pduHdr;
    uint8_t                         pduScope; /*!< PDUs affected; 0 implies all remaining in packet */
    uint8_t                         defaultAction; /*!< ignore/accept (see enAction enum) */
    uint16_t                        targetCount; /*!< Not required to be non-zero, but uninteresting if zero */
// implied field targetList
 /* implied field based on targetCount elements */
// implied field _pad

// implied field checkSumCopy


    const PDUheader & get_pduHdr() const OME_ALWAYS_INLINE {
        return (pduHdr);
    }

    void set_pduHdr(const PDUheader & _arg) OME_ALWAYS_INLINE {
        pduHdr = _arg;
    }

    const uint8_t get_pduScope() const OME_ALWAYS_INLINE {
        return (pduScope);
    }

    void set_pduScope(const uint8_t _arg) OME_ALWAYS_INLINE {
        pduScope = _arg;
    }
    
    const uint8_t get_defaultAction() const OME_ALWAYS_INLINE {
        return (defaultAction);
    }
        
    void set_defaultAction(const uint8_t _arg) OME_ALWAYS_INLINE {
        defaultAction = _arg;
    }
        
    const uint16_t get_targetCount() const OME_ALWAYS_INLINE {
        return (targetCount);
    }

    void set_targetCount(const uint16_t _arg) OME_ALWAYS_INLINE {
        targetCount = _arg;
    }

// function for implied field targetList
    const uint32_t getElement_targetList(const uint_fast32_t _subscript) const OME_ALWAYS_INLINE {
// implied element field uint32_t targetList
        int _attrOffset = 44;
        const unsigned char *_basePtr = reinterpret_cast<const unsigned char *>(this);
        const uint32_t * _resultPtr = reinterpret_cast<const uint32_t *>(_basePtr + _attrOffset); /* targetList */
        return (_resultPtr[_subscript]);
    }

    void setElement_targetList(const uint_fast32_t _subscript, const uint32_t _arg) OME_ALWAYS_INLINE {
// set implied element field uint32_t targetList
        int _attrOffset = 44;
        unsigned char *_basePtr = reinterpret_cast<unsigned char *>(this);
        uint32_t * _resultPtr = reinterpret_cast<uint32_t *>(_basePtr + _attrOffset); /* targetList */
        _resultPtr[_subscript] = _arg;
    }
    
// implied field targetList
    const uint32_t * get_targetList() const OME_ALWAYS_INLINE {
// implied field targetList
        int _attrOffset = 44;
        const unsigned char *_basePtr = reinterpret_cast<const unsigned char *>(this);
        const uint32_t * _resultPtr = reinterpret_cast<const uint32_t *>(_basePtr + _attrOffset); /* targetList */
        return (*_resultPtr);
    }

    void set_targetList(const uint32_t _arg[], size_t _len) OME_ALWAYS_INLINE {
// implied field uint32_t targetList
        int _attrOffset = 44;
        unsigned char *_basePtr = reinterpret_cast<unsigned char *>(this);
        uint32_t * _resultPtr = reinterpret_cast<uint32_t *>(_basePtr + _attrOffset); /* targetList */
        _resultPtr[_subscript] = _arg;
    }

// implied field checkSumCopy
    const uint32_t get_checkSumCopy() const OME_ALWAYS_INLINE {
// implied field checkSumCopy
        int _attrOffset = 44 + ((get_targetCount() * 4)) + _PAD_8_4(44 + ((get_targetCount() * 4))) ;
        const unsigned char *_basePtr = reinterpret_cast<const unsigned char *>(this);
        const uint32_t * _resultPtr = reinterpret_cast<const uint32_t *>(_basePtr + _attrOffset); /* checkSumCopy */
        return (*_resultPtr);
    }

    void set_checkSumCopy(const uint32_t _arg) OME_ALWAYS_INLINE {
// implied field uint32_t checkSumCopy
        int _attrOffset = 44 + ((get_targetCount() * 4)) + _PAD_8_4(44 + ((get_targetCount() * 4))) ;
        unsigned char *_basePtr = reinterpret_cast<unsigned char *>(this);
        uint32_t * _resultPtr = reinterpret_cast<uint32_t *>(_basePtr + _attrOffset); /* checkSumCopy */
        _resultPtr[_subscript] = _arg;
    }
    
    uint_fast16_t get_expectedSize() const OME_ALWAYS_INLINE {
        return (44 + ((get_targetCount() * 4)) + _PAD_8_4(44 + ((get_targetCount() * 4)))  + 4);
    }
}; // end IntendedForPDU

template <typename STREAMTYPE> inline STREAMTYPE & operator<<(STREAMTYPE &os, const IntendedForPDU &pdu) {
    char bfr[64];

    os << (StringInROM) "[IntendedForPDU={";
    os << (StringInROM) " pduHdr=" << pdu.get_pduHdr();
    os << (StringInROM) " pduScope=" << pdu.get_pduScope();
    os << (StringInROM) " defaultAction=" << pdu.get_defaultAction(); 
    os << (StringInROM) " targetCount=" << pdu.get_targetCount();
    for (uint_fast32_t _i=0;i < ((get_targetCount() * 4)); _i += 1) {
        os << (StringInROM) " targetList[" << _i << (StringInROM) "]=" << pdu.getElement_targetList(_i);
    } // end for _i loop
    os << (StringInROM) " checkSumCopy=" << pdu.get_checkSumCopy();
    os << (StringInROM) " }]";
    return (os);
}

/*! List of standard PDU types
*/
enum enPDUtype {
    enPDUtype_None=0,
    enPDUtype_HeartbeatPDU=1,
    enPDUtype_StartOfSequencePDU=2,
    enPDUtype_EndOfSequencePDU=3,
    enPDUtype_IntendedForPDU=4,
    enPDUtype_SubscribeRequestPDU=5,
    enPDUtype_EncapsulatedPacketPDU=6,
    enPDUtype_RequestAcknowledgementPDU=7,
    enPDUtype_AcknowledgementRangeEntry=8,
    enPDUtype_AcknowledgementPDU=9,
    enPDUtype_RespondPDU=10
};

/*!  Definition of ordering constraints
*/
enum enPDUorder {
    enPDUorder_NOT_IDEMPOTENT=1,            /*!<  Indicates repeats not OK */
    enPDUorder_NO_DUPLICATES=1,             /*!<  Alias for NOT_IDEMPOTENT */
    enPDUorder_GAPS_PERMITTED=2,            /*!<  Indicates if packet loss permitted */
    enPDUorder_ORDER_SIGNIFICANT=4,         /*!<  Indicates sequence 1,2 is different than 2,1 */
    enPDUorder_SKIP_OLD=6,                  /*!<  In order, duplicates and gaps permitted */
    enPDUorder_IN_STRICT_SEQUENCE=5,        /*!<  In order, no duplicates, no gaps */
    enPDUorder_IN_SEQUENCE_SKIP_MISSING=7,  /*!<  In order, no duplicates, but skip gaps */
    enPDUorder_FEC_RESEND=128               /*!<  PDU resent for forward error correction */
};

Last updated: 2024-04-14