FARGOS/VISTA Object Management Environment Core  ..
FARGOS/VISTA Object Management Environment Core Table of Contents
File_Buffer Class Reference

Implements an adjustable sliding buffer that minimizes data movement while enabling streams to process variable length content. More...

#include <read_file.hpp>

Public Types

enum  { EXTERNAL_SEGMENT = (1 << 0), IS_READ_ONLY = (1 << 1) }
 

Public Member Functions

 File_Buffer (size_t bfrSize=65536)
 Creates an empty buffer of the indicated size in the heap. More...
 
 File_Buffer (unsigned char *existingBuffer, size_t bfrSize, size_t currentLen=0, uint8_t ownershipFlags=EXTERNAL_SEGMENT)
 Accepts the use (and potentially transfers ownership) of an existing buffer that may already have some content. More...
 
 ~File_Buffer ()
 
void useRegionAsBuffer (unsigned char *existingBuffer, size_t bfrSize, size_t currentLen, uint8_t ownershipFlags) OME_ALWAYS_INLINE OME_ALWAYS_OPTIMIZE("-O3")
 Replace the existing buffer with a new region. More...
 
size_t spaceLeft () const OME_ALWAYS_INLINE
 Returns the amount of unused space in the buffer. More...
 
size_t spaceUsed () const OME_ALWAYS_INLINE
 Returns the amount of space current used in the buffer. More...
 
unsigned char * startBufferLocation () const OME_ALWAYS_INLINE
 Returns the start of the active content in the buffer. More...
 
unsigned char * endBufferLocation () const OME_ALWAYS_INLINE
 Returns just past the end of the active content in the buffer. Normally, this would be where new content would be appended. More...
 
void adjustBufferEnd (size_t amount) OME_ALWAYS_INLINE
 Adjusts the end of the buffer. This is typically called after new content has been added. More...
 
void adjustBufferStart (size_t amount) OME_ALWAYS_INLINE
 Adjusts the start of the active content in the buffer. More...
 
void discardInitialBytes (size_t bytes=0) OME_ALWAYS_INLINE OME_ALWAYS_OPTIMIZE("-O3")
 Discards bytes from the beginning of the active portion of the buffer; typically as a result of having processed them. More...
 
size_t appendDataToBuffer (const unsigned char *newData, size_t bytesToAdd, size_t startFromOffset=0) OME_ALWAYS_INLINE OME_ALWAYS_OPTIMIZE("-O3")
 Convenience routine to append data to the buffer. More...
 

Public Attributes

unsigned char * buffer
 base of the buffer segment More...
 
size_t bufferLen
 total length of buffer segment More...
 
size_t endBufferOffset
 offset of last byte used in buffer More...
 
size_t startBufferOffset
 offset of first byte used in buffer More...
 
uint8_t externalBufferFlags
 flags indicating read-only, external More...
 

Detailed Description

Implements an adjustable sliding buffer that minimizes data movement while enabling streams to process variable length content.

Typical usage patterns have an outer loop whose goal is to make calls to appendDataToBuffer() until all of the data has been appended. Within that loop, it is typical to first test if spaceLeft() is less than the amount of new data to be added. If so, a call to discardInitialBytes(0) is made in an attempt to free up all previously handled content in the front of the buffer. Next, a call to appendDataToBuffer() is made, which returns the number of bytes that were successfully appended. That result is tested to see if the number of bytes actually appended was the same as requested quantity. If not, depending on the application, this could be an immediately fatal error. For other scenarios, a sufficient quantity might now be in the buffer to allow intermediate records to be successfully processed. In those applications, there is an inner loop that processes all complete records that are in the buffer. Once those records have been processed, their corresponding data can be discarded from the buffer, which in turn would free up space to allow more of the remaining content to be appended on another pass of the outer loop.

Member Enumeration Documentation

◆ anonymous enum

anonymous enum
Enumerator
EXTERNAL_SEGMENT 

flag indicates buffer is in external segment

IS_READ_ONLY 

flag indicates buffer is read-only

Constructor & Destructor Documentation

◆ File_Buffer() [1/2]

File_Buffer::File_Buffer ( size_t  bfrSize = 65536)
inlineexplicit

Creates an empty buffer of the indicated size in the heap.

References buffer, bufferLen, endBufferOffset, externalBufferFlags, and startBufferOffset.

◆ File_Buffer() [2/2]

File_Buffer::File_Buffer ( unsigned char *  existingBuffer,
size_t  bfrSize,
size_t  currentLen = 0,
uint8_t  ownershipFlags = EXTERNAL_SEGMENT 
)
inline

Accepts the use (and potentially transfers ownership) of an existing buffer that may already have some content.

Parameters
existingBufferpoints to the existing buffer.
bfrSizeindicates the size of the existing buffer.
currentLenspecifies the length of the current content in the buffer.
ownershipFlagsspecifies a mask of ownership attributes. If the EXTERNAL_SEGMENT bit is set, the buffer will not be deleted. This arises when the region is owned by another object or is on the stack. If the IS_READ_ONLY bit is set, no attempts to modify the buffer will be honored. This is useful when the region is a read-only memory-mapped file that should not be modified.

References buffer, bufferLen, endBufferOffset, externalBufferFlags, and startBufferOffset.

◆ ~File_Buffer()

File_Buffer::~File_Buffer ( )
inline

Member Function Documentation

◆ adjustBufferEnd()

void File_Buffer::adjustBufferEnd ( size_t  amount)
inline

Adjusts the end of the buffer. This is typically called after new content has been added.

Parameters
amountspecifies the number of bytes to be added.

References endBufferOffset.

Referenced by Read_And_Process_File::readIntoFileBuffer().

◆ adjustBufferStart()

void File_Buffer::adjustBufferStart ( size_t  amount)
inline

Adjusts the start of the active content in the buffer.

Parameters
amountspecifies the number of bytes to be dropped from the beginning of the active portion of the buffer.

For efficiency, normally only the start of the buffer is adjusted rather than performing a block move to reorient the data in the buffer.

See discardInitialBytes() for the routine that is used to free up needed space.

References startBufferOffset.

Referenced by Read_And_Process_File::findAndProcessNextLine(), ReadBinaryLog::processBuffer(), and Read_And_Process_File::readAndProcessBlocksFromFile().

◆ appendDataToBuffer()

size_t File_Buffer::appendDataToBuffer ( const unsigned char *  newData,
size_t  bytesToAdd,
size_t  startFromOffset = 0 
)
inline

Convenience routine to append data to the buffer.

Parameters
newDatapoints to the content to be added.
bytesToAddspecifies the length of the content to be added.
startFromOffsetspecifies an optional offset relative to newData.

The effective source address of the new content is newData + offset.

Returns
The number of bytes added is returned. This may not be the same as bytesToAdd, particularly in the case where insufficient space remains in the buffer.

NOTE: this routine does not call discardInitialBytes() to free up space on demand; this is the responsibility of the caller.

References buffer, endBufferOffset, externalBufferFlags, IS_READ_ONLY, OME_EXPECT_FALSE, spaceLeft(), spaceUsed(), and useRegionAsBuffer().

Referenced by Extract_And_Process_Document_Stream::addIOblockThenProcess(), CSV_File::processPacket(), and Extract_And_Process_Document_Stream::scanAndProcessBuffer().

◆ discardInitialBytes()

◆ endBufferLocation()

unsigned char* File_Buffer::endBufferLocation ( ) const
inline

Returns just past the end of the active content in the buffer. Normally, this would be where new content would be appended.

References buffer, and endBufferOffset.

Referenced by Read_And_Process_File::readIntoFileBuffer().

◆ spaceLeft()

size_t File_Buffer::spaceLeft ( ) const
inline

◆ spaceUsed()

◆ startBufferLocation()

◆ useRegionAsBuffer()

void File_Buffer::useRegionAsBuffer ( unsigned char *  existingBuffer,
size_t  bfrSize,
size_t  currentLen,
uint8_t  ownershipFlags 
)
inline

Replace the existing buffer with a new region.

Parameters
existingBufferpoints to the existing buffer.
bfrSizeindicates the size of the existing buffer.
currentLenspecifies the length of the current content in the buffer.
ownershipFlagsspecifies a mask of ownership attributes. If the EXTERNAL_SEGMENT bit is set, the buffer will not be deleted. If the IS_READ_ONLY bit is set, no attempts to modify the buffer will be honored.

NOTE: should only be used if spaceUsed() is 0. Obviously, caution should be used to ensure no entity expects the buffer region to remain unchanged.

References buffer, bufferLen, endBufferOffset, EXTERNAL_SEGMENT, externalBufferFlags, IS_READ_ONLY, and startBufferOffset.

Referenced by appendDataToBuffer().

Member Data Documentation

◆ buffer

unsigned char* File_Buffer::buffer

◆ bufferLen

size_t File_Buffer::bufferLen

total length of buffer segment

Referenced by File_Buffer(), spaceLeft(), and useRegionAsBuffer().

◆ endBufferOffset

size_t File_Buffer::endBufferOffset

◆ externalBufferFlags

uint8_t File_Buffer::externalBufferFlags

flags indicating read-only, external

Referenced by appendDataToBuffer(), discardInitialBytes(), File_Buffer(), useRegionAsBuffer(), and ~File_Buffer().

◆ startBufferOffset

size_t File_Buffer::startBufferOffset

The documentation for this class was generated from the following file:
Generated: Tue Jul 28 2020 16:03:27
Support Information