Part of the FARGOS Development, LLC Utility Library
Part of the FARGOS Development, LLC Utility Library
Several of the FARGOS Development, LLC utilities exploit the SharedMemoryVariable class. A SharedMemoryVariable usually realizes either a fundamental primitive type, such as an integer or floating point number, or a string, but also exposes the content in a shared memory segment from which it can be viewed or modified by external processes. If the shared memory segment is maintained as a memory-mapped file, the data also survives the termination of the process. Thus the facility provides the ability for non-intrusive monitoring, adjustments on the fly, and post-mortem analysis of statistics or the reason why a process was stopped. These benefits are obtained with overhead no greater than a pointer dereference.
The SharedMemoryVariableManager class is responsible for maintaining a collection of SharedMemoryVariables within a memory segment. It is possible to have more than one SharedMemoryVariableManager active within a process, but admittedly this usage is uncommon. There is always at least one SharedMemoryVariableManager object automatically constructed and it has the well-known name of DEFAULT_sharedMemoryVariableManager. Although automatically constructed, it does not allocate any space: the application must initialize this variable appropriately if the facilities are to be used.
As SharedMemoryVariables are constructed, they register themselves with their corresponding SharedMemoryVariableManager, which by default is the previously mentioned DEFAULT_sharedMemoryVariableManager object. If the SharedMemoryVariableManager being used has not yet been initialized, it remembers such registrations and will later revisit each of the SharedMemoryVariables when it has finally been given ownership of a memory segment. This removes the dependency upon any order-of-initialization. The SharedMemoryVariableManager can be passed an existing unprepared memory region (via initializeSegment()) or told to construct one on its own (using createAndAttachStandardSegment()). It is also possible to access a previously prepared region by using attachExistingSegment(); this routine is used by external processes, such as smv_query, to access the shared memory segment and manipulate the existing data. There is also a low-level static routine, createSegment(), that can be used to create a memory-mapped file with a name of one's choice. The routine normally used is createAndAttachSegment(), which in turn calls createSegment(). Below is an example of usage in a main()routine, where allocation of the segment was controlled by a Boolean variable makeSMVsegment.
if (makeSMVsegment) {
size_t segLen = 4096 * ((enableLog == true) ? 4096 : 16);
DEFAULT_sharedMemoryVariableManager.createAndAttachStandardSegment(argv[0], &segLen);
}
There are two forms of SharedMemoryVariables: the smaller-sized class assumes that the corresponding SharedMemoryVariableManager will have been initialized before the SharedMemoryVariable is ever accessed. Note that "accessed" is not the same as "registered". The SharedMemoryVariable may be created and registered before the SharedMemoryVariableManager has memory available, but it cannot be read from or written to until the memory segment has been configured. This type of variable is appropriate when the developer knows the application will have created a segment as part of the program's startup. In contrast, it is not a good choice for usage in libraries where the developer has no control over the choices made by the user of the library. In those scenarios, the standalone subclass variants of SharedMemoryVariable should be used. They do consume more storage in the regular memory address space (usually 8 bytes per variable or the maximum possible string length), but not the shared memory segment. The significant benefit is that they are unaffected by the absence of an actual shared memory segment or early usage before the segment has been allocated.
Note: if in doubt, use the standalone variants. The standalone variants also have one extra capability that their regular counterparts lack: they can be initialized with a value.
For numeric types, the subclasses of interest are either SMV_Numeric<> or the standalone variant SMV_StandaloneNumeric<>. The template argument is the type of the variable. So to declare a simple counter, one simply does::
SMV_StandaloneNumeric<uint32_t> a32BitCounter("nameInSegment");
It can sometimes be convenient to use a macro to declare variables if the text name in the shared memory segment is to be the same as the variable name itself, as illustrated by this example:
#define DECLARE_SMV_COUNTER(name) SMV_Numeric<uint32_t> name(#name)
A numeric SharedMemoryVariable can be used much as a normal variable:
#include <utils/shared_vars/shared_variable.hpp> #include <iostream>
SMV_StandaloneNumeric<uint32_t> a32BitCounter("counter");
SMV_StandaloneString<128> aString("aString");
int main()
{
a32BitCounter += 1;
std::cout << "counter=" << a32BitCounter << std::endl;
++a32BitCounter;
std::cout << "counter=" << a32BitCounter << std::endl;
std::cout << "counter=" << a32BitCounter++ << std::endl;
std::cout << "counter=" << ++a32BitCounter << std::endl;
aString = "hi";
std::cout << "string=\"" << aString << "\"" << std::endl;
return (0);
}
yields output similar to:
counter=1 counter=2 counter=2 counter=4 string="hi"
Strings are a slightly more complicated situation because the maximum amount of space for the potential content needs to be allocated ahead of time. The classes SMV_StandaloneString and its peer SMV_String take a template argument that specifies the maximum possible length of the string. The first 3 standard variables illustrated below each allocate a maximum of 128 bytes for their respective string content:
SMV_StandaloneString<128> stopReason("stopReason");
SMV_StandaloneString<128> adminStopReason("adminStopReason");
SMV_StandaloneString<128> adminProcessLabel("adminProcessLabel");
SMV_StandaloneNumeric<uint32_t> OMEstopFlag("stopFlag");
The description of recommended administrative variables is provided later in this document.
Each SharedMemoryVariable has a simple name, which can lead to collisions if one wants to create a collection of objects and track the statistics of each individual object. The prototypical example is a set of counters on a price feed: one would like to be able to distinguish between packets being received on feed A from those being received on feed B. Rather than waste space with long variable names, a SharedMemoryVariable can be parented under a SharedMemoryVariableNode object. For the purposes of naming, the SharedMemoryVariableNode acts much like a directory.
The SharedMemoryVariableNode can itself be parented under another SharedMemoryVariableNode, which allows for arbitrary nesting. This tends to lead to class structures that appear similar to this fragment from IO_Processor_Statistics:
class IO_Processor_Statistics {
public:
SharedMemoryVariableNode namingNode;
// normally exposed as statistics variables
SMV_StandaloneNumeric<uint64_t> bytesRead; // updated by consume routine
SMV_StandaloneNumeric<uint64_t> bytesProcessed; // updated by processBlock routine
// normally exposed as statistics variables
SMV_StandaloneNumeric<uint32_t> packetsRead; // updated by consume routine
SMV_StandaloneNumeric<uint32_t> packetsProcessed; // updated by process routine SMV_StandaloneNumeric<uint32_t> productiveReadSpins;// updated by recvRoutine SMV_StandaloneNumeric<uint32_t> maxReadTimeout; // adjustable, used by recvRoutine //! max unproductive read attempts before thread blocks SMV_StandaloneNumeric<uint32_t> readAttemptsBeforeBlocking; // adjustable, used by recvRoutine
IO_Processor_Statistics(SharedMemoryVariableNode *parentNode) :
namingNode("io_stats", parentNode),
bytesRead("bytesRead", &namingNode),
bytesProcessed("bytesProcessed", &namingNode),
packetsRead("packetsRead", &namingNode),
packetsProcessed("packetsProcessed", &namingNode),
productiveReadSpins("productiveReadSpins", &namingNode),
maxReadTimeout("maxReadTimeout", &namingNode),
readAttemptsBeforeBlocking("readsBeforeBlock", &namingNode)
{
}
Caution: SharedMemoryVariables
should not be used in scenarios where the variables are
repeatedly allocated and then deleted as most of the
BufferRegion managers do not handle
deallocation.
The smv_query utility is able to access a shared memory segment and display its content, or modify the content.
$ smv_query --help
usage: smv_query [-a | {-t -n -v} | -q} [-z] [--info] -f fileName [-f fileName] ... [varName[=val]] ...
-t = output type label
-n = output variable name
-v = output value
-a = output all (i.e., type, name, value)
-q = quiet, no extraneous information displayed
-z = do not output records with 0 or null string values
--info = display segment statistics
-f fileName = specifies file name of a segment
If a filename ends in ".smv", the -f option flag can be omitted
Variable names act as filters.
Modification of an existing value is possible using the assignment syntax
There is a convenience cover script, smvq, that is easier to use for many cases because it will automatically find the memory-mapped file of interest (using recent_stats):
$ smvq --help usage: smvq [smv_query arguments] [-l N] [-component componentName] [{-date YYYYMMDD} | -today] [-host hostName] [-p dir] [-user userName] .. or smvq [smv_query arguments] {[-f] fileName} ...
An example using the -z flag, which ignores any values which are zero or the null string:
$ smvq -z
smv_query -f ./vista-20200709_045042-raspberrypi-14991.smv -z
/stdout/io_stats/bytesRead = uint64 5953
/stdout/io_stats/bytesProcessed = uint64 5953
/stdout/io_stats/packetsRead = uint32 25
/stdout/io_stats/packetsProcessed = uint32 25
/stderr/io_stats/bytesRead = uint64 1513
/stderr/io_stats/bytesProcessed = uint64 1513
/stderr/io_stats/packetsRead = uint32 10
/stderr/io_stats/packetsProcessed = uint32 10
/vista/vista_logMask = uint64 960
nameSpaceTotal = uint32 3
methodTotal = uint32 955
totalObjectsCreated = uint32 18508
totalObjectsDeleted = uint32 18460
totalThreadsDelayed = uint32 89534
totalThreadsAllowed = uint32 32
totalThreadsCreated = uint32 250161
totalThreadsDeleted = uint32 250146
slicesOnKernelThread-0 = uint32 573506
cpusAvailable = uint32 4
minWorkForMultiprocessing = uint32 3
maximumCPUs = uint32 128
totalTimeEventCalls = uint32 211946
totalWaitForIOcalls = uint32 211946
/oil2/oil2_logMask = uint64 960
IOmaxReadBuffer = uint32 65536
IOtruncReadBuffer = uint32 79185
classTotal = uint32 135
IOmaxVectors = uint32 128
IOtotalDescriptorsCreated = uint32 61
IOtotalDescriptorsDeleted = uint32 52
IOtotalFilesCreated = uint32 34
IOtotalFilesDeleted = uint32 29
IOtotalFileReads = uint32 33716
IOtotalFileWriteVectors = uint32 17
IOtotalFileSelectRead = uint32 33687
IOtotalSocketsCreated = uint32 27
IOtotalSocketsDeleted = uint32 23
IOtotalSocketsAccepted = uint32 23
IOtotalSocketRecvs = uint32 33
IOtotalSocketWriteVectors = uint32 230645
IOtotalSocketSelectRead = uint32 59
IOtotalSocketSelectWrite = uint32 78576
totalArrayDeepCopies = uint32 33932
totalSetDeepCopies = uint32 44
totalSetFastCompares = uint32 31
totalAssocDeepCopies = uint32 194
totalOIDsCreated = uint32 71035
totalOIDsDeleted = uint32 70971
processID = uint32 14991
bootTime = uint64 851775245
millisecondsUp = uint64 69118983
elapsedCPUticks = uint64 1327095961768
hostName = text "raspberrypi.shado.fargos.net"
debugFlag = uint32 128
stopReason = text "unexpected termination"
adminProcessLabel = text "FARGOSwebServer"
vista_major_version = uint32 8
vista_minor_version = uint32 2
vista_release_version = uint32 2
vista_os = text "Linux"
vista_cpu = text "aarch64"
app_logMask = uint64 960
io_logMask = uint64 896
coalesceIOVEC = uint32 35
slicesOnKernelThread-1 = uint32 147
slicesOnKernelThread-2 = uint32 70
slicesOnKernelThread-3 = uint32 22
There is a low-level utility script, recent_stats, which can be used to locate the most recent SMV file associated with an application. It is used by smvq and invokeAndRestart, for example, but can also be used by monitoring/management scripts that do not want to use smvq directly.
$ recent_stats --help
usage: [-c component] [{-d YYYYMMDD} | -today | -nodate] [{-t HHMMSS} | -notime] [{-host hostName} | -nohost] [-suffix fileSuffix=.smv] [{-p searchDir} ...] [-n maxWanted]
In contrast to the previously presented content that described available functionality, this section prescribes policy that is intended to provide structure that yields several benefits, but it can be ignored if those benefits are truly viewed as not worthwhile.
As displayed previously in the Recommended Standard Variables for Process Administration figure, there are a few strongly recommended variables that each process should maintain.
With these variables in place, typical usage will be similar to the fragments below. A signal handler is created to set the stopFlag and record that a signal was received in programStopReason:
#ifndef _WIN32
static void requestStop(int sig, siginfo_t *info, void *extra)
{
switch (sig) {
case SIGINT:
programStopReason = "INT signal received";
break;
case SIGTERM:
programStopReason = "TERM signal received";
break;
default:
programStopReason = "signal received";
break;
}
stopFlag = 1;
}
#endif
In main(), the argv list is examined for the ‑‑label argument and the adminProcessLabel is set, programStopReason is set with its initial value of "unexpected termination" and then a signal handler is set to alter stopFlag.:
for (i=1; i < argc; i += 1) {
if (commandOptionMatches("label", argv[i], true)) {
adminProcessLabel = argv[i+1];
i += 1;
continue;
}
// ...
}
programStopReason = "unexpected termination";
#ifndef _WIN32
struct sigaction stopSigData;
stopSigData.sa_sigaction = requestStop;
stopSigData.sa_flags = SA_SIGINFO;
sigemptyset(&(stopSigData.sa_mask));
sigaction(SIGTERM, &stopSigData, nullptr);
sigaction(SIGINT, &stopSigData, nullptr);
#endif
The program might be doing some activity that causes it to intentionally terminate, such as reaching the end of the data stream it was processing. In that case, stopReason gets updated to indicate the termination is expected:
if (OME_EXPECT_FALSE(orderedProcessor->isQueueEmpty())) {
stopFlag = 1;
LOG_CERR(info) << "EOF reached on data files" << LOG_ENDLINE;
programStopReason = "Normal Exit";
orderedProcessor->processWorkInProgress(stopFlag.getAddress());
}
Near the end of main(), an announcement as to why the program is intentionally terminating is output:
LOG_CERR(info) << "Program is terminating, reason=\"" << programStopReason << "\"";
const char *admin = (const char *) adminStopReason;
if (admin[0] != '\0') { // not null
logObj << " adminReason=\"" << admin << "\"";
}
logObj << LOG_ENDLINE;
While the examples above are showing LOG_CERR()
being used, normally the termination announcements are written to
the log file using something like LOG_WHEN().
Production services usually log to both places.
The invokeAndRestart convenience script is able to launch programs under the control of a variety of monitoring and profiling utilities, watch for program termination and inform relevant parties. It comes close to being a do-all cover script for starting long-running service daemons. It makes use of the stopReason, adminStopReason and adminProcessLabel variables when generating termination reports.
$ invokeAndRestart --help usage: invokeAndRestart [{--OPTIONS [--delay secs] [--attempts count] [--logdir dirName] [--name adminName] [--[no]syslog] [--symlink [prefix]] [--uniquedir [prefix]] [{--strace | --ltrace | --gdb | --gdbserver | --valgrind | --callgrind | --drd | -dhat | --helgrind | --massif}] --command] cmd [arg ...] The following are recognized between --OPTIONS and --command: --logdir - specifies the root log directory. Defaults to ./ --delay - specifies seconds between restart attempts. Defaults to 13 --attempts - specifies distinct number of invocation attempts. Defaults to 1 --name - specifies log file name prefix; normally inferred from executable name --merge - if present, merges standard out and error into single file. --symlink - enables automatically maintaining generic name for most recent log --syslog - enables sending unexpected termination notices via syslog --uniquedir - enables keeping log files in a unique directory per invocation This is useful when the program dumps unqualified file names into the current working directory The following are mutually exclusive. They are recognized and stripped
without the need for the --OPTIONS prefix: --strace - start via strace to trace system calls --ltrace - start via ltrace to trace library calls --gdb - start in the debugger --undergdb - start under control of the debugger --gdbserver - start using gdbserver mode to enable remote debugger NOTE: valgrind-related modes are selected with the mutually exclusive options below, but --gdbserver can be also used with the valgrind variants --valgrind - selected memcheck mode --callgrind - selects callgrind mode for function profiling Best used with kcachegrind for visualization --massif - selects massif mode to identify storage allocation hotspots --cachegrind - selects cachgrind mode for cache profiling (not frequently used) --drd - selects data race detector mode --dhat - selects dynamic heap analysis mode --helgrind - selects thread error detection mode See: http://www.valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.gdbserver-gdb Environment Variables: LIBC_FATAL_STDERR_ - default is 1. Set to 0 to inhibit the forcing of libc error to standard error rather than the TTY (not a great idea) CORE_LIMIT_LIMIT - default is \"unlimited\". Set to 0 to inhibit creation of core files. [NOTE: stopProcess automatically inhibits core file creation] INVOKE_AVOIDS_SYSLOG - default is 1. Set to 0 to default to sending sylog messages after program termination. Overridden by --syslog or --nosyslog
Along with invokeAndRestart, there is a stopProcess command that makes use of the shared memory variable segments. It will set adminStopReason with the value provided by the ‑‑reason argument.
$ stopProcess --help
usage: stopProcess [--display] [--schedstat] [{{--reason reasonMessage} | --eod | --halt | --test | --restart}] [--within microseconds] {{{-p pid} ...} | [{--name name}] [{--label adminLabel}]} ...
--display just lists process ids
--eod = End-Of-Day convenience reason
--halt = halt of system convenience reason
--test = testing system convenience reason
--restart = restart convenience reason
--schedstat = record per-task statistics before termination
--name NAME = selects process by NAME
--p PID = selects process by process id PID
--label = selects process by administrative label
Last updated: 2024-04-12