Earthworm Module: naqs2ew
Contributed by:
Function
Naqs2ew is an interface through which waveform data collected by the Nanometrics data acquisition software, NaqsServer, can be fed into an Earthworm system in near-real-time.
Details
Naqs2ew is an interface through which waveform data collected by the Nanometrics data acquisition software, NaqsServer, can be fed into an Earthworm system in near-real-time. Naqs2ew is a client of NaqsServer's "Online Data Streams" service which allows client programs to open a socket connection, subscribe to only the data they want, and receive the requested data over the same socket. The "online data streams" protocol and data formats are described in the Nanometrics Reference Manuals. Currently the only NaqsServer packet types that naqs2ew can process are: channel list messages, decompressed data packets, compressed data packets, and error messages. Naqs2ew is not required to run on the same computer as NaqsServer.
Naqs2ew is a single-threaded program. After reading its configuration file, allocating working buffers, and attaching to a transport ring, naqs2ew attempts to open a socket connection to NaqsServer, using a timeout of CONNECT_TIMEOUT seconds (currently 5 s). If the connection attempt fails, naqs2ew will attempt to connect every CONNECT_RETRY_DT seconds (currently 10 sec) forever. Naqs2ew will continue beating its own heart while trying to establish a NaqsServer connection.
Once it establishs a socket connection with NaqsServer, naqs2ew initiates the "online data streams" protocol. It sends a "connect message," reads and logs the list of available channels, and requests only the channels listed in its own configuration file. If some channels in the configuration file are not available, naqs2ew logs that fact and continues. While the socket connection remains open, naqs2ew reads a complete packet (blocking), converts it to one or more Earthworm TYPE_TRACEBUF messages, and writes the message(s) to the transport ring. Naqs2ew makes no attempt to reorder timeseries packets. It does attempt to produce one-second TYPE_TRACEBUF messages for each channel, buffering any fractional seconds between packets from NaqsServer if necessary. Time tears between packets will cause naqs2ew to produce TYPE_TRACEBUF packets shorter than one-second long. Naqs2ew also logs any error messages it receives from NaqsServer. After each packet is processed, naqs2ew enters its heartbeat function, and issues a heartbeat if its heartbeat timer has expired.
If naqs2ew experiences any socket errors or if NaqsServer breaks the connection, naqs2ew closes the socket and issues a TYPE_ERROR message stating the reason the connection was dropped. It then goes back to looping on reconnection attempts. When the connection is reestablished, naqs2ew will issue another TYPE_ERROR message (really an "un-error" message), stating that everything's OK again.
Configuration File Commands
On startup, naqs2ew reads the configuration file named on the command line. As always, the configuraton file contains comments:
# marks the line as a comment (example: # This is a comment).
Command names must be typed in the control file exactly as shown in this document (upper/lower case matters!).
EXAMPLE CONFIGURATION FILE
# naqs2ew config file MyModuleId MOD_NAQS2EW RingName WAVE_RING HeartBeatInt 30 LogFile 1 Debug 0 SocketDebug 0 NaqsServer 192.168.1.1 28000 # IP address and Port of NaqsServer MaxSamplePerMsg 200 TimeJumpTolerance 600 # (in seconds) This term is used to catch packets with # dubious timestamps. If a time gap is detected in an # incoming data stream, the new packet's timestamp is # compared to the system clock. If it is later than # the current system time by more than TimeJumpTolerance # seconds, naqs2ew assumes the packet timestamp is bogus # (it's in the future) and it ignores the entire packet. # NOTE: if you use this feature with small tolerances, # the PC's system clock must be kept pretty close to # network time!!! # Set to -1 if you never want to compare packet times # to the system clock. # Valid values are -1 or any number >= 0.0 # Set up list of channels you want to receive from NaqsServer using # a 'RequestChannel' command for each channel, with these arguments: # # sta station code you want to receive data from (no wildcards). # comp component (channel) of data you want to receive (no wildcards). # net 2-character network code to label this channel with (no wildcards). # NaqsServer supplies its own station and component codes to its clients, # but not a network code. The sta and comp codes you supply in this file # must match the codes supplied by NaqsServer. The TYPE_TRACEBUF messages # output to the Earthworm system will be labeled with the sta, comp, net # codes listed in this file. # # pinno pin number to assign to this channel (0-32767) # # delay short-term-completion time (in seconds) -1s <= s <= 300s. # When NaqsServer misses packets from the field, it will wait for the # given amount of time for the gap to be filled by re-transmitted # packets before sending data to naqs2ew. Specifying delay=0 will # guarantee that packets are in chronological order, without # waiting for missed data. Setting delay=-1 will deliver the packets # with no delay, in the order they were received (possibly out of # chronological order). # # format format in which you want to receive the data: # -1 = compressed packets (raw format from HRD) # 0 = uncompressed packets, original sample rate. # 0 < r = requested output sample rate. NaqsServer will accomodate. # # sendbuf Buffer flag: # 0 = do not send buffered packets for these channels. # 1 = send buffered packets for these channels. This effectively # moves the start of the data stream several packets into # the past. # WARNING: if naqs2ew is auto-restarted with sendbuf = 1, # duplicate data could enter the Earthworm system. # Therefore, we strongly recommend setting sendbuf = 0. # # sta comp net pinno delay(s) format sendbuf RequestChannel CSU VDZ NC 2001 30 0 0 RequestChannel CSU1 VDN NC 2002 30 0 0 RequestChannel CSU1 VDE NC 2003 30 0 0 RequestChannel CSU1 ADZ NC 2004 30 0 0 RequestChannel CSU1 ADN NC 2005 30 0 0 RequestChannel CSU1 ADE NC 2006 30 0 0 #
FUNCTIONAL COMMAND LISTING
Below are the configure commands recognized by naqs2ew, grouped by the function they influence. Command names must be typed in the control file exactly as shown in this document (upper/lower case matters!). Most of the commands are required.
Earthworm system setup: MyModuleId required RingName required HeartbeatInt required Waveform acquistion: NaqsServer required MaxSamplePerMsg required RequestChannel required TimeJumpTolerance required Output Control: LogFile required Debug optional SocketDebug optional
ALPHABETIC COMMAND LISTING & DESCRIPTION
In the following section, all configuration file commands are listed in alphabetical order. Listed along with the command (bold-type) are its arguments (in red), and the function within the module that the command influences. A detailed description of the command and is also given. Default values and example commands are listed after each command description.
The following list is organized by:
command [argument here]
Debug [flag]
Function: Output Control
Optional command to control the amount of logging done by naqs2ew. If flag is zero, debug-level logging is turned off; naqs2ew will do normal logging of start, stop, and error conditions. If flag is non-zero, naqs2ew will also make a line entry for every packet it sends and receives, and it will log the entire contents of every decompressed trace data packet.
Default: Debug 0
HeartbeatInt [nsec]
Function: Earthworm setup
Defines the minimum number of seconds nsec between TYPE_HEARTBEAT messages issued by naqs2ew. The heartbeat is implemented within both the connection-making and the socket-reading loops of naqs2ew.
Default: none Example: HeartbeatInt 30
LogFile [switch]
Function: Output Control
Sets the on-off switch for writing a log file to disk. If switch is 0, no log file will be written, but messages may go to stderr and/or stdout. If switch is 1, naqs2ew will write to stderr/stdout and to a daily log file(s) called naqs2ewxx.log_ccyymmdd where xx is naqs2ew's module id (set with "MyModuleId" command) and ccyymmdd is the current UTC date (ex: 19960123) on the system clock. The file(s) will be written in the EW_LOG directory (environment variable). If switch is 2, the log file will be written, but no messages will go to stderr or stdout.
Default: none Example: LogFile 1
MaxSamplePerMsg [nsamp]
Function: Waveform Acquisition
Sets the maximum length of each outgoing TYPE_TRACEBUF message to nsamp samples. The number of samples in each TYPE_TRACEBUF message depends on the sample rate of the channel. Normally naqs2ew loads one-second's worth of data into each TYPE_TRACEBUF message. However, if the sample rate for a channel is greater than nsamp samples per second, naqs2ew limits that channel's TYPE_TRACEBUF messages to nsamp samples each. In processing a packet from NaqsServer, naqs2ew will buffer any fractional second of data until it receives the next packet for that channel in an attempt to always send out equal length messages. If naqs2ew detects a time tear between packets from NaqsServer, it may output a message containing less than one second of data.
Default: none Example: MaxSamplePerMsg 200
MyModuleId [mod_id]
Function: Earthworm setup
Sets the module id for labeling all outgoing messages. mod_id is a character string (valid strings are listed in earthworm.d) that relates (in earthworm.d) to a unique single-byte number.
Default: none Example: MyModuleId MOD_NAQS2EW
NaqsServer [IPaddress port]
Function: Waveform Acquisition
Identifies the IPaddress and port number of the NaqsServer application to receive data from.
Default: none Example: NaqsServer 192.168.1.1 28000
RingName [ring]
Function: Earthworm setup
Tells naqs2ew which shared memory region to use for output. ring is a character string (valid strings are listed in earthworm.d) that relates (in earthworm.d) to a unique number for the key to the shared memory region.
Default: none Example: RingName WAVE_RING
RequestChannel [sta comp net pinno delay format sendbuf]
Function: Waveform Acquisition
Use one RequestChannel command for each channel of trace data that you want to request from NaqsServer (no hard-coded limit). Details of the arguments are given below:
sta station code you want to receive data from (up to 5 chars, no wildcards). comp component (channel) of data you want to receive (up to 3 chars, no wildcards). net network code to label this channel with (up to 2 chars,no wildcards). NaqsServer supplies its own station and component codes to its clients, but not a network code. The sta and comp codes you supply in this command must match the codes supplied by NaqsServer. The TYPE_TRACEBUF messages output to the Earthworm system will be labeled with the sta, comp, net codes listed in this command. pinno pin number to assign to this channel (0-32767) delay short-term-completion time (in seconds) -1s <= s <= 300s. When NaqsServer misses packets from the field, it will wait for the given amount of time for the gap to be filled by re-transmitted packets before sending data to naqs2ew. Specifying delay = 0 will guarantee that packets are in chronological order, without waiting for missed data. Setting delay = -1 will deliver the packets with no delay, in the order they were received (possibly out of chronological order). format format in which you want to receive the data from NaqsServer: -1 = compressed packets (raw format from HRD) 0 = uncompressed packets, original sample rate. 0 < r = requested output sample rate. NaqsServer will accomodate. Data is reformatted into Earthworm TYPE_TRACEBUF packets before it is written to the transport ring. sendbuf Buffer flag: 0 = do not send buffered packets for these channels. 1 = send buffered packets for these channels. This effectively moves the start of the data stream several packets into the past. WARNING: if naqs2ew is auto-restarted with sendbuf = 1, duplicate data could enter the Earthworm system (this is generally not a good thing). Therefore, we strongly recommend setting sendbuf = 0.
Default: none Example: RequestChannel CSU1 VDN NC 2002 30 0 0
SocketDebug [flag]
Function: Output Control
Optional command to control the amount of logging done by the socket library. If flag is zero, debug-level logging is turned off. If flag is non-zero, the socket library write LOTS of debugging info. WARNING: turing socket debugging on generates huge log files.
Default: SocketDebug 0
TimeJumpTolerance [flag]
Function: Waveform Acquisition
(in seconds) This term is used to catch packets with dubious timestamps. If a time gap is detected in an incoming data stream, the new packet's timestamp is compared to the system clock. If it is later than the current system time by more than TimeJumpTolerance seconds, naqs2ew assumes the packet timestamp is bogus (it's in the future) and it ignores the entire packet. NOTE: if you use this feature with small tolerances, the PC's system clock must be kept pretty close to network time!!! Set to -1 if you never want to compare packet times to the system clock. Valid values are -1 or any number >= 0.0