Earthworm Module: q2ew
Contributed by:
Function
A Quanterra/COMSERV data feeding program.
Details
q2ew is a Quanterra/COMSERV data feeding program for Earthworm. The utility was written for the Alaska Tsunami Warning Center's Quanterra Q730 digitizers, but should work with any Quanterra digitizer that is COMSERV compatible. COMSERV is a software freeware product developed by Quanterra and other Quanterra users. It is a middleware software layer between the digitizer and the data acquisition software (clients). In short, COMSERV insulates the developer/user from changes to the QSL communications protocol and acts as a multithreaded switch for client processes to access data from the digitizers.
COMSERV must be configured on the same host as the Earthworm system in order for this program to work. Berkeley is the ftp repository for the COMSERV software. Please refer to the COMSERV documentation on how to correctly install and configure this system.
q2ew is designed as a multi-threaded program. The main thread loops polling the COMSERV shared memory region, using COMSERV cs_scan() calls, for new packets arriving from digitizers. Once data arrives, the main thread converts the data into an earthworm trace buffer and ships the data off to the earthworm transport ring. The second thread, the heartbeat, checks that data are coming in from COMSERV in the main thread and issues regular heartbeats. Should there be a fatal problem with the COMSERV server for any of the stations, the heartbeat thread detects this and a message is sent to statmgr and the process dies. The cs_status() call is a critical one to adjust to fine tune the program to modify the severity of reactions to problems/changes observed with COMSERV. All of the fatal errors will cause q2ew to die, but some of the recoverable errors are now set to cause q2ew to die (in particular CSCR_DIED). As we gain more experience with the robustness of COMSERV, we can adjust the recoverable errors to simply provide status messages, but for now lets play it cautious.
The program q2ew has been tested on Solaris and Linux platforms.
Configuration File Commands
On startup, q2ew reads the configuration file named on the command line. As always, the configuraton file contains comments:
# marks the line as a comment (example: # This is a comment).
Command names must be typed in the control file exactly as shown in this document (upper/lower case matters!). q2ew supports one diagnostic command line option, -v is for verbose mode so that you can see packets coming in.
Example Configuration File
# # q2ew configuration file # # This code receives MiniSEED records from COMSERV, converts them into # Earthworm trace buf messages, and stuffs them into a wave ring. # # ModuleId MOD_Q2EW # module id for this import, RingName WAVE_RING # transport ring to use for input/output, HeartbeatInt 10 # Heartbeat interval in seconds # this should be half the q2ew.desc heartbeat! LogFile 1 # If 0, don't write logfile at all, LOG2LogFile 1 # write LOG chans to log file # this is useful for debugging timing or # data logger problems. TimeoutNoSend 5 # If no data for X seconds from COMSERV for # ALL stations, then kill process (HB and # main thread). # NOTE: this timeout does not take effect until # a trace data packet comes from comserv. That # is, LOG channels do not count! # # Iff TimeoutNoSend == 0, then this data # check is DISABLED and q2ew will only die # if EW or COMSERV dies. # For ATWC, set this next config param to CSQ_LAST ComservSeqBuf CSQ_LAST # This is a comserv sequence control value # that can have the following settings: # CSQ_NEXT = Get newer than already received # CSQ_LAST = Get first available data (latest) # CSQ_FIRST = Get all data in DAQ # This effects what data are returned. # These are optional for those EW users who want to have a pin number # instead of a SCNL. The keyword here is SCNL2pinmap. No duplicate checking # occurs. Note that SCNL2pinmap replaces SCN2pinmap (q2ew now supports location code) # Example # S C N L pin SCNL2pinmap Q003 HHZ AT -- 1 SCNL2pinmap Q003 HHN AT 00 2 SCNL2pinmap Q003 HHE AT 01 3
Functional Command Listing
Below are the configure commands recognized by q2ew, grouped by the function they influence. Most of the commands are required. Only SCNL2pinmap and LOG2LogFile are optional.
Earthworm system setup: MyModuleId required RingName required ComservSeqBuf required HeartbeatInt required TimeoutNoSend required Output Control: LogFile required LOG2LogFile optional SCNL2pinmap optional
Alphabetic Command Listing & Description
In the following section, all configuration file commands are listed in alphabetical order. Listed along with the command (bold-type) are its arguments (in red), the name of the subroutine that processes the command, and the function within the module that the command influences. A detailed description of the command and is also given. Default values and example commands are listed after each command description.
The following list is organized by:
command [argument here]
ComservSeqBuf [csqtype]
Processed by: main
Function: Main q2ew Thread
This command is used to define how q2ew gets data from COMSERV. csqtype is the name of a #define as found in COMSERV's comserv/include/service.h file. csqtype can have the following values:
CSQ_NEXT CSQ_LAST CSQ_FIRST CSQ_LAST is what is being used for ATWC since they are only interested in the latest available data from the digitizer or DA (digitizer=DA in Quanterras documents). CSQ_NEXT gets only the data that are newer than that already receieved by COMSERV and CSQ_FIRST gets all the data in the DA's buffer. Default: none Example: ComservSeqBuf CSQ_LAST
HeartbeatInt [beat]
Processed by: HeartBeat
Function: Earthworm setup
Beat the q2ew heart every beat seconds. This is implemented as a separate thread in q2ew.
LOG2LogFile [switch]
Processed by: logLOGchans
Function: Main q2ew Thread
Sets the on-off switch for writing Quanterra LOG channels to the LogFile. If switch is 0, no LOG channels will be written to the log. If switch is 1, q2ew will write all LOG channels to the LogFile. Turning this on will automatically turn on writing of a LogFile.
Default: 0 - do not use logit() for LOG channels
LogFile [switch]
Processed by: GetConfig
Function: Earthworm Setup
Sets the on-off switch for writing a log file to disk. If switch is 0, no log file will be written. If switch is 1, q2ew will write a daily log file(s) called q2ewxx.log_yymmdd where xx is q2ew's module id (set with "MyModuleId" command) and yymmdd is the current UTC date (ex: 960123) on the system clock. The file(s) will be written in the EW_LOG directory (environment variable).
Default: none
MyModuleId [mod_id]
Processed by: GetConfig
Function: Earthworm setup
Sets the module id for labeling all outgoing messages. mod_id is a character string (valid strings are listed in earthworm.d) that relates (in earthworm.d) to a unique single-byte number.
Default: none Calnet: MyModuleId MOD_Q2EW
RingName [ring]
Processed by: GetConfig
Function: Earthworm setup
This is the ring into which the waveforms and messages are sent.
Default: none, required option ATWC: RingName WAVE_RING
SCNL2pinmap [S C N L pin]
Processed by: insertSCNL
Function: Main q2ew Thread
This command is used to define how q2ew maps a Station Channel Name to a pin number for those installations who wish to define pins....This is a throwback to old digitizers and is not recommended since Quanterra digitizers can properly be configured to know SCNL information. Data from S C N L is compared to the SEED header of the waveform packets coming off the DA and the pin number is substituted into the tracebuf struct passed to Earthworm. S is for station name, C is for channel identifier, and N is for network code. Note that station, channel, and network names in Quanterra follow the SEED convention and can be 5, 3, and 2 chars in length respectively.
Default: none, optional command SCNL2pinmap Q003 HHZ AT -- 1 SCNL2pinmap Q003 HLN AT -- 2 SCNL2pinmap Q003 HLE AT -- 3
TimeoutNoSend [timeout]
Processed by: HeartBeat
Function: Earthworm setup
If no waveform packets come from the DA after timeout seconds, the q2ew module will notify statmgr and die. If timeout is set to 0, then the q2ew module will never check for this timeout. If set to 0, q2ew will only die if COMSERV dies, or if the Earthworm system calls for its termination. This timeout begins once the first Waveform packet is received from ANY DA served by COMSERV. It is not specific to any one DA.
Default: none, required option ATWC: TimeoutNoSend 0
Descriptor File Example
Here is a copy of the q2ew.desc file as implemented. Note that all causes of death for q2ew feed through the function q2ew_die() which logs all causes of death using logit() and sends a message to the statmgr utility.
modName q2ew modId MOD_Q2EW instId INST_ATWC restartMe # restart this sucker as it is critical to data collection # # # Heartbeat Specification. If the status manager does not receive # a heartbeat message every seconds from this module, an # error will be reported (client module dead). is the maximum # number of pager messages that will be reported and is the # maximum number of email messages that will be reported. If the # page or mail limit is exceeded, no further errors will be reported # until the status manager is restarted. # tsec: 20 page: 0 mail: 99 # these are the statmgr messages that q2ew will send before dieing # err: 0 nerr: 1 tsec: 0 page: 5 mail: 20 text: "COMSERV has a problem and died" # err: 1 nerr: 1 tsec: 0 page: 5 mail: 20 text: "COMSERV Recv NO DATA TIMEOUT and died" # err: 2 nerr: 1 tsec: 0 page: 5 mail: 20 text: "SIGNAL caused q2ew to die" # err: 3 nerr: 1 tsec: 0 page: 5 mail: 20 text: "EW tport_putmsg() failed, q2ew dead" # err: 4 nerr: 1 tsec: 0 page: 5 mail: 20 text: "EW TERMINATE recv'd, q2ew dead" # err: 5 nerr: 1 tsec: 0 page: 5 mail: 20 text: "EW config problems on startup, q2ew dead" # # BELOW: WARNINGS, NOT FATAL YET err: 6 nerr: 1 tsec: 0 page: 5 mail: 20 text: "qlib2 decompression errors" #