The Socket Server

The SocketServer allows live event streaming from any network device. Such inputs include Log4JSocketAppenders, SplunkForwarders or Snare Agents. The incoming stream of events is broken down into subdirectories which reflect the sourceIP and port. Each time a new a new connection is established the directory will be changed. i.e. 192.168.221.01_13000 indicates an incoming stream from the IP_Port combination.

To change the socket server port edit the boot.properties file and append/modify the following

sysprop=-Dsocket.server.port=123456

A bounce will be required for this to take effect.

Incoming Stream Data.

As the stream arrives the contents are inspected and demultiplexed using the mapping.csv file (as attached below). The mapping determines the set of rules which use used to breakdown the stream and write the contents into their relevant locations. This allows for a single stream of data to contain multiple types of data which can be imported into LogScape.

For Example

1. Recieving

   Aug 23 02:02:43 10.22.111.254 Aug 23 2015 02:03:07: %ASA-5-111008: User 'support' executed the 'enable 15' command.

2. Would send transfer to the following file

   .../logscape/SocketServer/172.16.101.101/172.16.101.101_50150/ASA/10.22.111.254/ ASA-11Aug23.log

   The Datatype would be assosciated with the record by running through the mapping.csv

   ASA,            timestamp=false,mline=false,appendNL=false,(\d+\.\d+\.\d+\.\d+) AND ASA-\d+

   Where

   • SocketServerRoot: is the location the Socket server is running. Generally in the logscape/work/directory. The location can be changed using: -Dsocket.server.root=/opt/logscape/SocketServer in the boot.properties and bouncing the system.
   • ClientIP (172.100.101): is the sender IP address. This might be a Forwarder or the device itself.
   • ClientIP_Port (172.12.100.101_50150): is the senders IP address and the port being sent from. Doing so allows for multiple senders from the same host, but also means that each session will go into a new directory.
   • ASA: is picked up from the mapping.csv as the matching type
   • 10.22.111.254: was extracted during the stream match, and the group used to put the contents into the device subdirectory. Mutliple groups can be used when extracting from a line.
   • ASA-11Aug23.log: is the name of the matching type with the current date appended. Doing so also allows for each days output to be placed into a new file and that files are manageable.
3. Other artifacts: raw-15Aug23.log and unknown-15Aug23.log
   • raw-XXXX.log is used to capture the raw state of the incoming stream. It is enabled by default and is the incoming stream written directly to disk with no processing. To disable raw output use:

   -Dsocket.server.dump.raw=false

   • unknown-XXXX.log is used to capture events which have not been classified.

Mapping CSV Rules

The following displays the first couple of lines from a mapping.csv file. The order of the mapping is evaluated from the top down - and is therefore important to ensure the precendence is correct. i.e. more specific rules at the top.

#type-key,#timestamp,#mline=t/f,#flush a NL after receiving lines,# log-line-match-expression (beware of white space!)
MSWinEventLog, timestamp=false,mline=false,appendNL=false,(*)\s+MSWinEventLog\s+*\s+(*)
ASA-session, timestamp=false,mline=false,appendNL=false,(\d+\.\d+\.\d+\.\d+) AND ASA-session

type-key	MSWinEventLog	Content will be placed into sub directory MSWinEventLog
timestamp	timestamp = false	if true - each line will be prepended with a timestamp as it arrives
mline	mline=true	true - Indicates whether or not to expect multiline events(Or very long line lenghts)
flush	appendNL	- true will append a NL
expression	(*)	The expression groups are used to recognize content and also determine the subdirectory. Each group () will form part of the path; hence allowing for hostnames, types to be used.

As the incoming stream is evaluated, the appropriate output files will be streamed into. Any stream of data that is not recognized will be output into an {timestamp}-unknown.log file. By inspecting this file you can see those types not recognized, update the mapping.csv file and then bounce the SocketServer (so it is reconfigured). By default the mapping.csv is located within the system-properties/vs-log-server directory. To make the required changes copy it into the root LogScape directory and edit the file there.