About Expression Based Event Filtering
In the Skyhigh Cloud Connector logprocessor.local.properties file, you can drop events based on certain criteria, if processing them with current options and preprocessor rules would end up with false positives.
To do this, use the property for custom event filter, called customEventFilter. This property holds a comma-separated list of Boolean MVEL expressions based on which the events can be dropped. An event is dropped if at least one expression executes to true.
For example, to drop all events where the client IP address is 192.168.0.1 and the HTTP method is connected, the value of the property is:
customEventFilter=((event.clienipaddress==192.168.0.1)&&(event.httpmethod=='CONNECT'))
You can also use a comma-separated list of expressions.
How to Filter
- Get a sample log line from the log. For example:
1509807030.874 63 10.100.176.166 TCP_DENIED/403 185 CONNECT tcp://ib.adnxs.com:443/ - DOMAIN\4kSPJATRbwDam -/- - - Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20Trident/7.0;%20MSBrowserIE;%20rv:11.0)%20like%20Gecko - --- OIP:107.154.102.111 AIP:170.74.248.22:8888 CAT:"A_PAusers_Whitelist;A_Review_Reminder_Whitelist;Technology/Internet" NSL:- OCT:- ORT:- XFF: <xff-ip redacted> PSP:0 4968 SSL-C:none SSL-V:- XVID:- 1509807030.874 63 10.100.176.167 TCP_DENIED/403 185 CONNECT tcp://ib.adnxs.com:443/ - DOMAIN\4kSPJATRbwMam -/- - - Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20Trident/7.0;%20MSBrowserIE;%20rv:11.0)%20like%20Gecko - --- OIP:107.154.102.112 AIP:170.74.248.21:8888 CAT:"A_PAusers_Whitelist;A_Review_Reminder_Whitelist;Technology/Internet" NSL:- OCT:- ORT:- XFF: <xff-ip redacted> PSP:0 4968 SSL-C:none SSL-V:- XVID:-
- In this example, the corresponding preprocessor configuration is:
{
"dateFormat": "AUTO",
"topRule": {
"type": "csv",
"on": " ",
"trim": "true"
},
"fields": {
"timestamp": {
"type": "select",
"index": "1"
},
"time_taken": {
"type": "select",
"index": "2"
},
"source_ip": {
"type": "priority_based",
"invalidValue": [
"",
"XFF"
],
"parsingRules": [
{
"type": "select",
"index": 22,
"replacePattern": "XFF:([^-]+)",
"replaceWith": "$1"
},
{
"type": "select",
"index": 3
}
]
},
"action": {
"type": "chain",
"rules": [
{
"type": "select",
"index": "4"
},
{
"type": "csv",
"on": "/"
},
{
"type": "select",
"index": "1"
}
]
},
"http_status": {
"type": "select",
"index": "4",
"between": {
"start": "/",
"end": " "
}
},
"destination_bytes": {
"type": "select",
"index": "5"
},
"method": {
"type": "select",
"index": "6"
},
"url": {
"type": "select",
"index": "7"
},
"source_user": {
"type": "select",
"index": "9"
},
"mime_type": {
"type": "select",
"index": "11"
},
"user_agent": {
"type": "select",
"index": "13"
},
"destination_ip": {
"type": "select",
"index": "16",
"replacePattern": "OIP:",
"replaceWith": ""
},
"source_bytes": {
"type": "select",
"index": "24"
},
"protocol_addOn": {
"type": "chain",
"rules": [
{
"type": "select",
"index": "17"
},
{
"type": "csv",
"on": ":"
},
{
"type": "select",
"index": "3"
}
]
}
}
}
- Filter the events that have a source IP address of 192.168.0.1 and the HTTP method is Connect.
- According to the preprocessor configuration above, index 3 is the source IP mapped to the source_ip parameter. Similarly, index 6 is the HTTP method and it is mapped to method.
- Now, look up source_ip and method in the following table. The corresponding values are event.clienipaddress and event.httpmethod
|
Preprocessor fields |
Event attribute |
|
|---|---|---|
| action | event.proxyaction | |
| custom0 |
Example:
This filters any event for which the first custom attribute value is 'HTTPS'. The index starts at 0. |
|
| destination_bytes | event.serivetoclientbytes | |
| destination_host | event.destinationhostip | |
| destination_ip | event.serverIP | |
| destination_port | event.port | |
| http_status | event.httpstatus | |
| method | event.httpmethod | |
| mime_type | event.contenttype | |
| protocol | event.protocol | |
| referral | event.referralUrl | |
| source_bytes | event.clienttoservicebytes | |
| source_ip | event.clienipaddress | |
| source_user | event.username | |
| time_taken | event.timetaken | |
| timestamp | event.timestamp | |
| url_path | event.uripath | |
| url_query | event.uriquery | |
| user_agent | event.useragent |
-
Use the values from the table in your filter. For example:
customEventFilter=((event.clienipaddress=='192.168.0.1')&&(event.httpmethod=='CONNECT')) customEventFilter=((event.Proxyaction=='Blocked')&& ((event.Httpmethod=='POST') || (event.Httpmethod=='GET'))) customEventFilter=((event.Username=='william.vogt@louisvilleky.gov')) customEventFilter=((event.Httpstatus=='200')) customEventFilter=((event.Clienipaddress=='10.120.1.165') && (event.Httpmethod=='GET') && (event.Httpstatus=='200'))/pre>
Convert UInt64 Hexadecimal Values
Skyhigh Cloud Connector can recognize and process hexadecimal, ocatdecimal, and binary values from log files, but you must configure the Additional Configuration field in Cloud Connector Configuration to make it work.
For example, if you have upload/download bytes in a UInt64/HEX (16 digit) value, Cloud Connector can parse these values.
In the following logline, all values for flowStatistics.bytesSent and flowStatistics.bytesReceived are in hexadecimal format. For example, 0000000000000300.
"Jan 4, 2019, 11:33:59 AM",Cisco FireSight @ 10.4.215.30,"DeviceType=Estreamer DeviceAddress=10.4.215.30 CurrentTime=1546619639904 recordType=RNA_FLOW_STATISTICS recordLength=700 timestamp=04 Jan 2019 11:33:59 netmapDomainRef=0 detectionEngineRef=9 ipAddress=0.0.0.0 MACAddress=00:00:00:00:00:00 hasIPv6=true eventSecond=0 eventMicroSecond=0 eventType=FLOW_STATISTICS fileNumber=B9862F5C filePosition=2A291101 ipV6Address=0:0:0:0:0:0:0:0 flowStatistics.initiatorIPAddress=10.6.101.13 flowStatistics.responderIPAddress=10.8.26.161 flowStatistics.originalClientIPAddress=0:0:0:0:0:0:0:0 flowStatistics.policyRevision=0000000000000000000000005C2ED025 flowStatistics.ruleId=268437521 flowStatistics.tunnelRuleId=0 flowStatistics.ruleAction=2 flowStatistics.ruleReason=0 flowStatistics.initiatorPort=8 flowStatistics.responderPort=0 flowStatistics.tcpFlags=0 flowStatistics.protocol=1 flowStatistics.netFlowIPAddress=0:0:0:0:0:0:0:0 flowStatistics.instanceId=18 flowStatistics.connectionCounter=25559 flowStatistics.firstPacketTimestamp=1546619238 flowStatistics.lastPacketTimestamp=1546619478 flowStatistics.packetsSent=000000000000000C flowStatistics.packetsReceived=000000000000000C flowStatistics.bytesSent=0000000000000300 flowStatistics.bytesReceived=0000000000000300 flowStatistics.initiatorPacketsDropped=0000000000000000 flowStatistics.responderPacketsDropped=0000000000000000 flowStatistics.initiatorBytesDropped=0000000000000000 flowStatistics.responderBytesDropped=0000000000000000 flowStatistics.qosAppliedInterface=00000000000000000000000000000000
To allow Cloud Connector to process hex values, update the Additional Configuration field in Cloud Connector Configuration with the conversions format. This is located on the Log Processing tab, in the Sub-configuration. See Cloud Connector Config Log Processing.
Example:
preprocessor.rules={"dateFormat":"dd MMM yyyy HH:mm:ss","topRule":{"type":"chain","rules":[
{"type":"csv","on":"\\t","trim":"true"}
]},"fields":{"date":{"type":"select","index":"0","between":
{"start":"timestamp","end":"\\t"}
,"trim":"true"},"source_ip":{"type":"select","index":"0","between":
{"start":"flowStatistics.initiatorIPAddress=","end":"\\t"}
,"trim":"true"},"destination_ip":{"type":"select","index":"0","between":
{"start":"flowStatistics.responderIPAddress=","end":"\\t"}
,"trim":"true"},"source_port":{"type":"select","index":"0","between":
{"start":"flowStatistics.initiatorPort=","end":"\\t"}
,"trim":"true"},"destination_port":{"type":"select","index":"0","between":
{"start":"flowStatistics.responderPort=","end":"\\t"}
,"trim":"true"},"url":{"type":"select","index":"0","between":
{"start":"flowStatistics.clientAppURL=","end":"\\t"}
,"trim":"true"},"user_agent":{"type":"select","index":"0","between":
{"start":"flowStatistics.userAgent=","end":"\\t"}
,"trim":"true"},"action":{"type":"select","index":"0","between":
{"start":"flowStatistics.ruleAction=","end":"\\t"}
,"trim":"true"},"http_status":{"type":"select","index":"0","between":
{"start":"flowStatistics.httpResponse=","end":"\\t"}
,"trim":"true"},"source_bytes":{"type":"select","index":"0","between":
{"start":"flowStatistics.bytesSent=","end":"\\t"}
,"trim":"true","base":16},"destination_bytes":{"type":"select","index":"0","between":
{"start":"flowStatistics.bytesReceived=","end":"\\t"}
,"trim":"true","base":16}}}
For hexadecimal values, use "base": 16 in the required field. For octa values, use "base": 8. For binary values, use "base": 2.
This is useful with selectors such as select, between, and more.