Configure log file pushing in SWG
Introduction
This document will explain how to configure the Skyhigh Security Web Gateway to push access log data to the Skyhigh Security Web Reporter for analysis.
Configuring Skyhigh Security Web Reporter
To properly configure Skyhigh Security Web Gateway for reporting purposes follow these steps.
- Logon to the Web Gateway admin user interface and navigate to:
Policy > Settings > Engines > File System Logging > Access Log Configuration. Expand“Settings for Rotation, Pushing, and Deletion”.
NOTE: DO NOT CONFIGURE log pushing from the Configuration > > Log File Manager section as this will result in unwanted logs getting sent to Web Reporter. See Troubleshooting section below.
- Under Auto Pushing select the “Enable auto pushing” check box and configure the URL
to the Web Reporter. - In the “Destination" field enter the Web Reporter log processing URL.
For example ftp://WebReporterIP:9121, http://WebReporterIP:9111/logloader.
- Create a username and password unique to this function and enter them under the “User name” section.
Note: The username and password defined here will be needed later in the Web Reporter configuration (below).
- It is recommended to setup the Web Gateway to automatically push the logs immediately after rotation. For that keep the “Enable pushing log files directly after rotation” checked.
If you would like to use time based push intervals instead, uncheck “Enable pushing log files directly after rotation” and set your “Push interval” hours and minutes.
Save Changes in the Web Gateway UI after configuring the Auto Pushing section.
Configuring Web Reporter
How to properly configure Skyhigh Security Web Reporter to accept these incoming logs.
- Logon to the Web Reporter admin user interface and navigate to: Administration > Setup > Log Sources. Click Add to create a new log source.
- Give this log source a name, note that there cannot be spaces in the name.
- Select “Accept incoming log files".
- In the log format drop-down make sure “Skyhigh Security Web Gateway (Webwasher) – Auto Discover” is selected.
- For the “Logon name” and “Password” fields use the same username and password created in the Web Gateway section (#4 above).
Validating your configuration
To confirm your Web Gateway to Web Reporter configuration is operating properly generate traffic until your next log push occurs. Alternatively, force a log push from the Web Gateway by clicking "Rotate and push logs" from the Configuration > > page. On Web
Reporter, check the Jobs section of your log source; under Administration > Setup > Log Sources > Jobs.
Also you should see information starting to show up under the Quick View section of the Web Reporter interface.
Common issues and Troubleshooting
Mismatched Password
The usernames and passwords must match exactly on both the Web Gateway and Web Reporter, for log pushing and reporting to operate properly. If you accidentally mistype the password you will not see new data coming into the Web Reporter.
Check the mwglogmanager.errors.log via the Web Gateway UI under Troubleshooting > Appliance name > Log Files > mwg errors > mwg-logmanager.errors.log and you will see entries like the following.
[06/Jun/2013:15:35:04 UTC] Cannot push '/opt/mwg/log/user-definedlogs/
access.log/access1306061535.log.gz' to 'ftp://10.10.76.16:9121/access1306061535-
10.10.76.10.log.gz'
Detailed reason(s):
command 'curl -g -q -f -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs -u wradmin:*****
-T /opt/mwg/log/user-defined-logs/access.log/access1306061535.log.gz
ftp://10.10.76.16:9121/access1306061...0.76.10.log.gz' failed with error code 67
Error output is 'curl: (67) Access denied: 530'
SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'
Note: You will not see errors on the Web Reporter as it is simply not receiving data via
the configured log source.
Note: You will not see errors on the Web Reporter as it is simply not receiving data via the configured log source.
Misconfigured Port
If the destination URLs port is entered incorrectly, such as port 9111 (Web Reporter http port) is entered for the ftp URL you will see the following in the mwg-logmanager.errors.log
[06/Jun/2013:16:04:02 UTC] Cannot push '/opt/mwg/log/user-definedlogs/
access.log/access1306061600.log.gz' to 'ftp://10.10.76.16:9111/access1306061600-
10.10.76.10.log.gz'
Detailed reason(s):
command 'curl -g -q -f -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs -u wradmin:*****
-T /opt/mwg/log/user-defined-logs/access.log/access1306061600.log.gz
ftp://10.10.76.16:9111/access1306061...0.76.10.log.gz' failed with error code 56
Error output is 'curl: (56) FTP response reading failed'
SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'
Global Log File pushing configured
Configuring auto pushing on the Web Gateway under 'Configuration > Log File Manager' instead of 'Policy > Settings > Engines > File System Logging' will result in unwanted files being sent to the Web Reporter of which cannot be reported. What you’ll see under
Administration > Setup > Log Sources > Jobs is that many of your jobs are failing. In the details of the job you can see that the log name was not "access........log". Only the access.logs from mwg can be imported into Web Reporter.
Note the File name here – mwg-monitor.errors1305290000-10.10.76.10.. etc – this is a log the Web Reporter cannot process.
If you have configured Log Pushing under Configuration > Log File Manager please refer to the steps at the beginning of this doc to properly configure log pushing for the access log only.
Log header does not match log lines
In case you see all your jobs completed as successful, but there is still no data in your reports, it is possible that the log data import failed due to mismatched log headers and log lines.
This sometimes happens when you try to modify your log file format (adding or removing columns) and the header does not line up with the fields that are being written.
On the Web reporter side you would see that the logs got uploaded and the header was detected (job successful), but when you look at the details of the job, you would see that all lines error out and were ignored.
Web Reporter ports not allowed
Traffic is not reaching the Web Reporter server at all. Assume that your firewall is not allowing ports 9121/9111/9112 you will not be able to logon to the Web Reporter interface from another host, log processing jobs/new report data will not show up and Comments in the mwg-logmanager.errors.log output you will see information like the following (similar to mismatched port configuration).
[06/Jun/2013:16:13:04 UTC] Cannot push '/opt/mwg/log/user-definedlogs/
access.log/access1306061605.log.gz' to 'ftp://10.10.76.16:9111/access1306061605-
10.10.76.10.log.gz'
Detailed reason(s):
command 'curl -g -q -f -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs -u
wradmin:***** -T /opt/mwg/log/user-defined-logs/access.log/access1306061605.log.gz
ftp://10.10.76.16:9111/access1306061...0.76.10.log.gz' failed with error code 56
Error output is 'curl: (56) FTP response reading failed'
SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'