Automated Diagnostic Data Collection
Overview
The mwgResInfo is a command-line tool to monitor resource consumption, track connection health, and automate actions, such as troubleshooting data collection or service restarts when specific thresholds require action in a Skyhigh Secure Web Gateway (SWG) environment.
The tool is available as a script in the path below : /opt/mwg/bin/mwgResInfo.sh.
Key Benefits
- Automated Troubleshooting. Configure intelligent triggers to execute diagnostic tasks (such as generating log bundles or core dumps) the moment an issue occurs.
- Active Monitoring. Continuously monitor critical health metrics including memory utilization, CPU load, thread counts, and number of connections.
Note: The tool is for Advanced Troubleshooting only and should not be continuously run for a prolonged period of time until the impact of it is carefully reviewed. Running this tool may impact SWG running processes, data processing, and resource availability depending on the data being collected.
Before scheduling trigger commands, please review your system conditions and resource limits:
- Disk Space: Ensure sufficient disk space is available before running commands like a core dump collection.
-
System Downtime: Commands such as feedback collection and core collection will pause the SWG process until the collection is complete. Please factor this into your scheduling.
Usage Syntax
mwgResInfo.sh [-i seconds] [-s seconds] [[-c number -l number -AND|-OR -c number -l number...] [-p number] -e command [-ot]
[-n]] [[-gl] [-gi]] [-noLog]
Task Reference
Use this table to identify the flags needed for specific monitoring goals.
| Task Goal | Flags | Description |
|---|---|---|
| Set Monitoring Timing | -i, -s |
|
| Specify Target Metrics | -c, -l, -p |
|
| Create Complex Triggers | -AND, -OR |
Chains multiple Note: Mixing -AND and -OR in the same command execution is not supported. Example for AND condition:
ALL specified conditions must be true simultaneously to trigger the action. Example for OR condition: If any of the above specified condition is true, an action will be triggered. |
| Automate Actions | -e, -ot, -n |
|
| Manage Logs and Reports | -noLog, -gl, -gi |
|
Condition (-c) Reference
Use these values with the -c flag to define which metric triggers the command specified by -e.
| Value | Monitored Metric (-c) |
|---|---|
| 1 | Virtual memory usage(Kilobytes) |
| 2 | Physical memory usage(Kilobytes) |
| 3 | Number of threads |
| 4 | Sockets in CLOSE_WAIT state |
| 5 | Machine load average(Float/Integer) |
| 6 | CPU usage(in percentage) |
| 7 | Working queue length |
Process (-p) Reference
Use these values with the -p flag to specify which application is evaluated against -c conditions.
Note: Background logging captures all processes regardless of this setting.
| Value | Supervised Process |
|---|---|
| 0 | All applications (Default) |
| 1 | mwg-core |
| 2 | mwg-coordinator |
| 4 | mwg-saas-connector |
| 5 | HSM agent |
| 6 | gamserver |
Logging
By default, the script logs the following statistics for all Skyhigh Secure Web Gateway applications every 180 seconds to the file ../opt/mwg/log/debug/mwgResInfo*.log in the following sequence:
- Virtual Memory Consumption: Total amount of memory address space an application has requested from the operating system, including both physical RAM and memory temporarily swapped to disk.
- Physical Memory Consumption: Amount of actual, physical RAM currently assigned to and actively used by the specific application being monitored (for example,
mwg-core). - Total Physical RAM in use by all processes: Total amount of physical memory currently being used by everything running on the system. This includes the operating system, all gateway services, and any other background tasks.
- Number of threads: Displays the total count of concurrent execution threads currently spawned by a specific process.
- Machine Load Average: Displays the system's computational workload over a period of time, representing the average number of processes actively using or waiting for CPU time.
- Number of connections in CLOSE_WAIT state: Tracks the network connections that the remote side has closed, but the local application has not yet finished shutting down.
- CPU percent of each service: Measures the percentage of processing power actively being consumed by an individual process, normalized per CPU core.
- Working queue length (TCP accepts queue depth on
MWGlistening ports—parameter applicable only formwg-coreservice): This represents the depth of the TCP accept queue on the gateway's listening ports (applicable only specifically to themwg-coreprocess). A growing queue length indicates that incoming network connections are arriving and piling up faster than the gateway can actively process them.
Use Cases and Examples
Monitor Memory Leaks
Capture logs and core files when virtual memory exceeds 16GB for mwg-core and automatically restart the service.
Command.
For 13.x onwards:
mwgResInfo.sh -p 1 -c 1 -l 16000000 -e "echo "Capture for reaching threshold" | /opt/Skyhigh/ldt/getlogs/getlogs.sh /opt/mwg/debug ; gcore -o /opt/mwg/log/debug/mwg_core_dump $(pgrep -n mwg-core); /etc/init.d/mwg restart" -n -noLog
The following list describes the commands used in the above example:
- Trigger Condition:
- -p 1: Instructs the script to specifically monitor the
mwg-coreprocess. - -c 1 -l 16000000: Sets the trigger condition to Virtual Memory usage (
-c 1) and the threshold to 16 GB (-l16000000 KB).
- -p 1: Instructs the script to specifically monitor the
- Executed actions:
- If the
mwg-coreprocess's virtual memory hits or exceeds 16 GB, it will sequentially execute three wrapped commands to capture debug data before restarting the service:/opt/Skyhigh/ldt/getlogs/getlogs.sh /opt/mwg/debug: Runs a system log collection script and saves the diagnostic logging output to the specified debug directory.gcore -o /opt/mwg/log/debug/mwg_core_dump $(pgrep -n mwg-core): Generates a core memory dump (gcore) of the currently runningmwg-coreprocess and saves it to your debug log directory. This is crucial for analyzing memory leaks or crashes./etc/init.d/mwg restart: Restarts the Secure Web Gateway service to clear the high memory condition and restore normal operations.
- If the
- Script Behavior Modifiers:
- -n (Run Until): Ensures that as soon as this 16 GB threshold is breached and the sequence of commands finishes executing, the script will completely terminate. It will not stay alive to monitor the system anymore.
-
-noLog: Prevents the script from writing its own continuous statistics text files (
mwgResInfo*.log) to the hard drive while it is waiting for the trigger, keeping disk activity silent until the event actually happens.
For 12.2.x:
mwgResInfo.sh -p 1 -c 1 -l 16000000 -e "echo "Capture for reaching threshold" | /opt/McAfee/ldt/getlogs/getlogs.sh /opt/mwg/debug ; gcore -o /opt/mwg/log/debug/mwg_core_dump $(pgrep -n mwg-core); /etc/init.d/mwg restart" -n -noLog
Detect Traffic Overload
Trigger feedback generation only when CPU usage is over 95% and working queue depth exceeds 50.
Command.
mwgResInfo.sh -p 1 -c 6 -l 95 -AND -c 7 -l 50 -e "./feedback.sh" -ot
Schedule Diagnostic Captures
Enable automated GetLogs capture for 48 hours while running standard log metrics every 5 minutes, terminating after 7 days.
Command.
mwgResInfo.sh -i 300 -gl -gi 172800 -s 604800
The following list describes the commands used in the above example:
- -i 300: Changes the script's standard metric checking interval. Instead of the default 180 seconds, it is changed to 300 seconds.
- -gl: Turns on the automated
GetLogsfeature to periodically capture full system diagnostic logs. - -gi 172800: Customizes the
GetLogsfrequency. It will gather and zip up the system logs every 48 hours (172,800 seconds) and save them to the debug directory. - -s 604800: Sets a hard stop for the script. It will run continuously in the background for exactly 7 days (604,800 seconds) and then automatically terminate.
Restart Service by Thread Count
Restart the SaaS connector service automatically if the number of threads exceeds 500.
Command.
mwgResInfo.sh -i 60 -p 4 -c 3 -l 500 -e "echo 'Saas-connector has hit 500+ threads. Will restart service ';/etc/init.d/mwg restart " -o
The following list describes the commands used in the above example:
- -i 60: Changes the script's standard metric checking interval to 60 seconds.
- -p4: Monitors only mwg-saas-connector
- -c3 -l500: Triggers -e if number of threads is greater than 500.
- -e: Prints status of saas-connector with echo command and then restarts the mwg services.
