Real-Time Log Streaming

Before you Begin

Make sure the following prerequisites are met:

IMPORTANT: To collect the logs in real-time, you must have the corresponding license. To obtain the license, select Get the Skyhigh Log Stream on the SIEM Integration (Inline) tab, and then click Contact. You will be redirect to the Skyhigh Support for further assistance.

Download and install Cloud Connector version 6.8.1 and above. To download the latest version of the Cloud Connector, see Download Skyhigh Cloud Connector. Before installing the Cloud Connector, make sure the Skyhigh Cloud Connector Prerequisites are met. However, you must have the additional prerequisites to install and configure the Log Stream.
You must have the Skyhigh Cloud Connector user role to install and configure Cloud Connector. For details, see About User Roles and Access Levels.
You must access Skyhigh CASB user interface from the same network on which your Cloud Connector is installed. Otherwise, an error message displays and you cannot enable the feature or configure settings. Error message: "SIEM Integration (Inline) setting cannot be configured, viewed or modified if accessed from an external network or the Cloud Connector is not reachable. You need to be inside your company's network and the Cloud Connector instance needs to be up and running."

Prerequisites

Cloud Connector Setup with Log Stream	Cores	RAM	Log Stream Max Memory Usage
Only Log Stream functionality in Cloud Connector (Recommended)	8 Core	16 GB	On logstream.vmoption file, set the system memory to Xmx8192m or more. On shnlps.vmoption file, set the system memory to Xmx4096m.
Log Stream with other Cloud Connector functionalities	8 Core (minimum for CC) + 4 Core (Log Stream)	8 GB (minimum for CC) + 8 GB (Log Stream)	No update required.

Cloud Connector Setup with Log Stream

Cores

RAM

Log Stream Max Memory Usage

Only Log Stream functionality in Cloud Connector (Recommended)

8 Core

16 GB

On logstream.vmoption file, set the system memory to Xmx8192m or more.

On shnlps.vmoption file, set the system memory to Xmx4096m.

Log Stream with other Cloud Connector functionalities

8 Core (minimum for CC) + 4 Core (Log Stream)

8 GB (minimum for CC) + 8 GB (Log Stream)

No update required.

NOTE: Do not upgrade Cores and RAM if you are not utilizing the Log Stream functionality.

Log Stream

The Log Stream (found under Settings > Infrastructure > Cloud Connector > SIEM Integration (Inline)) collects the near real-time Security Service Edge web access data within your network or feeds directly into your reporting and analytics tools. Real-time logs allow you to identify the issues as they occur. This helps to address the issues without any delay. You can save the logs to a local directory or send them to your third-party SIEM systems (Security Information and Event Management) through a Syslog server. You can use these files to investigate or to perform analysis with Skyhigh SSE.

The Log Stream can simultaneously download the data originating from different log types such as Secure Web Gateway (SWG), Remote Browser Isolation (RBI), Private Access, and Cloud Firewall which eliminates running multiple instances to collect data from different log types. A single Enterprise Cloud Connector accommodates downloading multiple log types at the same time.

Install Log Stream

When you install Cloud Connector, the Log Stream will get installed automatically. This unified application supports multiple platforms like Windows and Linux operating systems. After installing the Cloud Connector, start the Cloud Connector, and then configure the Log Stream.

Configure Log Stream

NOTES:

You can configure and edit eight log streaming configurations. Use the toggle button to disable or enable each configuration.
For each log stream configuration, you can add a maximum of four Syslog configurations.
By default, the API is set to the latest version, API version 13. However, the API version cannot be updated. For details about the related fields in version 13, see Reporting Fields.

To configure Log Stream, click Add New Log Stream Configuration, and then configure the settings based on the table below:

Fields	Description
Region	Choose a region depending on where your data is stored. By default, North America is the selected region: North America United Arab Emirates Europe United Kingdom Singapore India Australia Kingdom of Saudi Arabia
Log Type	You can simultaneously download different types of logs, including logs with data originating from the Secure Web Gateway (SWG), Remote Browser Isolation (RBI), Private Access, and Cloud Firewall. By default, the Log Type is selected as SWG. The Log Stream downloads log data based on the chosen log types. NOTE: To access Secure Web Gateway (SWG), Remote Browser Isolation (RBI), Private Access, and Cloud Firewall log files, you must have the corresponding licenses. For SSE license details, see About Skyhigh Security Service Edge, and for assistance, contact Skyhigh Support.
Save in Directory	Use this option to store the log files in your local directory. If you choose this option, configure the following: Directory Path. Choose the path to download the logs to your system. File Format. You can store the log files in two different formats: CSV. The logs are collected in the form of CSV format and stored in the local directory. By default, the CSV file format is selected. JSON. The logs are collected in the form of JSON format and stored in the local directory. Max File Size (MB). Specify the maximum file size (in megabytes) for storing logs in your directory. Each file can store log data ranging from 10 MB to 1024 MB (1 GB). By default, the Max File Size is 25 MB.
Send as Syslog	Use this option to send the log files to the Syslog server. If you choose to Send as Syslogs, configure the following options: Syslog Host. Enter the hostname or IP address of the Syslog server. By default, the host value is set to localhost. Syslog Port. Enter the port number of the Syslog server. By default, the port value is set to 513. Use Encryption. Choose an option whether to use Transport Layer Security (TLS) to transport logs. By default, the option is set to No. You can select the Yes option only if your Syslog server supports TLSv1.3. Syslog Protocol. Choose a protocol option for transport. It transfers the data to the Syslog server using TCP or UDP depending on your selection. By default, the protocol is set to TCP. . Click to add Syslog configuration. You can have maximum of four Syslog configurations.

You have successfully configured the Log Stream.

After configuring, you must start the Log Stream. You can start the Log Stream using Linux or Windows.

Start the Log Stream Using Linux or Windows

NOTE: Before starting the Log Stream service, make sure the Cloud Connector service is up and running.

Linux

To start the Log Stream service, go to the installer directory, and run the following command:

$ ./logstream start

Windows

You can start the Log Stream service using the Windows Command Prompt or Windows Services:

To start the Log Stream service using the Windows Command Prompt, go to the installer directory and run the below command:

.\logstream.exe /start

To start the Log Stream service using the Windows Services perform the steps below:

Go to Services from the Start Menu.
Select the Log Stream service.
Right-click the Log Stream service, and then select Start from the menu.

Check the Cloud Connector and Log Stream Status

After starting the Log Stream service, check the Cloud Connector and Log Stream status. Both the services should be up and running. To check the status, run the below commands:

Services	Linux	Windows
Cloud Connector	$ ./shnlps status	.\shnlps.exe /status
Log Stream	$ ./logstream status	.\logstream.exe /status

If you experience any issues with the configurations, contact Skyhigh Support for assistance.

NOTES:

When you configure the Log Stream, the configuration changes will appear in the application only after seven minutes.
If your directory is not responding or the Syslog server is down, data buffering will not occur. Only the real-time data is transferred. Hence, you must get the data from the Log Collector.
Each log event is assigned a time stamp upon entering the event streaming platform. The logs are retained in the event streaming platform for 24 hours. However, the logs older than 24 hours are automatically deleted.

Best Practices

► Click to view the best practices listed below to make sure seamless functioning of the Log Stream and prevent data loss:

Network Connectivity

We recommend calculating and upgrading the network and internet bandwidth based on the messages or events generated per second. For example, if you receive 10,000 messages per second and each message is 5 KB and the rate of compression is 10, then 5 MBps bandwidth is required. You must monitor the network usage and upgrade the network or bandwidth if it reaches 70% of the available bandwidth to prevent potential data loss.
Maintain a stable internet connection that can consistently reach Skyhigh domains like skyhigh.cloud, myshn.net, and myshn.eu.

System Resource Management

Monitor CPU, memory, and disk usage of the application and your overall system resources. If resource usage consistently approaches or exceeds 70%, upgrade your hardware or virtual machine to ensure sufficient resources to operate smoothly.

Maintain the Application

Always keep the Cloud Connector version updated to benefit from bug fixes and performance improvements.

Responding to Issues

If the LogStream application stops downloading and processing the logs, contact Skyhigh Support immediately to troubleshoot the problem. Resolving the issue within 24 hours will significantly minimize data loss.

NOTE: For detailed Cloud Connector and Log Stream debug logs, refer to directories located at <cloud_connector_install_dir>/logs and <cloud_connector_install_dir>/logstream/logs, respectively.