Load Balancer Best Practices
Introduction
Happy customers use load balancers. They allow for easy maintenance of your proxies and help reduce any proxy related issues from impacting your users. This guide can serve is a reference point for how Web Gateway can help with your load balancer strategy.
T1hank you for your input from the original Community discussion:
Load Balancer Modes
There are various modes that load balancers support. Below is a synopsis of the common modes and what to expect from them.
- Direct Routing / Server Return mode = Load balancer simply routes the packets directly to the upstream Web Gateway. Client IP is obtained from the source IP in the packets.
- Web Gateway Mode: Direct Proxy w/ Layer 2, Transparent Router
- NAT mode = Upstream Web Gateway must be in segmented network with load balancer as the default gateway. Client IP is obtained from source IP in packets.
- Web Gateway Mode: Direct Proxy w/ Layer 2, Transparent Router
- Source NAT mode = Load balancer acts as a proxy and sends traffic to upstream Web Gateway must parse X-Forwarded-For to get client IP. In some configurations the X-Forwarded-For can be unreliable.
- Web Gateway Mode: Direct Proxy
Web Gateway Modes
As it relates to a load balancer deployment, Web Gateway has three modes it might be deployed in when working with a load balancer:
Direct Proxy
- Typically used in source NAT mode because Web Gateway is just an upstream proxy.
Direct Proxy w/ Layer 2
- Typically used in high bandwidth deployments with more intelligent load balancers.
- Accepts packets where destination IP is not it's own -- No iptables modification required!
Transparent Router
- Typically used in deployments with less intelligent load balancers, bandwidth/throughput is a trade-off because Web Gateway must perform some redundancy functions which require use of the Skyhigh Security Network Driver.
- Skyhigh Security Network Driver (mfend) has a maximum throughput of 1Gbps
- Like Layer 2, it accepts packets where destination IP is not it's own -- No iptables modification required!
Load Balancer Health Checks
To check the health of the Web Gateway, a variety of health checks can be performed. These health checks could be as simple as a ping, or the load balancer could be checking the response body for certain text.
Ping
Pinging the Web Gateway simply validates that it's reachable, it doesn't tell us if traffic is flowing. As a result ping should be an auxiliary test performed, but not be a health check which carries too much weight.
TCP Port Check
Checking to see if the Web Gateway's proxy port is available is another test to validate the services are functioning on a very basic level.
HTTP Request
The ultimate test of whether or not the proxy is alive, is whether it can serve requests! For this purpose we've come up with some example health checks that can be used to check the status of the Web Gateway.
Internal Check
First example is an internal check, meaning we want to see if the Web Gateway is available WITHOUT connecting to the internet. When paired with the ruleset attached to this guide, the example request request for healthcheck.swginternal.com will be served only from the Web Gateway -- no DNS or connection to the internet is required. This could be useful in situations where there might be a problem with the outbound internet connection, but the Web Gateway is up and running just fine. The example requests demonstrate a GET and HEAD request (HEAD requests means no response body is returned).
# URL: http://healthcheck.swginternal.com/
# Load Balancer Request (\r\n = carriage return):
GET / HTTP/1.0\r\nHost: healthcheck.swginternal.com\r\nUser-Agent: Proxy Health Check Internal\r\nConnection: Close\r\n\r\n
# Example cURL Command:
curl -vi --proxy http://YOUR-SWG:9090 -H "Proxy-Connection: Close" -H "User-Agent: Proxy Health Check Internal" http://healthcheck.swginternal.com/
# Request from Load Balancer to Web Gateway :
GET / HTTP/1.0
Host: healthcheck.swginternal.com
User-Agent: Proxy Health Check Internal
Connection: Close
# Response Web Gateway to Load Balancer:
HTTP/1.0 200 OK
Connection: Close
Skyhigh Security {{webg}}
# URL: http://healthcheck.swginternal.com/
# Load Balancer Request (\r\n = carriage return):
HEAD / HTTP/1.0\r\nHost: healthcheck.swginternal.com\r\nUser-Agent: Proxy Health Check Internal\r\nConnection: Close\r\n\r\n
# Example cURL Command:
curl -vi --proxy http://YOUR-SWG:9090 -H "Proxy-Connection: Close" -H "User-Agent: Proxy Health Check Internal" -X HEAD http://healthcheck.swginternal.com/
# Request from Load Balancer to Web Gateway :
HEAD / HTTP/1.0
Host: healthcheck.swginternal.com
User-Agent: Proxy Health Check Internal
Connection: Close
# Response from Web Gateway to Load Balancer:
HTTP/1.0 200 OK
Connection: Close
External Checks
The second kind of check would be to see if the Web Gateway's outbound internet connection is alive. If it isn't maybe it's time to pull the Web Gateway from the pool, or alert in order to take corrective action. The below examples are requests to commonly used URLs, most are captive portal URLs used by Windows, Apple, or Skyhigh Security products to validate if the machine has access to the internet. Each of them return 200 response codes simple strings in the body that can be examined by the load balancer in order to determine a status.
Microsoft's Network Connectivity Status Indicator: https://technet.microsoft.com/enus/library/cc766017(v=ws.10).aspx
# URL: http://www.msftncsi.com/ncsi.txt
# Load Balancer Request (\r\n = carriage return):
GET /ncsi.txt HTTP/1.0\r\nHost: www.msftncsi.com\r\nUser-Agent: Proxy Health Check External\r\nConnection: Close\r\n\r\n
# Example cURL Command:
curl -vi --proxy http://YOUR-SWG:9090 -H "Proxy-Connection: Close" -H "User-Agent: Proxy Health Check External" http://www.msftncsi.com/ncsi.txt
# Request from Load Balancer to Web Gateway :
GET /ncsi.txt HTTP/1.0
Host: www.msftncsi.com
User-Agent: Proxy Health Check External
Connection: Close
# Response from Web Gateway to Load Balancer:
HTTP/1.0 200 OK
Content-Type: text/plain
Content-Length: 14
Connection: Close
Microsoft NCSI
Skyhigh Security's Captive Portal Check URL:
# URL: http://scp.webwasher.com/test/SCP.txt
# Load Balancer Request (\r\n = carriage return):
GET /test/SCP.txt HTTP/1.0\r\nHost: scp.webwasher.com\r\nUser-Agent: Proxy Health Check External\r\nConnection: Close\r\n\r\n
# Example cURL Command:
curl -vi --proxy http://YOUR-SWG:9090 -H "Proxy-Connection: Close" -H "User-Agent: Proxy Health Check External" http://scp.webwasher.com/test/SCP.txt
# Request from Load Balancer to Web Gateway :
GET /test/SCP.txt HTTP/1.0
Host: scp.webwasher.com
User-Agent: Proxy Health Check External
Connection: Close
# Response from Web Gateway to Load Balancer:
HTTP/1.0 200 OK
Content-Type: text/plain
Content-Length: 7
Connection: Close
{{corp}}
Apple's Captive Portal Check URL:
# URL: http://captive.apple.com/hotspot-detect.html
# Load Balancer Request (\r\n = carriage return)
GET /hotspot-detect.html HTTP/1.0\r\nHost: captive.apple.com\r\nUser-Agent: Proxy Health Check External\r\nConnection: Close\r\n\r\n
# Example cURL Command:
curl -vi --proxy http://YOUR-SWG:9090 -H "Proxy-Connection: Close" -H "User-Agent: Proxy Health Check External" http://captive.apple.com/hotspot-detect.html
# Request from Load Balancer to Web Gateway
GET /hotspot-detect.html HTTP/1.0
Host: captive.apple.com
Connection: Close
User-Agent: Proxy Health Check External
# Response from Web Gateway to Load Balancer
HTTP/1.0 200 OK
Content-Type: text/plain
Cache-Control: max-age=30, must-revalidate
Content-Length: 69
Connection: Close
<HTML><HEAD><TITLE>Success</TITLE></HEAD><BODY>Success</BODY></HTML>
ICAP Health Check
If you're using Web Gateway as an ICAP server it will also be useful to check the health of the ICAP service. No rules are required for this to function.
OPTIONS Request
To check basic availability we'll send an OPTIONS request to the Web Gateway, this will just tell us if the service is available, and what capabilities it supports.
# Load Balancer Request (\r\n = carriage return)
OPTIONS icap://YOUR-SWG ICAP/1.0\r\nUser-Agent: Proxy Health Check ICAP\r\n\r\n
# ICAP Request from Load Balancer to Web Gateway
OPTIONS icap://YOUR-SWG ICAP/1.0
User-Agent: Proxy Health Check ICAP
# ICAP Response from Web Gateway to Load Balancer
ICAP/1.0 200 OK
Methods: REQMOD, RESPMOD
Options-TTL: 3600
Encapsulated: null-body=0
Max-Connections: 100
Preview: 30
Service: {{corp}}{{webg}}7.7.2 build 24546
ISTag: "00005977-14.38.44-00008745"
Allow: 204
Health Check Rule Set
To ensure that the load balancer health checks receive priority when passing through the Web Gateway, we've created a Proxy Health Check Rule Set to handle the HTTP Request health checks.
Download the Proxy Health Check Rule Set:
Import the Rule Set and Template
- Create an "(Empty)" Template Collection and Import the "Proxy Health Check - Service Unavailable Template" zip file into the (Empty) collection.
- Import the Proxy Health Check Rule Set and move it to the top of your rules.
- Review the "Proxy Health Check URLs" list and add or remove any URLs as needed. The default URLs (as described in the health checks examples) are commonly used by other devices and are suitable for our needs here.
Maintenance Mode
The second rule in the ruleset is used for putting Web Gateway (single or multiple) into maintenance mode, meaning it will be pulled from the load balancer pool as long as it's in that list (and the load balancer is properly configured). Simply add the Web Gateway's hostname into the list, and it will send a 503 HTTP response code along with a different HTTP response body. Configuring maintenance mode allows for easy administration, as you won't need to pester the load balancer administrators everytime you need to update the Web Gateway's.
When a request is made to a Web Gateway in Maintenance Mode, the response would return a HTTP 503, with "Service Unavailable" as the response body:
# Request from Load Balancer to Web Gateway :
GET / HTTP/1.0
Host: healthcheck.swginternal.com
User-Agent: Proxy Health Check Internal
Connection: Close
# Response from Web Gateway to Load Balancer:
HTTP/1.0 503 ServiceUnavailable
Connection: Close
Content-Type: text/html
Cache-Control: no-cache
Content-Length: 19
Service Unavailable
Conclusion
We hope this helps make your life easier and better your understanding of how Web Gateway can work with your load balancer.