Considerations when Allowlisting HTTPS URLs
Introduction
This document will cover the configuration of how to allowlist or blocklist SSL (HTTPS) based websites while using the SSL scanner. There are many different configurations which a ruleset can be configured with however so we will provide some best practice ideas when implementing rules along with some information into how the SSL (HTTPS) process works.
Overview
In order to block or allow SSL traffic on the Web Gateway, you are going to need to have some basic knowledge of how SSL traffic works with proxies and what needs to be done to inspect SSL traffic. One major thing to take into consideration is that SSL traffic is different from HTTP traffic as more steps are needed to ensure the encryption of the traffic.
In HTTP, the full URL (www.domain.com/images/logo.gif) is transmitted in clear text and allowlisting (or blocking) can apply to the full URL.
For HTTPS requests, only the Host (www.domain.com) is transmitted in clear text. The full URL is encrypted inside the SSL Tunnel. To allowlist the full URL SSL inspection is needed and you have to ensure that the request is not blocked, before the full URL has been transmitted in the tunnel.
Here is an overview of the difference between the two:
This is what a HTTP connection looks like in Wireshark:
This is what happens during a HTTPs (SSL) “CONNECT” command;
Here is an example of a “CONNECT” attempt from the client which contains the HTTP header information and only the “Host”, not the full URL. This is what the Web Gateway will be able to see if the request is “tunneled” or the “SSL Scanner” is not used.
SSL scanning can be much more granularly controlled in the current versions (above 7.7.x) by using rule criteria associated with the SSL scanning ruleset.
For example, certificate check or content check bypasses can be based on any combination of criteria including certificates, user agent, destination IP, destination category, etc.
There are 2 options on how Web Gateway can use a client certificate:
- It is possible that Web Gateway uses a client certificate in the https connection to the server.
- This above statement stands true only if the connection will not be tunneled. It will also work if the client connection is HTTP and only the connection to the server is https.
- It selects a predefined certificate, independent from the client.
- Web Gateway will not send the client certificate in advance. It will send it only if it is configured to do so and if the server asks for it.
There are two different stages when a server may ask for a client certificate:
First during the initial SSL handshake
- Second during the re-negotiation sequence.
- The HTTPs connection goes through two cycles- the first being the "CONNECT CALL" and the second cycle being the "CERTVERIFY" cycle.
Therefore, if the HTTPs CONNECT call is allowed through, we should be passing the certs to fully establish the SSL communications between endpoints of the two tunnels "Client --- Tunnel1---> Web Gateway---Tunnel2---> Web Host".
After the Web Gateway gets the SSL Certificate, it will then process the certificate validation rules in the SSL Scanner ruleset against the server cert to determine if the connection should be allowed through to the site.
Two options:
- to use a client certificate list (1)
- to use a predefined certificate (2)
Both modes allow the customer to import certificates together with an RSA or DSA private key with a minimum size of 1024 bit. In a FIPS environment, the minimal size is 2048 bit. A certificate chain can/needs only to be imported for a predefined client certificate because in the other case the certificate chain sent by the client will be forwarded to the server.
When working with SSL traffic, there are two very important scenarios to consider:
- When working without SSL scanner you will need to keep the following things in mind;
- After the “CONNECT” occurs, the data will be encrypted.
- The “CONNECT” only contains the destination host and not the full URL (and very few other headers that can be read).
- The Steps below do NOT apply to setups without SSL Scanner. Without SSL Scanner, only hosts, not full URLs can be allowlisted
- When working with SSL scanner you will need to keep the following things in mind;
- The “CONNECT” only contains the destination host and not the full URL (and very few other headers that can be read).
- If the CONNECT is blocked, the full URL will never be visible to the Web Gateway
- If the CONNECT is NOT blocked, the traffic will be decrypted on the Web Gateway and at this time, the full url will visible.
- When using the SSL Scanner, there are going to be two different “Request” Cycles that will need to occur;
- One for command.name equals CONNECT
- One for command.name equals CERTVERIFY
(Both of these “Request Cycle” events need to be allowed for the full URL to be passed in the SSL Tunnel in order to allow the allowlisting to occur.)
Example Scenario
In our case, the website “csm-testcenter.org” is blocked based on the “Information Security” category but still users should be allowed to go to one specific URL on the site:
https://www.csm-testcenter.org/test?...nt_information
If this were to be an HTTP request you would block the category “Information Security” in your URL Filter block rules and then allowing this one specific URL in an exception before the blocking rule (for example, the global allowlist). This will not work with SSL/HTTPS traffic;
Allowlist:
Blocking Rule:
For HTTPS requests this results in the following. The CONNECT request is being blocked:
The Issue is, that the CONNECT request the client made for the https site, did NOT contain the full URL, but only the host www.csm-testcenter.org. As a result, your allowlist did not match, and the category block was applied.
To resolve this issue, you would need to make a rule to allow the “CONNECT” command and “CERTVERIFY” to pass through without getting blocked. We know what the URL.HOST is so you could make a rule like the following:
At this point it is very important to check your brackets. You can add brackets during the rule creation:
This will then allow the “CONNECT” part of the SSL traffic to pass through the Web Gateway allowing the tunnel to be established so that the full URL can be requested in the tunnel.
Note that no actual traffic (requests or responses) are exchanged during the CONNECT. Only the connections are established and certificates are exchanged. The “GET” request inside the tunnel, is still being filtered as normal with these new rules.
The Web Gateway will now be able to decrypt the traffic and we will be left with the following as seen in a connection trace using the SSL Scanner;
In conclusion, the site will now load correctly for the part of the site which we did not want to have blocked (Full URL added to your allowlist earlier) while yet blocking the rest of the site. Granted, it might be missing a style sheet and some pictures but this is because we only allowed the one very specific URL. The other URLs could be added to the allowlist as needed.
Please keep in mind that when trying to troubleshoot SSL traffic, you will have to use Connection Tracing to see inside the tunnel. This is enabled in the “Configuration > Troubleshooting” section and the output will be in the “Troubleshooting > Connection Tracing” section.
Additionally, if you plan to use the connection tracing for troubleshooting, please enable it but restrict it to one test client or you can risk filling up the Web Gateway.
Furthermore, the client-side of the tunnel will be the traces labeled like “-C.txt” and the Web Server side of the tunnel will be labeled like “-S.txt”.
Conclusion
With this information, you should now be able to allowlist and blocklist URLs on the Web Gateway. Furthermore, you should have some additional knowledge into some of the troubleshooting steps which can be implemented for troubleshooting SSL (HTTPs) traffic as well.