Troubleshooting NTLM

Category:

Authentication

Environment:

Secure Web Gateway

Problem:

Users are being prompted to authenticate.

Cause:

Rule misconfiguration
SWG to DC communication issues
Client to SWG communication issues
Time Skew
Performance issues
Known bugs and errors

Log Collection:

Client-side packet capture (-s 0 -i any host <client IP>)
DC-side packet capture (-s 0 -i any port 445 or port 53)
Rule trace
Feedback file (Enable authentication debugging logs)

Note: collect both packet captures at once by using -s 0 -i any host <client IP> or port 445 or port 53, keep in mind this will collect more traffic and might cause the packet capture to reach the UI limit of 200k frames.

Log Analysis:

Rule Misconfiguration

NTLM authentication can be implemented on SWGs deployed in either “explicit” or “transparent” mode. The ruleset that will be used will depend on which deployment method is being used. Below I have included screenshots of common rules available within the Rule Set Library.

Note: In explicit mode, ALL HTTP/HTTPS requests are authenticated, while in transparent mode it is common for authentication to occur on a time-base, once every hour, two, depending on the configuration.

Explicit Proxy

Transparent Proxy

For more information on NTLM authentication, review Choose the Right Authentication Method for your Deployment

SWG to DC Communication Issues

Ensure the web gateway is able to resolve the DC’s FQDN IP address and is able to communicate to the DC over port 445.

GUI: -s 0 -i any port 445 or port 53

CLI: tcpdump -s 0 -i any port 445 or port 53 -w SWG_DC.pcap

Packet capture over port 445

Client to SWG Communication Issues

Review the communication between the client and SWG. The packet capture between the client and SWG should show the following interaction:

Client sends an initial request
SWG responds with 407 AuthRequired
Client sends NTLM Negotiate
SWG responds with NTLM Challenge
Client sends NTLM Auth (contains credentials)
SWG responds with 200 OK

Explicit Proxy packet capture:

Note: The communication should all be within a single TCP stream. Load balancers in between the client and SWG are known to break this communication into two or more TCP streams, resulting in issues.

Time Skew

In the context of NTLM authentication, time skew refers to the allowable time difference between the clocks of the client and the server involved in the authentication process. In ideal circumstances, the time difference between the Domain Controllers and the SWG should be null. Beneficial checks:

service ntpd status ( Verify if the service is running )
timedatectl status ( Verify if NTP is active)
service ntpd stop

ntpd -gq > sends an immediately queries the NTP servers, makes necessary time adjustments, synchronizes the system clock, and then exits
ntpq -pn > outputs the NTP server system is synching with
service ntpd start

(These 3 commands will stop the ntp autosync, manually sync the time, then restarts the ntp autosync)

Performance Issues

Performance issues can range from SWG hardware issues to network latency, and anything in between. Reviewing the SWG counters can help determine whether the SWG is overloading and/or there is network latency experienced within the customer’s environment.

ResolveHostViaDNS - Average Time to Resolve Host via DNS (ms)

AuthNTLMRequests - NTLM Requests (number)

NTLMRequestProcTime - NTLM Request Average Processing Time (ms)

AuthNTLMCacheRequests - NTLM Requests from Cache (number)

Known bugs and Errors

Latest Secure Web Gateway (On-prem) Releases

Reference	Description	Resolved in

WP-5419	When the authentication process on Secure Web Gateway uses the basic NTLM authentication method, adding the default domain of the NTLM authentication server to the settings, no longer leads to a failure of the process.	11.2.12
WP-4245	An admin user can again log onto Web Gateway using NTLM authentication successfully	11.2