LSASS (Local Security Authority Subsystem Service) – the Windows process that is responsible for enforcing the security policy on the system - adopted its threading library from IIS to handle Windows socket communication, and uses the asynchronous thread queue to handle requests from Kerberos and LDAP.
Monitoring the asynchronous thread queue (ATQ) on an AD server will provide useful pointers to the request processing ability of the server. This test monitors the ATQ, reports the number and nature of requests queued in the ATQ, captures a steady growth (if any) in the length of the queue over time, and thus reveals potential processing bottlenecks on the AD server.
Target of the test : An Active Directory server
Agent deploying the test : An internal agent
Outputs of the test : One set of results for every Active Directory server being monitored
| Parameters | Description |
|---|---|
| Test period | This indicates how often should the test be executed. |
| Host | The IP address of the machine where the Active Directory is installed. |
| Port | The port number through which the Active Directory communicates. The default port number is 389. |
| Measurement | Description | Measurement Unit | Interpretation |
|---|---|---|---|
| ATQ estimated queue delay | Indicates how long the request has to wait in the queue. | Secs | This is the estimated time the next request will spend in the queue prior to being serviced by the directory service. |
| ATQ outstanding queued requests | Indicate the number of requests currently in the queue. | Number | A high level of queuing indicates that requests are arriving at the domain controller faster than they can be processed. This can also lead to a high latency in responding to requests. |
| ATQ request latency | Indicates the time it takes to process an enqueued request. | Secs | Since the type of requests can differ, the value of this measure is typically not significant, as its an average value. An expensive LDAP query that takes minutes to execute can be masked by hundreds of fast LDAP queries or KDC requests. The main use of this measure therefore is to monitor the wait time in queue. Any non-zero value indicates that DC has run out of threads. |
| ATQ threads ldap | Indicates the number of threads used by the LDAP server as determined by LDAP policy. | Number | This measure indicates the number of threads currently servicing LDAP requests. If the value of this measure is unusually high, then look for the following:
Large values for this measure are common but the thread count should remain less than the the value of the ATQ threads total measure. Also, note that this measure could also report abnormally high values for reasons that are initially triggered by LDAP but are ultimately affected by external reasons. Such reasons are as follows:
|
| ATQ thread others | Indicates the number of threads used by other component, in this case KDC. | Number | You can also have external dependencies generating requests that hit the Kerberos Key Distribution Center (KDC). One common operation is getting the list of global and universal groups from a DC that is not a Global Catalog (GC). A 2nd external and potentially intermittent root cause occurs when the Kerberos Forest Search Order (KFSO) feature has been enabled on Windows Server 2008 R2 and later KDCs to search trusted forests for SPNs that cannot be located in the local forest. The worst case scenario occurs when the KDC searches both local and trusted forests for an SPN that can’t be found either because the SPN does not exist or because the search focused on an incorrect SPN. Memory dumps from in-state KDCs will reveal a number of threads working on Kerberos Service Ticket Requests along with pending RPC calls to remote domain controllers. |
| ATQ threads total | Indicates the total number of threads that are currently allocated. | Number | This measure tracks the total number of threads from the ATQ threads ldap and ATQ threads other measures. The maximum number of threads that a given DC can apply to incoming workloads can be found my multiplying the product of MaxPoolThreads times the number of logical CPU cores. MaxPoolThreads defaults to a value of 4 in LDAP Policy and should not be modified without understanding the implications. Compare the value of this measure with the value of the ATQ threads ldap and ATQ threads other measures. If the ATQ threads ldap measure equals this measure in value then it implies that all of the LDAP listen threads are stuck processing LDAP requests currently. If the ATQ threads other measure equals this measure in value, then it means that all of the LDAP listen threads are busy responding to Kerberos related traffic. Similarly, note how close the current value for this measure is to the max value recorded in the trace and whether both values are using the maximum number of threads supported by the DC being monitored. If so, it’s a sure sign of a potential overload. |
