Introduction

A significant factor in improving the performance capacity and response time on microsoft.com was tuning the Microsoft® Internet Information Server (IIS) thread pool and the ASP request queue. After tuning, hits increased by about 20 million the following week, most likely due to better response time and increased capacity, which yielded a better user experience. This papers describes what you can do to improve IIS performance on your own site.

External dependencies (such as a database queries) tend to dominate the performance scene, as IIS is truly awesome at serving up .HTM and non-blocking ASPs (ASPs that finish their work more quickly than the rate at which they are being requested).

Queueing Defined

Under high load or blocked conditions, significant queueing can occur when IIS is servicing ASP Server.CreateObject requests. This happens when a component on which IIS is waiting gets called at a greater rate than the number of transactions per second the component can satisfy. When blocking occurs, incoming requests are placed in the IIS ASP Request queue and are processed in the order in which they are received. If blocking occurs only for several seconds, the queue smoothes out short fluctuations in load and all incoming requests are handled in a timely fashion.

However, when the spikes last for a long period of time (say 30 seconds), the queue builds and will peak at the IIS registry value (RequestQueueMax), which defaults to 500. At queue saturation, what happens next is determined by whether the exact queue size is at RequestQueueMax (in this case 500), or something less. When a request comes in and the number of queued requests exactly equals RequestQueueMax, IIS returns a "Server Too Busy" message. This is an accurate reflection of the current condition, because incoming requests cannot be served in a timely fashion.

If the number of queued requests is under 500, the user with the 499th request will see an hourglass as the request sits in the queue waiting for all the other requests to be filled. This is an acceptable condition if the queue clears rapidly (within 15 seconds). After about 15 seconds or so, most users probably get impatient and hit the Stop button thinking the server is hung. At this point they probably try again, which worsens the condition and gets the same or different results depending on which server picks up their next request in the DNS Round-Robin.

Queueing Philosophy

Any effort spent processing requests that have aged more than the amount of time the average user is willing to wait for the page is a waste of capacity. The user making the original request will not be around to see the results of the server's work as they will have given up and hit the Stop button. High queue latency also has the side effect of yielding poor performance for all non-.HTM requests for the entire time the queue is saturated, because all ASP requests share the same queue. Further exacerbating the condition is the fact that processor utilization tends to spike, lengthening the duration of the poor response time, probably due to managing the large queue.

Registry Settings to Focus On

There are two Registry settings for ASP that can have a significant impact on the performance and queueing of your site. They are ProcessorThreadMax and RequestQueueMax. To get a mental picture of what these do, think of traffic on a major highway that becomes congested during rush hours. ProcessorThreadMax essentially increases the total number of lanes and RequestQueueMax is like a flow control system that limits the number of cars that can enter the highway at any one time. ProcessorThreadMax is the maximum threads per processor. RequestQueueMax is the maximum size of the request queue.

These settings are found in the Registry at the following location:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3SVC\ASP\Parameters

Tuning ProcessorThreadMax

The goal of tuning ProcessorThreadMax is to get processor utilization above fifty percent (if possible) during peak load.

To get your money's worth out of your machines, you should be seeing better than fifty percent processor utilization during peak load or you may actually have more processing capability than you need given other bottlenecks in your system.

It's not possible to calculate what the appropriate number of threads should be because live sites are too dynamic. You should gather statistics using the Performance Monitor (PerfMon), including at least:

• Processor: %Processor Time (for each processor)
• ActiveServerPages: Requests/Sec, Requests Rejected, Total Queue Length
• HttpService: Connections/Second, Current Anonymous Users

Watch PerfMon counters on one of your live machines during peak load time as it is running (use a one-second Chart interval). Typically, if you have a busy site with a variety of work going on and minimal blocking, you will see the Total Queue Length counter go up and down. If the Total Queue Length never goes up and you are running at low processor utilization, this probably indicates you have a smoothly running site with more capacity (machines) than you need. Tuning in this scenario will not help you much unless you want to increase the efficiency of your machines in an effort to reduce the total number of machines you are using.

If your queue is going up and down and your processors are running below fifty percent, this is an indication that some of your requests are blocking and you can probably benefit by increasing your threads. Start by doubling the number of threads by changing ProcessorThreadMax from 10 (the default) to 20. Restart the machine (although you may be able to get away with just starting and stopping the service, this is not practical on a busy site). You should expect to see some increase in processor utilization and the queue will tend to go up and down more quickly. Increasing the number of threads improves response time for non-blocking operations as available processing power can be put to use satisfying more requests concurrently, improving overall response time. On a site with minor blocking (the queue only reaches RequestQueueMax occasionally), increasing the threads can have a dramatic effect and eliminate virtually all queueing as there are always enough threads to handle the blocking and non-blocking requests.

After increasing the threads, if you see the queueing get worse and processor utilization actually go down, this is typically a bad sign and an indication that you may have some serious blocking problems and that a significant percentage of your requests are blocking. Another side effect of increasing threads: You may be placing more concurrent pressure on blocking problems as more threads are blocked on the same external resource. If there are any bugs in any of the components, they will tend to surface more quickly as you increase the number of threads.

If you have a single point of entry for your site that has serious blocking problems (such as your home page or another highly popular page) you need to do some redesigning of these critical pages. Typically, you will find that a small percentage of your pages are hit much more frequently than all the rest and are causing all the blocking as you have included every available feature on them to make them look impressive. Sending out a press release pointing everyone to a single page on your site at a specific time is also a great way to generate some blocking.

As an example, a typical blocking problem might be caused by an ASP using ADO to access a SQL Server located on a separate machine. If the SQL request is running a query that takes five seconds to run, and if ten requests to run this same query arrive every second, soon all the threads will be allocated and each will wait the five seconds for the query to complete and will not be available to handle other incoming requests. Complicating matters is that increased pressure on the SQL Server focused on a specific query or table will often make the query take much longer, exacerbating the condition. This demonstrates the importance of ensuring that all activities are highly tuned and stay ahead of incoming requests.

If your queue stays down and processor utilization increases, continue increasing ProcessorThreadMax until you hit your target CPU utilization (stay below seventy percent and keep ProcessorThreadMax below 100). You may discover that the network I/O or another limiting factor is placing a limit on your performance, you have more machines than you really need, or you do not have any blocking and high CPU utilization cannot be achieved because IIS is extremely efficient for .HTM and non-blocking ASP.

Tuning RequestQueueMax

The goal of tuning RequestQueueMax is to ensure good response time while minimizing the Server Too Busy condition.

Now that you have your threads dialed in, you can focus attention on dialing in your RequestQueueMax. The goal here is simple. Determine your target response time (maybe 10 seconds) and keep the queue at a size that will never queue more than that many seconds of work: This is your upper queue limit. The ideal queue size would be below your response time limit, and still above your typical peak load queue size.

A queue size that's too small will yield too many Server Too Busy messages under load spikes; with a queue too large, the site will look hung and the user will give up. Watching the queue at peak time, you should see a natural pattern of ups and downs. Make note of the peaks and put the queue just above the peaks but under your response-time threshold. The point is to use the queue to handle short-term spikes, ensure response time, and throttle the system to avoid overload with sustained, unexpected spikes. You can calculate the response time by timing how long it takes for the queue to return to 0 after it goes up to the maximum value. On a smoothly running, non-blocking site, the queue will typically stay near 0 most of the time as requests are coming in and getting satisfied in a timely fashion.

Tip If you have no data, a good starting setting seems to be a one-to-one ratio of queue to total threads. Example: If you have ProcessorThreadMax set at 25 and 4 processors (100 threads), then start your RequestQueueMax at 100 and tune from there.

How to Minimize Queueing and Improve Response Time

1. Use .HTM files when possible (especially for special events that create overload conditions) as they never block and get maximum caching benefits.
2. Minimize the use of Server.CreateObject in your ASPs whenever possible.
3. Severely minimize external dependencies, such as anything not on the same machine as your Web server.
4. Maximize network performance and reliability to external dependencies such as back-end SQL databases or remote file shares.
5. Load test and rate all custom objects before deploying them on your site.
6. Make sure every component will execute faster than the rate at which it is called. If a component is called 20 times per second, it must complete each cycle in much less than 1/20 of a second or it will block. A single blocking component can ruin the performance of all ASP on your site.
7. Monitor PerfMon stats for your site each week. A single blocking object can impact your whole site.

On a smoothly running site, you should see very little queueing with all spikes staying well below your RequestQueueMax throughout the day as all components are able to handle the load placed on them. Processor utilization will hang around your target rate during peak load.

© 1997 Microsoft Corporation. All rights reserved. Terms of Use.