Spikes in latency

Hi All,

I am running a web farm of IIS7 on Server 2008 Web edition. We max out at 150 hits/sec at peak times load balanced across 4 servers.

One of our customers is complaining they are getting time outs and slow responses to one of our web services so I have been investigating.

I started by looking through the logs and found that most requests are serviced in sub 120ms but some hang on for seconds!

I followed up by running my own load tests from an external linux VPS using httping. This confirmed the behavior our client and the logs were showing.

To track down the issue I used powershell to write a quick test script * so I could test behind the firewall; all of the following figures are based upon script looping through 10,000 requests. I got the same behavior but less pronounced (some requests taking 1000+ms with an average of 18ms).

I thought maybe it was the Load balancer, so I ran the script on the web servers (one by one); Same behavior, less pronounced again ( long requests taking 25+ms avg 1ms) .

I reduced complexity and ran it against localhost with the default website; same behavior same times as above .

I took it to the test lab, clean (ish) install of 2k8 default website in IIS7; same results.

I am now perplexed? I have looked online for idea's on why I am getting seemingly random high latency spikes. I am sure that other things in my network such as Load balancers ect are amplifying the issue externally but to begin with I aim to get my max requests in my lab to a same order time as my average.

*SCRIPT WARNING

$client = new-object System.Net.WebClient
$slow=0
for($i=1; $i -le 10000; $i++)
{
$time = (measure-command { $client.DownloadFile( "http://localhost/", "C:\data\index.tmp")}).TotalMilliseconds
 if($time -gt 10)
 {
  echo "Time over 10 ms: " $time
 }
}

/SCRIPT

Obviously if you Tee-Object -file the output of the measure-command to a text file you can put it in a spreadsheet and Analise .

I've been getting a quick break down out of Excel that looks like this.

Any idea's on things I could monitor, tune, ect?

Regards

*PS I vote for an upgrade of the WYSIWYG editior so that it works in Opera.