Stressing nginx

[Maxim Dounin]

Hello!

On Thu, Mar 11, 2010 at 07:29:24AM -0500, stormdust wrote:

> Hi everybody.
> 
> I have been testing nginx with mono to see if we can go this 
> pass in development on the company.
> 
> I have made 4 simple pages:
> -Two static pages (html, one with 1k and another with 25k)
> -Two dinamic pages (one just compile on mono, the second compile 
> in mono and go to SQL database, both made cache on mono)
> 
> More relevant data: Runing Suse 11.2 with nginx 0.84 and fastcgi 
> with mono 2.6.

You probably mean nginx 0.8.34 instead?

> 
> Result
> http://img412.imageshack.us/img412/3068/graph4x.png
> 
> I had some problems ad-hoc during the test that i didn't manage 
> to resolve with the configurations in ngnix. I really need some 
> help to try identify the causes of the problems. Because the 
> problems are random it seems, and not being caused by stressing 
> the server.
> 
> I identify 3 major problems with nginx:
> - upstream sent unexpected FastCGI record: 3
> - upstream sent unsupported FastCGI protocol version: 116
> - upstream sent unsupported FastCGI protocol version: 60 while reading upstream
> 
> One maybe be causing by mono (check mono log).
> 
> Nginx log:
> 2010/03/05 18:27:45  5249#0: *16076 upstream sent unexpected FastCGI record: 3 while reading response header from upstream, client: 127.0.0.1, server: localhost, r$

Request was cancelled by backend via FASTCGI_END_REQUEST record.  
It's basically means that backend is unwilling to process request 
for some reason, and nginx considers this to be an error.

You have to look into detailed backend logs to find out what's 
the reason.

> 2010/03/05 18:27:45  5249#0: *16102 upstream sent unsupported FastCGI protocol version: 116 while reading upstream, client: 127.0.0.1, server: localhost, request: $

Backend sent some gargabe in fastcgi version field (it must be 1), 
or protocol happened to be out of sync.  Enabling debug logging in 
nginx may help to find out what's happending.

> 2010/03/05 18:27:45  5249#0: *16148 FastCGI sent in stderr: "Failed to parse parameter data" while reading response header from upstream, client: 127.0.0.1, server$
> 2010/03/05 18:27:45  5249#0: *16148 upstream sent unexpected FastCGI record: 3 while reading response header from upstream, client: 127.0.0.1, server: localhost, r$

Backend sent error to fastcgi's stderr and cancelled request via 
FASTCGI_END_REQUEST record.  See above.

(everything else seems to be the same, skipping)

> Mono log:
> 
> Parameter name: s
> [2010-03-05 18:27:45Z] Error   Aborting request 1. Reason follows:
> [2010-03-05 18:27:45Z] Error   Failed to parse parameter data.
> [2010-03-05 18:27:45Z] Error   Failed to process connection. Reason: Argument cannot be null.
> Parameter name: s
> [2010-03-05 18:27:45Z] Error   Aborting request 1. Reason follows:
> [2010-03-05 18:27:45Z] Error   Failed to parse parameter data.
> [2010-03-05 18:27:45Z] Error   Failed to process connection. Reason: Argument cannot be null.
> Parameter name: s
> [2010-03-05 18:27:47Z] Error   Aborting request 1. Reason follows:
> [2010-03-05 18:27:47Z] Error   Failed to parse parameter data.
> [2010-03-05 18:27:47Z] Error   Failed to process connection. Reason: Argument cannot be null.
> Parameter name: s
> [2010-03-05 18:27:48Z] Error   Failed to process connection. Reason: 8 bytes expected.
> Parameter name: record
> [2010-03-05 18:27:48Z] Error   Failed to process connection. Reason: 8 bytes expected.
> Parameter name: record
> [2010-03-05 18:27:48Z] Error   Failed to process connection. Reason: 8 bytes expected.
> Parameter name: record

[...]

This doesn't really explains anything as it lacks any details.

For now I suppose the problem is in mono, and most likely it 
manifests itself when full fastcgi record can't be read via single 
read() syscall as it happened to be fragmented into multiple 
packets and/or read() was interrupted by signal.  I may be wrong 
though.

Further analysis require more details - at least nginx debug log.  
Something like tcpdump (with packet contents) of problematic 
request to backend and more details from mono should be helpful, 
too.

Maxim Dounin