499 errors, fcgi sockets require restart python and custom proxy

[JacobSingh]

Hi folks,

I posted in the paid help about this too... here is the issue:

We currently have a setup where we have multiple app servers sitting behind NGINX. We want to provide HMAC based authentication comparing an hash of the content or headers of the request against an authenticator using a known key (which varies depending on which URL you are trying to access).

We receive on port 80.
There are 50 or so sockets setup as a backend which feed into a python script (gatekeeper.py).
gatekeeper.py has the logic to authenticate the request given params in the headers, etc
It then makes a request to port :81
:81 is setup in nginx to route to our application servers
The app servers return to nginx, nginx returns to gatekeeper.py and then gatekeeper.py returns back to nginx.
 
It has been mostly successful, and fast enough, but we're having some issues with fcgi threads timing out, and having to recreate them every so often. Having an integrated solution would be ideal. 

Problem is, we keep getting these in the log:

2009/04/04 20:59:28  29828#0: *754537 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 174.36.146.67, server: balancer1.search.xxxxxxxx.net, request: "GET /solr/xxxxxxxxxxx/select?qt=mlt&fl=nid%2Ctitle%2Curl&mlt.fl=title%2Ctaxonomy_names&mlt.mintf=1&mlt.mindf=1&mlt.minwl=3&mlt.maxwl=15&mlt.maxqt=30&version=1.2&wt=json&json.nl=map&q=id%3Acdd16c5b4c3f3d823192cc429892d2ed%2Fnode%2F1733&start=0&rows=4&request_id=49d8021634646 HTTP/1.0", upstream: "fastcgi://unix:/tmp/fcgi-gate-46.socket:", host: "search.xxxxxx.com"

And we get lots of them, and while the requests typically make it through after trying 4 or 5 threads, it gets worse and worse until I delete the sockets and recreate them, then it runs absolutely fine, given the same traffic for hours.

Any ideas?  This one is really killing me.  Totally open to paying someone to help with this too.

Best,
Jacob

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,808,808#msg-808