keepalive connection to fastcgi backend hangs

[Nicolas Franck]

Interesting!

I looks like there is nothing that managing the incoming connections
for the fcgi workers. Every fcgi worker needs to do this on its own, right?
So if there are more clients (i.e. nginx workers) than fcgi workers,
then it becomes unresponsive after a few requests, because all
the fcgi workers are holding on to a connection to an nginx worker,
and there seems to be no queue handling this. 

Is this correct? Just guessing here

> On 20 Dec 2021, at 15:11, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Sun, Dec 19, 2021 at 07:56:51PM +0000, Nicolas Franck wrote:
> 
>> In order to make nginx keep the tcp connections alive,
>> I've added the following settings:
>> 
>> * proxy_socket_keepalive on
>> * proxy_http_version 1.1;
>> * proxy_set_header Connection "";
> 
> Just a side note: you don't need any of these to keep FastCGI 
> connections alive.
> 
>> * fastcgi_keep_conn on;
>> * added an upstream "fgi":
>> 
>> upstream fcgi {
>>    keepalive 10;
>>    server myhost:9000;
>> }
>> 
>> * added a location block (only snippet given):
>> 
>> location /fcgi {
>>   fastcgi_pass_request_headers on;
>>   fastcgi_pass fcgi;
>>   fastcgi_keep_conn on;
>> }
>> 
>> What I see: after a couple of requests nginx "hangs" when I 
>> visit path "/fcgi".
>> 
>> This disappears when
>> 
>> * I remove the setting "keepalive" from the upstream (but that 
>> disables keepalive altogether)
>> * bind the fcgi application to a unix socket, and let nginx bind 
>> to that. But that requires nginx and the fcgi to be on the same 
>> server.
>> * reduce the number of nginx workers to exactly 1. Not sure why 
>> that works.
>> * I spawn the application with tool "supervisord" (a fcgi 
>> process manager written in python)
>> 
>> Does anyone know what is happening here?
>> Fcgi has little documentation on the web..
> 
> [...]
> 
>> Spawn: spawn-fcgi -f /path/to/fcgi_example.cpp -p 9000
> 
> The spawn-fcgi defaults to 1 child process, and each child process 
> can handle just one connection.  On the other hand, your 
> configuration instruct nginx to cache up to 10 connections per 
> nginx process.
> 
> As long as the only one connection your upstream server can handle 
> is cached in another nginx process, nginx won't be able able to 
> reach the upstream server, and will return 504 Gateway Timeout 
> error once the fastcgi_connect_timeout expires (60s by default).  
> Likely this is something you see as "hangs".
> 
> Obvious fix would be to add additional fastcgi processes.  Given 
> "keepalive 10;" in nginx configuration, you'll need at least 10 * 
> <number of nginx worker processes>.  Something like:
> 
> spawn-fcgi -F 20 -f /path/to/fcgi -p 9000
> 
> should fix things for up to 2 nginx worker processes.
> 
> Just in case, that's exactly the problem upstream keepalive 
> documentation warns about (https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnginx.org%2Fr%2Fkeepalive&data=04%7C01%7CNicolas.Franck%40ugent.be%7Ce982b61a75624979ba3708d9c3c2b12b%7Cd7811cdeecef496c8f91a1786241b99c%7C1%7C0%7C637756063709619074%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=mHb200pbVMC0oS0HrbLY31hX3QyLQV0WQLoyt%2Fh96eM%3D&reserved=0): "The 
> connections parameter should be set to a number small enough to 
> let upstream servers process new incoming connections as well".
> 
> -- 
> Maxim Dounin
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmdounin.ru%2F&data=04%7C01%7CNicolas.Franck%40ugent.be%7Ce982b61a75624979ba3708d9c3c2b12b%7Cd7811cdeecef496c8f91a1786241b99c%7C1%7C0%7C637756063709619074%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vgJOdIzYbvba1kqXCKAiMY%2FPyNx3RgyQonp9cbLXZ6Q%3D&reserved=0
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.nginx.org%2Fmailman%2Flistinfo%2Fnginx&data=04%7C01%7CNicolas.Franck%40ugent.be%7Ce982b61a75624979ba3708d9c3c2b12b%7Cd7811cdeecef496c8f91a1786241b99c%7C1%7C0%7C637756063709619074%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qNyDaV%2B0p4cuTlcFkBZeKN3gCT0A03xIMrb5qOOeBFk%3D&reserved=0