Problem running nginx in a container

Mon Feb 7 03:08:49 UTC 2022

Francis,  see my response inline:

Hi there,

There's lots of information here, and I'm not sure what specific parts
relate to an nginx issue.

I don't have a full answer, but there are some suggestions below that might
help point at where the problem and fix might be.

> I am running nginx in a container created with a docker-compose yaml file.

> Here is a sample docker-compose.yml and environment file:
> 
>
https://github.com/marioja/Mailu/tree/test-mailu/test-mailu-docker-compose.

I don't see "this is content of the nginx.conf file" on that page. I also
don't see "here is where nginx is started".

The nginx.conf can be seen here:
https://1drv.ms/u/s!ApcymW6zCVpnuwSp3gbKFP9U8mak?e=DKziGh

> the configuration entries in the
> /etc/nginx/http.d/rainloop.conf configuration file created when the 
> rainloop image starts from the 
> webmails/rainloop/config/nginx-rainloop.conf in the github branch 
> previously mentioned is not recognized by nginx when the webmail 
> container (using the rainloop image) is started by docker-compose up 
> -d.  However, if I restart the webmail container using docker-compose 
> restart webmail then the instructions in that rainloop.conf are then
recognized by nginx.

If you can find the place that starts nginx, perhaps adding a call to nginx
with all of the same arguments plus "-T", and catching the stdout and stderr
of that somewhere, will help?

I checked and the output when starting from docker-compose up -d or
docker-compose restart is identical. I include it here:
https://1drv.ms/u/s!ApcymW6zCVpnuwKQOFFbSCLiZhn9?e=bvrpJU

That should show what config files, with what content, this instance of
nginx is reading.

If that output differs between the "up -d" and the "restart webmail"
calls, that might point at the next place to investigate.

>  I tried to use nginx-debug with error_log set to debug but the output 
> I get is not understandable and does not refer to any config file path 
> or parsing.

The nginx debug log can be a bit overwhelming -- there's lots in it, and it
is not always immediately clear why what is in it, is in it.

But people who know what to look for, and what is "normal" in it, may be
able to interpret it for you.

Generally, you want something of the form:

I make *this* request
I get *this* response
I want *that* response instead

ideally with enough information to allow someone else to repeat what you are
doing.

And if the log file is especially big, it may be better to host it somewhere
and provide a link to it.

You can edit the file to hide "private" information -- but if you do change
things, please change them consistently. Best would be to build a test
system with no private information, and provide the full logs for that
system that shows the same problem.

I ran strace on nginx and I have to trace files.  This is the trace file
when start with docker-compose up -d:
https://1drv.ms/u/s!ApcymW6zCVpnun28ya8wIBQi9qEt?e=AqzHfI
This is the trace file when I start with docker-compose restart:
https://1drv.ms/u/s!ApcymW6zCVpnuwFnO5IT-2bERyHI?e=FzZxuD

I cannot see any difference except that for the response on line 80 contains
the prefix "X-Powered-By: PHP/7/4/26\r\n" in the docker-compose up -d case
but not in the docker-compose restart.  Does this shed some light?

Also the message I get from nginx (this is either stdout or stderr)  when
the error occurs is:

webmail_1    | 2022/02/07 02:08:18 [warn] 25#25: *145 a client request body
is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000005,
client: 192.168.200.6, server: , request: "POST
/?/Upload/&q[]=/_eiVamsWTAQ12l6UF7vBn495f4U5LAGxhZ0nSBHjARdRjXwgf35Z7yvyZwvKejLSOPeJBrY8J_XpMKC_ukp5jM8z9FT0uhT3hH51K4THDw7GWMitAjdQNx1SGw_qF04hNvcQ4_eWaUXxSPiR0vizLEVBScYBZIEXYDHWVTVKbVTYbETLW3kpq214FuLRHhwWLyxjeACQwoxxucaj_uLcm7oyzQjwwUOV9PQKA7l9BPkq0Oq63sFEBS_vVhIplQ6EmFq59lxuSrZiDlhSBOdLIW8h2xD1kz9Twb_-qzwelpHuF-84Lbdg8tUpE7BYbUyU76La5BcFSREyf7qvm8e_mszUfh_Zac-63pk8jrdwmgQEApa0OPJL4NAXTimMG-JmbS9CCPvoGzGmoj12afq0CcwrkmxJI2Z00N2HbfrIns69INt-G-x34tdDwo2k/
HTTP/1.1", host: "mailu.company.local:444", referrer:
"https://mailu.company.local:444/webmail/"

> The reason I am asking this in the nginx mailing list is because I 
> have exhausted all of the tests that can gather information from the 
> container side and I am wondering if there is anything that you guys 
> can thing that could explain this.  Here is the directory structure of 
> the files in the container where nginx is running before starting as a 
> result of the docker-compose up -d:

When nginx starts, it reads one configuration file.

"nginx -T" should show the contents of that, plus anything "include"'d from
it, that nginx sees.

It looks like your starting file might be from

https://git.alpinelinux.org/aports/tree/main/nginx/nginx.conf

with the error_log line (line 12) set to debug; and then your include'd file
re-sets it to "warn" and changes client_max_body_size, within that server{}
block.

>From what you describe, it sounds like the include'd file may not be present
when nginx first starts? Are there any requests that you make that do
succeed, before the one that fails?

The included file is present as show in the nginx -T output.

Where are you reading the error log? From /var/log/nginx/error.log, or from
stderr from the docker command? It might make a difference, if two error_log
directives are present.

I was reading from /var/log/nginx/error.log which only contains with warn:

2022/02/07 03:01:28 [notice] 23#23: using the "epoll" event method
2022/02/07 03:01:28 [notice] 23#23: nginx/1.20.2
2022/02/07 03:01:28 [notice] 23#23: OS: Linux 5.4.0-96-generic
2022/02/07 03:01:28 [notice] 23#23: getrlimit(RLIMIT_NOFILE): 1024:524288
2022/02/07 03:01:28 [notice] 23#23: start worker processes
2022/02/07 03:01:28 [notice] 23#23: start worker process 24
2022/02/07 03:01:28 [notice] 23#23: start worker process 25
2022/02/07 03:01:28 [notice] 23#23: start worker process 26
2022/02/07 03:01:28 [notice] 23#23: start worker process 27

> This is the content of the rainloop.conf in the webmail container at
> runtime:

Is that "according to something that runs before nginx starts", or something
else?

If files are being changed, the timing might matter.

The rainloop.conf file is created when the container image is created.  This
means that it is identical when docker-compose up -d or docker-compose
restart is used.  I have verified that.

>     # /dev/stdout (Default), <path>, off
>     access_log off;
> 
>     # /dev/stderr (Default), <path>, debug, info, notice, warn, error, 
> crit, alert, emerg
>     error_log /dev/stderr warn;

When problems are happening, turning on logging can be useful to see what
the system thinks is happening.

>     # set maximum body size to configured limit
>     client_max_body_size 58388608;

That's about 55 MB.

> Is there a way to tell nginx or nginx-debug to tell us what is going on? 

Don't turn off the logging.

> The error that occurs after docker-compse up -d is that a 5MB 
> attachment posted to the webmail container fails.

I'm guessing that's a HTTP 413 coming direct from nginx; not any message
from the fastcgi service; and not a HTTP 502 coming direct from nginx?

The specific details might matter.

(413 suggests that the "new" client_max_body_size was not used. 502 suggests
that php-fpm was not available. Another error might suggest something else.
nginx's error_log will usually indicate why it raised an error response, at
a suitable log level.)

>  That same 5MB attachment works after
> a docker-compose restart.  The statement that does not appear to take 
> effect until the docker-compose restart is the client_max_body_size
58388608.
> Please note that there is a client_max_body_size 1m directive in the 
> /etc/nginx/nginx.conf during both restart (docker-compose up and 
> docker-compose restart).
> 
> I have searched low and high for any information about this http.d 
> directory but I could not see anything.  I am using nginx version
> 1.20.2 installed from alpinelinux apk command.

Your nginx.conf line 103 might have "include /etc/nginx/http.d/*.conf;"

You are right, I don't know how I missed that.  DOH

which reads the matching file contents. The directory is only special to
this nginx because it is named in the conf file.

Good luck with it,

	f
-- 
Francis Daly        francis at daoine.org
_______________________________________________
nginx mailing list -- nginx at nginx.org
To unsubscribe send an email to nginx-leave at nginx.org

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,293592,293599#msg-293599