From gfrankliu at gmail.com Tue Oct 1 05:53:02 2019 From: gfrankliu at gmail.com (Frank Liu) Date: Mon, 30 Sep 2019 22:53:02 -0700 Subject: max pem file size Message-ID: When using ssl_client_certificate or ssl_trusted_certificate to verify client cert, is there a limit on the pem file size or number of CAs? How to tune that if we need to increase that size? What's the impact if that file is getting too large? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Oct 1 12:08:21 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 1 Oct 2019 15:08:21 +0300 Subject: max pem file size In-Reply-To: References: Message-ID: <20191001120820.GP1877@mdounin.ru> Hello! On Mon, Sep 30, 2019 at 10:53:02PM -0700, Frank Liu wrote: > When using ssl_client_certificate or ssl_trusted_certificate to verify > client cert, is there a limit on the pem file size or number of CAs? How to > tune that if we need to increase that size? What's the impact if that file > is getting too large? There are no limits, except due to amount of available memory. -- Maxim Dounin http://mdounin.ru/ From nginx-forum at forum.nginx.org Tue Oct 1 14:17:36 2019 From: nginx-forum at forum.nginx.org (lvic4594) Date: Tue, 01 Oct 2019 10:17:36 -0400 Subject: proxy http requests to different port numbers? In-Reply-To: <20190929082435.GC1914@daoine.org> References: <20190929082435.GC1914@daoine.org> Message-ID: Works with http but not with https: I tried to add second proxy_pass for https server { listen 443 ssl; server_name _; ssl_certificate my.crt; ssl_certificate_key my.key; root /usr/share/nginx/html; include /etc/nginx/default.d/*.conf; location /rest2/ { proxy_pass https://myserver.com:8444/; For https I have "resource not found /usr/share/nginx/html/rest2/" e.g. nginx is not trying to proxy https request.... Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285723,285768#msg-285768 From jlmuir at imca-cat.org Tue Oct 1 17:42:27 2019 From: jlmuir at imca-cat.org (J. Lewis Muir) Date: Tue, 1 Oct 2019 12:42:27 -0500 Subject: RHEL Yum repo instructions don't work on RHEL 7 Message-ID: <20191001174227.ztiy3almckars2ex@mink.imca.aps.anl.gov> Hello! I'd like to report a problem with the RHEL/CentOS instructions at https://nginx.org/en/linux_packages.html#RHEL-CentOS They don't work on RHEL 7. When I attempt to install the nginx package as root with yum install nginx I get an HTTP 404 error: http://nginx.org/packages/centos/7Workstation/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found The problem seems to be the use of the $releasever Yum variable in the baseurl key of the nginx.repo spec: baseurl=http://nginx.org/packages/centos/$releasever/$basearch/ On my RHEL 7 machine, that variable is evaluating to 7Workstation Looking at http://nginx.org/packages/centos/ the nginx repo is expecting $releasever to evaluate to just the release number without the variant/edition text. So, if in nginx.repo I change baseurl=http://nginx.org/packages/centos/$releasever/$basearch/ to baseurl=http://nginx.org/packages/centos/7/$basearch/ it works. There's more info at https://access.redhat.com/solutions/1256473 I have a default /etc/yum.conf and it does not have a line that sets the distroverpkg key. I think something in the documentation or the nginx repo needs to be changed. One fix would be to list the release version number explicitly in the baseurl key value like I did above. And just say in the documentation that you have to set the number to the version that matches your installed version of RHEL/CentOS. Another fix would be to add symlinks or redirects or whatever at http://nginx.org/packages/centos/ that would make the $releasever work. For example: 7Workstation -> 7 7Server -> 7 I don't know whether there are other release variants/editions. At some point, "Client" was a variant/edition, I think, but it seems like these are changing a fair amount from major release to major release, so it may be a moving target. And maybe there are other better fixes that I'm not aware of. Thanks! Lewis From thresh at nginx.com Wed Oct 2 10:21:37 2019 From: thresh at nginx.com (Konstantin Pavlov) Date: Wed, 2 Oct 2019 13:21:37 +0300 Subject: RHEL Yum repo instructions don't work on RHEL 7 In-Reply-To: <20191001174227.ztiy3almckars2ex@mink.imca.aps.anl.gov> References: <20191001174227.ztiy3almckars2ex@mink.imca.aps.anl.gov> Message-ID: <3a4ee95f-1f68-9e5e-3306-b82819263676@nginx.com> Hi Lewis, Thanks for bringing the issue to our attention. 01.10.2019 20:42, J. Lewis Muir wrote: > Another fix would be to add symlinks or redirects or whatever at > > http://nginx.org/packages/centos/ > > that would make the $releasever work. For example: > > 7Workstation -> 7 > 7Server -> 7 > > I don't know whether there are other release variants/editions. At some > point, "Client" was a variant/edition, I think, but it seems like these > are changing a fair amount from major release to major release, so it > may be a moving target. Yep, that seems like the easiest solution. I've put Client, Workstation and Server symlinks for respectable major releases to both stable and mainline repos - I'd appreciate if you try again and let me know if that fixes the issue for you. Thank you, -- Konstantin Pavlov https://www.nginx.com/ From francis at daoine.org Wed Oct 2 12:07:32 2019 From: francis at daoine.org (Francis Daly) Date: Wed, 2 Oct 2019 13:07:32 +0100 Subject: proxy http requests to different port numbers? In-Reply-To: References: <20190929082435.GC1914@daoine.org> Message-ID: <20191002120732.GE1914@daoine.org> On Tue, Oct 01, 2019 at 10:17:36AM -0400, lvic4594 wrote: Hi there, > Works with http but not with https: I tried to add second proxy_pass for > https > > server { > listen 443 ssl; > server_name _; > ssl_certificate my.crt; > ssl_certificate_key my.key; > For https I have "resource not found /usr/share/nginx/html/rest2/" e.g. > nginx is not trying to proxy https request.... Have you any evidence (logs, "curl -v" output) that your request is being handled in this nginx server{} block? f -- Francis Daly francis at daoine.org From jlmuir at imca-cat.org Wed Oct 2 15:29:46 2019 From: jlmuir at imca-cat.org (J. Lewis Muir) Date: Wed, 2 Oct 2019 10:29:46 -0500 Subject: RHEL Yum repo instructions don't work on RHEL 7 In-Reply-To: <3a4ee95f-1f68-9e5e-3306-b82819263676@nginx.com> References: <20191001174227.ztiy3almckars2ex@mink.imca.aps.anl.gov> <3a4ee95f-1f68-9e5e-3306-b82819263676@nginx.com> Message-ID: <20191002152946.cah3uqyenvh64r7y@mink.imca.aps.anl.gov> On 10/02, Konstantin Pavlov wrote: > Yep, that seems like the easiest solution. I've put Client, Workstation > and Server symlinks for respectable major releases to both stable and > mainline repos - I'd appreciate if you try again and let me know if that > fixes the issue for you. I confirm that using the $releasever Yum variable in the baseurl key value as given in the nginx documentation works now. Thanks! Lewis From nginx-forum at forum.nginx.org Thu Oct 3 13:14:35 2019 From: nginx-forum at forum.nginx.org (lvic4594) Date: Thu, 03 Oct 2019 09:14:35 -0400 Subject: proxy http requests to different port numbers? In-Reply-To: <20191002120732.GE1914@daoine.org> References: <20191002120732.GE1914@daoine.org> Message-ID: The same thing happens if i have both http and https in the same server block, which is the only one in config: server { listen 80; listen 443 ssl; server_name _; ssl_certificate myserver.crt; ssl_certificate_key myserver.key; root /usr/share/nginx/html; client_max_body_size 200M; # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; location /http { proxy_pass http://myserver:9092; } location /tls { proxy_pass https://myserver:8444; } Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285723,285776#msg-285776 From francis at daoine.org Thu Oct 3 15:22:04 2019 From: francis at daoine.org (Francis Daly) Date: Thu, 3 Oct 2019 16:22:04 +0100 Subject: proxy http requests to different port numbers? In-Reply-To: References: <20191002120732.GE1914@daoine.org> Message-ID: <20191003152204.GF1914@daoine.org> On Thu, Oct 03, 2019 at 09:14:35AM -0400, lvic4594 wrote: Hi there, > The same thing happens if i have both http and https in the same server > block, which is the only one in config: What "same thing", specifically? What request do you make; what response do you get; what response do you want instead? > include /etc/nginx/default.d/*.conf; That line might be doing anything. What happens if you leave it out? > location /http { > proxy_pass http://myserver:9092; Those lines should probably each have one more "/" in them, to match the original requirement. Cheers, f -- Francis Daly francis at daoine.org From nginx-forum at forum.nginx.org Thu Oct 3 15:36:27 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Thu, 03 Oct 2019 11:36:27 -0400 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL Message-ID: <362748953e3ea9e184f9774ecb0a7bb7.NginxMailingListEnglish@forum.nginx.org> I have the following server configuration. I have two problems that I would appreciate if someone can tell me what I am doing wrong: 1- this location is not respected since the configuration lowercases all uris without respecting these locations. Please note that api, contentAssets, categoryServlet are paths, not directories or folders. location ~ ^/(api|contentAsset|categoriesServlet|DotAjaxDirector|html|dwr|dA)/{} 2. It seems that if I am in a reverse proxy I have to input the proxy definition in every location because otherwise it does not work. Is the a more practical way to do this? Thank you for all your help! Lex Note: I have changed the proxy IP address and port for x and Is. server { listen 80; server_name example.com; ##Trailing Slash in URLContent ## It doest not work on http://example.com/real-estate/cccc/ #rewrite ^/(.*)/(.*)/$ /$1/$2 permanent; # matches any query beginning with /contentAsset or dA/ and halts searching, # so regular expressions will not be checked. location ~ ^/(api|contentAsset|categoriesServlet|dotAdmin|DotAjaxDirector|html|dwr|dA)/ { proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; } ## IT MUST BE FIRST TO LOWERCASE THE URI FIRST ## IF IT HAS UPPERCASE CHARACTERS location ~ [A-Z] { return 301 $scheme://$host$my_uri_to_lowercase; proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; } ## IT MUST BE FIRST TO LOWERCASE THE URI FIRST ## IF IT HAS UPPERCASE CHARACTERS location ~ [A-Z] { return 301 $scheme://$host$my_uri_to_lowercase; } location / { proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; } } Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285780,285780#msg-285780 From francis at daoine.org Thu Oct 3 15:55:46 2019 From: francis at daoine.org (Francis Daly) Date: Thu, 3 Oct 2019 16:55:46 +0100 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: <362748953e3ea9e184f9774ecb0a7bb7.NginxMailingListEnglish@forum.nginx.org> References: <362748953e3ea9e184f9774ecb0a7bb7.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191003155546.GG1914@daoine.org> On Thu, Oct 03, 2019 at 11:36:27AM -0400, Alex Med wrote: Hi there, > 1- this location is not respected since the configuration lowercases all > uris without respecting these locations. Please note that api, > contentAssets, categoryServlet are paths, not directories or folders. Which part of the configuration does the lowercasing? I don't see where $my_uri_to_lowercase is set. > 2. It seems that if I am in a reverse proxy I have to input the proxy > definition in every location because otherwise it does not work. Is the a > more practical way to do this? You need proxy_pass in each location{} where you want to reverse proxy things. But all of the other directives will inherit from the outer level (if not set in the location). > Thank you for all your help! If the problem remains, can you show a request that you make, the response that you get, and describe the response that you want instead? It may help make clearer what is happening. Thanks, f -- Francis Daly francis at daoine.org From nginx-forum at forum.nginx.org Thu Oct 3 16:21:29 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Thu, 03 Oct 2019 12:21:29 -0400 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: <20191003155546.GG1914@daoine.org> References: <20191003155546.GG1914@daoine.org> Message-ID: <19b9c16ca106723d6397691131718d49.NginxMailingListEnglish@forum.nginx.org> Dear Francis: Here are the answers to your questions. Thank YOU for helping! The location block are in the following order: Location #1 location ~ ^/(api|contentAsset|categoriesServlet|DotAjaxDirector|html|dwr|dA)/ {} Location #2 ----> This location does the lowercasing location ~ [A-Z] { return 301 $scheme://$host$my_uri_to_lowercase;} Location #3 location /{} Everything is lowercased. So I want some paths to be untouched. So I created location #1 to exclude some paths from lowercasing. The problem is that in some URLs containing the paths above are lowercased. For example: /dwr/interface/CategoryAjax.js, /dwr/interface/HostAjax.js are lowercased. Also, /html/images/languages/gh_TW.gif Any help will be appreciated! Note: The function definition for $my_uri_to_lowercase is inside my http definition, here is a copy of it: ## ## URL REWRITE FUNCTION: LOWERCASE ALL URIs ## START perl_set $my_uri_to_lowercase 'sub { my $r = shift; my $uri = $r->uri; $uri = lc($uri); return $uri; }'; Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285780,285782#msg-285782 From vbart at nginx.com Thu Oct 3 16:43:48 2019 From: vbart at nginx.com (Valentin V. Bartenev) Date: Thu, 03 Oct 2019 19:43:48 +0300 Subject: Unit 1.12.0 release Message-ID: <16190232.JLIOjcLzyV@vbart-workstation> Hi, I'm glad to announce a new release of NGINX Unit. This is an ad-hoc release that focuses on fixing several annoying bugs and adds compatibility with the upcoming PHP 7.4, scheduled for release on November 28, 2019. Changes with Unit 1.12.0 03 Oct 2019 *) Feature: compatibility with PHP 7.4. *) Bugfix: descriptors leak on process creation; the bug had appeared in 1.11.0. *) Bugfix: TLS connection might be closed prematurely while sending response. *) Bugfix: segmentation fault might have occurred if an irregular file was requested. Regarding our plans for the near future, see our earlier announcement: - https://mailman.nginx.org/pipermail/unit/2019-September/000167.html To know more about some features introduced recently, you can follow posts about Unit in the official blog: - https://www.nginx.com/blog/tag/nginx-unit/ We also updated our Docker images with an initialization script that significantly simplifies the initial configuration of Unit daemon inside a container. Please check the documentation for instructions: - https://unit.nginx.org/installation/#initial-configuration wbr, Valentin V. Bartenev From nginx-forum at forum.nginx.org Thu Oct 3 17:03:01 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Thu, 03 Oct 2019 13:03:01 -0400 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: <19b9c16ca106723d6397691131718d49.NginxMailingListEnglish@forum.nginx.org> References: <20191003155546.GG1914@daoine.org> <19b9c16ca106723d6397691131718d49.NginxMailingListEnglish@forum.nginx.org> Message-ID: I also would like to add that when I access: example.com/API/ it is lowercased to example.com/api/. According to my configuration it should not. So it means location #1 is not working properly. Alex Med Wrote: ------------------------------------------------------- > Dear Francis: > > Here are the answers to your questions. Thank YOU for helping! > > The location block are in the following order: > > Location #1 > location ~ > ^/(api|contentAsset|categoriesServlet|DotAjaxDirector|html|dwr|dA)/ > {} > > Location #2 ----> This location does the lowercasing > location ~ [A-Z] { return 301 $scheme://$host$my_uri_to_lowercase;} > > Location #3 > location /{} > > Everything is lowercased. So I want some paths to be untouched. So I > created location #1 to exclude some paths from lowercasing. The > problem is that > in some URLs containing the paths above are lowercased. For example: > /dwr/interface/CategoryAjax.js, /dwr/interface/HostAjax.js are > lowercased. > Also, /html/images/languages/gh_TW.gif > > Any help will be appreciated! > > > > Note: The function definition for $my_uri_to_lowercase is inside my > http definition, here is a copy of it: > ## > ## URL REWRITE FUNCTION: LOWERCASE ALL URIs > ## START > > perl_set $my_uri_to_lowercase 'sub { > my $r = shift; > my $uri = $r->uri; > $uri = lc($uri); > return $uri; > }'; Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285780,285785#msg-285785 From francis at daoine.org Thu Oct 3 17:37:06 2019 From: francis at daoine.org (Francis Daly) Date: Thu, 3 Oct 2019 18:37:06 +0100 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: <19b9c16ca106723d6397691131718d49.NginxMailingListEnglish@forum.nginx.org> References: <20191003155546.GG1914@daoine.org> <19b9c16ca106723d6397691131718d49.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191003173706.GH1914@daoine.org> On Thu, Oct 03, 2019 at 12:21:29PM -0400, Alex Med wrote: Hi there, > Location #1 > location ~ > ^/(api|contentAsset|categoriesServlet|DotAjaxDirector|html|dwr|dA)/ {} > > Location #2 ----> This location does the lowercasing > location ~ [A-Z] { return 301 $scheme://$host$my_uri_to_lowercase;} > > Location #3 > location /{} > > Everything is lowercased. So I want some paths to be untouched. So I > created location #1 to exclude some paths from lowercasing. The problem is > that > in some URLs containing the paths above are lowercased. For example: > /dwr/interface/CategoryAjax.js, /dwr/interface/HostAjax.js are lowercased. > Also, /html/images/languages/gh_TW.gif If I make my server{} have three location{}s: == location ~ ^/(api|contentAsset|categoriesServlet|DotAjaxDirector|html|dwr|dA)/ { return 200 "In the first location, uri $uri\n"; } location ~ [A-Z] { return 200 "In the A-Z location, uri $uri\n"; } location / { return 200 "In the / location, uri $uri\n"; } == then the requests for /dwr/interface/CategoryAjax.js, /dwr/interface/HostAjax.js, and /html/images/languages/gh_TW.gif are served from the first location, not the second. So - I'm not seeing what you are reporting. When you do a "curl -i" for /dwr/interface/CategoryAjax.js, what response do you get? HTTP 200, HTTP 301, something else? Thanks, f -- Francis Daly francis at daoine.org From francis at daoine.org Thu Oct 3 19:05:04 2019 From: francis at daoine.org (Francis Daly) Date: Thu, 3 Oct 2019 20:05:04 +0100 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: References: <20191003155546.GG1914@daoine.org> <19b9c16ca106723d6397691131718d49.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191003190504.GI1914@daoine.org> On Thu, Oct 03, 2019 at 01:03:01PM -0400, Alex Med wrote: Hi there, > I also would like to add that when I access: example.com/API/ it is > lowercased to example.com/api/. According to my configuration it should > not. So it means location #1 is not working properly. Why do you think that? My reading of your configuration is that a request for example.com/API/ should get a response of a http 301 to example.com/api/. Is there some part of your configuration that you are not showing here? If you test with exactly the configuration that you show here, do you see the same behaviour? f -- Francis Daly francis at daoine.org From wizard at bnnorth.net Fri Oct 4 03:54:43 2019 From: wizard at bnnorth.net (Ken Wright) Date: Thu, 3 Oct 2019 23:54:43 -0400 Subject: PR_END_OF_FILE_ERROR after kernel update Message-ID: I hope someone more knowledgeable than I am can help here. A day or two ago I ran some routine updates on my Ubuntu Server 18.04.? No big deal, right?? The only unusual thing was the kernel updated as well, but I've updated the kernel on this machine many times before with no problems.? This time, however, was different. The updates ran with no difficulty, I rebooted the server to start using the new kernel, and that's when the troubles began.? Ever since, when I try to go to the server from a browser, I get an error message that says "PR_END_OF_FILE_ERROR"? I Googled this error and the suggestion was made that it was a flaw with my SSL certificates.? I reinstalled those, but to no avail.? I'm still getting the same error. I'm sorry to have run on this long about the problem, but I wanted to give as much info as I have.? Any further information is available on request. Help! Ken -- Registered Linux user #483005 If you ever think international relations make sense, remember this: because a Serb shot an Austrian in Bosnia, Germany invaded Belgium. From lists at lazygranch.com Fri Oct 4 04:25:34 2019 From: lists at lazygranch.com (lists) Date: Thu, 03 Oct 2019 21:25:34 -0700 Subject: PR_END_OF_FILE_ERROR after kernel update In-Reply-To: Message-ID: You could test the cert using SSL labs. https://www.ssllabs.com/ You might have a drop your firewall if it doesn't work at first. It never hurts to do dumb stuff like boot the server again. ? Original Message ? From: wizard at bnnorth.net Sent: October 3, 2019 8:55 PM To: nginx at nginx.org Reply-to: nginx at nginx.org Subject: PR_END_OF_FILE_ERROR after kernel update I hope someone more knowledgeable than I am can help here. A day or two ago I ran some routine updates on my Ubuntu Server 18.04.? No big deal, right?? The only unusual thing was the kernel updated as well, but I've updated the kernel on this machine many times before with no problems.? This time, however, was different. The updates ran with no difficulty, I rebooted the server to start using the new kernel, and that's when the troubles began.? Ever since, when I try to go to the server from a browser, I get an error message that says "PR_END_OF_FILE_ERROR"? I Googled this error and the suggestion was made that it was a flaw with my SSL certificates.? I reinstalled those, but to no avail.? I'm still getting the same error. I'm sorry to have run on this long about the problem, but I wanted to give as much info as I have.? Any further information is available on request. Help! Ken -- Registered Linux user #483005 If you ever think international relations make sense, remember this:? because a Serb shot an Austrian in Bosnia, Germany invaded Belgium. _______________________________________________ nginx mailing list nginx at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx From wizard at bnnorth.net Fri Oct 4 12:51:49 2019 From: wizard at bnnorth.net (Ken Wright) Date: Fri, 4 Oct 2019 08:51:49 -0400 Subject: PR_END_OF_FILE_ERROR after kernel update In-Reply-To: References: Message-ID: <191bb86d-a9d9-03f6-1326-c1b7bd863d1d@bnnorth.net> On 10/4/19 12:25 AM, lists wrote: > You could test the cert using SSL labs. > https://www.ssllabs.com/ > You might have a drop your firewall if it doesn't work at first. > > It never hurts to do dumb stuff like boot the server again. Okay, I rebooted the server then tried testing the cert.? SSL Labs reports no secure protocols supported.? Once again, I'm lost.? I know I set up the server to use LetsEncrypt certs, and I've checked them; they're there.? I tried disabling the server's firewall and testing again, only to get the same message. Ken -- Registered Linux user #483005 If you ever think international relations make sense, remember this: because a Serb shot an Austrian in Bosnia, Germany invaded Belgium. From jlmuir at imca-cat.org Fri Oct 4 14:19:26 2019 From: jlmuir at imca-cat.org (J. Lewis Muir) Date: Fri, 4 Oct 2019 09:19:26 -0500 Subject: PR_END_OF_FILE_ERROR after kernel update In-Reply-To: <191bb86d-a9d9-03f6-1326-c1b7bd863d1d@bnnorth.net> References: <191bb86d-a9d9-03f6-1326-c1b7bd863d1d@bnnorth.net> Message-ID: <20191004141926.zmham3kpgazxxeqk@gromit.local> On 10/04, Ken Wright wrote: > Okay, I rebooted the server then tried testing the cert.? SSL Labs > reports no secure protocols supported.? Once again, I'm lost.? I know I > set up the server to use LetsEncrypt certs, and I've checked them; > they're there.? I tried disabling the server's firewall and testing > again, only to get the same message. Can you check your nginx server logs? Can you check that the same content can be served over plain HTTP? Can you check that all of your nginx and Let's Encrypt configuration files are still intact? Can you check that your file systems are not full? Can you share a curl command that shows your output that others could use to compare to your output? Lewis From j.keith at xsb.com Fri Oct 4 17:25:22 2019 From: j.keith at xsb.com (Jordan Keith) Date: Fri, 4 Oct 2019 12:25:22 -0500 (CDT) Subject: Doing a mass purge of the cache using a wildcard Message-ID: <728597595.1787657.1570209922987.JavaMail.zimbra@xsb.com> We are using nginx plus. We have caching configured for a few endpoints, one which is similar to the following: document/{id}/report I would like to purge all items cached under that endpoint, but executing a purge request like the following does not purge anything: document/*/report We have other endpoints where mass purges work fine, but they have the * at the end of the URL instead of the middle. So I have 2 questions: 1). Does nginx only support wildcards at the end of the URL for cache-purging purposes? 2). Is there some other way I can accomplish a mass purge here? -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Fri Oct 4 19:36:13 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Fri, 04 Oct 2019 15:36:13 -0400 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: <20191003190504.GI1914@daoine.org> References: <20191003190504.GI1914@daoine.org> Message-ID: Dear Francis: Thank you again for your help! I apologize. Perhaps, I should have started with what I want to accomplish with the configuration. I want the server to lowercased every URI except the ones that begins with any of the following paths: /api/ /contentAsset/ /categoriesServlet/ /DotAjaxDirector/ /html/ /dwr/ /dA/ /JsonTag/ Most of the above paths belong to a protected area that requires the user to login with an id and a password. So for example, if the server receives a request for https://example.com/API/ or https://example.com/contentAsset/ or https://example.com/DotAjaxDirector/ it should leave them intact. This is not happening in certain instances For anything else, it should lowercase any other uri. Now, to answer your questions: Your Question # 1 ----- then the requests for /dwr/interface/CategoryAjax.js, /dwr/interface/HostAjax.js, and /html/images/languages/gh_TW.gif are served from the first location, not the second. So - I'm not seeing what you are reporting. When you do a "curl -i" for /dwr/interface/CategoryAjax.js, what response do you get? HTTP 200, HTTP 301, something else? --- Answer to Your Question: I get a 401 error not authorized if I use curl. But if I go and log in the restricted area via a web browser and viewed the page with the inspertor's developer tool I no longer see some of the error on the console. This is probably because I was getting the errors from cache? I cleared the cache this morning so I stopped seeing some errors. I still see a: Failed to load resource:the server responded with a status of 404() to the following urls: example.com/jsontags example.comcategoriesservlet The errors are being generated because nginx is lowercasing JsonTags and CategoriesServlet. If I do a curl: curl -i http://example.com/JsonTags HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2019 19:21:32 GMT Content-Type: text/html Content-Length: 178 Connection: keep-alive Location: http://example.com/jsontags 301 Moved Permanently

301 Moved Permanently

nginx
curl -i http://example.com/CategoriesServlet HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2019 19:23:56 GMT Content-Type: text/html Content-Length: 178 Connection: keep-alive Location: http://example.com/categoriesservlet 301 Moved Permanently

301 Moved Permanently

nginx
For URIs outside of the restricted areas: If I write http://example.com/API/ or http://example.com/API they are lowercased, they should not be lowercase because it should be served by location #1 which makes the path case insensitive with the "~". curl -i http://example.com/API/ HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2019 19:24:56 GMT Content-Type: text/html Content-Length: 178 Connection: keep-alive Location: http://example.com/api/ 301 Moved Permanently

301 Moved Permanently

nginx
However, the url http://example.com/contentAsset is not lowercased and this uri is outside of the restricted area. curl -I http://example.com/contentAsset/image/ctfgs9b9b-fdcb-4a4a-807d-123x896fb0/fileAsset/filter/Resize,Jpeg/resize_w/1000/Jpeg_q/.8 HTTP/1.1 200 Server: nginx Date: Fri, 04 Oct 2019 19:27:10 GMT Content-Type: image/jpeg Content-Length: 83580 Connection: keep-alive Content-Disposition: inline; filename="scarlet-macaw-ara-macao.jpg" Expires: Sat, 03 Oct 2020 19:27:10 +0000 Cache-Control: public, max-age=31536000 Last-Modified: Sat, 28 Sep 2019 17:49:34 +0000 ETag: dot:ctfgs9b9b-fdcb-4a4a-807d-123x896fb0:1569692974000:83580 Accept-Ranges: bytes I also get this error: WebSocket connection to 'ws://example.com/api/ws/v1/system/events' failed: Unexpected response code: 200. This I have been told that I have to set the proxy pass to localhost rather than IP in order to work it properly. I have not tested it yet. Below is the current configuration for your consideration. Any ideas what I could be doing wrong? # SERVER DEFINITIONS ##BETA EXAMPLE HTTP SERVER CONFIGURATION START server { listen 80; server_name example.com; ## URL REWRITE FUNCTION THAT LOWERCASES ALL URIS ## START ## THIS IS OUTSIDE OF THE SERVER DEFINITION IN THE HTTP BLOCK - I PUT IT HERE SO YOU CAN SEE IT perl_set $my_uri_to_lowercase 'sub { my $r = shift; my $uri = $r->uri; $uri = lc($uri); return $uri; }'; ## URL REWRITE FUNCTION THAT LOWERCASES ALL URIS ## END ##Trailing Slash in URLContent ## It doest not work on http://beta.example.com/real-estate/properties #rewrite ^/(.*)/(.*)/$ /$1/$2 permanent; ## LOCATION #1 ## THIS LOCATION WILL PREVENT LOWERCASING IN ANY URI THAT BEGINS WITH /API/WHATEVER , /CONTENTASSET/WHATEVER, ETC. ## ~ MAKES IT CASEINSENSITIVE SO EITHER /api/ or /API/ will not be lowercased ## ^ MAKES IT MATCH WITH THE BEGINING OF THE "/" AND THE PATH WE DO NOT WANT TO LOWERCASE ## I ADDED THE proxy_pass as observation of David Francis on Nginx Forum ## IF URI MATCHES THIS LOCATION THEN IT WILL STOP SEARCHING FOR OTHER LOCATION MATCHES location ~ ^/(api|contentAsset|categoriesServlet|DotAjaxDirector|html|dwr|dA|JsonTags)/ { proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; } ## LOCATION #2 ## THIS LOCATION WILL LOWERCASE ANYTHING THAT HAS UPPCASE LETTERS ## EXCEPT THE PATHS STATED ON LOCATION #1 location ~ [A-Z] { return 301 $scheme://$host$my_uri_to_lowercase; proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; } ## LOCATION #3 ## THIS LOCATION IS THE DEFAULT LOCATION location / { proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; } } ##BETA EXAMPLE HTTP SERVER CONFIGURATION END Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285780,285797#msg-285797 From francis at daoine.org Fri Oct 4 20:49:49 2019 From: francis at daoine.org (Francis Daly) Date: Fri, 4 Oct 2019 21:49:49 +0100 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: References: <20191003190504.GI1914@daoine.org> Message-ID: <20191004204949.GJ1914@daoine.org> On Fri, Oct 04, 2019 at 03:36:13PM -0400, Alex Med wrote: Hi there, Thanks for the extra information. There are a few different things in here, but there is one that stands out as probably being a "quick fix" that should allow the rest to be tested separately... > Your Question # 1 > ----- > then the requests for /dwr/interface/CategoryAjax.js, > /dwr/interface/HostAjax.js, and /html/images/languages/gh_TW.gif are > served from the first location, not the second. > > So - I'm not seeing what you are reporting. > > When you do a "curl -i" for /dwr/interface/CategoryAjax.js, what response > do you get? > > HTTP 200, HTTP 301, something else? > --- > > Answer to Your Question: > > I get a 401 error not authorized if I use curl. Ok; that suggests that it *is* being handled in your location#1, which is what is wanted. That's good. > The errors are being generated because nginx is lowercasing JsonTags and > CategoriesServlet. > > If I do a curl: > > curl -i http://example.com/JsonTags > HTTP/1.1 301 Moved Permanently > Server: nginx > Date: Fri, 04 Oct 2019 19:21:32 GMT > Content-Type: text/html > Content-Length: 178 > Connection: keep-alive > Location: http://example.com/jsontags Yes - that is because that's what your config asks it to do. Because... > If I write http://example.com/API/ or http://example.com/API they are > lowercased, they should not be lowercase because it > should be served by location #1 which makes the path case insensitive with > the "~". No. "~" means "this is a regex location". "~*" would be "this is a case-insensitive regex location". http://nginx.org/r/location If you change your config to use "~*", does that make things do what you want? (Note: your location#1 will not match "/API", because that regex location needs two slashes. So /API will probably be lowercased to /api, and /api will probably be handled in location#3.) > I also get this error: WebSocket connection to > 'ws://example.com/api/ws/v1/system/events' failed: Unexpected response code: > 200. This I have been told > that I have to set the proxy pass to localhost rather than IP in order to > work it properly. I have not tested it yet. Websockets have not been mentioned in this mail yet, as far as I know. I suggest: get the rest working first; then sort out websockets. Good luck with it, f -- Francis Daly francis at daoine.org From nginx-forum at forum.nginx.org Sat Oct 5 16:44:30 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Sat, 05 Oct 2019 12:44:30 -0400 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: <20191004204949.GJ1914@daoine.org> References: <20191004204949.GJ1914@daoine.org> Message-ID: Dear David: I am very grateful for your help! Below is the final configuration working optimally. I even included the proxy configuration parameters for the websocket connection to work.I noticed that the location =/ does not support OR so I had create two additional locations to catch paths without the end forward slash. Is there a limit in the number of location that a configuration file should have? I do have some other additional configuration I want to do, but I will post another message with it. Thank YOU again for being in this forum and helping everyone! Regards, Alex # SERVER DEFINITIONS ##BETA EXAMPLE HTTP SERVER CONFIGURATION START server { listen 80; server_name example.com; ## URL REWRITE FUNCTION THAT LOWERCASES ALL URIS ## START -- THIS IS INSIDE THE HTTP, BUT IT IS HERE FOR ILLUSTRATION PURPOSES perl_set $my_uri_to_lowercase 'sub { my $r = shift; my $uri = $r->uri; $uri = lc($uri); return $uri; }'; ## URL REWRITE FUNCTION THAT LOWERCASES ALL URIS ## END ## LOCATION #1 ## THIS LOCATION WILL PREVENT LOWERCASING IN ANY URI THAT BEGINS WITH /API/WHATEVER , /CONTENTASSET/WHATEVER, ETC. ## ~ MEANS IT IS A REGEX ## ~* MAKES IT CASEINSENSITIVE SO EITHER /api/ or /API/ will not be lowercased ## ^ MAKES IT MATCH WITH THE BEGINING OF THE "/" AND THE PATH WE DO NOT WANT TO LOWERCASE ## I ADDED THE proxy_pass as observation of David Francis on Nginx Forum ## IF URI MATCHES THIS LOCATION THEN IT WILL STOP SEARCHING FOR OTHER LOCATION MATCHES location ~ ^/(api|contentAsset|categoriesServlet|DotAjaxDirector|html|dwr|dA|JsonTags)/ { proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; ### ### ADD THIS FOR WEBSOCKET SUPPORT ### proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; } ## LOCATION #2 ## THIS LOCATION WILL CATCH THE NECESSARY STRINGS WITHOUT A FORWARD SLASH ## SO THAT THE BACKEND WORKS PROPERTY location = /categoriesServlet { proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; ### ### ADD THIS FOR WEBSOCKET SUPPORT ### proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; } ##LOCATION #3 ## HANDLES ONE PATH WITHOUT FOWARD SLASH ## location = /JSONTags { proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; ### ### ADD THIS FOR WEBSOCKET SUPPORT ### proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; } ## LOCATION #4 ## THIS LOCATION WILL LOWERCASE ANYTHING THAT HAS UPPCASE LETTERS ## EXCEPT THE PATHS STATED ON LOCATION #1 location ~ [A-Z] { return 301 $scheme://$host$my_uri_to_lowercase; proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; } ## LOCATION #5 ## THIS LOCATION IS THE DEFAULT LOCATION location / { proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; } } ##BETA EXAMPLE HTTP SERVER CONFIGURATION END Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285780,285799#msg-285799 From nginx-forum at forum.nginx.org Sat Oct 5 16:50:46 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Sat, 05 Oct 2019 12:50:46 -0400 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: References: <20191004204949.GJ1914@daoine.org> Message-ID: <75a7503aa80a6e62a68d5d2c14ab9308.NginxMailingListEnglish@forum.nginx.org> Francis, I apologized. I wrote your name incorrectly in my previous message. I am not able to edit it. So, THANK YOU, FRANCIS! Alex Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285780,285800#msg-285800 From wizard at bnnorth.net Sun Oct 6 02:09:28 2019 From: wizard at bnnorth.net (Ken Wright) Date: Sat, 5 Oct 2019 22:09:28 -0400 Subject: PR_END_OF_FILE_ERROR after kernel update In-Reply-To: <20191004141926.zmham3kpgazxxeqk@gromit.local> References: <191bb86d-a9d9-03f6-1326-c1b7bd863d1d@bnnorth.net> <20191004141926.zmham3kpgazxxeqk@gromit.local> Message-ID: On 10/4/19 10:19 AM, J. Lewis Muir wrote: > Can you check your nginx server logs? My nginx error.log is ten thousand lines of: 2019/10/05 21:54:08 [error] 2034#2034: *8497989 no "ssl_certificate" is defined in server listening on SSL port while SSL handshaking, client: 192.168.44.1, server: 0.0.0.0:443 > Can you check that the same content can be served over plain HTTP? I've got everything set up to redirect to https.? Sorry. > Can you check that all of your nginx and Let's Encrypt configuration > files are still intact? All good. > Can you check that your file systems are not full? According to parted, there's plenty of room on the drive. > Can you share a curl command that shows your output that others could > use to compare to your output? Umm, now you've gone a bit beyond me.? Can you dumb it down a bit? Ken -- Registered Linux user #483005 If you ever think international relations make sense, remember this: because a Serb shot an Austrian in Bosnia, Germany invaded Belgium. From francis at daoine.org Sun Oct 6 11:14:35 2019 From: francis at daoine.org (Francis Daly) Date: Sun, 6 Oct 2019 12:14:35 +0100 Subject: Nginx Reverse Proxy Lowercase URL and some exceptions to the URL In-Reply-To: References: <20191004204949.GJ1914@daoine.org> Message-ID: <20191006111435.GK1914@daoine.org> On Sat, Oct 05, 2019 at 12:44:30PM -0400, Alex Med wrote: Hi there, Good that you have a configuration that works for you. If you want to spend more time on it, you may be able to remove some of the duplication; but that is not necessary. > I even included the proxy configuration parameters for the > websocket connection to work. Good stuff. I would expect that the extra "websockets" configuration is not necessary for non-websocket requests; so you might be able to add new location{}s for just-websocket-requests -- but again, if what you have works, there is no need to change anything. > I noticed that the location =/ does not > support OR so I had create two additional locations to catch paths without > the end forward slash. Correct. "location =" is for an exact string match. > Is there a limit in the number of location that a > configuration file should have? "As many as it needs". If it needs more than your hardware can handle, you need different hardware. Which is just a way of saying: no, there is no practical limit that you will meet. > Thank YOU again for being in this forum and helping everyone! You're welcome. > location ~ > ^/(api|contentAsset|categoriesServlet|DotAjaxDirector|html|dwr|dA|JsonTags)/ > { So - this location will match requests that start with "/X/" for each of the eight strings in the pattern -- matching case-sensitively, because you write "~" and not "~*". > proxy_pass http://xxx.xx.xx.xxx:IIII; That line says to proxy_pass these requests elsewhere. > proxy_set_header Host $host; > proxy_set_header X-Real-IP $remote_addr; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > proxy_connect_timeout 600; > proxy_send_timeout 600; > proxy_read_timeout 600; > proxy_redirect off; And those lines set some proxy-specific variables. Because those same extra lines appear in many location{}s, you *could* choose to put them at server{} level once, outside all location{}s, and they would inherit into those location{}s, per the normal nginx method. > location = /categoriesServlet { This location will match exactly that request. > proxy_pass http://xxx.xx.xx.xxx:IIII; That line is needed. > proxy_set_header Host $host; > proxy_set_header X-Real-IP $remote_addr; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > proxy_connect_timeout 600; > proxy_send_timeout 600; > proxy_read_timeout 600; > proxy_redirect off; > > ### > ### ADD THIS FOR WEBSOCKET SUPPORT > ### > > proxy_http_version 1.1; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "Upgrade"; Those lines could be removed from here, if they were all set at server{} level. > location = /JSONTags { > proxy_pass http://xxx.xx.xx.xxx:IIII; Same story here -- exactly one request handled, this capitalisation only. > location ~ [A-Z] { This will match any request that includes a capital letter. > return 301 $scheme://$host$my_uri_to_lowercase; And will return a redirect to the matching lowercased-request. > proxy_pass http://xxx.xx.xx.xxx:IIII; This line (and the rest of them below) will not do anything, because the "return" happens first. > location / { > proxy_pass http://xxx.xx.xx.xxx:IIII; This location will match any other requests, and will proxy_pass them to upstream. Cheers, f -- Francis Daly francis at daoine.org From francis at daoine.org Sun Oct 6 11:21:22 2019 From: francis at daoine.org (Francis Daly) Date: Sun, 6 Oct 2019 12:21:22 +0100 Subject: PR_END_OF_FILE_ERROR after kernel update In-Reply-To: References: <191bb86d-a9d9-03f6-1326-c1b7bd863d1d@bnnorth.net> <20191004141926.zmham3kpgazxxeqk@gromit.local> Message-ID: <20191006112122.GL1914@daoine.org> On Sat, Oct 05, 2019 at 10:09:28PM -0400, Ken Wright wrote: > On 10/4/19 10:19 AM, J. Lewis Muir wrote: Hi there, > > Can you check your nginx server logs? > My nginx error.log is ten thousand lines of: > > 2019/10/05 21:54:08 [error] 2034#2034: *8497989 no "ssl_certificate" is > defined in server listening on SSL port while SSL handshaking, client: > 192.168.44.1, server: 0.0.0.0:443 That looks like a fairly clear set of words for an error message. >From your configuration, can you see why nginx thinks that your server{} that does listen on port 443 for SSL does not have a "ssl_certificate" directive? Possibilities include: * there is one such server{}, and it has no "ssl_certificate" line. * there is one such server{}, and it has a "ssl_certificate" line, but it is commented out; or perhaps the previous line is like "server_name" and does not end in a semi-colon; * there is more than one such server, and the one that you think you have told nginx to use is not the one that you actually told nginx to use. Something like the output of nginx -T | grep -E -n 'configuration|server|listen|ssl' might show what relevant configuration your nginx is reading, and might help show what should be changed in order to make things work. Good luck with it, f -- Francis Daly francis at daoine.org From nginx-forum at forum.nginx.org Sun Oct 6 17:02:22 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Sun, 06 Oct 2019 13:02:22 -0400 Subject: Trailing Slash Redirect Loop Help In-Reply-To: <20170512212503.GD10157@daoine.org> References: <20170512212503.GD10157@daoine.org> Message-ID: <679f11efef292a22c0dac034b8c2c768.NginxMailingListEnglish@forum.nginx.org> Dear Francis: After so many years, I am back to the same questions I had many years ago, but never resolved on Nginx. So perhaps, it is time to change this and find a useful implementation. So, now what I want is to get rid off the trailing slashes for anything that is not a directory or a folder (because my tomcat app seems to add a trailing slash to it and that creates an infinite loop that does not allow the page to render.). I have used many things to address the issue. You can see my efforts below, but they do not work. I always get the infinite loop for directories and an error from the browser: The browser can not open the page "http://example.com/xxxx" The error is: "Load can not follow more than 20 redirections" (:0) As you stated it: "And: what's a file and what's a directory? Your initial config example used proxy_pass, which refers to remote urls, not files or directories. *This* nginx does not know whether an upstream url corresponds to a file or to something else. So that may want to be considered before designing the solution." What can I do? Thank you for your feedback! ----this is what I have tried ----- # 1 - The Evil If if ( !-e $request_filename ) { rewrite ^/(.*)/$ /$1 permanent; } # 2 - TRY_FILES location / { ##TRY_FILES WILL GET A $URI AND TEST IF IT IS A FILE AND SERVE IT IF THE FILE EXISTS ##ELSE IT WILL TEST IF THE URI IS A DIRECTORY $URI/ IF DIRECTORY EXISTS IT WILL SERVE IT, AND ELSE AND FINALLY ##IF IT IS NOT EITHER IT WILL PASS IT TO A LOCATION ##THAT WILL REWRITE TO LOWERCASE ## root - OUR APP REQUIRES NO ROOT FOR NGINX TO WORK WITH IT SO NO ROOT IS PROVIDED FROM THE TOMCAT APP. try_files $uri $uri/ @trimslash; proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; ### ### ADD THIS FOR WEBSOCKET SUPPORT ### proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; } location @trimslash { rewrite ^/(.*)/$ /$1 permanent; proxy_pass http://xxx.xx.xx.xxx:IIII; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; proxy_redirect off; ### ### ADD THIS FOR WEBSOCKET SUPPORT ### proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; } } ##BETA EXAMPLE HTTP SERVER CONFIGURATION END Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273964,285805#msg-285805 From wizard at bnnorth.net Sun Oct 6 21:03:51 2019 From: wizard at bnnorth.net (Ken Wright) Date: Sun, 6 Oct 2019 17:03:51 -0400 Subject: PR_END_OF_FILE_ERROR after kernel update In-Reply-To: <20191006112122.GL1914@daoine.org> References: <191bb86d-a9d9-03f6-1326-c1b7bd863d1d@bnnorth.net> <20191004141926.zmham3kpgazxxeqk@gromit.local> <20191006112122.GL1914@daoine.org> Message-ID: Okay, after checking per Francis' suggestion I found the problem:? I was able to add the ssl_certificates to the active server and now... I'm getting a different error. This one, however, I don't think is an nginx error.? When I point my browser to my website, I get a "The page isn't redirecting properly" message.? I'm going to contact my registrar and see what's going on now. Francis, thank you for pointing me in the right direction! Ken On 10/6/19 7:21 AM, Francis Daly wrote: > On Sat, Oct 05, 2019 at 10:09:28PM -0400, Ken Wright wrote: >> On 10/4/19 10:19 AM, J. Lewis Muir wrote: > -- Registered Linux user #483005 If you ever think international relations make sense, remember this: because a Serb shot an Austrian in Bosnia, Germany invaded Belgium. From nginx-forum at forum.nginx.org Mon Oct 7 06:07:52 2019 From: nginx-forum at forum.nginx.org (mmpl) Date: Mon, 07 Oct 2019 02:07:52 -0400 Subject: Error 502 Bad Gateway on nginx Message-ID: Error 502 Bad Gateway on nginx and php-fpm Posted by: mmpl I can't start php7-fpm in supervisor. The error log: unable to bind listening socket for address '127.0.0.1:9000': Address in > use (98) In browser: localhost:8090 is error: 502 Bad Gateway netstat -tulnp tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 13/nginx tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 13/nginx tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 13/nginx tcp 0 0 :::3000 :::* LISTEN 11/node tcp 0 0 :::7000 :::* LISTEN 10/node ps aux / # ps aux PID USER TIME COMMAND 1 root 0:00 {bootloader} /bin/sh /boot/bootloader 8 root 0:01 {supervisord} /usr/bin/python2 /usr/bin/supervisord 62 root 0:00 /bin/sh 92 root 0:00 nginx: master process /usr/sbin/nginx 93 root 0:00 nginx: worker process 94 root 0:00 nginx: worker process 122 root 0:00 ps aux php-fpm.d/www/conf [www] user = nobody group = nobody listen = 127.0.0.1:9000 pm = dynamic pm.max_children = 5 pm.start_servers = 2 pm.min_spare_servers = 1 pm.max_spare_servers = 3 /etc/nginx/sites-enabled/page1 server { listen 8090; root /usr/bin; server_name localhost; access_log /dev/null; error_log /dev/null; location / { proxy_pass http://127.0.0.0:7000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Fowarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Fowarded-Proto $scheme; proxy_cache_bypass $http_upgrade; try_files $uri $uri/ =404; } location ~ \.(gif) { root /var/lib; } } /etc/nginx/sites-enabled/page2 server { server_name localhost; root /www; index index.php; access_log /dev/null; error_log /dev/null; client_max_body_size 100M; fastcgi_read_timeout 1800; location / { try_files $uri $uri/ /index.php$query_string; } location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ { expires max; log_not_found off; access_log off; } location ~ \.php$ { try_files $uri =404; include fastcgi_params; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_pass 127.0.0.1:9000; } } Where is error? I don't just any idea. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285807,285807#msg-285807 From lahiruprasad at gmail.com Mon Oct 7 09:51:18 2019 From: lahiruprasad at gmail.com (Lahiru Prasad) Date: Mon, 7 Oct 2019 15:21:18 +0530 Subject: Error 502 Bad Gateway on nginx In-Reply-To: References: Message-ID: Hi, Does nc -l 9000 gives the same bind error ? If so, better to check netstat for :9000 sockets with all status, like netstat -anp | grep :9000 to get an idea. Regards, Lahiru Prasad. On Mon, Oct 7, 2019 at 11:38 AM mmpl wrote: > Error 502 Bad Gateway on nginx and php-fpm > Posted by: mmpl > > I can't start php7-fpm in supervisor. The error log: > > unable to bind listening socket for address '127.0.0.1:9000': Address in > > use (98) > > In browser: > > localhost:8090 is error: 502 Bad Gateway > > netstat -tulnp > > tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 13/nginx > tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 13/nginx > tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 13/nginx > tcp 0 0 :::3000 :::* LISTEN 11/node > tcp 0 0 :::7000 :::* LISTEN 10/node > ps aux > > / # ps aux > PID USER TIME COMMAND > 1 root 0:00 {bootloader} /bin/sh /boot/bootloader > 8 root 0:01 {supervisord} /usr/bin/python2 /usr/bin/supervisord > 62 root 0:00 /bin/sh > 92 root 0:00 nginx: master process /usr/sbin/nginx > 93 root 0:00 nginx: worker process > 94 root 0:00 nginx: worker process > 122 root 0:00 ps aux > php-fpm.d/www/conf > > [www] > > user = nobody > group = nobody > > listen = 127.0.0.1:9000 > > pm = dynamic > > pm.max_children = 5 > > pm.start_servers = 2 > > pm.min_spare_servers = 1 > > pm.max_spare_servers = 3 > /etc/nginx/sites-enabled/page1 > > server { > listen 8090; > root /usr/bin; > server_name localhost; > access_log /dev/null; > error_log /dev/null; > > location / { > proxy_pass http://127.0.0.0:7000; > proxy_http_version 1.1; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection 'upgrade'; > proxy_set_header Host $host; > proxy_set_header X-Real-IP $remote_addr; > proxy_set_header X-Fowarded-For $proxy_add_x_forwarded_for; > proxy_set_header X-Fowarded-Proto $scheme; > proxy_cache_bypass $http_upgrade; > > try_files $uri $uri/ =404; > } > > location ~ \.(gif) { > root /var/lib; > } > > } > /etc/nginx/sites-enabled/page2 > > server { > server_name localhost; > root /www; > index index.php; > access_log /dev/null; > error_log /dev/null; > > client_max_body_size 100M; > fastcgi_read_timeout 1800; > > location / { > try_files $uri $uri/ /index.php$query_string; > } > > location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ { > expires max; > log_not_found off; > access_log off; > } > > location ~ \.php$ { > try_files $uri =404; > include fastcgi_params; > fastcgi_index index.php; > fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; > fastcgi_pass 127.0.0.1:9000; > } > } > Where is error? I don't just any idea. > > Posted at Nginx Forum: > https://forum.nginx.org/read.php?2,285807,285807#msg-285807 > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Oct 7 12:38:29 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 7 Oct 2019 15:38:29 +0300 Subject: Doing a mass purge of the cache using a wildcard In-Reply-To: <728597595.1787657.1570209922987.JavaMail.zimbra@xsb.com> References: <728597595.1787657.1570209922987.JavaMail.zimbra@xsb.com> Message-ID: <20191007123828.GW1877@mdounin.ru> Hello! On Fri, Oct 04, 2019 at 12:25:22PM -0500, Jordan Keith wrote: > We are using nginx plus. Just to make sure you know: contacting support might be a better/faster option if you are using nginx-plus. > We have caching configured for a few endpoints, one which is similar to the following: > > document/{id}/report > > I would like to purge all items cached under that endpoint, but executing a purge request like the following does not purge anything: > > document/*/report > > We have other endpoints where mass purges work fine, but they have the * at the end of the URL instead of the middle. > > So I have 2 questions: > > 1). Does nginx only support wildcards at the end of the URL for cache-purging purposes? Quoting http://nginx.org/r/proxy_cache_purge: : If the cache key of a purge request ends with an asterisk (?*?), all cache : entries matching the wildcard key will be removed from the cache. Note "ends with" - only wildcards at the end of the cache key are supported. This doesn't mean "at the end of the URL" though, as cache keys can be arbitrary configured (see http://nginx.org/r/proxy_cache_key). > 2). Is there some other way I can accomplish a mass purge here? If you want to purge resources like this on a regular basis, re-structuring your proxy_cache_key might be an option. For example, you may split "document/{id}/report" into "document/{id}" and "/report", and put "/report" into the cache key first. -- Maxim Dounin http://mdounin.ru/ From roger at netskrt.io Mon Oct 7 19:07:29 2019 From: roger at netskrt.io (Roger Fischer) Date: Mon, 7 Oct 2019 12:07:29 -0700 Subject: Add header to response before response is cached Message-ID: <76C6E829-2D57-4250-9EA8-700A582302ED@netskrt.io> Hello, is there a way in an NGINX HTTP Proxy to add a header to the response before it is cached? I would like to capture some information from the request and add it to the cached response, so that all clients getting the cached response receive that info. Thanks? Roger From nginx-forum at forum.nginx.org Tue Oct 8 08:00:28 2019 From: nginx-forum at forum.nginx.org (jiluluogen18) Date: Tue, 08 Oct 2019 04:00:28 -0400 Subject: =?UTF-8?B?5Yqe55CG5Yqg5ou/5aSn55WZ5a2m6K+B5piO44CB5Yqe5rGk5aeG6YCK5aSn5a2m?= =?UTF-8?B?5pys56GV55WZ5a2m6K+B5piOUeW+ruS/oe+8mjk2MzE0NjM3NuWKnueQhlRS?= =?UTF-8?B?VeaIkOe7qeWNleOAgeWKnueQhuWKoOaLv+Wkp+eVmeWtpuaWh+WHreivgQ==?= =?UTF-8?B?5Lmm44CB5Yqe5rGk5aeG6YCK5aSn5a2m5oiQ57up5Y2V44CB5Yqe55CG5Yqg?= =?UTF-8?B?5ou/5aSn55WZ5a2m5paH5Yet6K+B5Lmm44CB5YqeVFJV5oiQ57up5Y2V44CB?= =?UTF-8?B?5Yqe55CG5Yqg5ou/5aSn55WZ5a2m5paH5Yet6K+B5Lmm44CB5Yqe5rGk5aeG?= =?UTF-8?B?6YCK5aSn5a2m55WZ5a2m6K+B5Lmm44CB5Yqe55CG5Yqg5ou/5aSn5q+V5Lia?= =?UTF-8?B?6K+B44CB5YqeVFJV55WZ5a2m6K+B5Lmm44CB5Yqe55CG5Yqg5ou/5aSn5q+V?= =?UTF-8?B?5Lia6K+B44CB5Yqe5rGk5aeG6YCK5aSn5a2m5a2m5L2N6K+B5Lmm44CB5Yqe?= =?UTF-8?B?55CG5Yqg5ou/5aSn55WZ5a2m6K+B5piO44CB5YqeVFJV5a2m5L2N6K+B5Lmm?= =?UTF-8?B?44CB5Yqe55CG5Yqg5ou/5aSn55WZ5a2m6K+B5piO44CB5Yqe5rGk5aeG6YCK?= =?UTF-8?B?5aSn5a2m5YGH5oiQ57up5Y2V44CB5Yqe55CG5Yqg5ou/5aSn55WZ5a2m5paH?= =?UTF-8?B?5Yet6K+B5Lmm44CB5YqeVFJV5YGH5pys56GV55WZ5a2m6K+B5piOUeW+rg==?= =?UTF-8?B?5L+h77yaOTYzMTQ2Mzc25Yqe55CG?= Message-ID: <2432eda3e06e79693f9bc13f522a533e.NginxMailingListEnglish@forum.nginx.org> 1 Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285816,285816#msg-285816 From francis at daoine.org Tue Oct 8 22:46:04 2019 From: francis at daoine.org (Francis Daly) Date: Tue, 8 Oct 2019 23:46:04 +0100 Subject: Trailing Slash Redirect Loop Help In-Reply-To: <679f11efef292a22c0dac034b8c2c768.NginxMailingListEnglish@forum.nginx.org> References: <20170512212503.GD10157@daoine.org> <679f11efef292a22c0dac034b8c2c768.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191008224604.GM1914@daoine.org> On Sun, Oct 06, 2019 at 01:02:22PM -0400, Alex Med wrote: Hi there, > So, now what I want is to get rid off the trailing slashes for anything that > is not a directory or a folder (because my tomcat app seems to add a > trailing slash to it and that creates an infinite loop that does not allow > the page to render.). That sounds like unusual behaviour for a tomcat app. Can I suggest that you may be better off fixing the app so that it works correctly? > I have used many things to address the issue. You can see my efforts below, > but they do not work. I always get the > infinite loop for directories and an error from the browser: For clarity: can you show the config you use, and one request that uses that config, and the response that you get? And, in case it is not obvious, can you show the response that you want instead? With a specific example, it may be clearer what needs to be configured. > The browser can not open the page "http://example.com/xxxx" The error is: > "Load can not follow more than 20 redirections" (:0) Rather than "a browser", if you use "curl -v" or "curl -i", it should be clearer what is happening. If the response is a 30x redirect to a Location that is different from the original request, perhaps make a new "curl -i" request for that Location. At some point in the cycle, the loop will become obvious. > # 1 - The Evil If > > if ( !-e $request_filename ) { rewrite ^/(.*)/$ /$1 permanent; } You want "request ends in slash, and does not correspond to a directory". Use "!-d". And - that "if" does not look evil to me. > # 2 - TRY_FILES > try_files $uri $uri/ @trimslash; > proxy_pass http://xxx.xx.xx.xxx:IIII; I'm not sure what this is trying to do. Does the proxy_pass upstream share the same filesystem as this nginx? > location @trimslash { > > rewrite ^/(.*)/$ /$1 permanent; > proxy_pass http://xxx.xx.xx.xxx:IIII; Once the "rewrite" happens, the "proxy_pass" will not apply. So I'm not sure what this is trying to do, either. f -- Francis Daly francis at daoine.org From wizard at bnnorth.net Wed Oct 9 05:52:11 2019 From: wizard at bnnorth.net (Ken Wright) Date: Wed, 9 Oct 2019 01:52:11 -0400 Subject: Odd behavior Message-ID: Sorry to be taking up so much bandwidth lately, but I'm seeing some weird behavior from nginx. When I enter my domain name in Firefox, I get a 404 status.? If I enter .com/index.nginx-debian.html I get the regular nginx welcome screen.? Just for fun I copied index.nginx-debian.html to index.html, and .com/index.html gets an "Opening index.php" window.? This one in particular has me stumped, because index.html and index.nginx-debian.html are identical, but the show different behavior. Like I said, weird.? Has anyone else seen anything like this? Ken -- Registered Linux user #483005 If you ever think international relations make sense, remember this: because a Serb shot an Austrian in Bosnia, Germany invaded Belgium. From peter_booth at me.com Wed Oct 9 07:39:01 2019 From: peter_booth at me.com (Peter Booth) Date: Wed, 9 Oct 2019 03:39:01 -0400 Subject: Odd behavior In-Reply-To: References: Message-ID: Is your web server on the internet? If so then see what redbot shows. It?s an amazing tool to debug nuanced http issues Sent from my iPhone > On Oct 9, 2019, at 1:52 AM, Ken Wright wrote: > > ?Sorry to be taking up so much bandwidth lately, but I'm seeing some > weird behavior from nginx. > > When I enter my domain name in Firefox, I get a 404 status. If I enter > .com/index.nginx-debian.html I get the regular nginx welcome > screen. Just for fun I copied index.nginx-debian.html to index.html, > and .com/index.html gets an "Opening index.php" window. This > one in particular has me stumped, because index.html and > index.nginx-debian.html are identical, but the show different behavior. > > Like I said, weird. Has anyone else seen anything like this? > > Ken > > -- > > Registered Linux user #483005 > > If you ever think international relations make sense, remember this: > because a Serb shot an Austrian in Bosnia, Germany invaded Belgium. > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From francis at daoine.org Wed Oct 9 08:27:25 2019 From: francis at daoine.org (Francis Daly) Date: Wed, 9 Oct 2019 09:27:25 +0100 Subject: Odd behavior In-Reply-To: References: Message-ID: <20191009082725.GN1914@daoine.org> On Wed, Oct 09, 2019 at 01:52:11AM -0400, Ken Wright wrote: Hi there, > When I enter my domain name in Firefox, I get a 404 status. Do your nginx logs show this request getting to nginx? If not, there is something outside of nginx that needs changing first. Perhaps if you test using "curl", that will eliminate any possibility of browser-side caching interfering in your tests. > If I enter > .com/index.nginx-debian.html I get the regular nginx welcome > screen.? Just for fun I copied index.nginx-debian.html to index.html, > and .com/index.html gets an "Opening index.php" window.? This > one in particular has me stumped, because index.html and > index.nginx-debian.html are identical, but the show different behavior. If your configuration includes something like location = /index.html { return 200 "Opening index.php\n"; } that might explain that difference. Your configuration probably does not include that location{} block; but only you know for sure. On your test system where you can replicate this unwanted behaviour: how many lines from the config can you remove, and still see the issue? If you can show a (small) config that someone else can use to see the issue, perhaps it will be clear what needs to be changed. Cheers, f -- Francis Daly francis at daoine.org From nginx-forum at forum.nginx.org Wed Oct 9 17:54:58 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Wed, 09 Oct 2019 13:54:58 -0400 Subject: Trailing Slash Redirect Loop Help In-Reply-To: <20191008224604.GM1914@daoine.org> References: <20191008224604.GM1914@daoine.org> Message-ID: Dear Francis: Thank you for your answers. Here is the full configuration with reverse proxy, pagespeed,and lowercasing. It all works well, but now that I incorporated removing the trailing slash it brought up new issues: https://pastebin.com/keQ239D4. Let me know if you prefer that I post the configurations in the post. Here are the answers to your questions: As stated before the "evil if" or try_files work well except when it comes to directories or folders. >>>>For clarity: can you show the config you use, and one request that >>>>Fuses that config, and the response that you get? And, in case it is not >>>>Fobvious, can you show the response that you want instead? This is with the "evil if" enable, I get the following response when it is a directory such as the blog directory: curl -i http://example.com/blog/ HTTP/1.1 301 Moved Permanently Server: nginx Date: Wed, 09 Oct 2019 15:06:04 GMT Content-Type: text/html Content-Length: 178 Location: http://example.com/blog Connection: keep-alive 301 Moved Permanently

301 Moved Permanently

nginx
curl -i http://example.com/blog HTTP/1.1 301 Server: nginx Date: Wed, 09 Oct 2019 15:06:48 GMT Content-Length: 0 Connection: keep-alive Location: /blog/ Cache-Control: s-maxage=10 If I disable the "evil if" and try with try_files, I get this: To answer your question: >>Does the proxy_pass upstream >>share the same filesystem as this nginx? No, they do not share the same file system. curl -i http://example.com/blog/ HTTP/1.1 301 Moved Permanently Server: nginx Date: Wed, 09 Oct 2019 15:13:23 GMT Content-Type: text/html Content-Length: 178 Location: http://example.com/blog Connection: keep-alive 301 Moved Permanently

301 Moved Permanently

nginx
curl -i http://example.com/blog HTTP/1.1 301 Server: nginx Date: Wed, 09 Oct 2019 15:13:30 GMT Content-Length: 0 Connection: keep-alive Location: /blog/ Cache-Control: s-maxage=10 Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273964,285830#msg-285830 From francis at daoine.org Wed Oct 9 21:33:08 2019 From: francis at daoine.org (Francis Daly) Date: Wed, 9 Oct 2019 22:33:08 +0100 Subject: Trailing Slash Redirect Loop Help In-Reply-To: References: <20191008224604.GM1914@daoine.org> Message-ID: <20191009213308.GO1914@daoine.org> On Wed, Oct 09, 2019 at 01:54:58PM -0400, Alex Med wrote: Hi there, > Here is the full configuration with reverse proxy, pagespeed,and > lowercasing. It all > works well, but now that I incorporated removing the trailing slash it > brought up new > issues: https://pastebin.com/keQ239D4. Let me know if you prefer that I > post the configurations > in the post. Thanks for this. Some of this configuration is not "stock nginx", but I guess that the extra modules do not affect the issue here. It would be convenient for history here, to have the config here; but the relevant part is basically the part shown in the previous mail. Most requests eventually use a proxy_pass to an upstream server. > This is with the "evil if" enable, I get the following response when it is a > directory such as the blog directory: I suspect that there may be an interpretation issue here. When I read "the blog directory", I understand something along the lines of "/usr/local/nginx/html/blog/index.html exists as a file that nginx can see". I now suspect that you intend something like "the upstream server considers it to be a directory". > curl -i http://example.com/blog/ > HTTP/1.1 301 Moved Permanently > Server: nginx > Date: Wed, 09 Oct 2019 15:06:04 GMT > Content-Type: text/html > Content-Length: 178 > Location: http://example.com/blog > Connection: keep-alive That's from nginx; your config says "if the request ends in / and does not represent a directory on this filesystem, redirect to no-slash". > curl -i http://example.com/blog > HTTP/1.1 301 > Server: nginx > Date: Wed, 09 Oct 2019 15:06:48 GMT > Content-Length: 0 > Connection: keep-alive > Location: /blog/ > Cache-Control: s-maxage=10 That's from your upstream server. Presumably it has decided that "/blog refers to a directory as far as it is concerned; please request /blog/ instead". And there is your infinite loop. So I guess the next question is: what response do you want from nginx? In this specific case: when you request /blog/, how would you like nginx to handle that request? And in your answer, also consider: how might nginx possibly be able to tell that you want the request handled differently from a request for /file/? > >>Does the proxy_pass upstream > >>share the same filesystem as this nginx? > No, they do not share the same file system. "if -d" looks on the filesystem. try_files looks on the filesystem. If the filesystem does not contain the information on whether "this" thing is a directory or not, looking on the filesystem will not be able to give a useful answer. So, what specific problem are you trying to solve? If you remove the "remove the slash" configuration from nginx, is there one request that gets a response that you do not want? Perhaps fixing the handling of *that* request will get you to a config that does what you want. f -- Francis Daly francis at daoine.org From nginx-forum at forum.nginx.org Wed Oct 9 21:55:31 2019 From: nginx-forum at forum.nginx.org (vz19) Date: Wed, 09 Oct 2019 17:55:31 -0400 Subject: Client certificate authentication with multiple CA but single CRL Message-ID: <1bab6e0061c40bcf541e1009a61c0474.NginxMailingListEnglish@forum.nginx.org> Hi, My current nginx configuration has 3 CAs concatenated into a PEM file called /etc/ngx_cas. However, only one of these CAs is configured with a CRL which is stored in /etc/ngx_crls. ssl_client_certificate /etc/ngx_cas; ssl_verify_client strict; ssl_crl /etc/ngx_crls; When I make a request using curl, I get a 400 Bad request with SSL certificate error, even though the client certificate is valid and signed by one of the 3 CAs recognized by NGINX. This CA does not have a CRL tied to it. Upon checking logs, this is the error that is thrown: 2019/10/09 21:40:28 [info] 25610#0: *1 client SSL certificate verify error: (3:unable to get certificate CRL) while reading client request headers, However, when I remove the configured CRL from NGINX, the curl request goes through and authentication is successful. Is there any limitation on CRLs where if one trustpoint's CRL is configured, then CRL for all should be configured as well? Please let me know if I am missing something. Thanks. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285834,285834#msg-285834 From wizard at bnnorth.net Wed Oct 9 23:53:44 2019 From: wizard at bnnorth.net (Ken Wright) Date: Wed, 9 Oct 2019 19:53:44 -0400 Subject: Odd behavior In-Reply-To: <20191009082725.GN1914@daoine.org> References: <20191009082725.GN1914@daoine.org> Message-ID: On 10/9/19 4:27 AM, Francis Daly wrote: > On Wed, Oct 09, 2019 at 01:52:11AM -0400, Ken Wright wrote: > > When I enter my domain name in Firefox, I get a 404 status. > Do your nginx logs show this request getting to nginx? > > If not, there is something outside of nginx that needs changing first. I'm sure I'm getting to nginx; the 404 message shows nginx as well. > Perhaps if you test using "curl", that will eliminate any possibility of > browser-side caching interfering in your tests. I tried using curl on both index.html and index.nginx-debian.html and got identical results for both. > > If you can show a (small) config that someone else can use to see the > issue, perhaps it will be clear what needs to be changed. Okay, here's the default config: # Default server configuration # upstream php-handler { ??? server unix:/run/php/php7.2-fpm.sock; } server { ??? if ($host = www.koalatyworks.com) { ??????? return 301 https://$host$request_uri; ??? } # managed by Certbot ??? if ($host = koalatyworks.com) { ??????? return 301 https://$host$request_uri; ??? } # managed by Certbot ??? listen 80; ??? listen [::]:80; ??????? server_name koalatyworks.com www.koalatyworks.com; ??? # SSL configuration ??? # ??? listen 443 ssl http2; ??? listen [::]:443 ssl http2; ??????? server_name koalatyworks.com www.koalatyworks.com; ??? # ??? root /var/www/html; ??? # Add index.php to the list if you are using PHP ??? index index.html index.htm index.nginx-debian.html index.php; ??? return 301 https://$host$request_uri; ??????? ssl_certificate /etc/letsencrypt/live/koalatyworks.com/fullchain.pem; # managed by Certbot ??????? ssl_certificate_key /etc/letsencrypt/live/koalatyworks.com/privkey.pem; # managed by Certbot ??? location / { ??? ??? # First attempt to serve request as file, then ??? ??? # as directory, then fall back to displaying a 404. ??? ??? try_files $uri $uri/ =404; ??? } } I hope that helps! Ken -- Registered Linux user #483005 If you ever think international relations make sense, remember this: because a Serb shot an Austrian in Bosnia, Germany invaded Belgium. From francis at daoine.org Thu Oct 10 08:31:54 2019 From: francis at daoine.org (Francis Daly) Date: Thu, 10 Oct 2019 09:31:54 +0100 Subject: Odd behavior In-Reply-To: References: <20191009082725.GN1914@daoine.org> Message-ID: <20191010083154.GP1914@daoine.org> On Wed, Oct 09, 2019 at 07:53:44PM -0400, Ken Wright wrote: > On 10/9/19 4:27 AM, Francis Daly wrote: > > On Wed, Oct 09, 2019 at 01:52:11AM -0400, Ken Wright wrote: Hi there, > > If you can show a (small) config that someone else can use to see the > > issue, perhaps it will be clear what needs to be changed. > > Okay, here's the default config: > server { > ??? if ($host = www.koalatyworks.com) { > ??????? return 301 https://$host$request_uri; > ??? } # managed by Certbot > > ??? if ($host = koalatyworks.com) { > ??????? return 301 https://$host$request_uri; > ??? } # managed by Certbot [snip] > ??? root /var/www/html; > ??? return 301 https://$host$request_uri; When using this config, is there one request that you can make that responds with something other than a http 301? I think that you said you get a 404, and maybe two different 200s. So I'm going to say that, using the config that you provide, I am unable to replicate the issue that you report. Perhaps I'm using a different version of nginx. Or perhaps the config that was in place when you saw the issue, is not the config that is here. Cheers, f -- Francis Daly francis at daoine.org From francis at daoine.org Thu Oct 10 08:40:48 2019 From: francis at daoine.org (Francis Daly) Date: Thu, 10 Oct 2019 09:40:48 +0100 Subject: Add header to response before response is cached In-Reply-To: <76C6E829-2D57-4250-9EA8-700A582302ED@netskrt.io> References: <76C6E829-2D57-4250-9EA8-700A582302ED@netskrt.io> Message-ID: <20191010084048.GQ1914@daoine.org> On Mon, Oct 07, 2019 at 12:07:29PM -0700, Roger Fischer wrote: Hi there, > is there a way in an NGINX HTTP Proxy to add a header to the response before it is cached? > > I would like to capture some information from the request and add it to the cached response, so that all clients getting the cached response receive that info. If you do not find a more direct way (either by config or by coding), then you could potentially add a new nginx server{} to modify the upstream response as you want. That is: * main nginx does proxy_pass to new nginx, including your extra information as proxy_set_header values. main nginx does proxy_cache the response. * new nginx does proxy_pass to upstream; does not proxy_cache the response; and does add_header with your extra information so that that extra information is in the response that goes to main nginx. "new nginx" can be a server{} that listens on a loopback port, or on a unix domain socket -- something that the world does not need to access directly. f -- Francis Daly francis at daoine.org From mdounin at mdounin.ru Thu Oct 10 08:58:33 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 10 Oct 2019 11:58:33 +0300 Subject: Client certificate authentication with multiple CA but single CRL In-Reply-To: <1bab6e0061c40bcf541e1009a61c0474.NginxMailingListEnglish@forum.nginx.org> References: <1bab6e0061c40bcf541e1009a61c0474.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191010085832.GH1877@mdounin.ru> Hello! On Wed, Oct 09, 2019 at 05:55:31PM -0400, vz19 wrote: > My current nginx configuration has 3 CAs concatenated into a PEM file called > /etc/ngx_cas. However, only one of these CAs is configured with a CRL which > is stored in /etc/ngx_crls. > > ssl_client_certificate /etc/ngx_cas; > ssl_verify_client strict; > ssl_crl /etc/ngx_crls; > > When I make a request using curl, I get a 400 Bad request with SSL > certificate error, even though the client certificate is valid and signed by > one of the 3 CAs recognized by NGINX. This CA does not have a CRL tied to > it. Upon checking logs, this is the error that is thrown: > 2019/10/09 21:40:28 [info] 25610#0: *1 client SSL certificate verify > error: (3:unable to get certificate CRL) while reading client request > headers, > > However, when I remove the configured CRL from NGINX, the curl request goes > through and authentication is successful. > > Is there any limitation on CRLs where if one trustpoint's CRL is configured, > then CRL for all should be configured as well? Yes, as long as ssl_crl is configured, nginx uses the X509_V_FLAG_CRL_CHECK_ALL flag, which implies that verification against CA certificates which does not have CRLs will fail. That is, you have to configure CRLs for all CA certificates you use (including intermediate CAs if there are any). -- Maxim Dounin http://mdounin.ru/ From dmitry at xdmitry.ru Thu Oct 10 14:12:40 2019 From: dmitry at xdmitry.ru (Dmitry Vereschaka) Date: Thu, 10 Oct 2019 17:12:40 +0300 (MSK) Subject: gzip compression for upstream requests Message-ID: Hi Does nginx support compression of proxied upstream requests (not responses)? I have configuration below and what I see is that responses are sent uncompressed to upstream server (and I'm getting compressed responses from upstream, as upstream is also nginx with gzip compression enabled). upstream someupstream { server someserver:3333; keepalive 32; } server { listen 3333; gzip on; gzip_proxied any; gzip_types *; gzip_min_length 10; gzip_vary on; gzip_http_version 1.1; location / { proxy_pass http://someupstream; proxy_read_timeout 84600s; proxy_send_timeout 84600s; proxy_http_version 1.1; proxy_set_header Connection ""; } } when I do curl --data-binary @file.txt http://localhost:3333/zxc I can see with tcpdump that following data is sent to upstream server POST /zxc HTTP/1.1 Host: someupstream Content-Length: 4301 Accept: */* Content-Type: application/x-www-form-urlencoded Regards From mdounin at mdounin.ru Thu Oct 10 15:05:08 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 10 Oct 2019 18:05:08 +0300 Subject: gzip compression for upstream requests In-Reply-To: References: Message-ID: <20191010150508.GI1877@mdounin.ru> Hello! On Thu, Oct 10, 2019 at 05:12:40PM +0300, Dmitry Vereschaka wrote: > Does nginx support compression of proxied upstream requests (not > responses)? No. > I have configuration below and what I see is that responses are sent Just a side note: you probably mean "requests" here. > uncompressed to upstream server (and I'm getting compressed responses from > upstream, as upstream is also nginx with gzip compression enabled). Quoting documentation of the gzip directive (http://nginx.org/r/gzip): : Enables or disables gzipping of responses. Similar wording can be found in the module description above. It's about responses, not requests. [...] -- Maxim Dounin http://mdounin.ru/ From dmitry at xdmitry.ru Thu Oct 10 15:30:03 2019 From: dmitry at xdmitry.ru (Dmitry Vereschaka) Date: Thu, 10 Oct 2019 18:30:03 +0300 Subject: gzip compression for upstream requests In-Reply-To: <20191010150508.GI1877@mdounin.ru> References: <20191010150508.GI1877@mdounin.ru> Message-ID: Thank you! On Thu, Oct 10, 2019 at 6:05 PM Maxim Dounin wrote: > Hello! > > On Thu, Oct 10, 2019 at 05:12:40PM +0300, Dmitry Vereschaka wrote: > > > Does nginx support compression of proxied upstream requests (not > > responses)? > > No. > > > I have configuration below and what I see is that responses are sent > > Just a side note: you probably mean "requests" here. > Yes, thank you! > > > uncompressed to upstream server (and I'm getting compressed responses > from > > upstream, as upstream is also nginx with gzip compression enabled). > > Quoting documentation of the gzip directive > (http://nginx.org/r/gzip): > > : Enables or disables gzipping of responses. > > Similar wording can be found in the module description above. > It's about responses, not requests. > > [...] > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From soupdiver at gmail.com Fri Oct 11 08:30:14 2019 From: soupdiver at gmail.com (=?utf-8?Q?Felix_Gl=C3=A4ske?=) Date: Fri, 11 Oct 2019 10:30:14 +0200 Subject: Behaviour of map changes when using it's result in limit_req/limit_req_zone Message-ID: <1311913D-FFED-43A1-A258-65B794F0BE36@gmail.com> I try to achieve the following: I want to throttle requests based on upstream response codes. If a user (repeatedly) fails to login, and therefor the upstream server responds with 403, the requests should be throttled. Not for 2xx or other response codes. Only 403. Online I found a solution using a `map` and `limit_req_zone`/`limit_req`. So far so good, I tried out the things separately and the mapping as well as the throttling works. But when putting them together things get weird, at least as far as I understand so far. My config: error_log /dev/stdout info; daemon off; events { worker_connections 4096; } http { map $status $bad_guy { ~^[2] 1; ~^[4] $remote_addr; default "default"; } map $status $wup { ~^[2] 1; ~^[4] $remote_addr; default "default"; } log_format main '$remote_addr - $status ->$wup<- ->$bad_guy<- '; access_log /dev/stdout main; limit_req_zone $bad_guy zone=by_status:10m rate=60r/m; server { listen 8000; location / { proxy_pass http://127.0.0.1:8888$request_uri; limit_req zone=by_status burst=2; } } } I have two identical maps. The only difference is that $bad_guy is used with "limit_req_zone". I would assume that in the log statement the values for $wup and $bad_guy are the same since they are based on maps with the same rules. However, this is _not_ the case: Scenario 1: Upstream reports 403 Log: 127.0.0.1 - 403 ->127.0.0.1<- ->default<- As we can see, only $wup has the expected value and $bad_guy is default for some reason. Scenario 2: Upstream reports 226 Log: 127.0.0.1 - 226 ->1<- ->default<- The same behaviour as before. Let's change the configuration a little bit and comment out the line "limit_req zone=by_status burst=2;" Scenario 3: Upstream reports 403 Log: 127.0.0.1 - 403 ->127.0.0.1<- ->127.0.0.1<- Ok, now the values seem all right. Scenario 4: Upstream reports 226 Log: 127.0.0.1 - 226 ->1<- ->1<- Ok, good values again. Final question: Why does using "limit_req" change the behaviour/outcome of the "map"? Maybe I'm not aware enough of the internals but this seems weird to me and also blocks me from reaching my goal to throttle failed login attempts. I hope someone can shine some light on that. Thanks! From francis at daoine.org Fri Oct 11 10:26:16 2019 From: francis at daoine.org (Francis Daly) Date: Fri, 11 Oct 2019 11:26:16 +0100 Subject: Behaviour of map changes when using it's result in limit_req/limit_req_zone In-Reply-To: <1311913D-FFED-43A1-A258-65B794F0BE36@gmail.com> References: <1311913D-FFED-43A1-A258-65B794F0BE36@gmail.com> Message-ID: <20191011102616.GR1914@daoine.org> On Fri, Oct 11, 2019 at 10:30:14AM +0200, Felix Gl?ske wrote: Hi there, > I want to throttle requests based on upstream response codes. It is unlikely to be useful to try to limit incoming requests based on something that is not in the incoming request. (More specifically: based on something that is not available at the time that the limiting should happen.) > If a user (repeatedly) fails to login, and therefor the upstream server responds with 403, the requests should be throttled. Not for 2xx or other response codes. Only 403. > > Online I found a solution using a `map` and `limit_req_zone`/`limit_req`. Could you link to this solution? Perhaps it is valid in some circumstances. > http { > map $status $bad_guy { > ~^[2] 1; > ~^[4] $remote_addr; > default "default"; > } > > map $status $wup { > ~^[2] 1; > ~^[4] $remote_addr; > default "default"; > } http://nginx.org/r/$status -- that is the response status. > log_format main '$remote_addr - $status ->$wup<- ->$bad_guy<- '; > access_log /dev/stdout main; > > limit_req_zone $bad_guy zone=by_status:10m rate=60r/m; > location / { > proxy_pass http://127.0.0.1:8888$request_uri; > limit_req zone=by_status burst=2; A request comes in. "limit_req" says "check if this should be limited". limit_req_zone says "what value has $bad_guy got right now?". map says "$status is empty, therefore $bad_guy is default". limiting happens based on "default". After the request completes, "log_format" wants the values of four variables. The nginx-built-in ones have their appropriate values. The extra ones have whatever value they have -- $bad_guy is "default" because that is what "map" set it to previously; "$wup" was not set previously, so now "map" sets it based on $status -- which now is not empty. You can mark $bad_guy "volatile" (http://nginx.org/r/map) if you want "map" to set it to a new value at log time; but that will not affect the fact that limit_req was based on the value "default". > I would assume that in the log statement the values for $wup and $bad_guy are the same since they are based on maps with the same rules. No. http://nginx.org/r/map: """ Since variables are evaluated only when they are used, the mere declaration even of a large number of ?map? variables does not add any extra costs to request processing. """ > Let's change the configuration a little bit and comment out the line "limit_req zone=by_status burst=2;" > > Scenario 3: Upstream reports 403 > Log: > 127.0.0.1 - 403 ->127.0.0.1<- ->127.0.0.1<- > > Ok, now the values seem all right. Same analysis as above, except in this config $bad_guy is not used before "log_format", so it has no value before logging, so "map" sets it based on the value of $status at logging time. > Final question: Why does using "limit_req" change the behaviour/outcome of the "map"? Timing. f -- Francis Daly francis at daoine.org From nginx-forum at forum.nginx.org Mon Oct 14 00:47:16 2019 From: nginx-forum at forum.nginx.org (Ansuel) Date: Sun, 13 Oct 2019 20:47:16 -0400 Subject: read request body with http2 Message-ID: <52db6c96fdd8b8be5d205652b32ee2bb.NginxMailingListEnglish@forum.nginx.org> Hello, i'm developing a custom module that needs to read the request body to get some data from it. To read the request body i'm using ngx_http_read_client_request_body and in the callback i use for (in = r->request_body->bufs; in; in = in->next) { len = ngx_buf_size(in->buf); ngx_memcpy(buffer + pos,in->buf->pos,len); pos += len; } To put the data in a char buffer Aside from the fact that put request body in a buffer is wrong because it can became very big... I notice that if i enable http2 connection, the request body doesn't get read at all.... I notice the function read_client_request_body is never called... Am i missing something or there are some specail way to wait/read the request body with http2? By disabling it (remove support from nginx.conf) the module works as it should and i can correctly read the request body. I really hope this can be solved as i don't want to remove http2 support for this... Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285859,285859#msg-285859 From nginx-forum at forum.nginx.org Mon Oct 14 06:56:54 2019 From: nginx-forum at forum.nginx.org (Hett) Date: Mon, 14 Oct 2019 02:56:54 -0400 Subject: max_ranges has no effect Message-ID: Hello, I want to disable resumable downloads. Added max_ranges directive: location ~ ^(/[^/]+/[^/]+/[^/]+) { set $zone_limit_key $binary_remote_addr$host; limit_req zone=req_free burst=10 nodelay; limit_conn conn_free 1; max_ranges 0; rewrite ^/([^/]+)/([^/]+)/([^/]+)$ /v1/AUTH_system/$1/$2/$3?type=free break; keepalive_requests 0; proxy_pass http://10.10.100.110:8080; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_intercept_errors on; } But it has no effect hett at hett-pc: $ curl -v -r 100-102 "https://......." 2>&1 > /dev/null | grep "<" < HTTP/1.1 206 Partial Content < Server: nginx < Date: Mon, 14 Oct 2019 06:41:13 GMT < Content-Type: application/octet-stream < Content-Length: 3 < Connection: close < Accept-Ranges: bytes < Last-Modified: Tue, 20 Aug 2019 12:16:22 GMT < Content-Range: bytes 100-102/466391882 Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285861,285861#msg-285861 From pluknet at nginx.com Mon Oct 14 11:39:30 2019 From: pluknet at nginx.com (Sergey Kandaurov) Date: Mon, 14 Oct 2019 14:39:30 +0300 Subject: max_ranges has no effect In-Reply-To: References: Message-ID: > On 14 Oct 2019, at 09:56, Hett wrote: > > Hello, > I want to disable resumable downloads. > Added max_ranges directive: > > > location ~ ^(/[^/]+/[^/]+/[^/]+) { > set $zone_limit_key $binary_remote_addr$host; > limit_req zone=req_free burst=10 nodelay; > limit_conn conn_free 1; > max_ranges 0; > rewrite ^/([^/]+)/([^/]+)/([^/]+)$ > /v1/AUTH_system/$1/$2/$3?type=free break; > keepalive_requests 0; > proxy_pass http://10.10.100.110:8080; > proxy_set_header Host $host; > proxy_set_header X-Real-IP $remote_addr; > proxy_set_header X-Forwarded-For $remote_addr; > proxy_intercept_errors on; > } > > But it has no effect This is because the range request is processed by the proxied server, and then nginx has nothing to do with the response. Instead, you may want to disable range processing on backend, by removing the Range request header while proxying requests: proxy_set_header Range ""; -- Sergey Kandaurov From mdounin at mdounin.ru Mon Oct 14 14:25:58 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 14 Oct 2019 17:25:58 +0300 Subject: read request body with http2 In-Reply-To: <52db6c96fdd8b8be5d205652b32ee2bb.NginxMailingListEnglish@forum.nginx.org> References: <52db6c96fdd8b8be5d205652b32ee2bb.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191014142558.GM1877@mdounin.ru> Hello! On Sun, Oct 13, 2019 at 08:47:16PM -0400, Ansuel wrote: > Hello, i'm developing a custom module that needs to read the request body to > get some data from it. > To read the request body i'm using > > ngx_http_read_client_request_body > > and in the callback i use > > for (in = r->request_body->bufs; in; in = in->next) { > len = ngx_buf_size(in->buf); > ngx_memcpy(buffer + pos,in->buf->pos,len); > pos += len; > } > > To put the data in a char buffer Note that unless you've specifically tuned client_body_buffer_size and client_max_body_size, there can be in-file buffers in r->request_body->bufs. > Aside from the fact that put request body in a buffer is wrong because it > can became very big... > > I notice that if i enable http2 connection, the request body doesn't get > read at all.... I notice the function read_client_request_body is never > called... > Am i missing something or there are some specail way to wait/read the > request body with http2? By disabling it (remove support from nginx.conf) > the module works as it should and i can correctly read the request body. There is no "read_client_request_body" function in nginx. If that's how you've called your handler function you pass into ngx_http_read_client_request_body(), and it's never called - probably there is something wrong with how you handle things. Consider looking at the examples at the examples at the Developer's Guide, probably you'll be able to find out what you've did wrong, and/or will be able to reimplement it correctly from scratch: http://nginx.org/en/docs/dev/development_guide.html#http_request_body > I really hope this can be solved as i don't want to remove http2 support for > this... Certainly reading request body works fine with HTTP/2, and it is used in many standard modules, including proxy. -- Maxim Dounin http://mdounin.ru/ From nginx-forum at forum.nginx.org Mon Oct 14 18:41:33 2019 From: nginx-forum at forum.nginx.org (Ansuel) Date: Mon, 14 Oct 2019 14:41:33 -0400 Subject: read request body with http2 In-Reply-To: <20191014142558.GM1877@mdounin.ru> References: <20191014142558.GM1877@mdounin.ru> Message-ID: this is what i have in the module handler function rc = ngx_http_read_client_request_body(r, ngx_http_test_read_req); if (rc != NGX_OK && rc != NGX_AGAIN) { return rc; } And this is what i have in ngx_http_test_read_req char *buffer = ngx_pcalloc(r->pool, cglcf->req_len); for (in = r->request_body->bufs; in; in = in->next) { len = ngx_buf_size(in->buf); ngx_memcpy(buffer + pos,in->buf->pos,len); pos += len; } Do you see anything wrong in how i access the request body? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285859,285870#msg-285870 From nginx-forum at forum.nginx.org Tue Oct 15 03:04:30 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Mon, 14 Oct 2019 23:04:30 -0400 Subject: Trailing Slash Redirect Loop Help In-Reply-To: <20191009213308.GO1914@daoine.org> References: <20191009213308.GO1914@daoine.org> Message-ID: <66f7c73102a2e96f21e63ebb3adf4aee.NginxMailingListEnglish@forum.nginx.org> Dear Francis: What I am perceiving from your answers is that if nginx can not know with (!-d or the try_files) that the uri is a directory or a file. Is there a way to tell nginx that if it gets a "/" from the upstream to leave it the way it is? I guess I will give up the idea of having all urls without a trailing slash. Regards, Alex P.S. I will look into having the APP remove the trailing slash since it should know better what it is a directory or a file. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273964,285872#msg-285872 From nginx-forum at forum.nginx.org Tue Oct 15 03:06:02 2019 From: nginx-forum at forum.nginx.org (Alex Med) Date: Mon, 14 Oct 2019 23:06:02 -0400 Subject: Trailing Slash Redirect Loop Help In-Reply-To: <66f7c73102a2e96f21e63ebb3adf4aee.NginxMailingListEnglish@forum.nginx.org> References: <20191009213308.GO1914@daoine.org> <66f7c73102a2e96f21e63ebb3adf4aee.NginxMailingListEnglish@forum.nginx.org> Message-ID: Dear Francis: What I am perceiving from your answers is that if nginx can not know with (!-d or the try_files) that the uri is a directory or a file, there is not much it can do about not removing slashes from the uris it gets. Is there a way to tell nginx that if it gets a "/" from the upstream to leave it the way it is? I guess I will give up the idea of having all urls without a trailing slash. Regards, Alex P.S. I will look into having the APP remove the trailing slash since it should know better what it is a directory or a file. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273964,285873#msg-285873 From nginx-forum at forum.nginx.org Tue Oct 15 03:07:43 2019 From: nginx-forum at forum.nginx.org (tiendungitd) Date: Mon, 14 Oct 2019 23:07:43 -0400 Subject: IOS keep asking password with nginx auth_basic Message-ID: Hi there, I'm using nginx auth_basic "Restricted area" to protect website. So users have to enter user/pass one time to access, it work fine in computer. However, When testing with Ipad, everytime access to particular page which page extension is .ivp, it will require a credential, once I submit user/password, I can continue to work in this website. I think my issue similar to https://stackoverflow.com/questions/6178507/mp4-in-safari-fails-with-htaccess-authentication, but I did not using .htaccess to try with the solution mentioned in this topic. As you can see in access log https://pastebin.com/6vWhtb3Y, when I submited a POST request, then browser will have GET request to ivp page, and got a 401 code which is require to enter user/password. Any idea of what might be causing this? Thanks. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285874,285874#msg-285874 From nginx-forum at forum.nginx.org Tue Oct 15 05:34:35 2019 From: nginx-forum at forum.nginx.org (Hett) Date: Tue, 15 Oct 2019 01:34:35 -0400 Subject: max_ranges has no effect In-Reply-To: References: Message-ID: <623ff6c35e0fd8299f82325a3ad943a3.NginxMailingListEnglish@forum.nginx.org> Thanks, I thought so, and I thought nginx will delete this header via `max_ranges` directive. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285861,285875#msg-285875 From nginx-forum at forum.nginx.org Tue Oct 15 05:43:04 2019 From: nginx-forum at forum.nginx.org (Hett) Date: Tue, 15 Oct 2019 01:43:04 -0400 Subject: max_ranges has no effect In-Reply-To: <623ff6c35e0fd8299f82325a3ad943a3.NginxMailingListEnglish@forum.nginx.org> References: <623ff6c35e0fd8299f82325a3ad943a3.NginxMailingListEnglish@forum.nginx.org> Message-ID: <8b13a4c86096a5e04a556e352275ea3c.NginxMailingListEnglish@forum.nginx.org> Next headers resolve my issue proxy_set_header Range ""; proxy_hide_header Accept-Ranges; add_header Accept-Ranges none; Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285861,285876#msg-285876 From danny at trisect.uk Tue Oct 15 12:40:51 2019 From: danny at trisect.uk (Danny Horne) Date: Tue, 15 Oct 2019 13:40:51 +0100 Subject: Problems with redirects Message-ID: <2aca6525-d7ae-a50d-1654-437b45c24135@trisect.uk> Hi all, I've been struggling to get redirects working the way I'd like, basically I want - http://trisect.uk > https://trisect.uk http://site1.trisect.uk > https://site1.trisect.uk http://site2.trisect.uk > https://site2.trisect.uk etc.... This is the server block doing the redirects - server { ??????? listen *:80; ??????? listen [::]:80; ??????? server_name trisect.uk *.trisect.uk; ??????? return 301 https://$server_name$request_uri; } I have a wildcard Let's Encrypt certificate and am using NginX 1.14.2 Hope someone can help, thanks for looking From nginx-forum at forum.nginx.org Tue Oct 15 12:56:15 2019 From: nginx-forum at forum.nginx.org (j94305) Date: Tue, 15 Oct 2019 08:56:15 -0400 Subject: Problems with redirects In-Reply-To: <2aca6525-d7ae-a50d-1654-437b45c24135@trisect.uk> References: <2aca6525-d7ae-a50d-1654-437b45c24135@trisect.uk> Message-ID: <1ac8ec442a9dee351c6b67d34ea14efb.NginxMailingListEnglish@forum.nginx.org> Hi Danny, two comments: 1) Don't forget about $is_args$args to also pass any arguments supplied with the URL. 2) You cannot redirect requests with a request body, most importantly POST and PUT, so your rule is only applicable for GET/HEAD requests. I have no idea what you are really struggling with as you don't mention the actual challenge. Regarding your server block: - "listen 80 default_server;" is sufficient. - You can specify server names, but your intention is to forward everything, so you may as well omit it or write a dummy such as "server_name _;" - You may want to add a different redirection if the request method is not GET (or HEAD), so users will end up on the home page or somewhere else meaningful and not have to face a funny error message the won't understand, because a POST/PUT request did not specify a body. Regarding the https service: - If you use one IP address to serve multiple domains, get a SAN certificate with all domain names added. This way, you can avoid using SNI. Cheers, --j. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285882,285884#msg-285884 From danny at trisect.uk Tue Oct 15 13:00:37 2019 From: danny at trisect.uk (Danny Horne) Date: Tue, 15 Oct 2019 14:00:37 +0100 Subject: Problems with redirects In-Reply-To: <1ac8ec442a9dee351c6b67d34ea14efb.NginxMailingListEnglish@forum.nginx.org> References: <2aca6525-d7ae-a50d-1654-437b45c24135@trisect.uk> <1ac8ec442a9dee351c6b67d34ea14efb.NginxMailingListEnglish@forum.nginx.org> Message-ID: On 15/10/2019 13:56, j94305 wrote: > I have no idea what you are really struggling with as you don't mention the > actual challenge. > The problem is, whatever URL I put in the browser, it redirects to https://trisect.uk From r at roze.lv Tue Oct 15 13:08:19 2019 From: r at roze.lv (Reinis Rozitis) Date: Tue, 15 Oct 2019 16:08:19 +0300 Subject: Problems with redirects In-Reply-To: References: <2aca6525-d7ae-a50d-1654-437b45c24135@trisect.uk> <1ac8ec442a9dee351c6b67d34ea14efb.NginxMailingListEnglish@forum.nginx.org> Message-ID: <001f01d58359$9d82a310$d887e930$@roze.lv> > The problem is, whatever URL I put in the browser, it redirects to > https://trisect.uk _______________________________________________ > server_name trisect.uk *.trisect.uk; > return 301 https://$server_name$request_uri; } For that I don't think you can use $server_name here because it will always be the first one picked no matter if the client passes site1.trisect.uk in Host header. Try with return 301 https://$host$request_uri; rr From danny at trisect.uk Tue Oct 15 14:00:08 2019 From: danny at trisect.uk (Danny Horne) Date: Tue, 15 Oct 2019 15:00:08 +0100 Subject: Problems with redirects In-Reply-To: <001f01d58359$9d82a310$d887e930$@roze.lv> References: <2aca6525-d7ae-a50d-1654-437b45c24135@trisect.uk> <1ac8ec442a9dee351c6b67d34ea14efb.NginxMailingListEnglish@forum.nginx.org> <001f01d58359$9d82a310$d887e930$@roze.lv> Message-ID: <25a9c0bc-9ea4-75f3-946d-5282e283b8fb@trisect.uk> On 15/10/2019 14:08, Reinis Rozitis wrote: >> The problem is, whatever URL I put in the browser, it redirects to >> https://trisect.uk _______________________________________________ >> server_name trisect.uk *.trisect.uk; >> return 301 https://$server_name$request_uri; } > For that I don't think you can use $server_name here because it will always be the first one picked no matter if the client passes site1.trisect.uk in Host header. > > Try with return 301 https://$host$request_uri; > > rr Ok, I think I'm there with the following server block, I think the main problem was that I'd shot myself in the foot by using 301 redirects, which Chrome was caching, testing with another browser showed everything working as desired.? Clearing the Chrome redirect cache has (I think) fixed it. server { ??????? listen *:80; ??????? listen [::]:80; ??????? server_name trisect.uk *.trisect.uk; ??????? return 301 https:/$host$request_uri; } Thanks for all replies From francis at daoine.org Tue Oct 15 14:34:09 2019 From: francis at daoine.org (Francis Daly) Date: Tue, 15 Oct 2019 15:34:09 +0100 Subject: Trailing Slash Redirect Loop Help In-Reply-To: References: <20191009213308.GO1914@daoine.org> <66f7c73102a2e96f21e63ebb3adf4aee.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191015143409.GU1914@daoine.org> On Mon, Oct 14, 2019 at 11:06:02PM -0400, Alex Med wrote: Hi there, > What I am perceiving from your answers is that if nginx can not know with > (!-d or the try_files) that the uri is a directory or a file, there is not > much it can do about not removing slashes from the uris it gets. Correct. More specifically: there is not much nginx can do about removing slashes based on whether the uri corresponds to a file or directory on a server that nginx cannot read. > Is there a way to tell nginx that if it gets a "/" from the upstream to > leave it the way it is? I don't understand the question. nginx will usually leave things the way they are, unless configured otherwise. (Aside: some default configuration options are "do change things".) If it still matters, can you describe a timeline of what you would like to have happen? Something like: * client requests /thing/ * nginx (does something, perhaps involving the filesystem or a proxy_pass)? * nginx responds http 301 to /thing, or responds 200 with some content. Or maybe responds http 301 to /thing/index? but include all the words at step 2 that will make it very clear what your desired end result is. > I guess I will give up the idea of having all urls > without a trailing slash. You can do "no trailing slash" if you want to; but you will have to configure all of your web servers to do what you want them to when they get a "trailing slash" url. I don't think that I know what you want them to do when they get a "trailing slash" url. Good luck with it, f -- Francis Daly francis at daoine.org From mdounin at mdounin.ru Tue Oct 15 14:56:05 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 15 Oct 2019 17:56:05 +0300 Subject: read request body with http2 In-Reply-To: References: <20191014142558.GM1877@mdounin.ru> Message-ID: <20191015145605.GT1877@mdounin.ru> Hello! On Mon, Oct 14, 2019 at 02:41:33PM -0400, Ansuel wrote: > this is what i have in the module handler function > > rc = ngx_http_read_client_request_body(r, ngx_http_test_read_req); > if (rc != NGX_OK && rc != NGX_AGAIN) { > return rc; > } The snippet provided is not enough to conclude if the handling is completely wrong and going to cause problems, but this at least differs from the proper pattern, and it is going to cause problems if not followed by "return NGX_DONE;", assuming the code is used in the content phase. Proper pattern is outlined in the development guide, (http://nginx.org/en/docs/dev/development_guide.html#http_request_body): rc = ngx_http_read_client_request_body(r, ngx_http_foo_init); if (rc >= NGX_HTTP_SPECIAL_RESPONSE) { return rc; } return NGX_DONE; The same pattern can be seen in all nginx modules calling ngx_http_read_client_request_body(). > And this is what i have in > ngx_http_test_read_req > > char *buffer = ngx_pcalloc(r->pool, cglcf->req_len); > > for (in = r->request_body->bufs; in; in = in->next) { > len = ngx_buf_size(in->buf); > ngx_memcpy(buffer + pos,in->buf->pos,len); > pos += len; > } > > > Do you see anything wrong in how i access the request body? Sure, see above. Further, I already wrote that assuming buffers are in memory is wrong unless you've specifically tuned configuration parameters. Note well that your code seems to assume that total request body size is less than cglcf->req_len, which is never checked. This can easily cause buffer overflow if the request body is actually bigger. -- Maxim Dounin http://mdounin.ru/ From nginx-forum at forum.nginx.org Wed Oct 16 09:21:41 2019 From: nginx-forum at forum.nginx.org (sachin.shetty@gmail.com) Date: Wed, 16 Oct 2019 05:21:41 -0400 Subject: limit_rate_after does not work inside location block Message-ID: <44acb522adb8f0b0afa42ec392cd340e.NginxMailingListEnglish@forum.nginx.org> Hi, I have a location block location ~ /get_file$ { limit_rate_after 500m; limit_rate 1m; ... ... } The limit_rate_after does not work when put inside the location block, if I move it right above the location line i.e. inside server block, it works. Any idea on how to make it work inside location and if block. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285895,285895#msg-285895 From nginx-forum at forum.nginx.org Wed Oct 16 09:24:01 2019 From: nginx-forum at forum.nginx.org (sachin.shetty@gmail.com) Date: Wed, 16 Oct 2019 05:24:01 -0400 Subject: Nginx proxy cache purge process does not clean up items fast enough for new elements Message-ID: <83461c5956ccc0aaace10735234f0861.NginxMailingListEnglish@forum.nginx.org> Hi, We have a nginx fronting our object storage which caches large objects. Objects are as large as 100GB. The nginx cache max size is set to about 3.5TB. When there is a surge of large object requests and disk quickly fills up, nginx runs into out of disk space error. I was expecting the cache manager to purge items based on LRU and make room for the new elements, but that does not happen. I can reproduce the problem with a simple test case: Config: proxy_cache_path /tmp/cache levels=1:2 keys_zone=cache_one:256m inactive=2d max_size=16G use_temp_path=off; Test: Run a request to download a file of 15GB, it is served correctly and stored in cache. Run a second request to download a different file of 10GB, it will fail with something like this: 2019/10/04 11:49:08 [crit] 20206#20206: *21 pwritev() "/tmp/cache/9/fa/a301d42ca6e5d4188c38ecf56aa3afa9.0000000001" has written only 221184 of 229376 while reading upstream, client: 127.0.0.1, server: eos_cache_filer, request: "GET... 2019/10/04 12:07:29 [crit] 21201#21201: *487 pwrite() "/tmp/cache/9/fa/a301d42ca6e5d4188c38ecf56aa3afa9.0000000002" failed (28: No space left on device) while reading upstream, client: 127.0.0.1, server: eos_cache_filer, request: Can I tune some cache_manager parameters to make this work? Is there a way to disable buffering in such case - ideally download should not fail, it should just disable caching and buffering. Thanks Sachin Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285896,285896#msg-285896 From nginx-forum at forum.nginx.org Wed Oct 16 09:28:38 2019 From: nginx-forum at forum.nginx.org (fenglin) Date: Wed, 16 Oct 2019 05:28:38 -0400 Subject: appEngine nginx not able to redirect to index.html when the url contains "-" dash Message-ID: hi , I have a problem with gcp AppEngine nginx. I am using the AppEngine flex env for running my angular application. My application use ng-route to control the flows. I have a url of the following format: /modelview/ in which p2 might contain dash "-" in the url. I have default appEngine nginx configuraiton. which I believe will return the route to index.html which is the entry. Now I discover if the p2 contains dashes "-" e.g., AB-cd-1242, the page returns a whiteLabel error page 404. if it is a string without "-" it no problem. I have tried the following ways: add Nginx-app.conf file with the following configure: location ^~ /modelview/ { try_files $uri $uri/ /index.html; } add extra handler in app.yaml file with the following configure: handlers: - url: /.* secure: always script: index.html - url: /modelview/.* secure: always script: index.html non of them can solve the problem. Is there any mysterious reasons for nginx not able to correctly redirect url with post fix containing "-"? We can make a walk around like adding extra post fix. e.g.,/modelview//view can solve the problem. I just want to figure out why? can any help to this? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285897,285897#msg-285897 From mdounin at mdounin.ru Wed Oct 16 12:39:50 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 16 Oct 2019 15:39:50 +0300 Subject: limit_rate_after does not work inside location block In-Reply-To: <44acb522adb8f0b0afa42ec392cd340e.NginxMailingListEnglish@forum.nginx.org> References: <44acb522adb8f0b0afa42ec392cd340e.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191016123950.GU1877@mdounin.ru> Hello! On Wed, Oct 16, 2019 at 05:21:41AM -0400, sachin.shetty at gmail.com wrote: > I have a location block > > location ~ /get_file$ { > limit_rate_after 500m; > limit_rate 1m; > ... > ... > } > > The limit_rate_after does not work when put inside the location block, if I > move it right above the location line i.e. inside server block, it works. > > Any idea on how to make it work inside location and if block. It does work, as long as the response is returned within the location in question. If it doesn't work, likely the response is returned elsewhere. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Wed Oct 16 12:56:49 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 16 Oct 2019 15:56:49 +0300 Subject: Nginx proxy cache purge process does not clean up items fast enough for new elements In-Reply-To: <83461c5956ccc0aaace10735234f0861.NginxMailingListEnglish@forum.nginx.org> References: <83461c5956ccc0aaace10735234f0861.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191016125649.GV1877@mdounin.ru> Hello! On Wed, Oct 16, 2019 at 05:24:01AM -0400, sachin.shetty at gmail.com wrote: > We have a nginx fronting our object storage which caches large objects. > Objects are as large as 100GB. The nginx cache max size is set to about > 3.5TB. > > When there is a surge of large object requests and disk quickly fills up, > nginx runs into out of disk space error. I was expecting the cache manager > to purge items based on LRU and make room for the new elements, but that > does not happen. > > I can reproduce the problem with a simple test case: > > Config: > > proxy_cache_path /tmp/cache levels=1:2 keys_zone=cache_one:256m inactive=2d > max_size=16G use_temp_path=off; > > Test: > > Run a request to download a file of 15GB, it is served correctly and > stored in cache. > Run a second request to download a different file of 10GB, it will fail > with something like this: > > 2019/10/04 11:49:08 [crit] 20206#20206: *21 pwritev() > "/tmp/cache/9/fa/a301d42ca6e5d4188c38ecf56aa3afa9.0000000001" has written > only 221184 of 229376 while reading upstream, client: 127.0.0.1, server: > eos_cache_filer, request: "GET... > 2019/10/04 12:07:29 [crit] 21201#21201: *487 pwrite() > "/tmp/cache/9/fa/a301d42ca6e5d4188c38ecf56aa3afa9.0000000002" failed (28: No > space left on device) while reading upstream, client: 127.0.0.1, server: > eos_cache_filer, request: > > Can I tune some cache_manager parameters to make this work? Is there a way > to disable buffering in such case - ideally download should not fail, it > should just disable caching and buffering. Cache manager works in parallel to worker processes which fill up cache. Further, with "max_size=" it only starts to clean things once max_size limit is reached. Hence it is possible that total size of the cache will be larger than max_size configured. It is recommended to keep max_size smaller than actual disk space available, and maintain the difference large enough for at least 10 seconds of filling up cache (10 seconds is how long cache manager will sleep if it has nothing to do), preferably more. In particular, the difference is expected to be larger than maximum size of a single cache item, or it is possible that adding one cache item will fail if max_size limit is not yet reached. This is probably what happens in your case. Note well that temporary files, regardless of whether you use "use_temp_path=off" or not, are not included into cache size. You have to reserve some space for temporary files as well. -- Maxim Dounin http://mdounin.ru/ From nginx-forum at forum.nginx.org Wed Oct 16 13:44:08 2019 From: nginx-forum at forum.nginx.org (sachin.shetty@gmail.com) Date: Wed, 16 Oct 2019 09:44:08 -0400 Subject: Nginx proxy cache purge process does not clean up items fast enough for new elements In-Reply-To: <20191016125649.GV1877@mdounin.ru> References: <20191016125649.GV1877@mdounin.ru> Message-ID: <1e48fd8fcf91f6828af280f8c9cad73d.NginxMailingListEnglish@forum.nginx.org> Thankyou Maxim, is there anyway I can make the cache manager a bit more aggressive in prune and purge? We already leave 20% of space free on the disks, but the concurrent request rate for large files can be huge and we still run in to this issue. What are your thoughts about disabling buffering on such issues? This is not a fatal error, so we should stop buffering and switch to streaming mode and let the request succeed with a error log line in error.log. Thanks again for your help on this. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285896,285900#msg-285900 From nginx-forum at forum.nginx.org Wed Oct 16 13:50:34 2019 From: nginx-forum at forum.nginx.org (sachin.shetty@gmail.com) Date: Wed, 16 Oct 2019 09:50:34 -0400 Subject: limit_rate_after does not work inside location block In-Reply-To: <20191016123950.GU1877@mdounin.ru> References: <20191016123950.GU1877@mdounin.ru> Message-ID: <8c6bcee7ba506b7b2ea36ceebd0a4f27.NginxMailingListEnglish@forum.nginx.org> Got it. You are right, response is returned by another block via X-Accel-Redirect. Thankyou! Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285895,285901#msg-285901 From nginx-forum at forum.nginx.org Wed Oct 16 13:55:40 2019 From: nginx-forum at forum.nginx.org (fenglin) Date: Wed, 16 Oct 2019 09:55:40 -0400 Subject: appEngine nginx not able to redirect to index.html when the url contains "-" dash In-Reply-To: References: Message-ID: forget about this, after investigation, I discover it is related with some angular "-" in the url parameter problem. angular does not allow "-" in url route parameter, it will cause 404 error. but it does not clearly say the value of the parameter contains problem, and if we add a extra postfix, the problem can be walk around. it is not related with nginx. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285897,285902#msg-285902 From hungnv at opensource.com.vn Wed Oct 16 13:56:17 2019 From: hungnv at opensource.com.vn (Hung Nguyen) Date: Wed, 16 Oct 2019 20:56:17 +0700 Subject: Nginx proxy cache purge process does not clean up items fast enough for new elements In-Reply-To: <1e48fd8fcf91f6828af280f8c9cad73d.NginxMailingListEnglish@forum.nginx.org> References: <1e48fd8fcf91f6828af280f8c9cad73d.NginxMailingListEnglish@forum.nginx.org> Message-ID: <669EC162-AC32-4064-941D-D404E8219631@opensource.com.vn> Hello, I Dont think the way you currently use nginx as cache proxy is best practice. Serving large file then store whole file into cache with large number of request is like burning your disk, even if nginx cache manager can delete and refill cache fast enough, it will keep write/delete file infinitely. Another option should be using nginx slice module, and serving large file using range request. Nginx cache will then work far better. -- H?ng > On Oct 16, 2019, at 20:44, sachin.shetty at gmail.com wrote: > > ?Thankyou Maxim, is there anyway I can make the cache manager a bit more > aggressive in prune and purge? We already leave 20% of space free on the > disks, but the concurrent request rate for large files can be huge and we > still run in to this issue. > > What are your thoughts about disabling buffering on such issues? This is not > a fatal error, so we should stop buffering and switch to streaming mode and > let the request succeed with a error log line in error.log. > > Thanks again for your help on this. > > Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285896,285900#msg-285900 > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From mdounin at mdounin.ru Wed Oct 16 16:48:38 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 16 Oct 2019 19:48:38 +0300 Subject: Nginx proxy cache purge process does not clean up items fast enough for new elements In-Reply-To: <1e48fd8fcf91f6828af280f8c9cad73d.NginxMailingListEnglish@forum.nginx.org> References: <20191016125649.GV1877@mdounin.ru> <1e48fd8fcf91f6828af280f8c9cad73d.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191016164838.GZ1877@mdounin.ru> Hello! On Wed, Oct 16, 2019 at 09:44:08AM -0400, sachin.shetty at gmail.com wrote: > Thankyou Maxim, is there anyway I can make the cache manager a bit more > aggressive in prune and purge? We already leave 20% of space free on the > disks, but the concurrent request rate for large files can be huge and we > still run in to this issue. Most likely you are hitting space limits due to temporary files, which aren't managed by cache manager at all. My recommendation would be to consider proxy_cache_lock and/or "proxy_cache_use_stale updating" to reduce number of concurrent requests trying to cache the same files, see here: http://nginx.org/r/proxy_cache_lock http://nginx.org/r/proxy_cache_use_stale Also you may want to tune things like proxy_cache_min_uses to reduce unneeded caching on sporadic requests. > What are your thoughts about disabling buffering on such issues? This is not > a fatal error, so we should stop buffering and switch to streaming mode and > let the request succeed with a error log line in error.log. Yes, in theory. But the question is how complex and error prone the resulting code would be. Hence the current choice is to treat such errors as fatal. -- Maxim Dounin http://mdounin.ru/ From nginx-forum at forum.nginx.org Thu Oct 17 07:44:59 2019 From: nginx-forum at forum.nginx.org (tiendungitd) Date: Thu, 17 Oct 2019 03:44:59 -0400 Subject: IOS keep asking password with nginx auth_basic In-Reply-To: References: Message-ID: <2802589e4ca0f210ccbbd079caf7465f.NginxMailingListEnglish@forum.nginx.org> Hi everyone, any idea about this issue? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285874,285908#msg-285908 From nginx-forum at forum.nginx.org Thu Oct 17 08:16:03 2019 From: nginx-forum at forum.nginx.org (gaoyan09) Date: Thu, 17 Oct 2019 04:16:03 -0400 Subject: [njs] Would njs js_content support dynamic script? Message-ID: <97a8433ee4415149f4baab68bd797a8e.NginxMailingListEnglish@forum.nginx.org> Njs js_content can only call js function for now. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285909,285909#msg-285909 From nginx-forum at forum.nginx.org Thu Oct 17 12:35:58 2019 From: nginx-forum at forum.nginx.org (sachin.shetty@gmail.com) Date: Thu, 17 Oct 2019 08:35:58 -0400 Subject: Nginx proxy cache purge process does not clean up items fast enough for new elements In-Reply-To: <20191016164838.GZ1877@mdounin.ru> References: <20191016164838.GZ1877@mdounin.ru> Message-ID: Thankyou, we use proxy_cache_lock as well, but in certain weird burst scenarios, it still ends up filling the disk. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285896,285910#msg-285910 From mdounin at mdounin.ru Thu Oct 17 12:57:37 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 17 Oct 2019 15:57:37 +0300 Subject: Nginx proxy cache purge process does not clean up items fast enough for new elements In-Reply-To: References: <20191016164838.GZ1877@mdounin.ru> Message-ID: <20191017125737.GD1877@mdounin.ru> Hello! On Thu, Oct 17, 2019 at 08:35:58AM -0400, sachin.shetty at gmail.com wrote: > Thankyou, we use proxy_cache_lock as well, but in certain weird burst > scenarios, it still ends up filling the disk. There are two timeouts for proxy_cache_lock to tune, proxy_cache_lock_age and proxy_cache_lock_timeout: http://nginx.org/r/proxy_cache_lock_age http://nginx.org/r/proxy_cache_lock_timeout Defaults are small enough and certainly aren't optimal for 100-gigabyte files. -- Maxim Dounin http://mdounin.ru/ From jlmuir at imca-cat.org Thu Oct 17 15:05:15 2019 From: jlmuir at imca-cat.org (J. Lewis Muir) Date: Thu, 17 Oct 2019 10:05:15 -0500 Subject: IOS keep asking password with nginx auth_basic In-Reply-To: <2802589e4ca0f210ccbbd079caf7465f.NginxMailingListEnglish@forum.nginx.org> References: <2802589e4ca0f210ccbbd079caf7465f.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191017150515.4f6wlfn25j7gbyr6@mink.imca.aps.anl.gov> On 10/17, tiendungitd wrote: > any idea about this issue? Are you serving over HTTPS? Are you positive that you are *not* serving any content over HTTP from HTTPS? Can you create an MCVE (minimal, complete, and verifiable example) for a ".txt" file over HTTP? For a ".txt" file over HTTPS? For a ".ivp" file over HTTP? For a ".ivp" file over HTTPS? In your Pastebin access log, I see occurrences of "$1" in your requests. Is that intentional, or is that a bug? What Content-Type header field are you sending in your response to the request for a resource ending in ".ivp"? Can you try adding a user name and password for the website on the iPad under Settings > Passwords & Accounts > Website & App Passwords? What happens if you try Firefox and Google Chrome on the iPad? Do they work? Lewis From anthony at mindmedia.com.sg Thu Oct 17 17:01:25 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Fri, 18 Oct 2019 01:01:25 +0800 Subject: How to do location /test/place?id=2 Message-ID: Hi, Currently have the following url, https://old.example.com/test/place?id=1 https://old.example.com/test/place?id=2 https://old.example.com/test/place?id=3 Need to redirect only id=2 to another url. Did the following and it works for id=2. Need id=1 and id=3 to continue normally without change. location = /test/place { if ($args = "id=2") { return 301 https://new.example.com/test/place?$args; } } Or is there a way to do the following? That would be ideal. location = /test/place?id=2 { return 301 https://new.example.com/test/place?id=2 } Unfortunately the above does not work. What is missing? -- P.V.Anthony From themadbeaker at gmail.com Thu Oct 17 17:49:49 2019 From: themadbeaker at gmail.com (J.R.) Date: Thu, 17 Oct 2019 12:49:49 -0500 Subject: Typo in documentation... Message-ID: Wasn't sure how to contact the webmaster as there is only a sales contact form on the nginx website.... Would it kill them to have a basic feedback form or email address? I believe there's a typo on the following page: https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-http/ Specifically, under "Limiting the Bandwidth", last box there is the following: location /download/ { limit_conn addr 1; limit_rate 1m; limit_rate 50k; } I believe the first "limit_rate" is actually supposed to be "limit_rate_after"... From yar at nginx.com Thu Oct 17 18:21:21 2019 From: yar at nginx.com (Yaroslav Zhuravlev) Date: Thu, 17 Oct 2019 21:21:21 +0300 Subject: Typo in documentation... In-Reply-To: References: Message-ID: <7F1B9110-0655-4146-8DCC-83EDD181D3F7@nginx.com> Hello! > On 17 Oct 2019, at 20:49, J.R. wrote: > > Wasn't sure how to contact the webmaster as there is only a sales > contact form on the nginx website.... Would it kill them to have a > basic feedback form or email address? > > I believe there's a typo on the following page: > > https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-http/ > > Specifically, under "Limiting the Bandwidth", last box there is the following: > > location /download/ { > limit_conn addr 1; > limit_rate 1m; > limit_rate 50k; > } > > I believe the first "limit_rate" is actually supposed to be > "limit_rate_after"? Thanks for reporting this issue, example corrected, the fix will appear shortly on the website. > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From nginx-forum at forum.nginx.org Thu Oct 17 20:42:54 2019 From: nginx-forum at forum.nginx.org (Ansuel) Date: Thu, 17 Oct 2019 16:42:54 -0400 Subject: read request body with http2 In-Reply-To: <20191015145605.GT1877@mdounin.ru> References: <20191015145605.GT1877@mdounin.ru> Message-ID: <2992cc52beb999c6663bc573b8418783.NginxMailingListEnglish@forum.nginx.org> I don't know if this is a bug or not but... yes you were right... All the work should be done in the ngx_http_read_client_request_body handler, as it does duplicate the r connection to another address What confused me is the fact that this is only done in http2 connections... I really don't know why but all the changes i do in the read_client_request handler reflect to the main ngx_http_request_t struct. I thought the code was cleaner by having the body read in a function and parse the data from it in another function. Well this doesn't work with http2 because all the changes done in read_client_request handler get lost and doesn't apply to the main struct... Think this is a bug as the behavior should be similar and not change across http versions... Sorry for the bad explanation... Also i think this should be noted in the wiki. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285859,285920#msg-285920 From jlmuir at imca-cat.org Thu Oct 17 21:15:13 2019 From: jlmuir at imca-cat.org (J. Lewis Muir) Date: Thu, 17 Oct 2019 16:15:13 -0500 Subject: How to do location /test/place?id=2 In-Reply-To: References: Message-ID: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> On 10/18, P.V.Anthony wrote: > Currently have the following url, > > https://old.example.com/test/place?id=1 > https://old.example.com/test/place?id=2 > https://old.example.com/test/place?id=3 > > Need to redirect only id=2 to another url. > > Did the following and it works for id=2. Need id=1 and id=3 to continue > normally without change. > > location = /test/place { > if ($args = "id=2") { > return 301 https://new.example.com/test/place?$args; > } > } You might want to use $arg_id here (i.e., the $arg_ variable for the argument). Otherwise, it won't work if any other arguments are given. > Or is there a way to do the following? That would be ideal. > > location = /test/place?id=2 { > return 301 https://new.example.com/test/place?id=2 > } I don't think that's allowed. > Unfortunately the above does not work. What is missing? What doesn't work? I would think your > location = /test/place { block would work, although not as shown, but I assume you just left out the part that normally handles the request. It would handle the requests for id=1 and id=3 as before, and it's just the id=2 case that gets redirected, right? Lewis From jeff.dyke at gmail.com Thu Oct 17 21:26:32 2019 From: jeff.dyke at gmail.com (Jeff Dyke) Date: Thu, 17 Oct 2019 17:26:32 -0400 Subject: How to do location /test/place?id=2 In-Reply-To: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> References: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> Message-ID: I know this is not an answer to your question, but it begs another, mainly due to the if statement. How many of these are you going to have? https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ You've likely considered this, but if not wanted to throw it out there. Even if you are moving domains, could you create a separate route that you could key off rather than arg_id? Random thoughts waiting for things to compile On Thu, Oct 17, 2019 at 5:15 PM J. Lewis Muir wrote: > On 10/18, P.V.Anthony wrote: > > Currently have the following url, > > > > https://old.example.com/test/place?id=1 > > https://old.example.com/test/place?id=2 > > https://old.example.com/test/place?id=3 > > > > Need to redirect only id=2 to another url. > > > > Did the following and it works for id=2. Need id=1 and id=3 to continue > > normally without change. > > > > location = /test/place { > > if ($args = "id=2") { > > return 301 https://new.example.com/test/place?$args; > > } > > } > > You might want to use $arg_id here (i.e., the $arg_ variable for > the argument). Otherwise, it won't work if any other arguments > are given. > > > Or is there a way to do the following? That would be ideal. > > > > location = /test/place?id=2 { > > return 301 https://new.example.com/test/place?id=2 > > } > > I don't think that's allowed. > > > Unfortunately the above does not work. What is missing? > > What doesn't work? > > I would think your > > > location = /test/place { > > block would work, although not as shown, but I assume you just left > out the part that normally handles the request. It would handle the > requests for id=1 and id=3 as before, and it's just the id=2 case that > gets redirected, right? > > Lewis > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Thu Oct 17 23:44:14 2019 From: nginx-forum at forum.nginx.org (ptcell) Date: Thu, 17 Oct 2019 19:44:14 -0400 Subject: ngx_http_catch_body_filter doesn't appear to work Message-ID: <1105b9e2345a8fb91ee07a8677a23c96.NginxMailingListEnglish@forum.nginx.org> I'm trying to use the ngx_http_catch_body_filter example to capture request bodies from post (etc.) requests. I started with the example here in the hg repo: https://www.nginx.com/resources/wiki/extending/examples/body_filter/ I changed the config to make it a dynamic nginx module. I can see the `ngx_http_catch_body_init()` method getting called in the logs (I put an error log in that method). When I try a proxied POST or file upload, `ngx_http_catch_body_filter()` never gets called. Here's my modified source code and config. What am I doing wrong? ---- nginx.conf ---- ... load_module modules/ngx_http_catch_body_filter_module.so; ... location /app1/ { catch_body on; proxy_pass http://localhost:8180/java_test_app/; } ---- config ---- # (C) Maxim Dounin # Configuration for ngx_http_catch_body_filter_module. ngx_addon_name="ngx_http_catch_body_filter_module" #HTTP_MODULES="$HTTP_MODULES \ # ngx_http_catch_body_filter_module" NGX_ADDON_SRCS="$NGX_ADDON_SRCS \ $ngx_addon_dir/ngx_http_catch_body_filter_module.c" ngx_module_type=HTTP_AUX_FILTER ngx_module_srcs=$NGX_ADDON_SRCS ngx_module_name=$ngx_addon_name . auto/module ---- ngx_http_catch_body_filter_module.c ---- /* * Copyright (C) Maxim Dounin */ #include #include #include #include typedef struct { ngx_flag_t enable; } ngx_http_catch_body_conf_t; static void *ngx_http_catch_body_create_conf(ngx_conf_t *cf); static char *ngx_http_catch_body_merge_conf(ngx_conf_t *cf, void *parent, void *child); static ngx_int_t ngx_http_catch_body_init(ngx_conf_t *cf); static ngx_command_t ngx_http_catch_body_commands[] = { {ngx_string("catch_body"), NGX_HTTP_MAIN_CONF | NGX_HTTP_SRV_CONF | NGX_HTTP_LOC_CONF | NGX_CONF_FLAG, ngx_conf_set_flag_slot, NGX_HTTP_LOC_CONF_OFFSET, offsetof(ngx_http_catch_body_conf_t, enable), NULL}, ngx_null_command }; static ngx_http_module_t ngx_http_catch_body_module_ctx = { NULL, /* preconfiguration */ ngx_http_catch_body_init, /* postconfiguration */ NULL, /* create main configuration */ NULL, /* init main configuration */ NULL, /* create server configuration */ NULL, /* merge server configuration */ ngx_http_catch_body_create_conf, /* create location configuration */ ngx_http_catch_body_merge_conf /* merge location configuration */ }; ngx_module_t ngx_http_catch_body_filter_module = { NGX_MODULE_V1, &ngx_http_catch_body_module_ctx, /* module context */ ngx_http_catch_body_commands, /* module directives */ NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ NULL, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING }; static ngx_http_request_body_filter_pt ngx_http_next_request_body_filter; static ngx_int_t ngx_http_catch_body_filter(ngx_http_request_t *r, ngx_chain_t *in) { ngx_log_error(NGX_LOG_NOTICE, r->connection->log, 0, "catch request body filter"); fprintf(stderr, "catch request body filter\n"); return ngx_http_next_request_body_filter(r, in); } static void * ngx_http_catch_body_create_conf(ngx_conf_t *cf) { ngx_http_catch_body_conf_t *conf; conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_catch_body_conf_t)); if (conf == NULL) { return NULL; } conf->enable = NGX_CONF_UNSET; return conf; } static char * ngx_http_catch_body_merge_conf(ngx_conf_t *cf, void *parent, void *child) { ngx_http_catch_body_conf_t *prev = parent; ngx_http_catch_body_conf_t *conf = child; ngx_conf_merge_value(conf->enable, prev->enable, 0); return NGX_CONF_OK; } static ngx_int_t ngx_http_catch_body_init(ngx_conf_t *cf) { ngx_log_error(NGX_LOG_NOTICE, cf->log, 0, "init catch request body filter"); ngx_http_next_request_body_filter = ngx_http_top_request_body_filter; ngx_http_top_request_body_filter = ngx_http_catch_body_filter; return NGX_OK; } Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285923,285923#msg-285923 From 201904-nginx at jslf.app Fri Oct 18 01:03:11 2019 From: 201904-nginx at jslf.app (Patrick) Date: Fri, 18 Oct 2019 09:03:11 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: References: Message-ID: <20191018010311.GA18308@haller.ws> On 2019-10-18 01:01, P.V.Anthony wrote: > Currently have the following url, > > https://old.example.com/test/place?id=1 > https://old.example.com/test/place?id=2 > https://old.example.com/test/place?id=3 > > Need to redirect only id=2 to another url. This looks like a gradual migration of content to a new server. Try using `rewrite' instead of `return'? map "$uri?$args" $is_new_site { /test/place?id=2 1; default 0; } server { ... if ( $is_new_site ) { rewrite ^ http://new.example.com/${uri}?${args}? last; } ... # all the locations go here } Patrick From jlmuir at imca-cat.org Fri Oct 18 02:07:38 2019 From: jlmuir at imca-cat.org (J. Lewis Muir) Date: Thu, 17 Oct 2019 21:07:38 -0500 Subject: Static content and Front Controller pattern under same base URI Message-ID: <20191018020738.nkjrfagtnhapgjln@mink.imca.aps.anl.gov> Hello, nginxers! What's the best way to server static content as well as dynamic content that uses the Front Controller pattern under the same base URI? I'm dealing with a web app partially written in PHP that expects to serve static content as well as dynamic PHP content, using the Front Controller pattern, both under the same base URI. It expects to serve static content if the file or directory exists. But if the file or directory does not exist, it expects request rewriting according to the Front Controller pattern like this: ---- Request Rewritten Request /foo/ /foo/index.php /foo/bar /foo/index.php/bar /foo/bar/baz /foo/index.php/bar/baz ---- The following config works correctly: ---- location = /foo { return 301 $uri/$is_args$args; } location /foo/ { alias /srv/www/localhost/app/foo/1.0.0/; index index.php; try_files $uri $uri/ @foo_front_controller; } location ~ ^/foo/.*?[^/]\.php(?:/|$) { error_page 418 = @foo_front_controller; return 418; } location @foo_front_controller { rewrite ^/foo(?!/index\.php)(/.+)$ /foo/index.php$1 last; root /srv/www/localhost/app/foo/1.0.0; fastcgi_split_path_info ^/foo(/.+?\.php)(/.*)?$; include php-fpm.conf; fastcgi_param REQUEST_URI $fastcgi_script_name$fastcgi_path_info$is_args$args; } ---- I'm not a big fan of the location that sets the 418 error_page to the @foo_front_controller named location, but I don't know of any other way to essentially do a "return @foo_front_controller". Is there a better way? The whole point of the @foo_front_controller named location is so that I can have one block that defines the Front Controller logic, and then use it in both the "/foo/" location block in the try_files directive as the fallback for when the file or directory does not exist and in the location block right below it for requests that are for .php files. Is this a good approach, or is there a better way? Example content and requests are shown below. Thank you! Lewis ---- $ ls -al /srv/www/localhost/app/foo/1.0.0 total 8 drwxr-xr-x 2 root root 38 Oct 17 17:04 . drwxr-xr-x 3 root root 19 Oct 17 16:37 .. -rw-r--r-- 1 root root 209 Oct 17 17:04 index.php -rw-r--r-- 1 root root 21 Oct 17 16:44 msg.txt $ cat /srv/www/localhost/app/foo/1.0.0/index.php hello, world hello, world

' . "\n"; ?> REQUEST_URI: ' . $_SERVER['REQUEST_URI'] . '

' . "\n"; ?> $ cat /srv/www/localhost/app/foo/1.0.0/msg.txt Use the Force, Luke. $ curl http://localhost/foo/ hello, world

hello, world

REQUEST_URI: /index.php

$ curl http://localhost/foo/index.php hello, world

hello, world

REQUEST_URI: /index.php

$ curl http://localhost/foo/bar hello, world

hello, world

REQUEST_URI: /index.php/bar

$ curl http://localhost/foo/bar/baz hello, world

hello, world

REQUEST_URI: /index.php/bar/baz

$ curl http://localhost/foo/msg.txt Use the Force, Luke. $ curl http://localhost/foo/msg-quux.txt hello, world

hello, world

REQUEST_URI: /index.php/msg-quux.txt

---- From anthony at mindmedia.com.sg Fri Oct 18 04:00:12 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Fri, 18 Oct 2019 12:00:12 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> References: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> Message-ID: On 18/10/19 5:15 am, J. Lewis Muir wrote: > You might want to use $arg_id here (i.e., the $arg_ variable for > the argument). Otherwise, it won't work if any other arguments > are given. Noted. I will change that. >> Or is there a way to do the following? That would be ideal. >> >> location = /test/place?id=2 { >> return 301 https://new.example.com/test/place?id=2 >> } > > I don't think that's allowed. Yah. Sad. >> Unfortunately the above does not work. What is missing? > > What doesn't work? I was hoping to get location = /test/place?id=2 { to work. Like you have mentioned, it is not allowed. > I would think your > >> location = /test/place { > > block would work, although not as shown, but I assume you just left > out the part that normally handles the request. It would handle the > requests for id=1 and id=3 as before, and it's just the id=2 case that > gets redirected, right? That is where I need help with. I need help with requests for id=1 and id=3. I need them to just pass through without any changes. I only need id=2 to be redirected and id=2 redirection is working. I cannot figure out how to pass through id=1 and id=3. With the follow block id=1 and id=3 gives a 404 error. location = /test/place { if ($args = "id=2") { return 301 https://new.example.com/test/place?$args; } } How to make id=1 and id=3 to work? P.V.Anthony From anthony at mindmedia.com.sg Fri Oct 18 04:02:51 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Fri, 18 Oct 2019 12:02:51 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: References: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> Message-ID: <8da93f3d-65eb-a21b-8a20-2eef00c10caf@mindmedia.com.sg> On 18/10/19 5:26 am, Jeff Dyke wrote: > I know this is not an answer to your question, but it begs another, > mainly due to the if statement.? How many of these are you going to > have? https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ > > You've likely considered this, but if not wanted to throw it out there. > Even if you are moving domains, could you create a separate route that > you could key off rather than arg_id? Oh no. I was not aware of the "if" danger. Thank you for bringing that up. I am not able to change the application. Is there another way without using "if"? P.V.Anthony From anthony at mindmedia.com.sg Fri Oct 18 04:12:02 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Fri, 18 Oct 2019 12:12:02 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <20191018010311.GA18308@haller.ws> References: <20191018010311.GA18308@haller.ws> Message-ID: <1c34f628-6231-1112-78a7-366b4553054c@mindmedia.com.sg> On 18/10/19 9:03 am, Patrick wrote: > This looks like a gradual migration of content to a new server. Try > using `rewrite' instead of `return'? > > > map "$uri?$args" $is_new_site { > /test/place?id=2 1; > default 0; > } > > server { > ... > > if ( $is_new_site ) { > rewrite ^ http://new.example.com/${uri}?${args}? last; > } > > ... > > # all the locations go here > } This looks cool. Thank you for sharing this solution. I am using ispconfig web control that only allows changes in the server block. I think map block is only allowed outside the server block. Is there some way of setting a variable in the server block so that I can use the config you have shared below? if ( $is_new_site ) { rewrite ^ http://new.example.com/${uri}?${args}? last; } P.V.Anthony From 201904-nginx at jslf.app Fri Oct 18 04:20:00 2019 From: 201904-nginx at jslf.app (Patrick) Date: Fri, 18 Oct 2019 12:20:00 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <1c34f628-6231-1112-78a7-366b4553054c@mindmedia.com.sg> References: <20191018010311.GA18308@haller.ws> <1c34f628-6231-1112-78a7-366b4553054c@mindmedia.com.sg> Message-ID: <20191018042000.GA28424@haller.ws> On 2019-10-18 12:12, P.V.Anthony wrote: > I am using ispconfig web control that only allows changes in the server > block. Without a map, try starting with: if ( $uri?$args = /test/place?id=2 ) { rewrite ^ http://new.example.com/${uri}?${args}? last; } Then as the site migration continues turn that `if' test into a regexp that will match the migrated components. Patrick From anthony at mindmedia.com.sg Fri Oct 18 05:17:52 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Fri, 18 Oct 2019 13:17:52 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <20191018042000.GA28424@haller.ws> References: <20191018010311.GA18308@haller.ws> <1c34f628-6231-1112-78a7-366b4553054c@mindmedia.com.sg> <20191018042000.GA28424@haller.ws> Message-ID: <2a318b7e-d289-7969-8aee-403f8edfe0a6@mindmedia.com.sg> On 18/10/19 12:20 pm, Patrick wrote: > Without a map, try starting with: > > if ( $uri?$args = /test/place?id=2 ) { > rewrite ^ http://new.example.com/${uri}?${args}? last; > } > > Then as the site migration continues turn that `if' test into a regexp > that will match the migrated components. I can get the following to work. if ( $uri = /test/place ) { rewrite ^ http://new.example.com${uri}?${args}? last; } and this also works. if ( $args = id=2 ) { rewrite ^ http://new.example.com${uri}?${args}? last; } Unfortunately I cannot get the suggested solution to work. Did I miss some setting or quotes? > if ( $uri?$args = /test/place?id=2 ) { > rewrite ^ http://new.example.com${uri}?${args}? last; > } Tried the following and it works but is it safe? if ( $request_uri = "/test/place?id=2" ) { rewrite ^ http://new.example.com${uri}?${args}? last; } P.V.Anthony From 201904-nginx at jslf.app Fri Oct 18 05:33:57 2019 From: 201904-nginx at jslf.app (Patrick) Date: Fri, 18 Oct 2019 13:33:57 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <2a318b7e-d289-7969-8aee-403f8edfe0a6@mindmedia.com.sg> References: <20191018010311.GA18308@haller.ws> <1c34f628-6231-1112-78a7-366b4553054c@mindmedia.com.sg> <20191018042000.GA28424@haller.ws> <2a318b7e-d289-7969-8aee-403f8edfe0a6@mindmedia.com.sg> Message-ID: <20191018053357.GA636@haller.ws> On 2019-10-18 13:17, P.V.Anthony wrote: > Tried the following and it works but is it safe? > > if ( $request_uri = "/test/place?id=2" ) { > rewrite ^ http://new.example.com${uri}?${args}? last; > } Awesome. By `safe' do you mean netsec, or memory-safe, or ? Patrick From anthony at mindmedia.com.sg Fri Oct 18 05:57:43 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Fri, 18 Oct 2019 13:57:43 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <20191018053357.GA636@haller.ws> References: <20191018010311.GA18308@haller.ws> <1c34f628-6231-1112-78a7-366b4553054c@mindmedia.com.sg> <20191018042000.GA28424@haller.ws> <2a318b7e-d289-7969-8aee-403f8edfe0a6@mindmedia.com.sg> <20191018053357.GA636@haller.ws> Message-ID: <0be97b6d-0f68-c38c-507b-905b0d0578e4@mindmedia.com.sg> On 18/10/19 1:33 pm, Patrick wrote: > Awesome. By `safe' do you mean netsec, or memory-safe, or ? Like netsec and using "if" in the config. I suppose for my case, I have to use "if". Thank you very much for helping to solve this. I was cracking my head on how to solve this. You really helped. Thank you again. P.V.Anthony From 201904-nginx at jslf.app Fri Oct 18 06:59:44 2019 From: 201904-nginx at jslf.app (Patrick) Date: Fri, 18 Oct 2019 14:59:44 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <0be97b6d-0f68-c38c-507b-905b0d0578e4@mindmedia.com.sg> References: <20191018010311.GA18308@haller.ws> <1c34f628-6231-1112-78a7-366b4553054c@mindmedia.com.sg> <20191018042000.GA28424@haller.ws> <2a318b7e-d289-7969-8aee-403f8edfe0a6@mindmedia.com.sg> <20191018053357.GA636@haller.ws> <0be97b6d-0f68-c38c-507b-905b0d0578e4@mindmedia.com.sg> Message-ID: <20191018065944.GA6543@haller.ws> On 2019-10-18 13:57, P.V.Anthony wrote: > Like netsec and using "if" in the config. The `if' part should be fine. The problem would be someone crafting a URL that skips past the `=' check and yet is still parsed as `id=2' by the underlying app. Can the underlying old app also be changed to log an attack, and throw a 444 when it gets an `id' set to 2? Patrick From nginx-forum at forum.nginx.org Fri Oct 18 10:01:56 2019 From: nginx-forum at forum.nginx.org (tiendungitd) Date: Fri, 18 Oct 2019 06:01:56 -0400 Subject: IOS keep asking password with nginx auth_basic In-Reply-To: <20191017150515.4f6wlfn25j7gbyr6@mink.imca.aps.anl.gov> References: <20191017150515.4f6wlfn25j7gbyr6@mink.imca.aps.anl.gov> Message-ID: <8d08ff99a7ecad55a5fdaa7b89eaa35a.NginxMailingListEnglish@forum.nginx.org> Hi Lewis, > > Are you serving over HTTPS? > Yes, all content over HTTPS > Can you create an MCVE (minimal, complete, and verifiable example) for > a > ".txt" file over HTTP? For a ".txt" file over HTTPS? For a ".ivp" > file > over HTTP? For a ".ivp" file over HTTPS? > Sorry, I don't know how to create this. > In your Pastebin access log, I see occurrences of "$1" in your > requests. > Is that intentional, or is that a bug? > Yes, that is intentional. > What Content-Type header field are you sending in your response to the > request for a resource ending in ".ivp"? > They are text/xml;charset=UTF-8 and application/json. > Can you try adding a user name and password for the website on the > iPad > under Settings > Passwords & Accounts > Website & App Passwords? > Yes, I already do this, but the browsers still promp password > What happens if you try Firefox and Google Chrome on the iPad? Do > they > work? > neither of them work in Ipad/Iphone. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285874,285934#msg-285934 From themadbeaker at gmail.com Fri Oct 18 13:31:40 2019 From: themadbeaker at gmail.com (J.R.) Date: Fri, 18 Oct 2019 08:31:40 -0500 Subject: Static content and Front Controller pattern under same base URI Message-ID: > I'm not a big fan of the location that sets the 418 error_page to the > @foo_front_controller named location, but I don't know of any other way > to essentially do a "return @foo_front_controller". Is there a better > way? Use "try_files" instead with a filename that will never exist (i.e. underscore) so it will always end up going to the named location: try_files _ @foo_front_controller; From jlmuir at imca-cat.org Fri Oct 18 13:47:39 2019 From: jlmuir at imca-cat.org (J. Lewis Muir) Date: Fri, 18 Oct 2019 08:47:39 -0500 Subject: How to do location /test/place?id=2 In-Reply-To: <8da93f3d-65eb-a21b-8a20-2eef00c10caf@mindmedia.com.sg> References: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> <8da93f3d-65eb-a21b-8a20-2eef00c10caf@mindmedia.com.sg> Message-ID: <20191018134739.l5msnagdqfnqdx4i@mink.imca.aps.anl.gov> On 10/18, P.V.Anthony wrote: > On 18/10/19 5:26 am, Jeff Dyke wrote: > > I know this is not an answer to your question, but it begs another, > > mainly due to the if statement.? How many of these are you going to > > have? https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ > > > > You've likely considered this, but if not wanted to throw it out there. > > Even if you are moving domains, could you create a separate route that > > you could key off rather than arg_id? > > Oh no. I was not aware of the "if" danger. Thank you for bringing that up. > > I am not able to change the application. Is there another way without using > "if"? The "if" behavior is weird, so it's good to avoid it if you can, but sometimes it really is the only way to do something. And at the top of that "If Is Evil" page it says: The only 100% safe things which may be done inside "if" in a location context are: * return ...; * rewrite ... last; And in your example, you were doing a return inside an "if" which is noted as being safe in a location context. Lewis From nginx-forum at forum.nginx.org Fri Oct 18 20:44:44 2019 From: nginx-forum at forum.nginx.org (ptcell) Date: Fri, 18 Oct 2019 16:44:44 -0400 Subject: ngx_http_catch_body_filter doesn't appear to work In-Reply-To: <1105b9e2345a8fb91ee07a8677a23c96.NginxMailingListEnglish@forum.nginx.org> References: <1105b9e2345a8fb91ee07a8677a23c96.NginxMailingListEnglish@forum.nginx.org> Message-ID: <7f22ba64705cb6aee7750f9370763b42.NginxMailingListEnglish@forum.nginx.org> Please disregard or delete this post. The problem was with my setup. It works fine. Thanks. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285923,285941#msg-285941 From anthony at mindmedia.com.sg Sat Oct 19 05:54:44 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Sat, 19 Oct 2019 13:54:44 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <20191018065944.GA6543@haller.ws> References: <20191018010311.GA18308@haller.ws> <1c34f628-6231-1112-78a7-366b4553054c@mindmedia.com.sg> <20191018042000.GA28424@haller.ws> <2a318b7e-d289-7969-8aee-403f8edfe0a6@mindmedia.com.sg> <20191018053357.GA636@haller.ws> <0be97b6d-0f68-c38c-507b-905b0d0578e4@mindmedia.com.sg> <20191018065944.GA6543@haller.ws> Message-ID: On 18/10/19 2:59 pm, Patrick wrote: > The `if' part should be fine. The problem would be someone crafting a > URL that skips past the `=' check and yet is still parsed as `id=2' by > the underlying app. > > Can the underlying old app also be changed to log an attack, and throw a > 444 when it gets an `id' set to 2? Will check it out. Thank you for the advice. P.V.Anthony From anthony at mindmedia.com.sg Sat Oct 19 05:55:48 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Sat, 19 Oct 2019 13:55:48 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <20191018134739.l5msnagdqfnqdx4i@mink.imca.aps.anl.gov> References: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> <8da93f3d-65eb-a21b-8a20-2eef00c10caf@mindmedia.com.sg> <20191018134739.l5msnagdqfnqdx4i@mink.imca.aps.anl.gov> Message-ID: <9a4ba4fa-a39e-4774-771c-94c5b19baa6a@mindmedia.com.sg> On 18/10/19 9:47 pm, J. Lewis Muir wrote: > And in your example, you were doing a return inside an "if" which is > noted as being safe in a location context. Phew! Thank you for the advice. P.V.Anthony From al-nginx at none.at Sat Oct 19 08:22:37 2019 From: al-nginx at none.at (Aleksandar Lazic) Date: Sat, 19 Oct 2019 10:22:37 +0200 Subject: How to do location /test/place?id=2 In-Reply-To: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> References: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> Message-ID: <62db5485-eb3e-4bf9-19e2-27ad3846060f@none.at> Hi P.V.Anthony. Am 17.10.19 um 23:15 schrieb J. Lewis Muir: > On 10/18, P.V.Anthony wrote: >> Currently have the following url, >> >> https://old.example.com/test/place?id=1 >> https://old.example.com/test/place?id=2 >> https://old.example.com/test/place?id=3 >> >> Need to redirect only id=2 to another url. >> >> Did the following and it works for id=2. Need id=1 and id=3 to continue >> normally without change. >> >> location = /test/place { >> if ($args = "id=2") { >> return 301 https://new.example.com/test/place?$args; >> } >> } > > You might want to use $arg_id here (i.e., the $arg_ variable for > the argument). Otherwise, it won't work if any other arguments > are given. Have you tried Lewis suggestion with $arg_id, it looks exactly what you searching for? Untested: location = /test/place { if ($arg_id = "2") { return 301 https://new.example.com/test/place?$args; } } The documentation for arg_ is here. http://nginx.org/en/docs/http/ngx_http_core_module.html#var_arg_ >> Or is there a way to do the following? That would be ideal. >> >> location = /test/place?id=2 { >> return 301 https://new.example.com/test/place?id=2 >> } > > I don't think that's allowed. > >> Unfortunately the above does not work. What is missing? > > What doesn't work? > > I would think your > >> location = /test/place { > > block would work, although not as shown, but I assume you just left > out the part that normally handles the request. It would handle the > requests for id=1 and id=3 as before, and it's just the id=2 case that > gets redirected, right? > > Lewis > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > From francis at daoine.org Sat Oct 19 09:07:59 2019 From: francis at daoine.org (Francis Daly) Date: Sat, 19 Oct 2019 10:07:59 +0100 Subject: IOS keep asking password with nginx auth_basic In-Reply-To: References: Message-ID: <20191019090759.GW1914@daoine.org> On Mon, Oct 14, 2019 at 11:07:43PM -0400, tiendungitd wrote: Hi there, > As you can see in access log https://pastebin.com/6vWhtb3Y, when I submited > a POST request, then browser will have GET request to ivp page, and got a > 401 code which is require to enter user/password. Any idea of what might be > causing this? The access log that I see there does not seem to show me what you describe. Am I missing something? Anyway -- the overall issue sounds like either an incorrect web server configuration, or a broken client. Can you show an nginx configuration that leads to the problem that you report? Does nginx just serve from the filesystem, or does it proxy_pass to an upstream server? If the problem is "wrong config", then possibly the config can be changed to be right. If it is "broken client", then you must decide whether you want to wait for the client to be fixed, or to change your config to work with this client. Cheers, f -- Francis Daly francis at daoine.org From anthony at mindmedia.com.sg Sun Oct 20 05:17:02 2019 From: anthony at mindmedia.com.sg (P.V.Anthony) Date: Sun, 20 Oct 2019 13:17:02 +0800 Subject: How to do location /test/place?id=2 In-Reply-To: <62db5485-eb3e-4bf9-19e2-27ad3846060f@none.at> References: <20191017211513.5qqxvqb6b7hogqem@mink.imca.aps.anl.gov> <62db5485-eb3e-4bf9-19e2-27ad3846060f@none.at> Message-ID: On 19/10/19 4:22 pm, Aleksandar Lazic wrote: > Have you tried Lewis suggestion with $arg_id, it looks exactly what you > searching for? > > Untested: > location = /test/place { > if ($arg_id = "2") { > return 301 https://new.example.com/test/place?$args; > } > } I did try but it does not work when the uri has id=1 where id=1 should remain and go to old.example.com. The solution that works for me is as follows. if ( $request_uri = "/test/place?id=2" ) { rewrite ^ http://new.example.com${uri}?${args}? last; } Works with "return" also. P.V.Anthony From themadbeaker at gmail.com Sun Oct 20 17:00:07 2019 From: themadbeaker at gmail.com (J.R.) Date: Sun, 20 Oct 2019 12:00:07 -0500 Subject: Revisiting the long-overdue "TODO always gunzip" in ngx_http_gunzip_filter_module.c Message-ID: Recently I was looking into having my upstream server gzip content that is sent to nginx (which is acting as a reverse proxy) to reduce local bandwidth. However, I needed to decompress the response so nginx could do some manipulation, then obviously it would get re-compressed (typically with brotli) before sending out to the client. After finding numerous posts saying that "that feature doesn't exist", then looking at the source of the gunzip module and finding the "TODO" saying the exact thing I was looking for, I managed to find this old post from 2013 where someone made a patch to do the exact thing needed: http://mailman.nginx.org/pipermail/nginx-devel/2013-January/003276.html It's only about 11 lines of code, attached is a newer patch I did against 1.17.4 since the gunzip source changed a bit. Basically the patch adds the option "gunzip_force on;" I do agree with the TODO comment in the source that having a way for other modules to request decompression would be great, but also having a configuration option (like this patch) is also a good feature. After implementing this patch, and configuring the upstream server to use gzip level 1 (figured maximize speed and minimize cpu usage), my local bandwidth usage between the two immediately dropped 80-85%, and I swear even the load levels dropped a hair, probably due to less networking overhead, despite the compressing & decompressing of content. I realize there is probably a mile-long TODO list for nginx features, but something so trivial like this patch has such a huge impact on network usage and I'm sure a better implementation would still only be a minimal code change. Just trying to give this a nudge so it's not forgotten, as I did come across quite a few posts online into how to do such a thing, so the demand is out there. Jason -------------- next part -------------- A non-text attachment was scrubbed... Name: 002_force_gunzip.patch Type: application/octet-stream Size: 1615 bytes Desc: not available URL: From nginx-forum at forum.nginx.org Mon Oct 21 06:56:41 2019 From: nginx-forum at forum.nginx.org (tiendungitd) Date: Mon, 21 Oct 2019 02:56:41 -0400 Subject: IOS keep asking password with nginx auth_basic In-Reply-To: <20191019090759.GW1914@daoine.org> References: <20191019090759.GW1914@daoine.org> Message-ID: > The access log that I see there does not seem to show me what you describe. Well, I just hightlight what I described, see http://prntscr.com/pm2qhl > Can you show an nginx configuration that leads to the problem that you report? Sure, you can see in https://pastebin.com/iagU4XFC > Does nginx just serve from the filesystem, or does it proxy_pass to an upstream server? It proxy_pass to the upstream server. I tried which some IOS devices, they all got this issues. Here is my nginx version nginx version: nginx/1.13.6 built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) built with OpenSSL 1.0.1f 6 Jan 2014 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285874,285950#msg-285950 From francis at daoine.org Mon Oct 21 11:18:49 2019 From: francis at daoine.org (Francis Daly) Date: Mon, 21 Oct 2019 12:18:49 +0100 Subject: IOS keep asking password with nginx auth_basic In-Reply-To: References: <20191019090759.GW1914@daoine.org> Message-ID: <20191021111849.GX1914@daoine.org> On Mon, Oct 21, 2019 at 02:56:41AM -0400, tiendungitd wrote: Hi there, > > The access log that I see there does not seem to show me what you > describe. > Well, I just hightlight what I described, see http://prntscr.com/pm2qhl Ok. In the first highlighted section, I see a POST which got a HTTP 200 reply. That is "success". In the second highlighted section, I see a GET which got a HTTP 401 reply, followed by a GET of the same url with a username includes, and a HTTP 302 reply. > > Can you show an nginx configuration that leads to the problem that you > report? > Sure, you can see in https://pastebin.com/iagU4XFC I think that the log file you show does not come from this config. This config has: location = /403.html location ~ /(ivy/error) location ~ /(ivy/wf/) location ~* /favicon(.*) location / location /ivy/ The log shows requests that start with //faces/ or //pro/, and do not include the string /ivy/. Those requests should be handled in the "location /" block, which just has "return 301 /ivy;" The log shows no 301 responses. I agree that something strange is happening. But I think that you have not shown any evidence that nginx is involved in the strange thing. Can you build a test system with a very small config, and see if that shows the same problems? f -- Francis Daly francis at daoine.org From mdounin at mdounin.ru Mon Oct 21 12:40:39 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 21 Oct 2019 15:40:39 +0300 Subject: read request body with http2 In-Reply-To: <2992cc52beb999c6663bc573b8418783.NginxMailingListEnglish@forum.nginx.org> References: <20191015145605.GT1877@mdounin.ru> <2992cc52beb999c6663bc573b8418783.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191021124039.GK1877@mdounin.ru> Hello! On Thu, Oct 17, 2019 at 04:42:54PM -0400, Ansuel wrote: > I don't know if this is a bug or not but... yes you were right... > > All the work should be done in the ngx_http_read_client_request_body > handler, as it does duplicate the r connection to another address > > What confused me is the fact that this is only done in http2 connections... > > I really don't know why but all the changes i do in the read_client_request > handler reflect to the main ngx_http_request_t struct. I thought the code > was cleaner by having the body read in a function and parse the data from it > in another function. > > Well this doesn't work with http2 because all the changes done in > read_client_request handler get lost and doesn't apply to the main struct... > > > Think this is a bug as the behavior should be similar and not change across > http versions... > > Sorry for the bad explanation... Also i think this should be noted in the > wiki. It looks like you don't understand how things work. First of all, when the ngx_http_read_client_request_body() function returns, it doesn't mean that the request body was successfully read, and the body handler function was already called. You may want to start with reading more about event-driven programming in general. Also, as previously suggested, you may want to read Development guide, it has enough information about how things work and how they are expected to be handled. Also, if you want to better understand how nginx works, consider reading nginx source code - it is the only direct source of the information. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Mon Oct 21 17:07:35 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 21 Oct 2019 20:07:35 +0300 Subject: Revisiting the long-overdue "TODO always gunzip" in ngx_http_gunzip_filter_module.c In-Reply-To: References: Message-ID: <20191021170735.GP1877@mdounin.ru> Hello! On Sun, Oct 20, 2019 at 12:00:07PM -0500, J.R. wrote: > Recently I was looking into having my upstream server gzip content > that is sent to nginx (which is acting as a reverse proxy) to reduce > local bandwidth. However, I needed to decompress the response so nginx > could do some manipulation, then obviously it would get re-compressed > (typically with brotli) before sending out to the client. > > After finding numerous posts saying that "that feature doesn't exist", > then looking at the source of the gunzip module and finding the "TODO" > saying the exact thing I was looking for, I managed to find this old > post from 2013 where someone made a patch to do the exact thing > needed: > > http://mailman.nginx.org/pipermail/nginx-devel/2013-January/003276.html > > It's only about 11 lines of code, attached is a newer patch I did > against 1.17.4 since the gunzip source changed a bit. Basically the > patch adds the option "gunzip_force on;" > > I do agree with the TODO comment in the source that having a way for > other modules to request decompression would be great, but also having > a configuration option (like this patch) is also a good feature. > > After implementing this patch, and configuring the upstream server to > use gzip level 1 (figured maximize speed and minimize cpu usage), my > local bandwidth usage between the two immediately dropped 80-85%, and > I swear even the load levels dropped a hair, probably due to less > networking overhead, despite the compressing & decompressing of > content. > > I realize there is probably a mile-long TODO list for nginx features, > but something so trivial like this patch has such a huge impact on > network usage and I'm sure a better implementation would still only be > a minimal code change. Just trying to give this a nudge so it's not > forgotten, as I did come across quite a few posts online into how to > do such a thing, so the demand is out there. Just in case, my position hasn't changed since then: http://mailman.nginx.org/pipermail/nginx-devel/2013-January/003284.html Also note that if you really need to force gunziping for some reason, you can do so out of the box by using an additional local proxying layer with appropriate "proxy_set_header Accept-Encoding". -- Maxim Dounin http://mdounin.ru/ From themadbeaker at gmail.com Mon Oct 21 17:49:38 2019 From: themadbeaker at gmail.com (J.R.) Date: Mon, 21 Oct 2019 12:49:38 -0500 Subject: Revisiting the long-overdue "TODO always gunzip" in ngx_http_gunzip_filter_module.c Message-ID: > Also note that if you really need to force gunziping for some reason, > you can do so out of the box by using an additional local proxying > layer with appropriate "proxy_set_header Accept-Encoding". Yes, that is how I had it configured before patching, all content between nginx and the upstream servers was uncompressed using the directive you mentioned, but the goal was to have the data between the two servers compressed to reduce network traffic. Yes, I also read your comments from 2013, but after looking over the current gunzip module code, I can't find any flag that another module could set to force decompression? Or did I miss it buried somewhere? Thanks for the quick response! Jason From mdounin at mdounin.ru Mon Oct 21 18:39:09 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 21 Oct 2019 21:39:09 +0300 Subject: Revisiting the long-overdue "TODO always gunzip" in ngx_http_gunzip_filter_module.c In-Reply-To: References: Message-ID: <20191021183909.GQ1877@mdounin.ru> Hello! On Mon, Oct 21, 2019 at 12:49:38PM -0500, J.R. wrote: > > Also note that if you really need to force gunziping for some reason, > > you can do so out of the box by using an additional local proxying > > layer with appropriate "proxy_set_header Accept-Encoding". > > Yes, that is how I had it configured before patching, all content > between nginx and the upstream servers was uncompressed using the > directive you mentioned, but the goal was to have the data between the > two servers compressed to reduce network traffic. Well, it looks like I've failed to explain. You can have things compressed between servers and then decompressed on the frontend server. To do so, you can configure additional proxying on the frontend server, for example: server { listen 80; server_name example.com; location / { proxy_pass http://127.0.0.1:8080; proxy_set_header Accept-Encoding ""; } } server { listen 8080; server_name gunzip.example.com; location / { proxy_pass http://backend; proxy_set_header Accept-Encoding "gzip"; gunzip on; } } With such a configuration all traffic between the frontend server and the backend servers can be compressed using gzip. Yet everything in the first server block isn't compressed, and can be processed by sub filter and so on. > Yes, I also read your comments from 2013, but after looking over the > current gunzip module code, I can't find any flag that another module > could set to force decompression? Or did I miss it buried somewhere? It wasn't implemented. Rather, it is something I think should be implemented if we want to process/modify compressed responses within nginx. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Tue Oct 22 15:34:56 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 22 Oct 2019 18:34:56 +0300 Subject: nginx-1.17.5 Message-ID: <20191022153456.GV1877@mdounin.ru> Changes with nginx 1.17.5 22 Oct 2019 *) Feature: now nginx uses ioctl(FIONREAD), if available, to avoid reading from a fast connection for a long time. *) Bugfix: incomplete escaped characters at the end of the request URI were ignored. *) Bugfix: "/." and "/.." at the end of the request URI were not normalized. *) Bugfix: in the "merge_slashes" directive. *) Bugfix: in the "ignore_invalid_headers" directive. Thanks to Alan Kemp. *) Bugfix: nginx could not be built with MinGW-w64 gcc 8.1 or newer. -- Maxim Dounin http://nginx.org/ From xeioex at nginx.com Tue Oct 22 15:49:16 2019 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 22 Oct 2019 18:49:16 +0300 Subject: njs-0.3.6 Message-ID: Hello, I'm glad to announce a new release of NGINX JavaScript module (njs). This release proceeds to extend the coverage of ECMAScript specifications. Notable new features: - Function constructor: : > var sum = new Function('a, b', 'return a + b'); : undefined : > sum(2, 3) : 5 You can learn more about njs: - Overview and introduction: http://nginx.org/en/docs/njs/ - Presentation: https://youtu.be/Jc_L6UffFOs Feel free to try it and give us feedback on: - Github: https://github.com/nginx/njs/issues - Mailing list: http://mailman.nginx.org/mailman/listinfo/nginx-devel Changes with njs 0.3.6 22 Oct 2019 nginx modules: *) Improvement: getting special headers from r.headersIn. Core: *) Feature: added new Function() support. *) Feature: added Number.prototype.toFixed(). *) Feature: added Number.prototype.toPrecision(). *) Feature: added Number.prototype.toExponential(). *) Improvement: making "prototype" property of function instances writable. *) Improvement: limiting recursion depth while compiling. *) Improvement: moving global functions to the global object. *) Bugfix: fixed prototype mutation for object literals. *) Bugfix: fixed heap-buffer-overflow while parsing regexp literals. *) Bugfix: fixed integer-overflow while parsing exponent of number literals. *) Bugfix: fixed parseFloat(). *) Bugfix: fixed Array.prototype functions according to the specification. The following functions were fixed: every, includes, indexOf, filter, find, findIndex, forEach, lastIndexOf, map, pop, push, reduce, reduceRight, shift, some, unshift. *) Bugfix: fixed handing of accessor descriptors in Object.freeze(). *) Bugfix: fixed String.prototype.replace() when first argument is not a string. *) Bugfix: fixed stack-use-after-scope in Array.prototype.map(). *) Bugfix: Date.prototype.toUTCString() format was aligned to ES9. *) Bugfix: fixed buffer overflow in Number.prototype.toString(radix). *) Bugfix: fixed Regexp.prototype.test() for regexps with backreferences. *) Bugfix: fixed Array.prototype.map() for objects with nonexistent values. *) Bugfix: fixed Array.prototype.pop() and shift() for sparse objects. *) Bugfix: fixed Date.UTC() according to the specification. *) Bugfix: fixed Date() constructor according to the specification. *) Bugfix: fixed type of Date.prototype. Thanks to Artem S. Povalyukhin. *) Bugfix: fixed Date.prototype.setTime(). Thanks to Artem S. Povalyukhin. *) Bugfix: fixed default number of arguments expected by built-in functions. *) Bugfix: fixed "caller" and "arguments" properties of a function instance. Thanks to Artem S. Povalyukhin. From nginx-forum at forum.nginx.org Wed Oct 23 13:54:25 2019 From: nginx-forum at forum.nginx.org (bvek1) Date: Wed, 23 Oct 2019 09:54:25 -0400 Subject: High memory usage Message-ID: <2e9ec7734500d6cd058f454b2172a81d.NginxMailingListEnglish@forum.nginx.org> Hello, I need your help, I have a problem with my Nginx server : all the Nginx processes are consuming a huge amount of memory and it sets off the OOM Killer. I don't understand why Nginx uses this much memory, it seems odd, I expected the CPU to be the first problem instead of memory. My Nginx server is just a reverse proxy, to all kind of backend. SSL can be offloaded or not, it depends on the backend. I have a huge amount of vhosts with SSL on this Nginx (~15K) and the server have a lot of trafic (for me at least !) . The /stub_status tell me the following thing at this moment : Active connections: 3867 server accepts handled requests 2350039 2350039 5489648 Reading: 0 Writing: 332 Waiting: 3605 Netdata (https://github.com/netdata/netdata) tell me that it handle between 300 and 600 req/s. [root at nginxsrv nginx]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) [root at nginxsrv nginx]# nginx -V nginx version: nginx/1.17.3 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) built with OpenSSL 1.1.0k 28 May 2019 TLS SNI support enabled configure arguments: --add-module=/usr/local/src/ngx_brotli --add-module=/usr/local/src/kyprizel-testcookie-nginx-module --with-threads --with-openssl=/usr/local/src/openssl --with-http_v2_module --with-http_flv_module --with-ipv6 --with-http_mp4_module --sbin-path=/usr/local/sbin --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_realip_module --with-http_ssl_module --http-client-body-temp-path=/eacc/nginx_client --http-proxy-temp-path=/eacc/nginx_proxy --http-fastcgi-temp-path=/eacc/nginx_fastcgi --with-http_stub_status_module The server have a lot of ram, 120G and the CPU is an Intel Xeon E5-2697 CPU. The CPU is not used that much. A typical vhost file is quite simple, it's just a proxy_pass to a backend, with some hack to do a retry is we first get and error message (based on proxy_intercept_errors). http2 is on. I don't use any caches. Things I have tested or I suspect to be an issue : - I reduced the worker_processes to 6. I had issue with the frequent reloading of Nginx with an upper value. I suspect that it's more difficult for Nginx to reload properly if a lot of processes are spawned (correct me if i'm wrong) - I have gzip on and gzip_proxied to any. I suspect the compression to be an issue, I reduced the buffers value to gzip_buffers 32 4k; and gzip_comp_level to 1 - I reduced the ssl_session_cache from 10m to 1m - I reduced the keep_alive timeout too (30) - Reloading Nginx often seems to be a problem too, the memory usage seems higher during the reloading task. The reloading is quite slow too, probably because of the number of vhosts file to load (if someone know how to speed up this, it interest me !) - I have a lot of limit_conn configuration - ssl_stapling is off, I had issue with it with the DNS resolution. Do you have idea on what can cause a high memory usage on Nginx ? Thanks, Alexis Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285979,285979#msg-285979 From kworthington at gmail.com Wed Oct 23 14:22:03 2019 From: kworthington at gmail.com (Kevin Worthington) Date: Wed, 23 Oct 2019 10:22:03 -0400 Subject: [nginx-announce] nginx-1.17.5 In-Reply-To: <20191022153502.GW1877@mdounin.ru> References: <20191022153502.GW1877@mdounin.ru> Message-ID: Hello Nginx users, Now available: Nginx 1.17.5 for Windows https://kevinworthington.com/nginxwin1175 (32-bit and 64-bit versions) These versions are to support legacy users who are already using Cygwin based builds of Nginx. Officially supported native Windows binaries are at nginx.org. Announcements are also available here: Twitter http://twitter.com/kworthington Thank you, Kevin -- Kevin Worthington kworthington *@* (gmail] [dot} {com) https://kevinworthington.com/ https://twitter.com/kworthington On Tue, Oct 22, 2019 at 11:35 AM Maxim Dounin wrote: > Changes with nginx 1.17.5 22 Oct > 2019 > > *) Feature: now nginx uses ioctl(FIONREAD), if available, to avoid > reading from a fast connection for a long time. > > *) Bugfix: incomplete escaped characters at the end of the request URI > were ignored. > > *) Bugfix: "/." and "/.." at the end of the request URI were not > normalized. > > *) Bugfix: in the "merge_slashes" directive. > > *) Bugfix: in the "ignore_invalid_headers" directive. > Thanks to Alan Kemp. > > *) Bugfix: nginx could not be built with MinGW-w64 gcc 8.1 or newer. > > > -- > Maxim Dounin > http://nginx.org/ > _______________________________________________ > nginx-announce mailing list > nginx-announce at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-announce > -------------- next part -------------- An HTML attachment was scrubbed... URL: From themadbeaker at gmail.com Wed Oct 23 16:12:00 2019 From: themadbeaker at gmail.com (J.R.) Date: Wed, 23 Oct 2019 11:12:00 -0500 Subject: Revisiting the long-overdue "TODO always gunzip" in ngx_http_gunzip_filter_module.c Message-ID: > Well, it looks like I've failed to explain. You can have things > compressed between servers and then decompressed on the frontend > server. To do so, you can configure additional proxying on the > frontend server, for example: Thanks for the sample configuration, that makes sense with the second proxy. I will have to give that setup a try and see how it compares. Always appreciate the prompt replies! Jason From themadbeaker at gmail.com Wed Oct 23 16:36:11 2019 From: themadbeaker at gmail.com (J.R.) Date: Wed, 23 Oct 2019 11:36:11 -0500 Subject: High memory usage Message-ID: > A typical vhost file is quite simple, it's just a proxy_pass to a backend, > with some hack to do a retry is we first get and error message (based on > proxy_intercept_errors). http2 is on. I don't use any caches. Are you running PHP FPM? If so, check all your process manager settings. I've seen when misconfigured that can eat up a ton of ram quickly. I've always left the worker_processes to auto, as the docs say it should equal the number of cores of the system. Though it sounds like 6 is adequate if you are not experiencing high CPU load. Though I'm not sure if more cores would allow for faster reading / processing of your configuration files, maybe bump up to 12 since that is the number of actual cores for that CPU. Why do you suspect the gzip module? If anything I would think it would eat CPU cycles, not memory... Unless you are trying to compress some *really* large content. I know there is a minimum size setting, I don't know if there is a maximum one too... I see you have the brotli compression module installed too... Have you set brotli_comp_level (among the other settings) ??? The point of the SSL session cache is to reduce CPU usage by having to constantly re-negotiate a SSL session... Think about how long a person will read a page on your site(s)... You would really want at minimum like 5 minutes, otherwise the cache really isn't saving you any processing and only eating up memory. Depending on your log level, the error log would let you know if your cache gets full and needs to be increased (or the TTL lowered) as you would see something like, "could not allocate new session in SSL session shared cache...." Are you using a shared cache across *all* your virtual hosts for the ssl_session_cache & limit_conn, or does each vhost have their own? I could see how that would eat up memory real quick with the number of vhosts you have. Whenever possible you should try to use settings at the http level so that it's only creating that instance once. If you are using the same thing under every server directive, then each server instance has its own copy and thus using more memory. Can you post your configuration settings? It's okay to strip out any personal identifiable information. Otherwise it's just a guessing game... From mrbonezz75 at gmail.com Wed Oct 23 17:24:35 2019 From: mrbonezz75 at gmail.com (mrbonezz75) Date: Wed, 23 Oct 2019 12:24:35 -0500 Subject: unsubscribing Message-ID: i followed your unsubscribe directions they dont seem to be working on my phone i have been receiving your emails for ever and i never signed up for it no password dont even know what nginx is please make it go awaySent from my T-Mobile 4G LTE Device -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Thu Oct 24 09:30:16 2019 From: nginx-forum at forum.nginx.org (bvek1) Date: Thu, 24 Oct 2019 05:30:16 -0400 Subject: High memory usage In-Reply-To: References: Message-ID: <37eafeca1426dad55f9036e3a2635586.NginxMailingListEnglish@forum.nginx.org> Hello, Thanks for your time ! I'm running PHP-FPM for the API I created that dynamicaly create Nginx vhosts (that's why I need to reload Nginx quite often). It's not really use, like < 1000 requests / day. 99.999% of the trafic is just is associated with the 15K domain and Nginx just do a proxy_pass to a backend server. Nginx listen to 100+ IPs (if that's important). For the GZIP module, it's just a guest, for me it's logical that GZIP will use a lot of CPU but I thought that it could use some memory too for the buffering or some sort of cache. I'm aware that it's unlikely the problem. I used to have the ssl_session_cache on each vhosts but I moved it to the http level. The memory consuption haven't change. But, I have some limit_conn in each vhosts (included) so I think you're onto something with this ! Bellow the config I can share, with some comment : ### Begin of nginx.conf ### user nginx; worker_processes 6; # Doing some test on this, previous value was 8, also tested auto worker_rlimit_nofile 65535; error_log /var/log/nginx/err.crit error; pid /run/nginx.pid; events { worker_connections 16192; multi_accept on; accept_mutex off; use epoll; } http { # I define a few geo config and maps too (~10) # Used to block bad UA / domain / bot and apply rate limits geo $whitelist { default 0; x.x.x.x 1; # a few IP range } map $whitelist $map_host { 0 $host; 1 ""; } etag off; if_modified_since off; # I have a few rate limit too ~6-7 based on differents things limit_conn_zone $map_perip zone=perip:1m; limit_req_zone $map_perip zone=global1:1m rate=250r/s; limit_req zone=global1 burst=300; limit_req_status 429; server_names_hash_bucket_size 512; log_format upstreamlog '[$time_local] $remote_addr - $request - $server_name TO $upstream_addr : status=$upstream_status - response_time=$upstream_response_time - msec=$msec - request_time=$request_time'; log_format custom_server '$host - $server_name - $remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent $request_length $bytes_sent $request_time $server_port ' '"$http_referer" "$http_user_agent" '; # debug err map $status $loggable { ~^[234] 0; default 1; } map $upstream_status $upstream_loggable { ~^[234] 0; default 1; } sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 30; types_hash_max_size 2048; client_max_body_size 256M; aio threads; include /etc/nginx/mime.types; default_type application/octet-stream; gzip on; gzip_vary on; gzip_disable "MSIE [1-6]\."; gzip_proxied any; gzip_http_version 1.0; gzip_min_length 1000; gzip_comp_level 1; # Doing some test, initial value was 9; gzip_buffers 32 4k; # Doing some test, initial value was 64 8k; gzip_types text/plain text/xml text/css application/x-javascript application/xml application/javascript application/xml+rss text/javascript application/atom+xml application/xhtml+xml image/svg+xml application/x-font-woff application/font-woff2; brotli on; brotli_static on; brotli_types text/plain text/xml text/css application/x-javascript application/xml application/javascript application/xml+rss text/javascript application/atom+xml application/xhtml+xml image/svg+xml application/x-font-woff application/font-woff2; brotli_comp_level 9; ssl_session_cache shared:SSLCACHE:1m; # Doing some test, initial value was 10m and was defined in each vhosts server { listen x.x.x.x:80; # + ~100 listen for ~100 ips listen 80 default_server; listen [::]:80 default_server; server_name _; root /opt/web/default; location /stub_status { stub_status; allow 127.0.0.1; deny all; } } server { listen x.x.x.x:443 ssl http2; # + ~100 listen for ~100 ips listen 443 ssl http2 default_server; listen [::]:443 ssl http2 default_server; server_name _; root /opt/web/default; ssl_certificate "/etc/pki/nginx/nginx.crt"; ssl_certificate_key "/etc/pki/nginx/private/nginx.key"; } server { listen x.x.x.x:445 ssl http2; # + ~100 listen for ~100 ips listen 445 ssl http2 default_server; listen [::]:445 ssl http2 default_server; server_name _; root /opt/web/nginx/public; ssl_certificate "/etc/pki/nginx/nginx.crt"; ssl_certificate_key "/etc/pki/nginx/private/nginx.key"; access_log /var/log/nginx/api_access_log custom_server; error_log /var/log/nginx/api_error_log warn; location / { allow x.x.x.x/21; deny all; try_files $uri /index.php$is_args$args; } location ~ \.php$ { try_files $uri =404; fastcgi_index index.php; fastcgi_pass unix:/var/run/php-fpm-nginx.sock; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include /etc/nginx/fastcgi_params; } } include /etc/nginx/vhosts/*.conf; proxy_buffering on; proxy_buffers 256 16k; proxy_buffer_size 128k; proxy_busy_buffers_size 256k; proxy_temp_file_write_size 256k; proxy_connect_timeout 300s; proxy_read_timeout 300s; proxy_send_timeout 300s; open_file_cache max=2000 inactive=5m; open_file_cache_valid 2m; open_file_cache_min_uses 2; open_file_cache_errors on; } ### End of nginx.conf ### ### Begin of a typical vhosts ### server { listen x.x.x.x:443 ssl http2; listen x.x.x.x:4430 ssl http2; server_name xxxx.com www.xxxx.com; ssl_certificate /etc/nginx/ssl/xxxx.eu.crt; ssl_certificate_key /etc/nginx/ssl/xxxx.eu.key; ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA'; #ssl_session_cache shared:SSL:10m; # cf. nginx.conf ssl_dhparam /etc/nginx/ssl/dhparam.pem; ssl_session_timeout 5m; proxy_intercept_errors on; recursive_error_pages on; location / { proxy_intercept_errors on; proxy_pass https://z.z.z.z:4430; proxy_ssl_verify off; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-Port 443; proxy_set_header X-SSL yes; include security-ssl.inc; error_page 502 503 = @fallback1; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Custom-IP yes; } # Fallback 1, in case of 502/503, retry the proxy_pass location @fallback1 { proxy_intercept_errors off; proxy_pass https://z.z.z.z:4430; proxy_ssl_verify off; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-Port 443; proxy_set_header X-SSL yes; include security-ssl.inc; proxy_set_header X-Retry 1; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Custom-IP yes; } } ### End of vhosts ### ### security inc ### open_log_file_cache max=1000 inactive=20s valid=1m min_uses=2; access_log /var/log/nginx/web custom_server; # Debug #access_log /var/log/nginx/vhosts_5xx custom_server if=$loggable; #access_log /var/log/nginx/vhosts_upstream_5xx upstreamlog if=$upstream_loggable; limit_rate 4096k; limit_conn perip 15; limit_conn max_conn_to_vhost 150; if ($limit_forbidbot) { return 444; } if ($abusehost) { return 403; } if ($abuseuri) { return 403; } if ($request_method !~ ^(GET|PURGE|HEAD|OPTIONS|POST|PUT|DELETE)$ ) { return 444; } ### end of security inc ### Thanks, Alexis Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281689,285985#msg-285985 From themadbeaker at gmail.com Sun Oct 27 17:40:06 2019 From: themadbeaker at gmail.com (J.R.) Date: Sun, 27 Oct 2019 12:40:06 -0500 Subject: High memory usage Message-ID: Few things... brotli_comp_level should probably be something like 5, setting to 9 would be rather CPU intensive (much more than gzip 9). Some of your proxy buffer settings seem rather excessive... Something like this would probably be sufficient unless you have some *really* large files coming from upstream. proxy_buffer_size 8k; proxy_busy_buffers_size 16k; proxy_buffers 64 4k A good read for proxy settings that I've found is: https://www.getpagespeed.com/server-setup/nginx/tuning-proxy_buffer_size-in-nginx You should set your SSL settings at the 'http' level, including default certificates. Then in each 'server' directive you can re-define the certificates when necessary. You can also set all the common proxy_set_header vars at the 'http' level instead of duplicating for each server. Maybe also look into using a common upstream zone instead of always defining the proxy. Finally, just because I would check your PHP-FPM settings (not in nginx)... Make sure that at the very least: pm = ondemand pm.max_request = 1000 I've seen at times where not setting a max_request leads to eating up ram in Apache, could possibly be an issue with Nginx. Nothing else really stands out. I would just try to decrease buffers where you can, especially those that are created for each server instance. With all the vhosts you have that can really add up quickly. Jason From nginx-forum at forum.nginx.org Mon Oct 28 02:05:05 2019 From: nginx-forum at forum.nginx.org (liuyinhl) Date: Sun, 27 Oct 2019 22:05:05 -0400 Subject: Full Filename Directory Listing In-Reply-To: References: <8bde2327465f7f11e202fe40328c3d00.NginxMailingListEnglish@forum.nginx.org> Message-ID: Example: 1.cd /usr/local/nginx-1.16.1/src/thhp/modules 2.vi ngx_http_autoindex_module.c 3.cd /usr/local/nginx-1.16.1 4.make 5.make install Posted at Nginx Forum: https://forum.nginx.org/read.php?2,124400,286011#msg-286011 From nginx-forum at forum.nginx.org Mon Oct 28 15:42:59 2019 From: nginx-forum at forum.nginx.org (henrywilliam463) Date: Mon, 28 Oct 2019 11:42:59 -0400 Subject: High memory usage In-Reply-To: <2e9ec7734500d6cd058f454b2172a81d.NginxMailingListEnglish@forum.nginx.org> References: <2e9ec7734500d6cd058f454b2172a81d.NginxMailingListEnglish@forum.nginx.org> Message-ID: Hey, Have you got the solution of this? I also want to send mails for my website https://apkstreet.com Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285979,286021#msg-286021 From nginx-forum at forum.nginx.org Mon Oct 28 15:44:32 2019 From: nginx-forum at forum.nginx.org (henrywilliam463) Date: Mon, 28 Oct 2019 11:44:32 -0400 Subject: IP Transparency in NGINX In-Reply-To: <425cd65d485a0760508731a9f5426b9e.NginxMailingListEnglish@forum.nginx.org> References: <425cd65d485a0760508731a9f5426b9e.NginxMailingListEnglish@forum.nginx.org> Message-ID: Have you got the solution? I also want to know this about for my gaming website https://apkstreet.com where I publish MOD APKs Posted at Nginx Forum: https://forum.nginx.org/read.php?2,284810,286022#msg-286022 From roger at netskrt.io Mon Oct 28 23:08:45 2019 From: roger at netskrt.io (Roger Fischer) Date: Mon, 28 Oct 2019 16:08:45 -0700 Subject: proxy_store: is it used much? Message-ID: <7E9A9071-F182-4384-A2C7-3FBDC48F1024@netskrt.io> Hello, proxy_store seems to be a much simpler alternative to ?cache" pseudo-static resources. But there is very little discussion of it on the Internet or nginx forum (compared to proxy_cache). Is there anything non-obvious that speaks agains the use of proxy_store? Thanks? Roger From r at roze.lv Tue Oct 29 08:26:54 2019 From: r at roze.lv (Reinis Rozitis) Date: Tue, 29 Oct 2019 10:26:54 +0200 Subject: proxy_store: is it used much? In-Reply-To: <7E9A9071-F182-4384-A2C7-3FBDC48F1024@netskrt.io> References: <7E9A9071-F182-4384-A2C7-3FBDC48F1024@netskrt.io> Message-ID: <001e01d58e32$9f115580$dd340080$@roze.lv> > proxy_store seems to be a much simpler alternative to ?cache" pseudo-static resources. > > Is there anything non-obvious that speaks agains the use of proxy_store? Depends on how you look at "much simpler". proxy_store doesn't have a cache manager so space limitation/cleaning is up to you. Also nginx doesn't save any headers (contrary to proxy_cache) with the stored file so next requests won't return expire / content-type etc what came originally from backend but based on the (local) proxy configuration. The advantage is that there are no additional checks (like if the object is expired) or any additional requests to upstream. Also files are named and have the exact path as on backend (with proxy_cache nginx adds additional data in the beginning of each cached file). Other than that there is nothing wrong or bad with proxy_store. I use it for a lazy repository mirror where I know the package/iso files won't change and the cached directory/file structure is more human readable / easy to inspect. rr From mouseless at free.fr Tue Oct 29 12:45:58 2019 From: mouseless at free.fr (Vincent M.) Date: Tue, 29 Oct 2019 13:45:58 +0100 Subject: Don't log 404 error Message-ID: <80f4e78f-3078-a8e9-1800-3ad54beb0835@free.fr> Hello, How not to log 404 errors: 2019/10/29 12:42:09 [error] 469#469: *16382 open() "/var/www/website/fr/telecharger" failed (2: No such file or directory), In my config I have: error_log /var/log/nginx/website-error.log; Because I use 404 for doing url rewriting and so they are not errors... error_page 404 = /url_rewriting.php; Thanks, Vincent. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pluknet at nginx.com Tue Oct 29 12:59:00 2019 From: pluknet at nginx.com (Sergey Kandaurov) Date: Tue, 29 Oct 2019 15:59:00 +0300 Subject: Don't log 404 error In-Reply-To: <80f4e78f-3078-a8e9-1800-3ad54beb0835@free.fr> References: <80f4e78f-3078-a8e9-1800-3ad54beb0835@free.fr> Message-ID: <2C243EF8-931D-44D3-9E2B-1CFFB528CE2C@nginx.com> > On 29 Oct 2019, at 15:45, Vincent M. wrote: > > Hello, > > How not to log 404 errors: > > 2019/10/29 12:42:09 [error] 469#469: *16382 open() "/var/www/website/fr/telecharger" failed (2: No such file or directory), > > In my config I have: > > error_log /var/log/nginx/website-error.log; > > Because I use 404 for doing url rewriting and so they are not errors... > > error_page 404 = /url_rewriting.php; > http://nginx.org/r/log_not_found -- Sergey Kandaurov From nginx-forum at forum.nginx.org Tue Oct 29 13:49:29 2019 From: nginx-forum at forum.nginx.org (awagner) Date: Tue, 29 Oct 2019 09:49:29 -0400 Subject: Input filters in Nginx? Message-ID: <50965ba63f14fc458233be3e06debd75.NginxMailingListEnglish@forum.nginx.org> Hi all. I did some search on the web and the archives here and did not find an conclusive answer to this question. Apache has both input a filters (work on the request before content generation) and output filters (work on generated content). As I am exploring the possibility to port some Apache modules that transform data in both directions in a proxy-setup to Nginx, I am wondering whether Nginx has input filters as well. What I have read about Nginx modules seem to indicate that it does not. Is that correct? I am aware that I could hack something into the proxy module, but I would like to avoid that approach. Thanks for any insights or pointers. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286038,286038#msg-286038 From nginx-forum at forum.nginx.org Tue Oct 29 13:52:00 2019 From: nginx-forum at forum.nginx.org (skzai) Date: Tue, 29 Oct 2019 09:52:00 -0400 Subject: high memory usage In-Reply-To: References: Message-ID: Hi Anoop, there was a similar problem to me. This is really stressful :/ But I found a solution here: https://bit.ly/2pS0QW1 Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281689,286039#msg-286039 From mouseless at free.fr Tue Oct 29 15:05:14 2019 From: mouseless at free.fr (Vincent M.) Date: Tue, 29 Oct 2019 16:05:14 +0100 Subject: Don't log 404 error In-Reply-To: <2C243EF8-931D-44D3-9E2B-1CFFB528CE2C@nginx.com> References: <80f4e78f-3078-a8e9-1800-3ad54beb0835@free.fr> <2C243EF8-931D-44D3-9E2B-1CFFB528CE2C@nginx.com> Message-ID: Le 29/10/2019 ? 13:59, Sergey Kandaurov a ?crit?: >> On 29 Oct 2019, at 15:45, Vincent M. wrote: >> >> Hello, >> >> How not to log 404 errors: >> >> 2019/10/29 12:42:09 [error] 469#469: *16382 open() "/var/www/website/fr/telecharger" failed (2: No such file or directory), >> >> In my config I have: >> >> error_log /var/log/nginx/website-error.log; >> >> Because I use 404 for doing url rewriting and so they are not errors... >> >> error_page 404 = /url_rewriting.php; >> > http://nginx.org/r/log_not_found > Yes perfect! So easy. Sorry didn't think that would be so easy within an option. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Oct 29 16:19:58 2019 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 29 Oct 2019 19:19:58 +0300 Subject: Input filters in Nginx? In-Reply-To: <50965ba63f14fc458233be3e06debd75.NginxMailingListEnglish@forum.nginx.org> References: <50965ba63f14fc458233be3e06debd75.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191029161958.GC12894@mdounin.ru> Hello! On Tue, Oct 29, 2019 at 09:49:29AM -0400, awagner wrote: > Hi all. I did some search on the web and the archives here and did not find > an conclusive answer to this question. > > Apache has both input a filters (work on the request before content > generation) and output filters (work on generated content). As I am > exploring the possibility to port some Apache modules that transform data in > both directions in a proxy-setup to Nginx, I am wondering whether Nginx has > input filters as well. What I have read about Nginx modules seem to indicate > that it does not. Is that correct? > > I am aware that I could hack something into the proxy module, but I would > like to avoid that approach. Depending on what you are trying to do, either phase handlers or request body filters might work for you. If they won't, you are probably looking for either proxy-level modifications with proxy module configuration options (such as proxy_set_header and proxy_set_body), or for a proxy-like module which will do appropriate modifications. -- Maxim Dounin http://mdounin.ru/ From nginx-forum at forum.nginx.org Tue Oct 29 17:57:43 2019 From: nginx-forum at forum.nginx.org (skwok) Date: Tue, 29 Oct 2019 13:57:43 -0400 Subject: unknown directive "geoip_country" for openresty/nginx 1.9.7.4 Message-ID: <1fd39f7b44d6735395dce32f7b375ce2.NginxMailingListEnglish@forum.nginx.org> Hi, I've run into an issue which is very similar to https://forum.nginx.org/read.php?2,266453,266453#msg-266453, but with a different version such that the solution for that (directive `load_module`) is not available yet. I am building a docker container of nginx with modsecurity and geoip on Ubuntu Xenial, and I'm stuck with the geoip part as I can't find the instructions for version 1.9.7.4, which doesn't support dynamic module. Can someone please help? I've tried a few things as in the following: 1. Add `--with-http_geoip_module` as an option for configure 2. `apt-get install ngx_http_geoip_module`, and it returns Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package ngx_http_geoip_module 3. Declaring 'load_module "modules/ngx_http_geoip_module.so";' and that is not allowed as the load_module directive is not available in that version 4. Declaring geoip_country /usr/share/GeoIP/GeoIP.dat; results in `nginx: [emerg] unknown directive "geoip_country" ...' Thank you. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286043,286043#msg-286043 From francis at daoine.org Tue Oct 29 18:46:51 2019 From: francis at daoine.org (Francis Daly) Date: Tue, 29 Oct 2019 18:46:51 +0000 Subject: unknown directive "geoip_country" for openresty/nginx 1.9.7.4 In-Reply-To: <1fd39f7b44d6735395dce32f7b375ce2.NginxMailingListEnglish@forum.nginx.org> References: <1fd39f7b44d6735395dce32f7b375ce2.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20191029184651.GB1914@daoine.org> On Tue, Oct 29, 2019 at 01:57:43PM -0400, skwok wrote: Hi there, > I am building a docker container of nginx with modsecurity and geoip on > Ubuntu Xenial, and I'm stuck with the geoip part as I can't find the > instructions for version 1.9.7.4, which doesn't support dynamic module. Can > someone please help? > > I've tried a few things as in the following: > > 1. Add `--with-http_geoip_module` as an option for configure That should work. But you do need to be sure that you are using the nginx that was built after this ./configure run. > 4. Declaring geoip_country /usr/share/GeoIP/GeoIP.dat; results in `nginx: > [emerg] unknown directive "geoip_country" ...' That error message means that the nginx that is being used, does not include the geoip module. If you call the same nginx binary with a "-V" argument, it should show you the configure arguments that were used when building that binary. Does that show the full "configure" line that you expect? If not, then something has happened between your "make" and your deployment. (I've not tested with openresty, but I believe that it should be the same as stock nginx from this perspective.) f -- Francis Daly francis at daoine.org From roger at netskrt.io Tue Oct 29 20:45:30 2019 From: roger at netskrt.io (Roger Fischer) Date: Tue, 29 Oct 2019 13:45:30 -0700 Subject: Check if a resource is in the cache Message-ID: <323CDD2C-C78F-476C-9A16-7D3FD5E00FB8@netskrt.io> Hello, is there a way to check if a requested resource is in the cache? For example, ?if? has the option ?-f?, which could be used to check if a static file is present. Is there something similar for a cached resource? Thanks? Roger From igor at sysoev.ru Tue Oct 29 23:58:46 2019 From: igor at sysoev.ru (Igor Sysoev) Date: Wed, 30 Oct 2019 02:58:46 +0300 Subject: proxy_store: is it used much? In-Reply-To: <7E9A9071-F182-4384-A2C7-3FBDC48F1024@netskrt.io> References: <7E9A9071-F182-4384-A2C7-3FBDC48F1024@netskrt.io> Message-ID: <7A595243-12D0-4008-A5B7-AEEF726154FB@sysoev.ru> > On 29 Oct 2019, at 02:08, Roger Fischer wrote: > > Hello, > > proxy_store seems to be a much simpler alternative to ?cache" pseudo-static resources. But there is very little discussion of it on the Internet or nginx forum (compared to proxy_cache). > > Is there anything non-obvious that speaks agains the use of proxy_store? proxy_store is useful for forever static files mirroring. It does not track file changing and removal at origin. -- Igor Sysoev http://nginx.com From nginx-forum at forum.nginx.org Wed Oct 30 00:13:35 2019 From: nginx-forum at forum.nginx.org (skwok) Date: Tue, 29 Oct 2019 20:13:35 -0400 Subject: unknown directive "geoip_country" for openresty/nginx 1.9.7.4 In-Reply-To: <20191029184651.GB1914@daoine.org> References: <20191029184651.GB1914@daoine.org> Message-ID: <50441016fa173ef1f2f8e350c4297cc8.NginxMailingListEnglish@forum.nginx.org> Hi Francis, Thank you for you response. It is assuring and it's helped me find a bug later in the dockerfile that's replaced the nginx. Best, skwok Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286043,286048#msg-286048 From r at roze.lv Wed Oct 30 06:26:41 2019 From: r at roze.lv (Reinis Rozitis) Date: Wed, 30 Oct 2019 08:26:41 +0200 Subject: Check if a resource is in the cache In-Reply-To: <323CDD2C-C78F-476C-9A16-7D3FD5E00FB8@netskrt.io> References: <323CDD2C-C78F-476C-9A16-7D3FD5E00FB8@netskrt.io> Message-ID: <000901d58eea$fe4872f0$fad958d0$@roze.lv> > Hello, > > is there a way to check if a requested resource is in the cache? > > For example, ?if? has the option ?-f?, which could be used to check if a static > file is present. > > Is there something similar for a cached resource? Depending on what you want to achieve you could check $upstream_cache_status for 'HIT' ( http://nginx.org/en/docs/http/ngx_http_upstream_module.html?#var_upstream_cache_status ) rr From nginx-forum at forum.nginx.org Thu Oct 31 13:25:57 2019 From: nginx-forum at forum.nginx.org (awagner) Date: Thu, 31 Oct 2019 09:25:57 -0400 Subject: Input filters in Nginx? In-Reply-To: <20191029161958.GC12894@mdounin.ru> References: <20191029161958.GC12894@mdounin.ru> Message-ID: <7ad27ca8f6abb25c2aa53403822647ac.NginxMailingListEnglish@forum.nginx.org> Maxim Dounin Wrote: ------------------------------------------------------- [...] > > Depending on what you are trying to do, either phase handlers or > request body filters might work for you. Just working on the body will not do it. A phase handler, basically what Apache offers with its "hooks" from what I can see, looks promising though. > If they won't, you are probably looking for either proxy-level > modifications with proxy module configuration options (such as > proxy_set_header and proxy_set_body), or for a proxy-like module > which will do appropriate modifications. I do want to avoid writing my on proxy-like module or modifying an existing one. I do understand this is a possibility, but the people I am looking at this for will want to keep the functionality separate from the actual proxying. Thanks for the pointer to the phase handlers! Regards, Arno Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286038,286060#msg-286060